Running the Pipeline

After setting up the project, you can now run your Dagster pipeline:

1. Start the Dagster Dev Server: In your terminal, navigate to the root directory of your project and run:

uv run dg dev

The uv run command ensures that dg dev runs within the project's isolated environment defined in pyproject.toml. No manual venv activation is needed.

After executing the command, the Dagster logs will be displayed in the terminal. Once you see a message similar to:

2025-02-04 09:15:46 +0530 - dagster - INFO - Serving Dagster UI on http://127.0.0.1:3000

The Dagster web server is running successfully.

Note: dg dev creates an ephemeral instance by default. To persist your runs and assets across sessions, set the DAGSTER_HOME environment variable before running uv run dg dev:

Windows (PowerShell):

$env:DAGSTER_HOME="$env:USERPROFILE\.dagster_home"
uv run dg dev

macOS/Linux:

export DAGSTER_HOME=~/.dagster_home
uv run dg dev

1. Access the Dagster UI:

Open a web browser and navigate to http://127.0.0.1:3000. This will open the Dagster UI where you can manage and monitor your pipelines.

1. Run the Pipeline:

2. In the left navigation of the Dagster UI, click on Lineage.

   - Click Materialize all to execute the pipeline.

3. Monitor the Run:

The Dagster UI allows you to visualize the pipeline's progress, view logs, and inspect the status of each step. You can switch between different views to see the execution logs and metadata for each asset.