Running the Pipeline

After setting up the project and preparing the data, we can now run our Dagster pipeline:

Start the Dagster Dev Server:

Run the following command from the project root:

uv run dg dev

The uv run command ensures that dg dev runs within the project's isolated environment defined in pyproject.toml. No manual venv activation is needed.

After executing the command, the Dagster logs will be displayed in the terminal. Once you see a message similar to:

2025-02-04 09:15:46 +0530 - dagster - INFO - Serving Dagster UI on http://127.0.0.1:3000

The Dagster web server is running successfully.

Note: dg dev creates an ephemeral instance by default. To persist your runs and assets across sessions, set the DAGSTER_HOME environment variable before running uv run dg dev:

Windows (PowerShell):

$env:DAGSTER_HOME="$env:USERPROFILE\.dagster_home"
uv run dg dev

macOS/Linux:

export DAGSTER_HOME=~/.dagster_home
uv run dg dev

Access the Dagster UI:

Open a web browser and navigate to http://127.0.0.1:3000. This will open the Dagster UI where you can manage and monitor your pipelines.

In the Dagster UI, in the jobs tab you will see the following:

The job example_job is displayed, along with the associated assets.
The assets are organized under the "default" asset group.
In the middle, you can view the lineage of each @op, showing its dependencies and how each operation is related to others.

Launch the Job:

Go to the "Launchpad" tab. Your configuration values (host, user, password, database, AWS credentials) are already populated from the environment variables you set earlier. Simply click "Launch Run" to start the pipeline execution.

Tip: If you need to override any values at runtime, you can edit them in the Launchpad before launching. However, for most cases, the environment variables you set in step "Set Environment Variables" will be sufficient.

The Dagster UI allows us to visualize the pipeline's progress, view logs, and inspect the status of each step.