Set Up Environment Variables

Before defining the pipeline, configure the environment variables that the Teradata resource will use to connect to your Teradata instance. Create a .env file in the root of your dagster-quickstart project with the following content:

TERADATA_HOST=your_teradata_host
TERADATA_USER=your_teradata_username
TERADATA_PASSWORD=your_teradata_password
TERADATA_DATABASE=dagster_pipeline_db

Replace the placeholder values with your actual Teradata connection details: - TERADATA_HOST: The hostname or IP address of your Teradata instance - TERADATA_USER: Your Teradata username - TERADATA_PASSWORD: Your Teradata password - TERADATA_DATABASE: The database name (use dagster_pipeline_db if you created it as shown in the prerequisites)

The next step is to configure the pipeline by defining the necessary resources and jobs.

Edit the definitions.py File: Modify src/dagster_quickstart/definitions.py and define your Dagster pipeline as follows:

import os
from dotenv import load_dotenv

from dagster import Definitions
from dagster_teradata import TeradataResource

from .defs import read_csv_file, read_table, create_table, drop_table, insert_rows

# Load environment variables from .env file
load_dotenv()

# Configure Teradata resource with connection details from environment variables
td_resource = TeradataResource(
    host=os.getenv("TERADATA_HOST"),
    user=os.getenv("TERADATA_USER"),
    password=os.getenv("TERADATA_PASSWORD"),
    database=os.getenv("TERADATA_DATABASE"),
)

# Define the pipeline and resources
defs = Definitions(
    assets=[read_csv_file, read_table, create_table, drop_table, insert_rows],  
    resources={
        "teradata": td_resource,
    }
)

This code sets up a Dagster project that interacts with Teradata by defining assets and resources:

1. It imports necessary modules, including Dagster and dagster-teradata.
2. It imports asset functions (read_csv_file, read_table, create_table, drop_table, insert_rows) from the defs module.
3. It configures the TeradataResource with connection details from environment variables.
4. It registers these assets with Dagster using Definitions, allowing Dagster to track and execute them.