Airflow setup

Airflow is at the center of this tutorial. It is responsible for orchestrating Airbyte to move data to Teradata and triggering dbt to transform the data.

1. Rename the file from .env.example to .env.
2. On Linux/WSL, ensure AIRFLOW_UID in .env matches your host user id (for example, 1000).
3. For Airflow 3, set the following two variables in .env (they are included in .env.example as placeholders):
   • AIRFLOW__API_AUTH__JWT_SECRET
   • AIRFLOW__API__SECRET_KEY
4. Why these are required:
   • AIRFLOW__API_AUTH__JWT_SECRET is used to sign and validate Execution API JWTs exchanged between Airflow services (webserver, scheduler, worker, dag-processor).
   • AIRFLOW__API__SECRET_KEY is used to sign API/session-related data consistently across services.
   • If these values are missing or inconsistent between containers, tasks can fail with errors such as Invalid auth token or JWT signature validation failures.
5. How to define them:
   • Use long random values and keep them identical for all Airflow services in the same deployment.
   • Example generation commands:

openssl rand -hex 64
openssl rand -hex 64

• Then set the first output as AIRFLOW__API_AUTH__JWT_SECRET and the second as AIRFLOW__API__SECRET_KEY in .env.

• Build the custom Airflow image

docker compose build

1. Launch the Airflow container

docker compose up

This might take a few minutes initially as it sets up necessary databases and metadata.

1. Setting up Airflow Connections

Both Airbyte and dbt require connections to be set up in Airflow:

• Access the Airflow UI by navigating to http://localhost:8080 in your browser.
• The Username is admin, the password is retrieved by searching on docker desktop logs.

• Go to Admin > Connections.

• Create Airbyte Connection

• Go to Admin > Connections.

  Click the + Add Connection button to create a new connection and fill in the following details:

  • Connection Id: A unique ID for the Airbyte connection that will be used in DAGs to trigger Airbyte syncs. Name it airbyte_connection.
  • Connection Type: Select Airbyte.
  • Server URL: The URL of the Airbyte instance. For local setups, use http://host.docker.internal:8000/api/public/v1. For remote instances, use the instance URL.
  • Client ID: Copy from the Airbyte credentials generated above.
  • Client Secret: Copy from the Airbyte credentials generated above.
  • Token URL: For local deployments, use http://host.docker.internal:8000/api/public/v1/applications/token.

  Click Save.

  

• Create Teradata Connection

  Click the + button to create a new connection and fill in the following details: * Connection Id: A unique ID for the Teradata connection. Name it teradata_connection. * Connection Type: Select Teradata. * Database Server URL (required): The Teradata instance hostname to connect to. * Username (required): Your Teradata username. * Password (required): Your Teradata password. * Database (optional): The name of the database to connect to. Specify ecommerce.

  Click Save.

  

• Link Airbyte connection to the Airflow DAG

The last step before executing the DAG in Airflow is to link the connection_id from Airbyte:

• Visit the Airbyte UI at http://localhost:8000/.
• In the Connections tab, select the Sample Data to Teradata connection and copy its connection ID from the URL.
• Update the connection_id in the extract_data task within orchestration/airflow/dags/elt_dag.py with this ID.

That's it! Airflow is now configured to work with dbt and Airbyte.