Define a Data Pipeline

Define the data source, destination, and connection within the main.tf file. Open the newly created main.tf file in Visual Studio Code or any preferred code editor.

If using Visual Studio Code, install HashiCorp Terraform Extensions to add autocompletion and syntax highlighting.

Terraform Extensions on Visual Studio Code

Populate the main.tf file with the template provided:

# Provider Configuration
terraform {
  required_providers {
    airbyte = {
      source = "airbytehq/airbyte"
      version = "0.13.0"
    }
  }
}
provider "airbyte" {
  // Use client credentials so the provider refreshes access tokens automatically.
  client_id     = var.airbyte_client_id
  client_secret = var.airbyte_client_secret
  token_url     = var.airbyte_token_url
}

# Teradata Destination Configuration
# For optional parameters visit https://registry.terraform.io/providers/airbytehq/airbyte/latest/docs/resources/destination_teradata 
resource "airbyte_destination_teradata" "my_destination_teradata" {
  configuration = {
    host   = var.host
    schema = "airbyte_td_two"
    ssl    = false
    ssl_mode = {
      allow = {}
    }
    logmech = {
      td2 = {
        username = var.username
        password = var.password
      }
    }
  }
  name         = "Teradata"
  workspace_id = var.workspace_id
}
# Connection Configuration 
resource "airbyte_connection" "googlesheets_teradata" {
  name           = "Google Sheets - Teradata"
  source_id      = airbyte_source_google_sheets.my_source_google_sheets.source_id
  destination_id = airbyte_destination_teradata.my_destination_teradata.destination_id

  schedule = {
    schedule_type   = "cron"
    cron_expression = "0 */15 * * * ?" # every 15 minutes
  }
}
# Google Sheets Source Configuration
resource "airbyte_source_google_sheets" "my_source_google_sheets" {
  configuration = {
    spreadsheet_id = var.google_sheets_spreadsheet_id
    credentials = {
      service_account_key_authentication = {
        service_account_info = var.google_service_account_info
      }
    }
  }
  name         = "Google Sheets Source"
  workspace_id = var.workspace_id
}

Note that this example uses a cron expression to schedule the data transfer to run every 15 minutes.

In our main.tf file, we reference variables that are held in the variables.tf file, including the API key, workspace ID, Google Sheets ID, Google private key, and Teradata credentials. We will populate sensitive credentials to a terraform.tfvars file that we will not commit to version control.