Terraform-Docker-Airflow-GCP

Terraform Files to Create a GCP VM Install Docker Docker-Compose and Run Airflow

Walkthrough Video for Setup

Create a Project in GCP

Create a Service Account and Download the Key

Create a Service Account and give it Compute Engine Admin Permissions
Generate and download a key for the Service Account

Note: The variables.tf files are looking for a key titled terra-airflow.json in a .google directory in the root of the projects file structure.

Create a ./google directory and add your .json key there
Name it terra-airflow or update the varaibles.tf file variable

Update the Variables.tf File

Update the variables in variables.tf to match your project.

A good way to do this is to look at the Dashboard for the project name

Go to Compute Engine and click the Create Instance button.

The default shown for Region and Zone are most likely what you want to use.

Generate an SSH Key Pair

ssh-keygen -t rsa -f ~/.ssh/KEY_FILENAME -C USERNAME

The files as is will be looking for a private and public key pair with the name "GCP" in your users .ssh directoy.

Create Service Account for Airflow

You will now need to generate a Service Account with permissions to upload to a GCP Bucket.

Create a key file for the service acount and place it at .google/credentials/google_credentials.json

Deploy to GCP

From either the
file_user-script
or inline_user-script
directories run

terraform plan

to check for errors

terraform apply

to deploly to GCP