Using the Airflow CLI in Astronomer Enterprise

Although Airflow's UI supports the majority of the behavior you would need when setting up your environment and building DAGs, there are cases when you want to use the Airflow CLI.

Configuring kubectl

Before interacting directly with the pods that are created with each Airflow deployment, it is necessary to first configure the Kubernetes command line kubectl. Instructions on how to install kubectl can be found: here. Connecting to your Kubernetes cluster will be different depending on where it is installed.

Switching to the correct context

Once connected to your cluster, you will need to set the appropriate context. We highly recommend using kubectx to easily switch between Kubernetes clusters and namespaces.

$ kubectx

$ kubectx gke_astronomer-demo_us-west2-a_astronomer-demo-ltgzkozwnr
Switched to context "gke_astronomer-demo_us-west2-a_astronomer-demo-ltgzkozwnr".

Switching to the correct namespace

After setting the context, you will want to switch into the appropriate namespace. Using the kubens command, you will see a separate namespace for each airflow instance you have created in Astronomer.

$ kubens

$ kubens astronomer-demo-meteoric-meteorite-9699
Context "gke_astronomer-demo_us-west2-a_astronomer-demo-ltgzkozwnr" modified.
Active namespace is "astronomer-demo-meteoric-meteorite-9699".

Retrieving pod names

Once in the desired namespace, you can list the pod names. For our purposes, we need to make sure that we take a pod that has Airflow installed on it. This means the scheduler, webserver, or workers.

$ kubectl get pods
NAME                                                 READY     STATUS    RESTARTS   AGE
meteoric-meteorite-9699-flower-67b5c956bb-d2k27      1/1       Running   0          7m
meteoric-meteorite-9699-pgbouncer-874d47799-2j74x    2/2       Running   0          7m
meteoric-meteorite-9699-redis-0                      1/1       Running   0          7m
meteoric-meteorite-9699-scheduler-7f5ccc7fdb-pfdq5   1/1       Running   0          7m
meteoric-meteorite-9699-statsd-69548ddcbc-n8cgv      1/1       Running   0          7m
meteoric-meteorite-9699-webserver-684b7b56d4-zr7ff   1/1       Running   0          7m
meteoric-meteorite-9699-worker-0                     2/2       Running   0          7m


Once you have retrieved a pod name (remember, it needs to be one that has airflow installed!) your can exec into it and then execute the relevant airflow commands as below.

Adding Connections

$ kubectl exec -it meteoric-meteorite-9699-scheduler-7f5ccc7fdb-pfdq5 -- airflow connections --add --conn_id a_new_connection  --conn_type ' ' --conn_login etl --conn_password my_password
	Successfully added `conn_id`=a_new_connection :  ://etl:my_password@:

Adding Variables

$ kubectl exec -it meteoric-meteorite-9699-scheduler-7f5ccc7fdb-pfdq5 -- airflow variables --set my_key my_value

Ready to build your data workflows with Airflow?

Astronomer is the data engineering platform built by developers for developers. Send data anywhere with automated Apache Airflow workflows, built in minutes...