We’ve long had a vision at Astronomer to open source our core modules, and we’re excited to announce that we’ve begun that process. These Docker images will be directly utilized in core Astronomer Enterprise Edition and the next generation of Astronomer Cloud, and will be well-maintained.
We’ve put together Docker images for Airflow that include Airflow webserver + scheduler, Celery, Flower (to monitor Celery), with metrics flowing to Prometheus and dashboards in Grafana. We view this as a minimal production setup for Airflow.
It’s super easy to test this out:
Clone the repository by running:
git clone https://github.com/astronomerio/astronomer.gitand
Pop open a terminal and run
cd examples/airflow && docker-compose up. This will spin up the containers in the same configuration as a live Astronomer environment.
First, we spin up a Postgres container for the Airflow metadata database, and a Redis container to back Celery, which Airflow will use for its task queue. Once the storage containers have started, we start the Airflow Scheduler, Airflow Webserver, a Celery worker, and the Flower UI to monitor the Celery task queue. Once everything is up and running, open a browser tab and visit http://localhost:8080 for the Airflow UI and http://localhost:5555 for the Celery UI.
Sweet! You’re up and running with Apache Airflow and well on your way to automating all your data pipelines. Check out the docs to get started with your first pipelines, or get your existing pipelines running on the Astronomer Platform.
- For Astronomer Open Edition, it’s up to you to come up with your own deployment strategy. If you need help, we have services team and support contracts available.
- Astronomer Enterprise Edition is in beta as a commercial product, and packages machinery to easily deploy these modules to the major clouds: AWS, Google Cloud, Azure, IBM Bluemix, and Digital Ocean. You bring your own data stores.
- Astronomer Cloud Edition provides all infrastructure and data stores, but for that reason it's a little less flexible — but it's a great way to get started with the platform.
We’d love more Github stars on the project — and if you run into a problem, drop us a Github issue or pull request.