Supervisor operators and configuration for long-running tasks
Project description
airflow-supervisor
Apache Airflow utilities for running long-running or always-on jobs with supervisord
Overview
This library provides a configuration-driven way of generating supervisor configurations and airflow operators/sensors for long-running or always-on jobs. Configuration is managed by Pydantic, Hydra, and OmegaConf.
How To: Use in Airflow
airflow-supervisor
can be installed in your airflow server environment and imported in your dag files. It provides two convenient top level DAG subclasses:
SupervisorLocal
: creates a DAG representing a local supervisor instance running on the airflow worker node (underlying task will usePythonOperator
andBashOperator
to communicate between airflow and supervisor)SupervisorRemote
: creates a DAG representing a remote supervisor instance running on another machine (underlying tasks will useSSHOperator
to communicate between airflow and supervisor)
We expose DAGs composed of a variety of tasks and sensors, which are exposed as a discrete pipeline of steps:
- Setup
supervisord
configuration - Start the
supervisord
daemon - Start the supervised programs with
supervisorctl
- Start sensors to query the programs' state via supervisor's XML-RPC API
- Evaluate and take action according to the program's state changes
- Restart programs if necessary
- Tear down the sensors from (4)
- Stop the supervised programs from (3)
- Stop the
supervisord
daemon from (2) - Remove configuration from (1)
This setup provides maximal configureability with a minimal requirements on the machine (for example, no requirements on an existing supervisord
daemon via e.g. systemd
). It also lets you hook your own tasks into any step of the process. For example, if we detect a process has died in step (5), you could configure your own task to take some custom action before/instead of the default restart of step 6.
Here is a nice overview of the DAG, with annotations for code paths and the actions taken by Supervisor:
More docs and code examples coming soon!
How To: Use as a supervisord configuration frontend
This library can be used outside airflow as a generic supervisord configuration framework, with the static typing benefits that entails. For an example, look at the hydra configuration test. This example generates a supervisor configuration file by composing independent hydra configs.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for airflow_supervisor-0.2.0-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | b6c73f166632b91ddfff3d56c493f7d08dfb95c0c47b53e8adc37d01faddde5d |
|
MD5 | e2895b8eda3d30fada11994402fe5f5c |
|
BLAKE2b-256 | 4a15cfe0238eb0502cf9ea54cc0f83c72885531a99eb084831f03ad4991fb7ab |