What is Airflow?
Apache Airflow is an open-source platform designed to facilitate the orchestration and automation of complex workflows. Developed by Airbnb, it provides a flexible and scalable solution for managing, scheduling, and monitoring tasks within a data pipeline. Airflow uses directed acyclic graphs (DAGs) to represent workflows, allowing users to define and execute tasks with ease.
Airflow Architecture

- Web Server:
Users utilize the Airflow Web Server to manage metadata, monitor processes, and trigger job executions—a centralized interface for seamless workflow control.
- Scheduler:
The scheduler orchestrates DAGs, triggers tasks, and interacts directly with the metadata database, ensuring efficient execution of scheduled workflows.
- Metadata Database:
The repository for DAGs, task statuses, variables, and connections. Though defaulting to SQLite, Airflow accommodates various database servers supported by SQL Alchemy for flexible metadata storage.
- Executor:
Task execution relies on three types of executors—local, sequential, and Celery. While the default is sequential, the Celery executor is recommended for scalability and parallel execution, shaping task behavior in Airflow.
Airflow Features:
- Extensibility:
Airflow’s modular design allows the integration of custom operators and hooks, extending its capabilities to suit diverse use cases.
- Dynamic Workflow Configuration:
Parameters can be dynamically set during runtime, providing flexibility in adapting to changing requirements.
- Versioning and Backfilling:
Airflow supports versioning of DAGs, enabling easy management of changes, and allows backfilling to rerun or catch up on historical data.
- Rich Set of Operators:
The platform includes a variety of built-in operators for common tasks, such as BashOperator, PythonOperator, etc.
- Community and Support:
With a vibrant community, Airflow benefits from continuous development and a wealth of shared knowledge and resources.
Airflow Use Cases:
ETL Automation:
Airflow is widely employed to automate Extract, Transform, Load (ETL) processes. It efficiently orchestrates data workflows, ensuring seamless and scheduled execution of tasks for data extraction, transformation, and loading into destination systems.
Batch Processing:
Organizations leverage Airflow for batch processing tasks, such as nightly aggregations, report generation, or data cleansing. The platform’s scheduling capabilities enable the execution of recurring batch jobs, ensuring timely and accurate data processing.
Machine Learning Pipelines:
Airflow plays a crucial role in orchestrating end-to-end machine learning pipelines. It schedules and monitors tasks involved in data preprocessing, model training, and deployment, facilitating the automation of machine learning workflows and ensuring reproducibility.
Cloud Resource Management:
Airflow is employed for managing cloud resources efficiently. It orchestrates tasks related to provisioning, scaling, and decommissioning resources in cloud environments, optimizing resource utilization and reducing operational overhead.
Defining a Workflow in Airflow:
To define a workflow in Apache Airflow, you create a Directed Acyclic Graph (DAG) that represents the flow and dependencies of tasks.
DAG in Airflow:
In Airflow, a DAG is a collection of tasks with defined dependencies, forming a logical workflow. It represents the sequence of operations to be executed.
from airflow import DAG
dag = DAG('my_sample_dag', schedule_interval='@daily')
The code above creates a DAG named ‘my_sample_dag’ with a daily execution schedule, specifying that the tasks within the DAG should be triggered once per day.
Operator in Airflow:
An operator in Airflow defines the execution of a task. It encapsulates a single, ideally idempotent, unit of work.
from airflow.operators.python_operator import PythonOperator
from airflow.operators.databricks_operator import DatabricksSubmitRunOperator
1. Python Operator runs a python function.
def task1_function():
print("Executing task 1...")
task1 = PythonOperator(task_id='task_1', python_callable=task1_function, dag=dag)
2. DatabricksSubmitRunOperator submits a run to Databricks.
task2 = DatabricksSubmitRunOperator(
task_id='task_2',
databricks_conn_id='databricks_default',
notebook_task={'notebook_path': '/path/to/notebook'},
dag=dag)
Task Linking:
Tasks are linked using the bitshift operators (>>). For instance, task1 >> task2 denotes that task2 depends on the successful completion of task1. This establishes the order and dependencies in the workflow, creating a sequential execution structure.

Mian Ali Shah
Associate Consultant