Airflow Connection Introduction

In Airflow, a Connection refers to the information required to connect to an external system or service. These external systems can include databases, APIs, FTP servers, and more.

A Connection includes the following pieces of information:

• Connection ID: A string used to uniquely identify this Connection within Airflow.
• Connection Type: The type of connection, such as database, SSH, or FTP.
• Host: The hostname or IP address of the target system.
• Schema: The schema name for a database connection.
• Login: The username used to authenticate with the target system.
• Password: The password required for authentication.
• Port: The port number on which the service is listening.
• Extra: Additional parameters, such as SSL configuration or OAuth tokens.

In Airflow, Connections can be configured via the Web UI or through Python code within DAGs. Within a DAG, Connections are utilized to interact with external systems—for example, extracting data from a database or sending requests to an API. Connection details can be accessed using Hook objects in the DAG and are parameterized within the DAG code, allowing different connection information to be used across various environments.

Examples

A common type of Connection is one that connects to a MySQL database. In Airflow, you can configure a MySQL Connection via the Web UI or through Python code within your DAGs, as shown below:

from airflow import DAG
from airflow.providers.mysql.hooks.mysql import MySqlHook
from datetime import datetime

dag = DAG(
    'my_dag',
    start_date=datetime(2023, 6, 24),
    schedule_interval='0 0 * * *'
)

mysql_conn_id = 'my_mysql_conn'
mysql_hook = MySqlHook(mysql_conn_id)

query = 'SELECT * FROM my_table'
result = mysql_hook.get_records(query)

for row in result:
    print(row)

In this example, we first define a DAG and then use the MySqlHook to connect to a MySQL database. The variable mysql_conn_id represents the ID of the MySQL Connection defined within Airflow. By invoking the get_records() method on the MySqlHook, we execute an SQL query and store the resulting data in the result variable. Finally, we iterate through the results for processing; in this specific example, each row is simply printed to the console.

It should be noted that in this example, the Connection information is hardcoded directly within the DAG code. In practice, to ensure better maintainability and reusability, it is standard procedure to configure Connection details in Airflow's configuration files (e.g., airflow.cfg or environment variables) and access them using parameterized approaches within the DAG code.