Canal Core Components and Concepts

Open-sourced by Alibaba, Canal is a widely adopted incremental subscription and consumption component based on MySQL bin

Open-sourced by Alibaba, Canal is a widely adopted incremental subscription and consumption component based on MySQL binary logs (binlog). It is heavily utilized in modern data architectures for scenarios such as cache invalidation, heterogeneous database synchronization, and real-time Elasticsearch indexing. Mastering its core components and foundational concepts is the prerequisite for robust deployments and secondary development.

The core philosophy of Canal revolves around mimicking a MySQL slave node.

  1. Canal initiates a dump protocol request to the MySQL master.
  2. The master streams binlog events back to Canal upon receiving the request.
  3. Canal parses the raw binary logs and transforms them into structured data formats.
  4. Downstream clients subscribe to and consume these incremental changes.

The overall data pipeline flows as follows:

MySQL Master → EventParser → EventSink → EventStore → Canal Client / MQ → Downstream Systems

Canal Server and Instance

  • Canal Server: An independent runtime process responsible for maintaining connections with MySQL, fetching, and parsing binlogs. A single server instance can host multiple concurrent tracking tasks.
  • Instance: The fundamental logical unit. Each instance corresponds to a dedicated subscription task for a specific MySQL database, serving as the smallest isolation boundary for data synchronization.

The Four Internal Modules of an Instance

  • EventParser: Acting as the event parser, it simulates the slave protocol to fetch binlogs from the master database, translates them into structured Event objects, and handles position tracking and breakpoint resumption.
  • EventSink: Positioned between the parser and the store, the event processing pipeline handles data filtering, routing, and transformation, such as splitting events by database-table rules or merging multi-instance streams.
  • EventStore: The event storage layer. By default, it utilizes an in-memory RingBuffer queue to temporarily cache parsed data awaiting client consumption.
  • MetaManager: The metadata manager records operational states such as consumption offsets and subscription relationships, supporting storage backends like memory, local files, and ZooKeeper.

Surrounding Ecosystem Components

  • Canal Client: A lightweight client library that pulls data from the server and executes custom business logic.
  • Canal Adapter: An out-of-the-box plugin designed to route data directly into targets like Kafka, RocketMQ, Elasticsearch, and HBase.
  • Canal Admin: A centralized web management console providing unified configuration, monitoring, and operational capabilities for instances.
  • ZooKeeper (Optional): Used in cluster deployments to achieve High Availability (HA). Only one node runs the active instance at any given moment while standby nodes wait in readiness. Offsets are also persisted in ZooKeeper to allow seamless failover recovery.

Core Concepts

Position Tracking

  • journalName + position: The specific binlog file name combined with the internal byte offset, precisely locating the consumption coordinate.
  • timestamp: A time-based rollback mechanism allowing consumption to start from a specific historical time point.
  • GTID: When MySQL operates under Global Transaction Identifier (GTID) mode, consumption positions can be determined directly through transaction identifiers.

Entry and Message

Canal serializes binlog events into Entries using the Protocol Buffers format. The header contains metadata such as the database name, table name, event type, and execution timestamp, while the store value preserves row-level image changes before and after execution. Supported event types span INSERT, UPDATE, DELETE, QUERY, DDL statements, and transaction boundary markers like BEGIN and COMMIT. Clients pull a Message batch at a time, which carries a unique batchId along with a collection of entries.

Consumption Semantics: Get, Ack, and Rollback

Client consumption relies on a message-queue-style acknowledgment workflow:

  1. get: Pulls a batch of data without confirming receipt, leaving the data safely retained in the store.
  2. ack: Submits a confirmation after successful business processing, safely advancing the consumption offset.
  3. rollback: Triggers a rollback upon processing failures, allowing the batch to be re-fetched and retried.

FlatMessage

While raw entries are encoded in binary Protocol Buffers, data dispatched to message queues can be serialized into a cleaner JSON format known as a FlatMessage. It exposes structured fields such as database, table, type, data, and old, making it straightforward for language-agnostic downstream consumers to parse.

Quick Start Example

Instance Configuration (conf/example/instance.properties)

# Master database connection address
canal.instance.master.address=127.0.0.1:3306
# Credentials for dumping binlogs, requiring REPLICATION SLAVE privileges
canal.instance.dbUsername=canal
canal.instance.dbPassword=canal
# Regular expression filtering target databases and tables
canal.instance.filter.regex=.*\\..*

Client Consumption Code Example

CanalConnector connector = CanalConnectors.newSingleConnector(
        new InetSocketAddress("127.0.0.1", 11111),
        "example", "", "");
connector.connect();
connector.subscribe(".*\\..*");
connector.rollback();

while (true) {
    Message message = connector.getWithoutAck(100); // Batch fetch
    long batchId = message.getId();
    if (batchId == -1 || message.getEntries().isEmpty()) {
        Thread.sleep(1000);
        continue;
    }
    try {
        printEntries(message.getEntries()); // Parse entries and process business logic
        connector.ack(batchId);             // Confirm progress upon success
    } catch (Exception e) {
        connector.rollback(batchId);        // Rollback upon failure for reprocessing
    }
}

Summary

The architecture of Canal can be concisely summarized as follows: Instances carry synchronization tasks, parsers handle decoding, sinks manage filtering and routing, stores buffer data, and clients consume it. Combined with ZooKeeper for high availability, alongside reliable offset management and ack/rollback mechanisms, Canal ensures zero data loss and replay capabilities. Mastering these concepts empowers developers to effortlessly construct high-performance data pipelines bridging MySQL with message queues and search engines.