# Change data capture: log-based CDC, Debezium and the outbox pattern

> Change data capture streams each committed insert, update and delete from a database log as events. How Debezium, replication slots and the outbox work.

- URL: https://computese.com/change-data-capture/
- Author: Duong Quan Nguyen, CEO, Computese
- Published: 2026-10-10
- Updated: 2026-10-10
- Topics: Data, Integrations

## In short
- Change data capture (CDC) turns each committed insert, update and delete into an event. Log-based CDC reads the transaction log, such as the PostgreSQL WAL or the MySQL binlog, so it captures deletes and the intermediate changes that polling misses.
- Debezium is open source (Apache Software License 2.0). It runs as Kafka Connect source connectors, as Debezium Server without Kafka, or embedded in a Java application, and each event carries before, after, op and source fields.
- A PostgreSQL replication slot keeps the WAL its consumer has not confirmed, even with nothing connected, and can fill the disk. Cap it with max_slot_wal_keep_size (PostgreSQL 13+), add a heartbeat, alert on retained WAL, and use one slot per connector.
- Delivery is at-least-once, so events can repeat after a failure. With the primary key as the message key, each row's changes stay in order in one partition; consumers upsert by key and compare LSNs so repeats and late events do no harm.
- For events that other services consume, use the outbox pattern: write the event to an outbox table in the same transaction as the change, and let Debezium's outbox event router publish it, keyed by aggregate ID.

Change data capture (CDC) is a way to record every insert, update and delete a database commits and publish each one as an event for warehouses, caches and other services. The dependable way to do it is log-based: read the database's own transaction log, as Debezium does, instead of polling tables for changed rows.

The examples use [Debezium 3.7](https://debezium.io/releases/), released 29 September 2026 and the latest stable version as of October 2026. CDC is one of several [system integration approaches](https://computese.com/system-integration/) and often feeds [data analytics software](https://computese.com/building-data-analytics-software/).

## How do the change data capture methods compare?

| Method              | How it finds changes                                                                                 | Deletes  | Changes between reads         | Cost to the source                         |
| ------------------- | ---------------------------------------------------------------------------------------------------- | -------- | ----------------------------- | ------------------------------------------ |
| Query-based polling | A scheduled `SELECT ... WHERE updated_at > :last_run`                                                | Missed   | Missed: only the latest state | A query per table per run                  |
| Triggers            | A trigger copies each change into a shadow table                                                     | Captured | Captured                      | Extra work in every writing transaction    |
| Log-based           | Reads the WAL (PostgreSQL), binlog (MySQL), redo logs (Oracle) or log-fed change tables (SQL Server) | Captured | Captured, in order            | Reading the log, and keeping it until read |

Polling needs a column like `LAST_UPDATE_TIMESTAMP` on every captured table, kept correct by every writer. A query cannot return a row that no longer exists, so deletes need a soft-delete flag or a side table, and a row inserted and deleted between polls is [never captured at all](https://debezium.io/blog/2018/07/19/advantages-of-log-based-change-data-capture/).

Triggers see every change, but a PostgreSQL trigger [runs in the same transaction](https://www.postgresql.org/docs/18/trigger-definition.html) as the statement that fired it: every write pays for the capture, and an error in the trigger rolls back the business change. Microsoft says custom tracking built from triggers, timestamp columns and extra tables [often carries a high performance overhead](https://learn.microsoft.com/en-us/sql/relational-databases/track-changes/track-data-changes-sql-server?view=sql-server-ver17).

Reading the log gets every change in its exact order of application, with no extra columns, and a stopped reader resumes where it left off. The cost moves to the log, which the database must keep until the reader has it.

![A cutaway database cylinder with its log running out as a long strip of small document boxes. An orange connector at the end of the strip reads the newest box and sends copies to a queue and two servers.](https://computese.com/images/blog/change-data-capture/log-reader.dc948525dd-1536.webp)

*Read the log rather than the tables, and a delete, or a row that lived for a second, is still on the record.*

## How does Debezium capture changes?

Debezium is open source (Apache Software License 2.0), and version 3.7 is tested with PostgreSQL 14 to 18, MySQL 8.0, 8.4 and 9.7, SQL Server 2017, 2019 and 2022, and Oracle 19c, 21c, 23ai and 26ai. It [runs in three ways](https://debezium.io/documentation/reference/3.7/architecture.html):

- **On Kafka Connect**, the most common. By default each table's changes go to a topic named after the table, and sink connectors carry them to other systems.
- **As [Debezium Server](https://debezium.io/documentation/reference/3.7/operations/debezium-server.html)**, an application that streams changes without Kafka to sinks such as Amazon Kinesis, Google Cloud Pub/Sub, Apache Pulsar, JDBC databases and Apache Iceberg tables.
- **As the Debezium engine**, a library inside your own Java application.

Each change becomes a record keyed by the row's primary key, with an envelope as its value. An update to order 1042 (illustrative values, fields trimmed):

```json
{
  "before": { "id": 1042 },
  "after": { "id": 1042, "status": "paid", "currency": "CAD" },
  "source": {
    "version": "3.7.0.Final",
    "connector": "postgresql",
    "db": "orders",
    "schema": "public",
    "table": "orders",
    "snapshot": false,
    "ts_ms": 1791536400000,
    "txId": 88123,
    "lsn": 3049721912
  },
  "op": "u",
  "ts_ms": 1791536400412
}
```

`op` is `c` (create), `u` (update), `d` (delete), `r` (snapshot read), `t` (truncate) or `m` (message). `before` holds only `id` because the table's `REPLICA IDENTITY` is `DEFAULT`; `FULL` adds every column. `source` carries the transaction ID and the log sequence number (LSN), and its `ts_ms` is when the database made the change. The top-level `ts_ms` is when the connector processed it, so the gap, 412 ms here, is the capture lag.

A delete produces a `d` event (`before` set, `after` null), then a tombstone, the same key with a null value, so Kafka compaction can remove the key (`tombstones.on.delete`, `true` by default). A primary key change produces a delete and a tombstone for the old key, then an event with the new key.

### Snapshots and signals

A new connector first snapshots the existing rows as `r` events, then streams from the log. The default `snapshot.mode`, `initial`, snapshots only when no offsets are stored; `when_needed` also snapshots when the stored position is gone from the server; `no_data` never snapshots. A connector stopped mid-snapshot starts a new snapshot on restart.

For a large table, or to re-read one later, use an incremental snapshot: Debezium reads the table in chunks (1,024 rows by default) while streaming continues, and resumes after an interruption. Start one with a [signal](https://debezium.io/documentation/reference/3.7/configuration/signalling.html), a row inserted into the table named in `signal.data.collection`; the connector's user needs `INSERT` on it.

![A tall cabinet of table rows with one drawer pulled out in orange. A connector takes that drawer's rows while the log strip below keeps feeding it, and sends both on into one queue.](https://computese.com/images/blog/change-data-capture/incremental-snapshot.82fcb51071-1536.webp)

*The table is re-read one chunk at a time while the log keeps flowing, so capture never pauses.*

```sql
CREATE TABLE public.debezium_signal (
  id   varchar(42) PRIMARY KEY,
  type varchar(32) NOT NULL,
  data varchar(2048) NULL
);

INSERT INTO public.debezium_signal (id, type, data)
VALUES ('resnap-orders-1', 'execute-snapshot',
        '{"data-collections": ["public.orders"], "type": "incremental"}');
```

PostgreSQL's logical decoding does not carry DDL, so the connector cannot report schema changes as events, and it does not support them at all during an incremental snapshot. Coordinate migrations with consumers.

## How do you set up PostgreSQL for CDC?

Debezium reads PostgreSQL through logical decoding, which extracts committed changes from the write-ahead log (WAL) through an output plug-in. Use `pgoutput`, always present since PostgreSQL 10; the connector's default, `decoderbufs`, is maintained by the Debezium community.

**1. Server settings** in `postgresql.conf`:

```ini
wal_level = logical                     # read only at server start
max_slot_wal_keep_size = 102400         # MB; example cap, default -1 (unlimited)
idle_replication_slot_timeout = 172800  # seconds; example, default 0 (off); PostgreSQL 18
```

[`wal_level`](https://www.postgresql.org/docs/18/runtime-config-wal.html) can only be set at server start, so book a restart. On Amazon RDS, set `rds.logical_replication` to 1 instead (the instance may need a restart) and grant the `rds_replication` role; Cloud SQL requires `pgoutput`.

**2. A role, grants and the publication.** Create the publication yourself and set `publication.autocreate.mode` to `disabled`: the default, `all_tables`, runs `CREATE PUBLICATION ... FOR ALL TABLES` when none exists, which needs a superuser.

```sql
CREATE ROLE debezium REPLICATION LOGIN;
GRANT SELECT ON public.orders, public.order_lines TO debezium;   -- for snapshots

CREATE TABLE public.debezium_heartbeat (id int PRIMARY KEY, ts timestamptz NOT NULL);
GRANT INSERT ON public.debezium_signal TO debezium;
GRANT INSERT, UPDATE ON public.debezium_heartbeat TO debezium;

CREATE PUBLICATION dbz_orders FOR TABLE
  public.orders, public.order_lines, public.debezium_signal, public.debezium_heartbeat;
```

> [!WARNING]
> If a publication publishes updates and deletes, PostgreSQL [disallows `UPDATE` and `DELETE`](https://www.postgresql.org/docs/18/sql-createpublication.html) on its tables that lack a replica identity, and `REPLICA IDENTITY DEFAULT` on a table without a primary key [behaves like `NOTHING`](https://www.postgresql.org/docs/18/sql-altertable.html). Give each captured table a primary key, `USING INDEX` or `FULL` before you publish it, or the application's writes start failing.

`FULL` logs every old column value. Use it where consumers need complete `before` images or large values in every update: PostgreSQL moves large values out of the row into TOAST storage and leaves an unchanged TOAST value out of the change unless it is in the replica identity, so Debezium sends `__debezium_unavailable_value`.

**3. The connector:**

```json
{
  "name": "orders-cdc",
  "config": {
    "connector.class": "io.debezium.connector.postgresql.PostgresConnector",
    "database.hostname": "orders-db.internal",
    "database.dbname": "orders",
    "database.user": "debezium",
    "database.password": "${fileProvider:/etc/kafka-connect/orders-db.properties:dbPassword}",
    "topic.prefix": "orders",
    "plugin.name": "pgoutput",
    "slot.name": "debezium_orders",
    "publication.name": "dbz_orders",
    "publication.autocreate.mode": "disabled",
    "table.include.list": "public.orders,public.order_lines",
    "signal.data.collection": "public.debezium_signal",
    "heartbeat.interval.ms": "60000",
    "heartbeat.action.query": "INSERT INTO public.debezium_heartbeat (id, ts) VALUES (1, now()) ON CONFLICT (id) DO UPDATE SET ts = EXCLUDED.ts"
  }
}
```

Topics are named `topicPrefix.schemaName.tableName`, here `orders.public.orders`. Give every connector its own `slot.name` (the default is `debezium`). Kafka's [FileConfigProvider](https://kafka.apache.org/43/configuration/configuration-providers/), registered on the workers as `fileProvider`, reads the password from a file and keeps it out of the connector config. The connector runs one task, whatever `tasks.max` says.

## How do you keep a replication slot from filling the disk?

A replication slot is PostgreSQL's bookmark for one consumer: slots [persist across crashes and know nothing about the state of their consumers](https://www.postgresql.org/docs/18/logicaldecoding-explanation.html), and PostgreSQL keeps the WAL and catalog rows a slot requires, even with nothing connected. Stop the connector for a long weekend, or delete it and forget the slot, and WAL piles up until the disk fills; in extreme cases the database shuts down to prevent transaction ID wraparound.

![A server drawn in cutaway holds a tall disk column that fills with stacked segments. A bookmark holds the orange upper block in place while the cable to the connector hangs unplugged.](https://computese.com/images/blog/change-data-capture/idle-slot-disk.1a483126d9-1536.webp)

*Nobody reads the slot, yet the WAL it pins keeps growing until a cap or a full disk stops it.*

A running connector holds WAL too when its tables change rarely and others change often; the heartbeat above fixes that, and its table must be in the publication. Two settings cap the damage, at the price of the slot:

- [`max_slot_wal_keep_size`](https://www.postgresql.org/docs/18/runtime-config-replication.html), added in [PostgreSQL 13](https://www.postgresql.org/docs/13/release-13.html), limits the WAL slots may retain at checkpoint time (default -1, unlimited); a slot that would need more is marked invalid.
- `idle_replication_slot_timeout`, added in [PostgreSQL 18](https://www.postgresql.org/docs/18/release-18.html), invalidates a slot inactive for longer than the timeout (default 0, off).

A lost slot is no longer usable, so recovery means a new slot and a new snapshot. Set the cap above the WAL your longest tolerable outage writes, and alert well before it:

```sql
SELECT slot_name, active, wal_status, inactive_since, invalidation_reason,
       pg_size_pretty(pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn)) AS retained_wal,
       pg_size_pretty(safe_wal_size) AS headroom
FROM pg_replication_slots
WHERE slot_type = 'logical';
```

In [`pg_replication_slots`](https://www.postgresql.org/docs/18/view-pg-replication-slots.html), `wal_status` is `reserved`, `extended`, `unreserved` (some required WAL goes at the next checkpoint) or `lost`; `safe_wal_size` is the WAL that can still be written before the slot is lost (null without a cap). Alert on growing retained WAL, on `unreserved` and on any inactive slot.

Two rules from the [Debezium PostgreSQL documentation](https://debezium.io/documentation/reference/3.7/connectors/postgresql.html):

- **One slot per connector, never dropped while in use.** When connectors share a slot, each change reaches only one of them, with no warning, and a connector's stored position exists only while its slot does.
- **Plan failovers and upgrades.** Through PostgreSQL 15, logical slots exist only on the primary; 16 can create them on replicas, synchronized by hand; 17 and later support failover slots (`slot.failover=true`, plus `synchronized_standby_slots` on the primary). For a major upgrade, block writes, let the connector drain, stop it, drop the slot, upgrade, and recreate the slot before writes resume, or the connector silently skips the changes in between.

## What do MySQL, SQL Server and Oracle need?

**MySQL.** The [MySQL connector](https://debezium.io/documentation/reference/3.7/connectors/mysql.html) reads the binlog in row format with full row images. In [MySQL 8.4](https://dev.mysql.com/doc/refman/8.4/en/replication-options-binary-log.html), binary logging is on by default, in row format, and `binlog_row_image` defaults to `full` (`binlog_format` itself is deprecated), but check your server:

```ini
# my.cnf, as in Debezium's MySQL documentation
server-id                  = 223344
log_bin                    = mysql-bin
binlog_format              = ROW
binlog_row_image           = FULL
binlog_expire_logs_seconds = 864000   # 10 days; the MySQL 8.4 default is 2592000 (30 days)
```

The connector's user needs `SELECT, RELOAD, SHOW DATABASES, REPLICATION SLAVE, REPLICATION CLIENT`; `binlog_row_value_options` must not be `PARTIAL_JSON`; on Amazon RDS, the binlog needs automated backups enabled. The connector keeps the DDL it reads from the binlog in a schema history topic for its own use. If the binlog it needs was purged while it was down, it fails and asks for a new snapshot (or takes one with `snapshot.mode=when_needed`), so keep retention longer than any outage you plan to survive.

**SQL Server.** Debezium reads the change tables that SQL Server's [change data capture](https://learn.microsoft.com/en-us/sql/relational-databases/track-changes/about-change-data-capture-sql-server?view=sql-server-ver17) fills from the transaction log through SQL Server Agent jobs; the [connector](https://debezium.io/documentation/reference/3.7/connectors/sqlserver.html) needs SQL Server 2016 SP1 or later, Standard or Enterprise. To [enable CDC](https://learn.microsoft.com/en-us/sql/relational-databases/track-changes/enable-and-disable-change-data-capture-sql-server?view=sql-server-ver17), a sysadmin runs `sys.sp_cdc_enable_db` in the database, then a db_owner runs `sys.sp_cdc_enable_table` for each table, with `@role_name` naming a gating role (created if missing) or `NULL` for none.

The cleanup job runs daily at 2 A.M. and by default keeps 4,320 minutes (3 days) of changes, so a connector stopped for longer finds its oldest changes gone. The log cannot truncate until the capture job has gathered the changes, even in the simple recovery model. A new column does not reach an existing change table: a DBA refreshes the capture table, and a second capture instance can carry the new structure beside the old one.

**Oracle.** The [Oracle connector](https://debezium.io/documentation/reference/3.7/connectors/oracle.html) needs archive log mode (Debezium's example enables it from mount state) and supplemental logging. It reads redo and archive logs through LogMiner by default, or OpenLogReplicator or XStream (`database.connection.adapter`). XStream is a commercial component of Oracle GoldenGate, so check your licence.

## What does CDC guarantee about delivery and order?

Debezium provides [at-least-once delivery](https://debezium.io/documentation/reference/3.7/configuration/eos.html): no change is missed, but some can be delivered more than once. After a crash, the connector resumes from the last offset Kafka Connect recorded and re-sends what came after it; how many repeats depends on the offset flush period and the write volume just before the crash.

Kafka Connect 3.3.0 and later, in distributed mode, can run Debezium's PostgreSQL, MySQL, SQL Server, Oracle, MariaDB and MongoDB connectors exactly once (`exactly.once.source.support=enabled` on every worker, `exactly.once.support=required` on the connector), but Debezium says it remains unclear whether that implementation is fully correct. Keep consumers idempotent either way.

Order holds per key. The record key is the primary key (`message.key.columns` overrides it), and Kafka writes records with the same key to the [same partition](https://kafka.apache.org/43/getting-started/introduction/), which every consumer reads in the order written. All changes to order 1042 arrive in order; different rows, tables and topics share no order. Truncate events have a null key, so they keep their place only on single-partition topics.

![Three parallel lanes seen from above, with envelopes moving in single file. Every envelope with a square stamp goes into the orange middle lane, and a server at the end of each lane reads it.](https://computese.com/images/blog/change-data-capture/per-key-order.8feb98873f-1536.webp)

*Order is promised per key, not per topic: each row's changes stay in one partition, in sequence.*

Idempotent consumers make repeats harmless:

- **Upsert and delete by key.** Debezium's changes are idempotent: a sequence of events always ends in the same state.
- **Skip what you have seen.** Store the last LSN applied per key and skip events at or below it, which also guards side effects such as emails. The [dead letter queue guide](https://computese.com/dead-letter-queue/) covers idempotency keys and safe replay.
- **Compact topics that copy a table.** Kafka [keeps at least the last value for each key](https://kafka.apache.org/43/design/design/) and never reorders records, but removes tombstones after `delete.retention.ms` (24 hours by default), so a consumer that lags longer misses deletes.

## How does the outbox pattern use CDC?

Table-level CDC publishes your schema, so a renamed column becomes every consumer's problem. For business events such as `OrderPaid`, the tempting code updates the database, then publishes a message: a dual write, where a failure in one of the two [might leave the data inconsistent](https://docs.aws.amazon.com/prescriptive-guidance/latest/cloud-design-patterns/transactional-outbox.html). The transactional outbox removes it: the application writes the event to an outbox table in the same transaction as the change, and a connector publishes the outbox rows. Both commit or neither does.

![An orange outline wraps an order record and an envelope on their way into a database. A connector beside the database passes only the envelope on to a queue.](https://computese.com/images/blog/change-data-capture/outbox-transaction.b48d39c573-1536.webp)

*One commit covers the order and its event, so neither can exist without the other.*

Debezium's [outbox event router](https://debezium.io/documentation/reference/3.7/transformations/outbox-event-router.html) expects this table by default:

```sql
CREATE TABLE public.outbox (
  id            uuid         PRIMARY KEY,  -- unique event ID, sent as the "id" header
  aggregatetype varchar(255) NOT NULL,     -- routes to topic outbox.event.<aggregatetype>
  aggregateid   varchar(255) NOT NULL,     -- the message key: one order's events stay in order
  type          varchar(255) NOT NULL,
  payload       jsonb
);

BEGIN;
UPDATE public.orders SET status = 'paid' WHERE id = 1042;
INSERT INTO public.outbox (id, aggregatetype, aggregateid, type, payload)
VALUES (gen_random_uuid(), 'order', '1042', 'OrderPaid',
        '{"orderId": 1042, "amount": "129.00", "currency": "CAD"}');
COMMIT;
```

Run the router in its own connector, with its own slot and a publication holding only the outbox; Debezium says a connector that applies it should capture outbox tables only.

```ini
table.include.list=public.outbox
tombstones.on.delete=false
transforms=outbox
transforms.outbox.type=io.debezium.transforms.outbox.EventRouter
transforms.outbox.table.expand.json.payload=true
value.converter=org.apache.kafka.connect.json.JsonConverter
```

The router warns on an `UPDATE` (`table.op.invalid.behavior=warn`) and filters out `DELETE` operations, so a scheduled job can remove published rows, and `tombstones.on.delete=false` keeps those deletes from emitting tombstones. It is not compatible with the MongoDB connector, which has its own router. Event-driven architecture shows where these events fit, and CDC can also keep a legacy database and its replacement in step during a strangler fig migration.

## How do you land change events in a lake or warehouse?

Land the raw events first: append every event, unchanged, to a bronze table, the raw layer of a [medallion architecture](https://computese.com/medallion-architecture/), so anything downstream can be rebuilt by replay. Then merge into a current-state table.

Debezium's [new record state extraction](https://debezium.io/documentation/reference/3.7/transformations/event-flattening.html) turns the envelope into a row: it keeps `after` and adds the metadata you list. By default it drops delete events, because most consumers cannot yet handle them; `rewrite` instead turns each delete into a row of the old values with `__deleted` set to `true`:

```ini
transforms=unwrap
transforms.unwrap.type=io.debezium.transforms.ExtractNewRecordState
transforms.unwrap.delete.tombstone.handling.mode=rewrite
transforms.unwrap.add.fields=op,table,lsn,source.ts_ms
```

Records then carry `__op`, `__table`, `__lsn` and `__source_ts_ms`. A batch can hold several changes to one row, or the same change twice, and Iceberg's `MERGE INTO` [throws an error](https://iceberg.apache.org/docs/latest/spark-writes/) when more than one source row updates a target row. Keep the latest change per key, and compare LSNs so an older event cannot overwrite newer data:

![Cards for the same rows arrive in small piles at the top. An orange funnel lets one card from each pile through into a neat table below, and the extra cards drop into a grey tray.](https://computese.com/images/blog/change-data-capture/lake-merge.c85e5607ae-1536.webp)

*Many changes per row go in and one current row comes out, so a replayed or older change never wins.*

```sql
-- Spark SQL on Iceberg
MERGE INTO lake.silver.orders t
USING (
  SELECT * FROM (
    SELECT b.*, row_number() OVER (PARTITION BY id ORDER BY __lsn DESC) AS rn
    FROM lake.bronze.orders_cdc_batch b
  ) latest
  WHERE rn = 1
) s
ON t.id = s.id
WHEN MATCHED AND s.__lsn > t.lsn THEN
  UPDATE SET t.status = s.status, t.lsn = s.__lsn, t.deleted = (s.__op = 'd')
WHEN NOT MATCHED THEN
  INSERT (id, status, lsn, deleted) VALUES (s.id, s.status, s.__lsn, s.__op = 'd');
```

Deletes become a flag, so the key keeps its LSN and a replayed older event cannot bring the row back; readers use a view that filters out `deleted`. Bronze keeps the history; where analysts need every version, model it, for example as data vault satellites. A bank's [regulatory data platform](https://computese.com/work/banking-regulatory-data/) learned this with append-only Iceberg raw tables fed partly by CDC: without de-duplication to the latest version above them, the first correction is counted twice.

Managed services can replace the connector. [AWS DMS](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Task.CDC.html) runs full load plus CDC, or CDC only; on PostgreSQL it reads through logical replication slots with the `test_decoding` plug-in, so the slot rules apply, and AWS states there are no SLAs for CDC latency, which can increase to several minutes or longer. [Google Datastream](https://docs.cloud.google.com/datastream/docs/overview) is a serverless CDC and replication service into BigQuery or Cloud Storage, and [Microsoft Fabric Mirroring](https://learn.microsoft.com/en-us/fabric/mirroring/overview) continuously replicates sources such as Azure SQL Database and Azure Database for PostgreSQL into OneLake.

## What breaks in production, and how do you fix it?

| Symptom                                    | Likely cause                                                                | Fix                                                                               |
| ------------------------------------------ | --------------------------------------------------------------------------- | --------------------------------------------------------------------------------- |
| `pg_wal` fills the disk                    | An inactive or lagging slot, or no heartbeat on a quiet database            | Fix or retire the consumer, add a heartbeat, set `max_slot_wal_keep_size`         |
| The connector asks for a new snapshot      | Its position is gone: binlog purged, slot invalidated or lost in a failover | `when_needed` snapshots, longer binlog retention, failover slots (PostgreSQL 17+) |
| `UPDATE` and `DELETE` fail on the source   | A table without a replica identity joined the publication                   | A primary key, `USING INDEX` or `FULL` before publishing                          |
| `__debezium_unavailable_value` in a column | An unchanged TOAST value is not in the change                               | `REPLICA IDENTITY FULL`, or a merge that keeps the stored value                   |
| Totals counted twice                       | Events replayed after a connector crash                                     | Upserts by key, LSN checks, de-duplication before `MERGE`                         |
| A big snapshot starts over after a failure | Initial snapshots restart from scratch                                      | `snapshot.mode=no_data`, then an incremental snapshot                             |
| Deleted rows linger in the lake            | The flattening default drops deletes, or tombstones expired                 | `rewrite` mode; consumers that keep up within `delete.retention.ms`               |
| A consumer breaks after a release          | A renamed, dropped or retyped column                                        | Additive changes, coordinated releases, the outbox for public contracts           |
| Rows missing after a major upgrade         | The connector resumed on a new slot                                         | Debezium's procedure: drain, stop, upgrade, recreate the slot before writes       |

## When should you not use change data capture?

- **Small or slow-changing tables.** A reference table of a few thousand rows that changes weekly is cheaper to copy in full each night, deletes included, with no slot or connector to run.
- **The event already exists.** If the source already publishes what you need through an API, a queue or a [webhook](https://computese.com/what-is-a-webhook/), subscribe to it.
- **Consumers need business meaning.** Rows describe storage; `OrderShipped` describes the business. Use the outbox or the application's own events.
- **No owner.** A CDC pipeline is a production system, with slots to watch, connectors to upgrade and schema changes to coordinate. Without someone on call, a nightly extract that fails loudly is safer than a stream that fails quietly.

## Roll out your first CDC pipeline

1. **Pick one flow.** One database, a few tables, one target, and a named owner on each side.
2. **Prepare the source.** Check `wal_level`, replica identities and disk headroom (PostgreSQL), binlog settings and retention (MySQL) or the edition and SQL Server Agent (SQL Server), and book any restart. Grant the minimum, and set the WAL cap, heartbeat and alerts before the connector starts.
3. **Rehearse with production-like volume.** Time the snapshot, stop the connector for an hour and watch retained WAL, kill it mid-snapshot, and count duplicates after a restart.
4. **Build the consumer for duplicates and deletes.** Raw events in bronze, `rewrite` for deletes, de-duplication by LSN and idempotent merges.
5. **Go live and reconcile.** Compare row counts and checksums between source and target daily until they have matched for several weeks.

Computese's [data platform service](https://computese.com/services/data-platform/) runs change data capture on Debezium with schema evolution handled, and the [integrations service](https://computese.com/services/integrations/) adds durable queues ordered per key, idempotent consumers and dead-letter queues with safe replay.

## Key terms
- **Change data capture (CDC)**: Capturing the inserts, updates and deletes a database commits and delivering them to other systems as a stream of change events, so targets stay current without full reloads.
- **Log-based CDC**: CDC that reads changes from the database's transaction log (the PostgreSQL WAL, the MySQL binlog, Oracle redo logs) instead of querying tables or adding triggers. It captures deletes and intermediate changes and needs no extra columns.
- **Write-ahead log (WAL)**: PostgreSQL's transaction log. With wal_level set to logical, it carries the information logical decoding needs to extract committed row changes through an output plug-in such as pgoutput. The setting takes effect only at server start.
- **Replication slot**: A PostgreSQL object that tracks how far one consumer has read and keeps the WAL it still needs, even when no client is connected. Each Debezium PostgreSQL connector needs a slot of its own.
- **Publication**: A PostgreSQL object that lists the tables whose changes the pgoutput plug-in streams. A table in a publication that publishes updates and deletes must have a replica identity, or those operations are disallowed on it.
- **Replica identity**: The table setting that decides which old values PostgreSQL logs for updated and deleted rows: the primary key (DEFAULT), a unique index (USING INDEX), every column (FULL) or nothing (NOTHING).
- **Binary log (binlog)**: MySQL's log of changes, read by replicas and by CDC tools. Debezium needs row-based logging (binlog_format=ROW) with full row images (binlog_row_image=FULL).
- **Incremental snapshot**: A Debezium snapshot that reads a table in chunks, 1,024 rows by default, while streaming continues, and resumes where it stopped after an interruption. It is started by a signal, such as a row inserted into a signalling table.
- **Tombstone**: A Kafka record with a key and a null value. Debezium emits one after each delete so that log compaction can remove every earlier record with that key.
- **Transactional outbox**: A table that an application writes events into in the same transaction as its data change. A CDC connector reads the table and publishes the events, so the change and the message cannot disagree.

## Common questions

### What is change data capture (CDC)?

Change data capture records the inserts, updates and deletes a database commits and delivers them to other systems as events. Log-based CDC reads the database's transaction log, such as the PostgreSQL WAL or the MySQL binlog, so it needs no extra columns or triggers and captures deletes. Debezium, AWS DMS and SQL Server's built-in change data capture all read the log.

### Does Debezium need Kafka?

No. Debezium is most often deployed on Kafka Connect, but Debezium Server streams changes without Kafka to systems such as Amazon Kinesis, Google Cloud Pub/Sub, Apache Pulsar or Apache Iceberg tables, and the Debezium engine runs the connectors as a library inside a Java application.

### How does change data capture handle deletes?

Debezium emits a delete event, with op set to d and the row's old values (at least its key) in before, followed by a tombstone: the same key with a null value, so Kafka compaction can drop the row. Downstream, apply the delete explicitly. Debezium's new record state extraction transform drops delete events by default, and Kafka removes tombstones after delete.retention.ms (24 hours by default), so a consumer that lags longer can miss them.

### How do I enable change data capture in SQL Server?

A sysadmin runs sys.sp_cdc_enable_db in the database, then a db_owner runs sys.sp_cdc_enable_table for each table. SQL Server Agent must be running, because its capture job reads the transaction log into change tables and its cleanup job removes entries older than 3 days by default. Debezium's SQL Server connector reads those change tables.

### What happens to PostgreSQL if the CDC connector stops?

Its replication slot stays and keeps every WAL file the connector still needs, so disk use grows until the connector catches up or the slot is removed. max_slot_wal_keep_size (since PostgreSQL 13) caps the retained WAL and idle_replication_slot_timeout (since PostgreSQL 18) invalidates idle slots. Both protect the disk at the cost of the slot, and recovery then needs a new slot and a new snapshot.

### What is the outbox pattern, and when do I need it?

It removes the dual write, where an application updates its database and then publishes a message, and a failure in one of the two can leave the data inconsistent. The application writes the event to an outbox table in the same transaction as the change, and a CDC connector publishes the outbox rows. Use it when other services need business events rather than copies of your tables.

### Is change data capture exactly once?

Not by default. Debezium documents at-least-once delivery: no change is missed, but some can be delivered more than once after a failure. Kafka Connect 3.3.0 and later can run Debezium source connectors with exactly-once support, but Debezium notes it remains unclear whether that implementation is fully correct, so keep consumers idempotent.

## Sources
1. [Debezium releases overview](https://debezium.io/releases/), Debezium
2. [Five advantages of log-based change data capture](https://debezium.io/blog/2018/07/19/advantages-of-log-based-change-data-capture/), Debezium Blog
3. [Overview of trigger behavior (PostgreSQL 18)](https://www.postgresql.org/docs/18/trigger-definition.html), PostgreSQL Documentation
4. [Track data changes (SQL Server)](https://learn.microsoft.com/en-us/sql/relational-databases/track-changes/track-data-changes-sql-server?view=sql-server-ver17), Microsoft Learn
5. [Debezium architecture (Debezium 3.7)](https://debezium.io/documentation/reference/3.7/architecture.html), Debezium
6. [Debezium Server (Debezium 3.7)](https://debezium.io/documentation/reference/3.7/operations/debezium-server.html), Debezium
7. [Sending signals to a Debezium connector (Debezium 3.7)](https://debezium.io/documentation/reference/3.7/configuration/signalling.html), Debezium
8. [Debezium connector for PostgreSQL (Debezium 3.7)](https://debezium.io/documentation/reference/3.7/connectors/postgresql.html), Debezium
9. [Write ahead log settings (PostgreSQL 18)](https://www.postgresql.org/docs/18/runtime-config-wal.html), PostgreSQL Documentation
10. [CREATE PUBLICATION (PostgreSQL 18)](https://www.postgresql.org/docs/18/sql-createpublication.html), PostgreSQL Documentation
11. [ALTER TABLE (PostgreSQL 18)](https://www.postgresql.org/docs/18/sql-altertable.html), PostgreSQL Documentation
12. [Configuration providers (Apache Kafka 4.3)](https://kafka.apache.org/43/configuration/configuration-providers/), Apache Kafka
13. [Logical decoding concepts (PostgreSQL 18)](https://www.postgresql.org/docs/18/logicaldecoding-explanation.html), PostgreSQL Documentation
14. [Replication settings (PostgreSQL 18)](https://www.postgresql.org/docs/18/runtime-config-replication.html), PostgreSQL Documentation
15. [Release 13](https://www.postgresql.org/docs/13/release-13.html), PostgreSQL Documentation
16. [Release 18](https://www.postgresql.org/docs/18/release-18.html), PostgreSQL Documentation
17. [pg_replication_slots (PostgreSQL 18)](https://www.postgresql.org/docs/18/view-pg-replication-slots.html), PostgreSQL Documentation
18. [Debezium connector for MySQL (Debezium 3.7)](https://debezium.io/documentation/reference/3.7/connectors/mysql.html), Debezium
19. [Binary logging options and variables (MySQL 8.4 Reference Manual)](https://dev.mysql.com/doc/refman/8.4/en/replication-options-binary-log.html), MySQL Documentation
20. [What is change data capture (CDC)? (SQL Server)](https://learn.microsoft.com/en-us/sql/relational-databases/track-changes/about-change-data-capture-sql-server?view=sql-server-ver17), Microsoft Learn
21. [Debezium connector for SQL Server (Debezium 3.7)](https://debezium.io/documentation/reference/3.7/connectors/sqlserver.html), Debezium
22. [Enable and disable change data capture (SQL Server)](https://learn.microsoft.com/en-us/sql/relational-databases/track-changes/enable-and-disable-change-data-capture-sql-server?view=sql-server-ver17), Microsoft Learn
23. [Debezium connector for Oracle (Debezium 3.7)](https://debezium.io/documentation/reference/3.7/connectors/oracle.html), Debezium
24. [Exactly-once delivery (Debezium 3.7)](https://debezium.io/documentation/reference/3.7/configuration/eos.html), Debezium
25. [Introduction (Apache Kafka 4.3)](https://kafka.apache.org/43/getting-started/introduction/), Apache Kafka
26. [Design: log compaction (Apache Kafka 4.3)](https://kafka.apache.org/43/design/design/), Apache Kafka
27. [Transactional outbox pattern](https://docs.aws.amazon.com/prescriptive-guidance/latest/cloud-design-patterns/transactional-outbox.html), AWS Prescriptive Guidance
28. [Outbox event router (Debezium 3.7)](https://debezium.io/documentation/reference/3.7/transformations/outbox-event-router.html), Debezium
29. [New record state extraction (Debezium 3.7)](https://debezium.io/documentation/reference/3.7/transformations/event-flattening.html), Debezium
30. [Spark writes (Apache Iceberg 1.12.0)](https://iceberg.apache.org/docs/latest/spark-writes/), Apache Iceberg
31. [Creating tasks for ongoing replication using AWS DMS](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Task.CDC.html), Amazon Web Services
32. [Datastream overview](https://docs.cloud.google.com/datastream/docs/overview), Google Cloud
33. [Mirroring in Microsoft Fabric](https://learn.microsoft.com/en-us/fabric/mirroring/overview), Microsoft Learn