Join / merge upsert · All operations

Capture changes from a source

A variant of Merge into a table

Read the rows a source has inserted, updated or deleted since the last read, from its log or change feed.

Status

In arcform

Not built yet

arcform does not do this yet. Its place on the roadmap

What other tools call it

Meridian's classification; an independent blind rating agreed on 95.7% of 324 rows (variant, version 3).

Azure Data Factory (1)
  • Change data capture resourceread 2026-09-27

    Top-level resource that runs continuously, reading changed rows from sources and writing them to targets

Bruin (1)
  • ingestr: CDC (cdc: true)read 2026-09-27

    Capture row-level changes from Postgres, MySQL family, MongoDB or SQL Server logs and merge them

Duckle (2)
  • DuckLake CDCread 2026-09-25

    change feed since the last consumed DuckLake snapshot via ducklake_table_changes

  • PostgreSQL CDCread 2026-09-25

    log-based change capture from a PostgreSQL replication slot, pgoutput decoded in Rust

Talend (1)

Background

Change data capture (CDC) is a set of software design patterns used to determine and track the data that has changed, so that action can be taken using it.[1] CDC is an approach to data integration based on capturing and delivering the changes made to enterprise data sources, and can be used for incremental loading.[1] CDC occurs often in data warehouses, since capturing and preserving the state of data across time is one of a data warehouse's core functions.[1]

When to use it

A table whose changes must be captured may have a last-changed timestamp column, and any row whose timestamp is more recent than the last capture is considered to have changed.[1] In PostgreSQL, a logical replication slot emits each change once in normal operation, but after a crash it might return to an earlier position and send recent changes again.[2] In PostgreSQL, logical decoding clients are responsible for avoiding ill effects from handling the same message more than once.[2] Apply captured changes with a step that gives the same table when a change arrives twice, such as a merge on the key, rather than a plain append. (Meridian's view)

In PostgreSQL, a replication slot prevents removal of the write-ahead log it requires even when no connection is using it, which consumes storage, so a slot no longer required should be dropped.[2] In Bruin, CDC merges apply source deletes as soft deletes by default, keeping the rows with _cdc_deleted set to true, and remove them only when hard deletes are chosen.[3] Decide whether a deletion in the source should remove the row from the target or only mark it, because an analysis of what used to exist needs the marked row. (Meridian's view)

The method

The structure, contents and use of a transaction log are specific to each database management system, and unlike data access, no standard exists for transaction logs.[1] CDC based on transaction log files has advantages including minimal impact on the database and a change stream that replays the original transactions in the order they were committed.[1] Another CDC technique uses triggers on a table to log the events that happen to it into a queue table that can later be played back.[1]

In PostgreSQL, logical decoding turns the contents of the write-ahead log into an application-specific form such as a stream of tuples or SQL statements.[2] In PostgreSQL, a logical replication slot represents a stream of changes that can be replayed to a client in the order they were made on the origin server.[2]

DuckLake's table_changes function returns the changes made to a table between two snapshots, with a change_type column of insert, update_preimage, update_postimage or delete.[4] In DuckLake's change feed, an update is split into two rows: the update_preimage holds the row before the update and the update_postimage the row after it.[4] In DuckLake's change feed, a row inserted and deleted within the same transaction is not reported.[4]

Compared with the alternatives

CDC and slowly changing dimensions (SCDs) are similar in that both methods can detect changes in a data set.[1]

Sources

  1. [1]Change data capture - Wikipediasecondary source
  2. [2]PostgreSQL: Documentation: 18: 47.2. Logical Decoding Conceptsdocumentation
  3. [3]Ingestr Assets | Bruin CLIdocumentation
  4. [4]Data Change Feed – DuckLakedocumentation