Publish / describe · All operations

Document pipelines and their lineage

A variant of Describe a dataset

Generate documentation of pipelines, or rows of lineage saying which tables and assets each pipeline reads and writes.

Status

In arcform

Not built yet

arcform does not do this yet. Its place on the roadmap

What other tools call it

Meridian's classification; an independent blind rating agreed on 95.7% of 324 rows (variant, version 3).

Apache Hop (2)
  • Build documentationread 2026-09-27

    Generate documentation for the pipelines and workflows of a project

  • Database impact inputread 2026-09-27

    Emit the database tables and fields each pipeline reads or writes, as lineage rows

Bruin (5)
  • Dashboard assets (family)read 2026-09-27

    15 BI tool types (tableau, looker, metabase, powerbi, superset, …): no-op assets kept for lineage

  • Source assets (family)read 2026-09-27

    14 platform types (bq.source, pg.source, …): no-op assets documenting existing tables for lineage

  • quicksight.dashboardread 2026-09-27

    Represent a QuickSight dashboard; no-op, for lineage

  • tableau.dashboardread 2026-09-27

    Represent a Tableau dashboard; no-op, for lineage

  • tableau.worksheetread 2026-09-27

    Represent a Tableau worksheet; no-op, for lineage

Pentaho Data Integration (1)
RapidMiner Studio (1)
  • Scan Processesread 2026-09-27

    Parse the stored processes under a repository folder and list every operator they use

Background

Data lineage is the tracking of how data is generated, transformed, transmitted and used across systems over time.[1] Data lineage documents the origins, transformations and movements of data, giving visibility into its life cycle.[1] Data lineage makes errors in data analytics workflows easier to find, by letting users trace issues back to their root causes.[1]

When to use it

Lineage information underpins impact analysis and debugging of data pipelines, and supports regulatory reporting and audit.[1] A backward tracing query finds the inputs that derived an output and is useful for debugging, while a forward tracing query finds the outputs an input produced and is useful for tracking error propagation.[1] Ask backward questions before trusting a surprising number, and forward questions before changing a table other work reads. (Meridian's view)

The method

Lineage captured for a whole job over its files is called coarse-grain lineage, and lineage captured for each task over its records is called fine-grain lineage.[1] Eager lineage systems capture the entire lineage of a data flow at run time, while lazy systems typically capture only coarse-grain lineage and replay the data flow to answer fine-grain tracing queries.[1] Eager fine-grain lineage collection costs more to capture than lazy collection, but enables sophisticated replay and debugging.[1]

OpenLineage is an open standard for lineage metadata collection, designed to record metadata for a job in execution.[2] The OpenLineage standard defines a generic model of dataset, job and run entities, extended through facets of user-defined metadata.[2]

Apache Hop's Database impact input transform takes the file names of pipelines and reports which transforms read, write, update or delete information, and in which table.[3] RapidMiner's Scan Processes operator parses each stored process under a repository location and returns every operator in it, in order of execution, with its name, type and whether it is enabled.[4] Apache Hop's Build documentation action, a marketplace plugin rather than part of the standard Hop client, builds documentation of a project's Hop files into a target folder.[5] In Bruin, dashboard assets represent the reports and dashboards fed by a pipeline, to keep an accurate overview of the pipeline's lineage.[6]

Compared with the alternatives

In industry usage, data lineage typically denotes the end-to-end flow of datasets and transformations across systems, while data provenance emphasises the derivations and attribution of specific data items, and the two are complementary.[1] Reading pipeline definitions tells you what a pipeline is written to touch, which suits impact analysis before a change; capturing lineage at run time tells you what one run actually did, which suits debugging an output. (Meridian's view)

Sources

  1. [1]Data lineage - Wikipediasecondary source
  2. [2]About OpenLineage | OpenLineagedocumentation
  3. [3]Database impact input | Apache Hopdocumentation
  4. [4]Scan Processes - Rapidminer Documentationdocumentation
  5. [5]Build documentation | Apache Hopdocumentation
  6. [6]Dashboard Assets | Bruin CLIdocumentation