Control / subflow · All operations

Reuse steps inside the flow

Embed a saved group of steps, a sub-pipeline or a named subquery in the flow so rows pass through it as if its steps were written in place.

Status

In arcform

Not built yet

arcform does not do this yet; it would live in how a Protocol runs. Its place on the roadmap

What other tools call it

Meridian's classification; an independent blind rating agreed on 95.7% of 324 rows (variant, version 3).

Apache DataFusion (3)

Common Table Expressions (CTE)read 2026-09-27
Name a subquery in a WITH clause and reference it in the query
Subqueriesread 2026-09-27
Use a query inside SELECT, FROM, WHERE or HAVING
WITH RECURSIVEread 2026-09-27
Recursive common table expression

Apache Hop (2)

Multi Mappingread 2026-09-27
Run a reusable child pipeline with any number of input, output, info and target streams
Simple Mappingread 2026-09-27
Run a reusable sub-pipeline with one input and one output inside the pipeline

Apache NiFi (1)

Process Groupread 2026-09-27
Group of components handled as one unit of the dataflow

Azure Data Factory (1)

Flowletread 2026-09-27
Include a reusable block of transformation logic

dbt-utils (1)

ephemeralread 2026-09-27
Build nothing; interpolate the model into dependent models as a CTE

Pentaho Data Integration (3)

Mappingread 2026-09-27
Run a sub-transformation whose interface is set by mapping input and output steps
Simple Mapping (sub-transformation)read 2026-09-27
Wrap a reusable sequence of steps as a sub-transformation
Single Threaderread 2026-09-27
Run a sequence of steps in a single thread

RapidMiner Studio (4)

Subprocessread 2026-09-27
Group a chain of operators into a nested subprocess
Subprocess (Caching)read 2026-09-27
Subprocess whose results are cached and reused on later runs
Subprocess (In Database)read 2026-09-27
Group a chain of in-database operators into a nested subprocess
Work on Subsetread 2026-09-27
Run a nested subprocess on a chosen subset of columns and merge the result back

SQLMesh (2)

@WITHread 2026-09-27
Include a common table expression WITH clause only when the argument is true
EMBEDDEDread 2026-09-27
Build nothing; inject the query as a subquery into each downstream model

Tableau Prep (1)

Insert Flowread 2026-09-27
Insert steps saved from another flow as a reusable fragment

Background

In Azure Data Factory (ADF), a flowlet is a reusable container of activities that can be created from an existing mapping data flow or started from scratch.^[1] In ADF, reusing a flowlet prevents logic duplication and applies the same logic across many mapping data flows.^[1] In Apache Hop, the Simple Mapping transform re-uses a series of transforms in the form of a re-usable pipeline, also known as a mapping.^[2]

When to use it

Apache Hop's Simple Mapping transform is meant for logic repeated multiple times over several different pipelines.^[2] In Apache Hop, a reusable piece that needs more than one input or output calls for the Multi Mapping transform rather than Simple Mapping.^[2]

In dbt, an ephemeral model is not built into the database; dbt interpolates its code into dependent models as a common table expression (CTE).^[3] In dbt, an ephemeral model cannot be selected from directly, and overuse of ephemeral materialisation can make queries harder to debug.^[3] dbt suggests the ephemeral materialisation for very light-weight transformations that are used in only one or two downstream models and need not be queried directly.^[3] In SQLMesh, an EMBEDDED model has no table or view in the warehouse, and its query is injected as a subquery into each downstream model that references it.^[4]

In Tableau Prep Builder, reusable steps saved with a parameter keep that parameter as a static value taken from its current value.^[5] In Tableau Prep Builder, saved steps inserted between existing steps may need to be moved into place and any errors fixed.^[5] After inserting a saved fragment, check the columns it expects against the columns where it lands. (Meridian's view)

The method

In Structured Query Language (SQL), a CTE is a temporary named result set defined within the execution scope of a SELECT, INSERT, UPDATE or DELETE statement.^[6] In Apache DataFusion, a WITH clause defines CTEs that can be referenced by name in the rest of the query.^[7] CTEs can be thought of as alternatives to derived tables (subqueries), views and inline user-defined functions.^[6]

In ADF, a flowlet's input defines the columns it expects, and the calling mapping data flow maps columns from its stream onto them.^[1] In Apache Hop, the Simple Mapping transform's Input tab names the fields passed into the mapping, field names can be mapped, and its Output tab renames the fields returned.^[2]

In RapidMiner, when a Subprocess operator is reached, all operators inside it are executed, and the flow then returns to the parent process.^[8] In RapidMiner, nested subprocesses execute in the same order as a depth-first search through a tree.^[8]

Compared with the alternatives

In Apache Hop, data entering a mapping's Mapping output transform is streamed to the parent pipeline, whereas the Workflow action executes a separate workflow as a child of the current one.^[9]^[10]

Sources

[1]Flowlets in mapping data flows - Azure Data Factory & Azure Synapse | Microsoft Learndocumentation
[2]Simple Mapping | Apache Hopdocumentation
[3]Materializations | dbt Developer Hubdocumentation
[4]sqlmesh/docs/concepts/models/model_kinds.md at 23b237cea7218cc8f338b89c14f0957394a0003b · SQLMesh/sqlmesh · GitHubdocumentation
[5]Copy steps, actions and fieldsdocumentation
[6]Hierarchical and recursive queries in SQLsecondary source
[7]SELECT syntax — Apache DataFusion main documentationdocumentation
[8]Subprocess - Rapidminer Documentationdocumentation
[9]Mapping output | Apache Hopdocumentation
[10]Workflow | Apache Hopdocumentation