Control / flow · All operations

Run branches in parallel

A variant of Branch or loop

Run several branches or partitions of the data at the same time and gather their results.

Status

In arcform

Not built yet

arcform does not do this yet; it would live in how a Protocol runs. Its place on the roadmap

What other tools call it

Meridian's classification; an independent blind rating agreed on 95.7% of 324 rows (variant, version 3).

Apache NiFi (1)
  • DistributeLoadread 2026-09-27

    Distribute FlowFiles across downstream relationships by a strategy such as weighted round robin

Duckle (1)
  • Parallelizeread 2026-09-25

    run the downstream branches concurrently over one input snapshot

KNIME Analytics Platform (1)
RapidMiner Studio (1)
Talend (4)
  • tCollectorread 2026-09-27

    Feed the parallel execution processes with the threads tPartitioner generated

  • tDepartitionerread 2026-09-27

    Assemble the outputs of the parallel execution processes for tRecollector

  • tParallelizeread 2026-09-27

    Run several subJobs at once and synchronise another subJob with them

  • tRecollectorread 2026-09-27

    Output the results of parallel execution that tDepartitioner assembled

Background

In the fork–join model, execution branches off in parallel at designated points in a program and joins at a later point to resume sequential execution.[1] The fork–join model can be considered a parallel design pattern, and it was formulated as early as 1963.[1] RapidMiner's Subprocess (Parallel) operator runs several embedded processes in parallel and awaits the completion of all of them before continuing.[2]

When to use it

Run branches in parallel when they do not depend on each other and the machine or service has room for more than one at a time; otherwise the branches only queue for the same resource. (Meridian's view) A race condition is when a system's behaviour depends on the sequence or timing of uncontrollable events, leading to unexpected or inconsistent results.[3] Race conditions can occur especially in concurrent or distributed software programs.[3] Keep parallel branches from writing to the same table or file, or give each its own output and combine them after the join. (Meridian's view) Inputs connected to RapidMiner's Subprocess (Parallel) are visible to all its embedded processes, and several processes can safely work on the same input.[2]

The method

Implementations of the fork–join model typically fork lightweight tasks rather than operating-system threads or processes, and run them on a pool of threads.[1] Task parallelism runs many different tasks at the same time on the same data, while data parallelism runs the same task on different parts of the data.[4] A data-parallel sum of n elements on 4 processors takes about a quarter of the sequential time, plus the overhead of merging the results.[5]

In Talend, tPartitioner partitions the input data, tCollector feeds the parallel execution processes, tDepartitioner assembles their outputs and tRecollector outputs the results.[6] Apache NiFi's DistributeLoad processor distributes FlowFiles, NiFi's single pieces of data, to downstream processors by a distribution strategy.[7][8] In DistributeLoad's round robin strategy, each destination has a weighting of 1 by default, so FlowFiles are evenly distributed.[7]

Compared with the alternatives

A common type of task parallelism is pipelining, which moves a single set of data through a series of separate tasks that can each run independently of the others.[4] Parallel branches are task parallelism; splitting one table into chunks and running the same steps on each is data parallelism, and the two answer different bottlenecks. (Meridian's view)

Sources

  1. [1]Fork–join model - Wikipediasecondary source
  2. [2]Subprocess (Parallel) - Rapidminer Documentationdocumentation
  3. [3]Race condition - Wikipediasecondary source
  4. [4]Task parallelism - Wikipediasecondary source
  5. [5]Data parallelism - Wikipediasecondary source
  6. [6]Orchestration (Integration) components | Talend Components for Jobs Helpdocumentation
  7. [7]DistributeLoad - Apache NiFidocumentation
  8. [8]Apache NiFi User Guidedocumentation