Control operations
Operations that run the pipeline around the data, rather than changing the data. Among them are branching, looping and waiting, handling errors and retries, setting parameters, running scripts and commands, calling a reusable workflow, testing the pipeline and logging what happened. Control sits outside the five stages a dataset passes through: none of its operations is a data operation.
14 operations, each with its status in arcform today.
Not a data operation
Create or change tablesRuns as a SQL step
Create, alter or maintain tables and indexes (2 variants with a page)
Plan and run queriesA feature, not a step
Query planning, optimisation, memory, caching, partitioning, parallelism (2 variants with a page)
Handle errors and retriesA feature, not a step
Error handling, retry, abort, dead-letter routing (2 variants with a page)
Run an external commandA feature, not a step
Run an external job or process, trigger another pipeline, version control (6 variants with a page)
Branch or loopNot built yet
Branch, loop, iterate, conditional execution, wait (10 variants with a page)
Undo and restoreA feature, not a step
Undo, redo, snapshot, freeze, replay a recorded path
Run a step inside a databaseNot built yet
Push a step down into a database rather than the tool's own engine
Build an app interfaceNot built yet
User-interface widgets for building apps on a workflow
Log and annotate a runA feature, not a step
Logging, audit, metrics, annotations, comments (3 variants with a page)
Set parametersA feature, not a step
Variables, parameters, context, configuration (8 variants with a page)
Check before runningA feature, not a step
Check that a service, file or connection is available before running
Run a scriptA feature, not a step
Run user code: Python, R, JavaScript, Java, shell (6 variants with a page)
Call a reusable workflowNot built yet
Call a sub-workflow, macro, mapping or reusable fragment (4 variants with a page)
Test a pipelineNot built yet
Unit-test a pipeline against expected output