Control / error · All operations

Send failed rows aside

A variant of Handle errors and retries

Route the rows that failed a step to a separate output or dead-letter file while the rest carry on.

Status

In arcform

Not built yet

arcform does not do this yet; it would live in how a Protocol runs. Its place on the roadmap

What other tools call it

Meridian's classification; an independent blind rating agreed on 95.7% of 324 rows (variant, version 3).

AWS Glue (1)
  • ErrorsAsDynamicFrameread 2026-09-27

    Return the error records that arose while the source frame was created

Duckle (1)
Talend (1)
  • tJobRejectread 2026-09-27

    Receive the rows rejected during task processing

Background

In extract, transform, load (ETL) processing, data that fails the validation rules is rejected entirely or in part.[1] Rejected data is ideally reported back to the source system, so that incorrect records can be identified and put right.[1] In Duckle, validators split their input: passing rows continue on the main port, while failures go to a reject port for quarantine or re-processing.[2]

When to use it

In message queueing, routing messages that cannot be delivered to a dead letter queue (DLQ) enables analysis of common fault patterns and potential software problems.[3] A message can be routed to a DLQ for reasons such as having been read and rejected too many times.[3] Managing a DLQ typically involves monitoring and alert systems, and organisations may add handlers for automated remediation or manual processing of its messages.[3]

In AWS Glue, transforms such as filter and join take a stageThreshold and a totalThreshold, numbers of errors at which processing should error out, and both default to zero, meaning it does not error out.[4] A limit that is off by default lets rejected records pile up unnoticed; set one deliberately, at the share of bad rows you would not accept in a finished table. (Meridian's view)

The method

In AWS Glue, a DynamicFrame recognises malformation issues and turns malformed lines into error records that can be handled individually.[4] In AWS Glue, the errorsAsDynamicFrame method returns a DynamicFrame with the error records nested inside, so the failed records can be viewed.[4] In Duckle, the Dead Letter Queue component is a terminal sink for rejected rows, written as Parquet or CSV at a configurable path and conventionally wired to an upstream node's reject port.[5] In Talend Cloud, the tJobReject component receives data rejected after task processing.[6]

Compared with the alternatives

In message queueing, a consumer that receives a message it considers invalid can forward it to an Invalid Message Channel, keeping application-level faults apart from delivery failures.[3] Setting rows aside lets the run finish on the rows that passed, while stopping the run keeps every row out until the cause is fixed; choose the first when a partial load is still useful and the second when it would mislead. (Meridian's view)

Sources

  1. [1]Extract, transform, load - Wikipediasecondary source
  2. [2]duckle/website/docs/components.html at f85f8960edfb6efbf4cb89d58a7b32e16df0e867 · slothflowlabs/duckle · GitHubdocumentation
  3. [3]Dead letter queue - Wikipediasecondary source
  4. [4]DynamicFrame class - AWS Gluedocumentation
  5. [5]duckle/frontend/src/workflow-ui/palette-data.ts at f85f8960edfb6efbf4cb89d58a7b32e16df0e867 · slothflowlabs/duckle · GitHubdocumentation
  6. [6]Talend Cloud components | Talend Components for Jobs Helpdocumentation