Handle errors and retries

Status

Variants

Return a fault to a caller

Send a failure back as a fault message to the service or route that called the job.

Inspect errors

Open the error records or tracebacks of a run, or raise and trace error records in code.

Retry a failed step

Run a step again after a failure, up to a limit, with a backoff.

Roll back on failure

Group database steps in a transaction that commits on success and rolls back on failure.

Catch a failure and run a fallback

Wrap steps so that when one fails, a fallback branch or alternative runs instead of the run stopping.

Warn and carry on

Raise a warning, or force a successful result, without stopping the run.

What other tools call it

Meridian's classification; an independent blind rating agreed on 86.2% of 312 rows (exact class, version 2).

Background

When to use it

In extract, transform, load (ETL) processing, many transformations can raise exceptions, for example when a code translation meets an unknown code in the extracted data.^[2] Which conditions count as exceptions is decided by the programmer: division by zero may be treated as an exception, or given a behaviour such as returning zero.^[1] So the first decision is the error policy: for each check, whether a failure should stop the run, set the offending rows aside, try again, or only leave a warning. (Meridian's view)

In RapidMiner, the Handle Exception operator runs its Try subprocess and, if there is an error, runs its Catch subprocess instead of stopping the process.^[3] In RapidMiner, the Handle Exception operator also covers unexpected errors, so it is meant to be used with care.^[3] Wrap only the steps whose failure has a known, safe answer; a fallback around the whole pipeline hides the errors nobody expected. (Meridian's view) Keep retries for failures that are likely to pass, such as a busy or briefly unreachable service, and let a failure caused by the data itself stop the run or set its rows aside. (Meridian's view)

The method

Each procedure has a precondition, and an exception handling mechanism lets the procedure raise an exception when that precondition is violated, which the mechanism then handles.^[1] Software exception handling developed in the 1960s and 1970s, and many programming languages adopted it widely from the 1980s onward.^[1]

An exponential backoff algorithm reduces the rate of a process in response to adverse events, such as a failure to connect or an error response from a service.^[4] In exponential backoff, the pause before each new attempt is multiplied by a fixed amount, so a client might try again after 1 second, then 2 seconds, then 4.^[4] In Apache NiFi, the RetryFlowFile processor checks a retry count carried on each item against a configured maximum, and routes the item to an output named retries_exceeded once the count exceeds it.^[5]

A database transaction must be atomic: it either completes in its entirety or has no effect whatsoever.^[6] In KNIME, the DB Transaction End node commits the transaction if its body ran successfully, and otherwise rolls it back, returning the database to its state at the start of the transaction.^[7] In data warehousing, a big ETL process is usually split into smaller pieces, and tagging each piece with a run identifier helps roll back and rerun a failed piece.^[2]

In Talend, the tWarn component triggers a warning that is often caught by tLogCatcher, a component that collects and passes on log data.^[8]

Sources

1. [1]Exception handling - Wikipediasecondary source
2. [2]Extract, transform, load - Wikipediasecondary source
3. [3]Handle Exception - Rapidminer Documentationdocumentation
4. [4]Exponential backoff - Wikipediasecondary source
5. [5]RetryFlowFile - Apache NiFidocumentation
6. [6]Database transaction - Wikipediasecondary source
7. [7]DB Transaction Enddocumentation
8. [8]Logs and errors (Integration) components | Talend Components for Jobs Helpdocumentation

Used in