Test a pipeline

Status

What other tools call it

Meridian's classification; an independent blind rating agreed on 86.2% of 312 rows (exact class, version 2).

Background

When to use it

In dbt, unit tests validate Structured Query Language (SQL) modelling logic on a small set of static inputs before the full model is materialised in production.^[3] dbt Labs recommends a unit test when a model's SQL contains complex logic, such as regular expressions, date maths, window functions or a CASE WHEN with many branches.^[3] dbt Labs also recommends one for logic that has had bugs reported before, for edge cases not yet seen in the actual data, and before refactoring transformation logic.^[3] Test the logic you wrote rather than the engine underneath it: the business rule, the date arithmetic, the join that decides which rows survive. (Meridian's view)

Apache Hop unit tests can speed up development when input data does not exist yet, is still in development, or sits in a source system with no direct access.^[1]

Regression testing re-runs tests after a change to ensure that previously developed and tested software still performs as expected.^[4] Apache Hop unit tests support regression testing, to make sure that old issues that were once fixed remain fixed.^[1] In most software development, it is considered good coding practice to record a test that exposes a fixed bug and re-run it regularly after later changes.^[4]

In Apache Hop, a transform is a single unit of data processing inside a pipeline.^[5] In Apache Hop, the transform a golden data set is attached to is not executed, so checking a Calculator transform's output means attaching the golden data set to the transform after it.^[1] Unit testing only tests the functionality of the units themselves, so it will not catch integration errors or broader system-level errors such as performance.^[2] If a unit test's initial conditions are not set correctly, the test does not exercise the code in a realistic context, which diminishes the value of its results.^[2] Unit tests are harder to create when code mainly interacts with something external, and a mock-up of database interactions will probably not be as comprehensive as the real ones.^[2] Build a test's input rows to include the awkward cases, such as the null, the duplicate key and the date at the end of a month, because a fixture of tidy rows proves little. (Meridian's view) A golden data set captured from the pipeline's own run records what the pipeline did, not what it should have done, so read it closely before trusting it. (Meridian's view)

The method

In software testing, a test oracle provides the correct output for a test case's input, and testing compares the actual results with the oracle's expected results.^[6] The term test oracle was first introduced in a paper by William E. Howden.^[6] A test oracle can be used before a test is run, with the expected results encoded into the test logic.^[6] Regression test suites are an example of a derived test oracle, built on the assumption that a previous version's result can serve as the oracle for a future version.^[6]

In 1964, the software of the Mercury project was described as having individual units, developed by different programmers, undergo unit tests before being integrated.^[2] In 1989 Kent Beck described a testing framework for Smalltalk, later called SUnit, and in 1997 Kent Beck and Erich Gamma released JUnit.^[2]

In Apache Hop, a data set is a set of rows with a certain layout, stored as comma-separated values (CSV), called an input data set when fed in and a golden data set when used to validate output.^[1] An Apache Hop unit test combines a pipeline with zero or more input and golden data sets and with tweaks, which remove a transform or bypass it during the test.^[1] When an Apache Hop unit test runs, each transform marked with an input data set is replaced by an Injector transform that injects the data set's static rows instead.^[1] In an Apache Hop unit test, each transform marked with a golden data set is replaced by a Dummy transform, which does nothing, so the test does not write to a file or a database.^[1] Apache Hop makes a unit test's replacements on a copy of the pipeline, held in memory unless a file location is given, and the original pipeline on disk is not changed.^[1] After an Apache Hop unit test runs, the rows that flowed into each golden-data-set transform are compared with the golden data set and the result is logged.^[1] An Apache Hop unit test fails on a row count that differs from the golden data set's, or on a row and field whose value does not correspond to the data set's value.^[1] Apache Hop compares Number fields as strings formatted to the field's length and precision, so values that round to the same string are considered equal.^[1] Where length or precision is not set, Apache Hop compares numbers with a tolerance of 0.000001.^[1] An Apache Hop unit test can swap a database connection for a test-specific one on the in-memory test copy, without changing the project connection.^[1] Apache Hop's Write rows to data set action runs the current pipeline and writes its data to a data set.^[1]

Apache Hop's Run pipeline unit tests workflow action runs a series of selected pipeline unit tests, and is successful if all of them run without error.^[7] Apache Hop's Execute unit tests transform runs the current project's unit tests and returns the tested pipeline, the golden data set used and an error status of Y or N as output fields.^[8]

In Talend, the tCollectAndCheck component receives the output of a component being tested, loads reference files and returns whether the output matches the expected result.^[9] Talend's tCollectAndCheck is available only in a test case about a given component, and is added automatically to that test case.^[9]

Compared with the alternatives

In dbt, data tests assess the quality of input data or the structure of resulting datasets, and historically could only be executed after building a model.^[3] A unit test asks whether the pipeline's logic is right on inputs chosen in advance; a data test asks whether the data that actually arrived is fit to use, and a pipeline that matters needs both. (Meridian's view)

Integration with peripheral units belongs in integration tests rather than unit tests, and an elaborate hierarchy of unit tests does not equal integration testing.^[2]

In Talend, the tAssert component evaluates an assertive statement about a Talend Job's execution and reports Ok when it succeeds, or Fail when the result does not match the expectation or an error occurred.^[10]

Sources

1. [1]Pipeline Unit Testing | Apache Hopdocumentation
2. [2]Unit testing - Wikipediasecondary source
3. [3]Unit tests | dbt Developer Hubdocumentation
4. [4]Regression testing - Wikipediasecondary source
5. [5]Concepts | Apache Hopdocumentation
6. [6]Test oracle - Wikipediasecondary source
7. [7]Run pipeline unit tests | Apache Hopdocumentation
8. [8]Execute unit tests | Apache Hopdocumentation
9. [9]tCollectAndCheck | Talend Components for Jobs Helpdocumentation
10. [10]tAssert | Talend Components for Jobs Helpdocumentation