Operations guide
You know what is wrong with your data, or what you want from it, but not what the step is called. Find your problem here, work out which version of it you have, and follow it to the operations that answer it.
How operations are arranged
We arrange operations by the stages a dataset passes through on its way from a source to a result. There are five, in the order a pipeline meets them:
- Acquire: reading the data in, from files, databases and the web.
- Shape: cleaning, inspecting and rearranging one table.
- Join: bringing two or more tables together.
- Model: analysing the data or learning from it.
- Publish: sending the result out, as a file, a table or a message.
Control sits outside the five. Its operations run the pipeline around the data: they order and repeat steps, handle errors, set parameters, run scripts and commands, test the pipeline and log what happened, rather than changing the data.
Within a stage, we group operations by the shape of what you get back, compared with the table you started from:
- A new table, from no input table.
- One row out per row in, with columns added or changed.
- Fewer rows, same columns.
- More rows, each traceable to one input row.
- A new grain: each row stands for a group of input rows.
- The same cells, rearranged.
- One table from two or more.
- A table describing the input rather than holding its rows.
- A model or a file, not a table.
- A model or file and a table in, the table out with columns added.
- The table sent out, with nothing new to look at.
Once you can say which stage you are in and what shape you want back, the operation follows. Say you want the repeated rows of a table gone: you are shaping one table, and you want fewer rows back with the same columns. That leaves a handful of operations, and Deduplicate rows is the one.
Start from your problem
Each problem below comes in more than one version. Find the sentence that is true of your data, and follow it to the operations that answer it, in the order you would run them.
My customer list has the same company three times.
Repeated companies are at least three problems, and each needs a different fix.
The three rows match exactly, cell for cell, or on the key you care about.
The rows are one company whose name is spelt differently, such as Acme Ltd and ACME Limited, and otherwise agree.
The records describe one company in different ways, with no shared key and different details in each.
I've been sent a file and need it as a table.
Which reader you need depends on how the file lays out its values.
It is text with a separator between the fields, such as a comma, a tab or a pipe.
It is text whose columns sit at fixed character positions, with nothing between them.
It is a workbook from a spreadsheet program, such as an .xlsx file.
It is a .zip, .tar or .gz file with the data inside it.
I don't know what's in this file, or whether to trust it.
Before you change anything, you can ask what the file holds, and then whether it holds what it should.
You don't yet know its columns or what type each one is.
You know the columns, but not how many values are blank or distinct in each.
You know what the data should look like and want each rule checked, such as no blank customer IDs.
My dates won't sort or filter as dates.
The column holds dates in a form the table treats as something else, and the form tells you the fix.
The dates are text, such as 03/11/2026, and you know how they are written.
The values are large whole numbers, such as 1762128000, counting the time since 1 January 1970.
Every column came in as text, dates among them, and you want each set to the type its values fit.
My table is the wrong way round for what I need.
The cells are all there; what changes is whether months, or any repeated category, run across the columns or down the rows.
You have a column for each month and want one row for each month, with the month as a value.
You have a row for each month and want a column for each month, as in a cross-tab.
You want the rows and columns swapped wholesale, each row becoming a column.
My two tables should join, but the keys don't match.
Keys fail to match for different reasons, and each reason has its own kind of join.
The keys are the same values written differently, with stray spaces or different capitals.
The keys are close but not equal, such as slightly different spellings of a name.
One side holds a range, such as a price's start and end dates, and the other a value that falls inside it.
You want each row matched to the latest row of the other table at or before its time, such as a trade and the rate in force.
I need to know what changed between last month's file and this month's.
You may need a yes or no, every difference, or only the rows that came or went.
You only need to know whether the two files hold the same data.
You want every difference listed, cell by cell or column by column.
You want the rows whose key is in one file and not the other.
I need to write this month's results into a table that holds the earlier months.
Writing into a table that already has rows means deciding what happens to the rows already there.
Some of the rows are already in the table and should be updated, and the rest are new.
The new rows cover whole months, which should replace those months in the table.
You must keep the old version of each changed row as well as the new one.
The table should hold only the latest result, with the earlier months gone.
I want to put my customers into groups.
Whether the groups are already in your data decides whether you summarise or model.
The groups are already a column, such as region or plan, and you want figures for each.
There is no such column, and you want customers who are alike put together.
I want to know what next month's numbers will look like.
A forecast reads a series of periods, so what matters first is whether your rows already are one.
You have one row for each period, such as each week's sales, with none missing.
You have one row for each period, but some periods are missing.
Your rows are individual events, such as orders, with a time on each.
I need to hand the result to someone else.
The right file depends on what they will open it with.
They will open it in a spreadsheet program.
Any tool should be able to read it, and plain text is enough.
They will read it with another data tool, and the column types should arrive intact.
Others need to know what my dataset holds and where it came from.
Where the description goes depends on where they will look for it.
They will read a file beside the data, such as a data dictionary or a schema.
They look datasets up in a data catalogue service your organisation runs.
They need to know which pipelines read and write each table.
Operations by stage
Each stage's page lists its operations by the shape of the result, each with its status in arcform today.
Acquire15 operations
Reading the data in, from files, databases and the web.
Shape51 operations
Cleaning, inspecting and rearranging one table.
Join15 operations
Bringing two or more tables together.
Model37 operations
Analysing the data or learning from it.
Publish17 operations
Sending the result out, as a file, a table or a message.
Control14 operations
Running the pipeline around the data, rather than changing the data.
Arcform
Arcform runs a data pipeline from one arcform.yaml. It reads each SQL step to order the steps, and skips the ones whose inputs and SQL have not changed.
Dovetail
Dovetail prints the DuckDB SQL that loads an unfamiliar file, then finds which columns join which tables, and checks each candidate key against the data.