Arcform

A data job is rarely one query. You download a few files, load them into DuckDB, model them in SQL and export the result, and a Makefile or a folder of shell scripts runs the lot. Nothing in that folder knows which step reads what another one wrote, so after a one-line change you run everything again, downloads included.

Arcform runs the same chain from one file, arcform.yaml. It works out the order of the steps from what each SQL step reads and writes, and on the next run it skips the steps that have nothing new to do. The arcform.yaml and the SQL files beside it are a Protocol: the whole pipeline, which you can read and run again.

What Arcform does

Arcform's repository carries a reference Protocol, examples/brewtrend, which ranks Homebrew packages by their install counts. It downloads six files from Homebrew's API (its two package catalogues and four sets of install counts), loads them into DuckDB, scores each package by how its daily installs in the two months before the last 30 days compare with its daily installs in the last 30 days, ranks the packages on that score and on their installs in the last 30 days, and exports the ranking as Parquet. Every example on this page runs it. Its arcform.yaml, abridged:

# examples/brewtrend/arcform.yaml (abridged)
name: brewtrend
engine: duckdb
engine_version: ">=2.0"
db: trends.db

params:
  trend_threshold:
    default: "10"

steps:
  # Download one file, at most once a day
  - name: fetch_cask_30d
    op: http_fetch@1
    with:
      url: https://formulae.brew.sh/api/analytics/cask-install/homebrew-cask/30d.json
      out: data/cask/30d.json
    preconditions:
      - modified_after: { path: data/cask/30d.json, period: 24h }
  # … five more downloads like it …

  - name: load
    sql: models/load_data.sql
    depends_on:
      - data/cask/30d.json
      # … and the other five files …

  - name: trending
    sql: models/trending.sql

  - name: rank
    sql: models/full_ranking.sql

trending and rank name no step they depend on. Arcform reads each SQL file and finds the tables it reads and writes: load_data.sql creates casks, core and brew_analytics, trending.sql reads them, and full_ranking.sql reads what trending.sql wrote and ends by writing the file:

-- models/full_ranking.sql (its last statement)
COPY ranking TO 'data/ranking.parquet';

So the order comes from the SQL, not from a list you keep by hand. At the end of a run Arcform prints that graph. Here is its entry for casks from a run on 3 November 2026; the row count moves as Homebrew's catalogue does:

  casks [table, 7805 rows]
      produced by  load
      feeds        trending, rank

Each download carries a precondition, modified_after with a period of 24h, so a file fetched in the last day isn't fetched again. A SQL step whose SQL and inputs haven't changed since its last run is skipped. You'll see both under Use it, where the second run skips nine of the ten steps.

The example also carries a Data Package descriptor, datapackage.json, describing the Parquet file it writes: its fields, types and source. The descriptor describes the data; arc run runs arcform.yaml.

What running a Protocol costs you

Arcform runs your steps on your machine, as you. There is no sandbox and no container, and that is deliberate. Running a Protocol you did not write is equivalent to running a shell script you did not read.

A step's command: runs as sh -c <command>. So does a command: on a precondition or a hook, and a precondition's runs while Arcform works out what is stale, even when the step it guards is then skipped. A sql: step is passed to the DuckDB CLI, which reads and writes whatever the SQL directs. An op: step runs a catalog operator inside arc, or a script arc embeds: no shell, and no confinement either. Each of these inherits your environment, and arc registry run fetches a Protocol and runs it by the same path.

So read a Protocol's arcform.yaml before you run one someone sent you. Its command:, sql: and op: lines are what will run as you.

Installation

curl -fsSL https://install.meridian.online/arc | bash

This installs a release build for macOS (Apple silicon and Intel) or Linux (x86_64 and aarch64, glibc 2.35 or later) and links arc into ~/.local/bin. To pin a release, pass its tag: curl -fsSL https://install.meridian.online/arc | bash -s -- v0.2.2. The README has the build from source.

Arcform runs a Protocol's SQL steps with the DuckDB CLI, so duckdb must be on your PATH at run time: v2.0.0 recommended, >=2.0, <3 accepted. It must be a build that includes DuckDB's autocomplete extension, as DuckDB's release build does. When no duckdb is on your PATH, or the one there lacks that extension, the installer goes on to install DuckDB's v2.0.0 release build into ~/.duckdb/cli/2.0.0 and says what to put on PATH; set ARC_SKIP_DUCKDB=1 to leave DuckDB alone. A 1.x duckdb is refused, and Arcform v0.2.1 is the release for DuckDB 1.5. The Finetype steps need finetype 0.6.54 or later, and the Python operators need uv.

Use it

Clone Arcform's repository and go to the example:

git clone https://github.com/meridian-online/arcform
cd arcform/examples/brewtrend

Run the Protocol. It downloads the six files into data/, loads them into trends.db, ranks the packages, writes data/ranking.parquet and prints the top twenty:

arc run

It prints a line as each of the ten steps starts, [1/10] fetch_cask_catalogue ... to [10/10] show_top20 ..., then the table. This is the top of the table it printed on 3 November 2026; the figures are Homebrew's, and change daily:

| rank | category |       name       | installs_30d | trend |
|-----:|----------|------------------|-------------:|------:|
| 1    | formula  | gh               | 292812       | 0.774 |
| 2    | formula  | node             | 275544       | 0.537 |
| 3    | formula  | uv               | 204269       | 0.821 |
| 4    | cask     | codex            | 258471       | 0.446 |
| 5    | formula  | azure-cli        | 3466         | 7.647 |
| 6    | formula  | uutils-findutils | 510          | 8.86  |
| 7    | formula  | git-flow         | 340          | 7.894 |
| 8    | formula  | awscli           | 155487       | 0.98  |

The rest of the table follows, then the graph of every table and file the run made, and the last line, ✓ 10/10 steps succeeded.

Run it again:

arc run

This time the downloads are less than a day old and no SQL has changed, so only the last step, which prints the table, runs. The output ends ✓ 1/10 steps succeeded, 9 skipped (fresh)., and it starts:

[1/10] fetch_cask_catalogue [skip: precondition_modified_after]
[2/10] fetch_formula_catalogue [skip: precondition_modified_after]
[3/10] fetch_cask_30d [skip: precondition_modified_after]
[4/10] fetch_cask_90d [skip: precondition_modified_after]
[5/10] fetch_formula_30d [skip: precondition_modified_after]
[6/10] fetch_formula_90d [skip: precondition_modified_after]
[7/10] load [skip: hash_clean]
[8/10] trending [skip: hash_clean]
[9/10] rank [skip: hash_clean]
[10/10] show_top20 ...

trend_threshold is how many installs a day, averaged over the last 30 days, a package needs for a trend score; below it the score is 0. Raise it to 50 with --param. In Arcform v0.2.2 a changed parameter doesn't by itself make a step run again, so add --force, which runs every step:

arc run --force --param trend_threshold=50

In that run, uutils-findutils and git-flow averaged fewer than 50 installs a day, so their trend score became 0 and they dropped out of the first eight rows. This is the top of the table that run printed:

| rank | category |       name       | installs_30d | trend |
|-----:|----------|------------------|-------------:|------:|
| 1    | formula  | gh               | 292812       | 0.774 |
| 2    | formula  | node             | 275544       | 0.537 |
| 3    | formula  | uv               | 204269       | 0.821 |
| 4    | cask     | codex            | 258471       | 0.446 |
| 5    | formula  | azure-cli        | 3466         | 7.647 |
| 6    | formula  | awscli           | 155487       | 0.98  |
| 7    | formula  | lcov             | 2342         | 6.797 |
| 8    | formula  | ffmpeg           | 136186       | 1.102 |

Commands

Each command below, except arc init, runs in the brewtrend folder after the runs above.

arc init scaffolds a new Protocol in a folder of its name: an arcform.yaml with no steps yet, with models/ and sources/ folders beside it. Run it outside the brewtrend folder, such as in the folder you cloned arcform into:

arc init my_pipeline

Initialized project 'my_pipeline' with:
  arcform.yaml
  models/
  sources/
  .gitignore
  .gitattributes

For a complete, runnable example — SQL + command steps, preconditions,
retries, and parameters — see examples/brewtrend in the arcform repo.

arc run runs the Protocol in the current folder. --force runs every step, fresh or not, and --param KEY=VALUE sets a parameter; repeat it for several.

arc table describe prints a table's row count, its columns with their types and null counts, and up to five rows, as JSON. Here jq keeps the row count and the column names, as they were after the runs above:

arc table describe ranking | jq -c '{row_count, columns: [.columns[].name]}'
# → {"row_count":18977,"columns":["rank","category","name","installs_30d","trend_index","description","homepage"]}

arc sql query runs read-only SELECT statements against the Protocol's database and prints what they return as JSON. Here jq keeps the rows:

arc sql query 'SELECT rank, name, installs_30d FROM ranking ORDER BY rank LIMIT 3' | jq -c '.results[0].rows[]'
# → [1,"gh",292812]
# → [2,"node",275544]
# → [3,"uv",204269]

arc mcp serves arc to an AI agent, as the next section describes. arc --help lists every command, and arc <command> --help gives a command's options. arc registry is scaffolding for a future catalogue of Protocols, and isn't connected to a live index yet.

Using arc from an agent

arc mcp serves arc to an AI agent over the Model Context Protocol. Through it an agent can create a Protocol, record SQL steps and operations into it, run it, describe its tables and query them read-only, and every change it makes to arcform.yaml is a version in arc's history. It also serves Finetype's infer, profile, taxonomy, validate and generate, each run through the finetype binary. The release build the installer gives you includes it. To add it to Claude Code:

claude mcp add arc -- arc mcp
# → Added stdio MCP server arc with command: arc mcp to local config

Claude Code then prints a second line naming the configuration file it changed.

An agent started from a desktop app may not see your shell's PATH: give the absolute path of arc (which arc prints it) and a PATH that holds duckdb, finetype and uv. The README has the configuration for other clients.

Where to go next

• examples/brewtrend: the whole Protocol, with its SQL models and its README.
• Operations: the operations a data job is made of, from reading a file to publishing a table, each with its own page saying whether Arcform runs it today and what other tools call it.
• Protocols: the Protocols that build Meridian's datasets, and examples over well-known public datasets, with the commands that run them where Meridian has run them in full.
• Finetype: names the type of each column your data holds.
• Dovetail: works out how an unfamiliar file loads and how tables relate.

Arcform's source is on GitHub, with its changelog.