Schedule B — U.S. Export Classification Codes

17 stepsmanifest declares duckdb >=2.0b4/1

This Protocol builds the dataset Meridian maintains from the U.S. Census Bureau’s public files. Meridian holds the census namespace until the U.S. Census Bureau claims it.

The U.S. Census Bureau publishes Schedule B, its classification of exported goods, as a yearly bulk file of ten-digit commodities. This Protocol downloads that file and Census's index page for the year, takes the heading and subheading text from the U.S. International Trade Commission's HTS export, and builds Meridian's census.schedule_b dataset from them: a row for each commodity and for each chapter, heading and subheading its codes imply, each with its parent.

producescensus.schedule_bSchedule B — U.S. Export Classification Codes

Checked against a finetype schema

Once the Parquet is built, this Protocol’s validate step checks it against schema.finetype.json, a finetype schema kept in the repository: each column’s type, its length bounds, numeric ranges, closed lists of values and finetype labels. The step writes nothing, so the Parquet is unchanged, and a run whose data no longer fits the schema stops there.

The flow

Every step runs in order. Each step is one of three kinds: an op step runs an operator built into arc, a sql step runs a DuckDB query from the repository, and a command step runs a shell command, which arc runs as written without knowing what it reads or writes.

Operations used: Fetch from the web, Unpack an archive, Run a script, Read a JSON file, Cross join tables, Join tables on a key, Append tables, Change a column's type, Add a column by condition, Add a calculated column, Combine columns into one, Fill in missing values, Parse nested values, Match with a regular expression, Choose and rename columns, Clean up text, Deduplicate rows, Split a list into rows, Filter rows, Number or rank rows, Summarise by group, Calculate across rows, Read another file format, Handle errors and retries, Set parameters, Split a column, Write a columnar file, Describe a dataset, Check data quality, Write to a database, Upload to cloud storage.

  1. 01fetch_schedule_bop: http_fetch@1

    Fetch Census's Schedule B bulk file for the year (sbsdf.zip).

    producesbuild/sbsdf.zip
    with
    url: https://www.census.gov/foreign-trade/schedules/b/2026/sbsdf.zip
    out: build/sbsdf.zip
    headers:
      User-Agent: Meridian Protocol (open-analytics; research@meridian.online)
  2. 02extract_schedule_bop: archive_extract@1

    Extract econcord.txt, the commodity file, from the zip.

    readsbuild/sbsdf.zipproducesbuild/sbsdf/econcord.txt
    with
    archive: build/sbsdf.zip
    members: [econcord.txt]
    dest: build/sbsdf
  3. 03date_schedule_bop: uv@1

    Read econcord.txt's date from the zip's own directory, since the extract step does not keep it on the file it writes. The release column carries this date.

    readsbuild/sbsdf.zipproducesbuild/econcord_date.json
    with
    script: ../../scripts/member_date.py
    sha256: 98a42fee4c99e3eee02881ed21368034e725f1609333faa73c0e46a721bd456c
    args: [--archive, build/sbsdf.zip, --member, econcord.txt, --out, build/econcord_date.json]
    reads: [build/sbsdf.zip]
    produces: [build/econcord_date.json]
  4. 04fetch_indexop: http_fetch@1

    Fetch Census's Schedule B index page for the year, for the chapter titles and sections.

    producesbuild/schedule_b_index.html
    with
    url: https://www.census.gov/foreign-trade/schedules/b/2026/index.html
    out: build/schedule_b_index.html
    headers:
      User-Agent: Meridian Protocol (open-analytics; research@meridian.online)
  5. 05fetch_release_beforeop: http_fetch@1

    Fetch the name of USITC's current HTS revision, before the schedule is downloaded.

    producesbuild/release_before.json
    with
    url: https://hts.usitc.gov/reststop/currentRelease
    out: build/release_before.json
    headers:
      User-Agent: Meridian Protocol (open-analytics; research@meridian.online)
  6. 06fetch_htsop: http_fetch@1

    Fetch USITC's HTS export as JSON (about 12.6 MB): the current revision, which is the only one it serves.

    producesbuild/hts.json
    with
    url: https://hts.usitc.gov/reststop/exportList?from=0100&to=9999&format=JSON&styles=false
    out: build/hts.json
    headers:
      User-Agent: Meridian Protocol (open-analytics; research@meridian.online)
  7. 07fetch_release_afterop: http_fetch@1

    Fetch the revision name again after the download. The load step fails the run if the two differ, so a schedule that changed mid-fetch is never published under one revision's name.

    producesbuild/release_after.json
    with
    url: https://hts.usitc.gov/reststop/currentRelease
    out: build/release_after.json
    headers:
      User-Agent: Meridian Protocol (open-analytics; research@meridian.online)
  8. 08outlinesql: ../hts/models/outline.sql

    Read USITC's rows in order, with every code prefix's text, by running the HTS Protocol's own outline file.

    readsbuild/hts.json
    model
    ../hts/models/outline.sql
  9. 09loadsql: models/load.sql

    Build the ladder from chapter to commodity, with its text, the line columns and the release. Fails the run on a revision that changed mid-fetch, on a Schedule B year that is not USITC's HS edition, and where USITC names fewer than 99% of the headings or subheadings Schedule B implies.

    readsbuild/econcord_date.jsonbuild/release_after.jsonbuild/release_before.jsonbuild/sbsdf/econcord.txtbuild/schedule_b_index.html
    model
    models/load.sql
  10. 10packagesql: models/package.sql

    Materialise the terminal table (adds the search corpus).

    producesschedule_b_out
    model
    models/package.sql
  11. 11export_schedule_bop: parquet_export@1

    Write the unstamped Parquet to build/stage/schedule_b.parquet, which the describe, coverage, contract, gate and stamp steps read. order_by code (the primary key) is a total order → reproducible bytes.

    readsschedule_b_outproducesbuild/stage/schedule_b.parquet
    with
    input: schedule_b_out
    dest: build/stage/schedule_b.parquet
    compression: zstd
    order_by: code
  12. 12describeop: datapackage_describe@1

    Describe the staged Parquet into build/stage/described.json — finetype types every column; descriptor.overrides.json overlays the curated identity, licenses, sources, per-field prose, primaryKey, the code-list declaration and the NAICS join. Hard-fails on drift.

    readsbuild/stage/schedule_b.parquetdescriptor.overrides.jsonnominations.finetype.jsonproducesbuild/stage/described.json
    with
    parquet: build/stage/schedule_b.parquet
    overrides: descriptor.overrides.json
    nominations: nominations.finetype.json
    out: build/stage/described.json
  13. 13derive_join_coverageop: uv@1

    Run the NAICS join the descriptor declares against this build's Parquet and the published census/naics_2022 file, and write what it measures — the rows in scope and the rows that find a partner — as the join's coverage into build/stage/measured.json, a copy of the staged descriptor.

    reads../../datasets/edgar/datapackage.json../../datasets/edgar_gleif/datapackage.json../../datasets/gleif/datapackage.json../../datasets/hts/datapackage.json../../datasets/naics/datapackage.jsonbuild/stage/described.jsonbuild/stage/schedule_b.parquetproducesbuild/stage/measured.json
    with
    script: ../edgar_gleif/scripts/crosswalk_join.py
    sha256: d16e99e888ea74206d930214d8ccf8b9ad377327a6f9026b1487a70e611eab26
    deps: [duckdb==2.0.0]
    args: [--write, --package, schedule_b, --datasets-dir, ../../datasets, --descriptor, schedule_b=build/stage/described.json, --out, build/stage/measured.json, --local, schedule_b=build/stage/schedule_b.parquet]
    reads: [build/stage/schedule_b.parquet, build/stage/described.json, ../../datasets/edgar/datapackage.json, ../../datasets/edgar_gleif/datapackage.json, ../../datasets/gleif/datapackage.json, ../../datasets/hts/datapackage.json, ../../datasets/naics/datapackage.json]
    produces: [build/stage/measured.json]
  14. 14check_code_listop: uv@1

    Hold the built Parquet to the shared code-list contract: refuse a file with a parent that is missing or not one level up, a depth that is not its level's position, or a code that is repeated or empty.

    readsbuild/stage/schedule_b.parquetproducesbuild/code_list_check.json
    with
    script: ../../scripts/check_code_list.py
    sha256: a6d4fd3c42aca53591f7f94bfef625f9c4f503b2d4331b64bea8103db88f7102
    deps: [duckdb==2.0.0]
    args: [--parquet, build/stage/schedule_b.parquet, --levels, chapter,heading,subheading,commodity, --json, build/code_list_check.json]
    reads: [build/stage/schedule_b.parquet]
    produces: [build/code_list_check.json]
  15. 15validatefinetype gateop: finetype_validate@1

    Gate the built Parquet against the self-derived finetype contract. CHECK-ONLY: types every column and checks it against schema.finetype.json — a pass-through today, a fail-closed drift tripwire for future runs. Writes nothing; the Parquet stays byte-identical.

    readsbuild/stage/schedule_b.parquetschema.finetype.json
    with
    parquet: build/stage/schedule_b.parquet
    schema: schema.finetype.json
  16. 16stamp_descriptorop: uv@1

    Read the staged Parquet and the measured descriptor, and write the published pair: build/schedule_b.parquet, with the finished descriptor in its own footer so the file describes itself from its URL alone, and datapackage.json, with the stamped file's size and hash. When the published file already holds the same data pages, the descriptor keeps its earlier created date.

    readsbuild/stage/measured.jsonbuild/stage/schedule_b.parquetproducesbuild/schedule_b.parquetdatapackage.json
    with
    script: ../../scripts/stamp_descriptor.py
    sha256: a9356ac8d5412d94a940ebeef1e2508eaed6b77f1d923ada88edbacccf9d9b81
    deps: [duckdb==2.0.0]
    args: [stamp, --descriptor, build/stage/measured.json, --parquet, build/stage/schedule_b.parquet, --out, build/schedule_b.parquet, --out-descriptor, datapackage.json, --carry-created]
    reads: [build/stage/measured.json, build/stage/schedule_b.parquet]
    produces: [build/schedule_b.parquet, datapackage.json]
  17. 17publishop: ducklake_write@1

    Write the table into the open lake as one snapshot: the catalogue, the data file and the table's readable Parquet copy at openlake.meridian.online. It reads the stamped table and its descriptor, and any other file the Protocol uploads beside them, so it runs last. Without the three R2 credentials set, a run publishes into build/lake/ on your machine instead.

    readsbuild/schedule_b.parquetdatapackage.json
    with
    public_base: https://openlake.meridian.online/
    catalog: catalog/open.ducklake
    data_path: lake/
    runs: runs/
    manifest: manifest.json
    local_catalog: build/lake/open.ducklake
    tables: [[object Object]]
    credential:
      type: r2
      bucket: meridian-open
      env:
        key_id: R2_ACCESS_KEY_ID
        secret: R2_SECRET_ACCESS_KEY
        account_id: R2_ACCOUNT_ID