Schedule B — U.S. Export Classification Codes
17 stepsmanifest declares duckdb >=2.0b4/1This Protocol builds the dataset Meridian maintains from the U.S. Census Bureau’s public files. Meridian holds the census namespace until the U.S. Census Bureau claims it.
The U.S. Census Bureau publishes Schedule B, its classification of exported goods, as a yearly bulk file of ten-digit commodities. This Protocol downloads that file and Census's index page for the year, takes the heading and subheading text from the U.S. International Trade Commission's HTS export, and builds Meridian's census.schedule_b dataset from them: a row for each commodity and for each chapter, heading and subheading its codes imply, each with its parent.
producescensus.schedule_bSchedule B — U.S. Export Classification CodesChecked against a finetype schema
Once the Parquet is built, this Protocol’s validate step checks it against schema.finetype.json, a finetype schema kept in the repository: each column’s type, its length bounds, numeric ranges, closed lists of values and finetype labels. The step writes nothing, so the Parquet is unchanged, and a run whose data no longer fits the schema stops there.
The flow
Every step runs in order. Each step is one of three kinds: an op step runs an operator built into arc, a sql step runs a DuckDB query from the repository, and a command step runs a shell command, which arc runs as written without knowing what it reads or writes.
Operations used: Fetch from the web, Unpack an archive, Run a script, Read a JSON file, Cross join tables, Join tables on a key, Append tables, Change a column's type, Add a column by condition, Add a calculated column, Combine columns into one, Fill in missing values, Parse nested values, Match with a regular expression, Choose and rename columns, Clean up text, Deduplicate rows, Split a list into rows, Filter rows, Number or rank rows, Summarise by group, Calculate across rows, Read another file format, Handle errors and retries, Set parameters, Split a column, Write a columnar file, Describe a dataset, Check data quality, Write to a database, Upload to cloud storage.
- 01fetch_schedule_bop: http_fetch@1
Fetch Census's Schedule B bulk file for the year (sbsdf.zip).
producesbuild/sbsdf.zipwithurl: https://www.census.gov/foreign-trade/schedules/b/2026/sbsdf.zip out: build/sbsdf.zip headers: User-Agent: Meridian Protocol (open-analytics; research@meridian.online) - 02extract_schedule_bop: archive_extract@1
Extract econcord.txt, the commodity file, from the zip.
readsbuild/sbsdf.zipproducesbuild/sbsdf/econcord.txtwitharchive: build/sbsdf.zip members: [econcord.txt] dest: build/sbsdf - 03date_schedule_bop: uv@1
Read econcord.txt's date from the zip's own directory, since the extract step does not keep it on the file it writes. The release column carries this date.
readsbuild/sbsdf.zipproducesbuild/econcord_date.jsonwithscript: ../../scripts/member_date.py sha256: 98a42fee4c99e3eee02881ed21368034e725f1609333faa73c0e46a721bd456c args: [--archive, build/sbsdf.zip, --member, econcord.txt, --out, build/econcord_date.json] reads: [build/sbsdf.zip] produces: [build/econcord_date.json] - 04fetch_indexop: http_fetch@1
Fetch Census's Schedule B index page for the year, for the chapter titles and sections.
producesbuild/schedule_b_index.htmlwithurl: https://www.census.gov/foreign-trade/schedules/b/2026/index.html out: build/schedule_b_index.html headers: User-Agent: Meridian Protocol (open-analytics; research@meridian.online) - 05fetch_release_beforeop: http_fetch@1
Fetch the name of USITC's current HTS revision, before the schedule is downloaded.
producesbuild/release_before.jsonwithurl: https://hts.usitc.gov/reststop/currentRelease out: build/release_before.json headers: User-Agent: Meridian Protocol (open-analytics; research@meridian.online) - 06fetch_htsop: http_fetch@1
Fetch USITC's HTS export as JSON (about 12.6 MB): the current revision, which is the only one it serves.
producesbuild/hts.jsonwithurl: https://hts.usitc.gov/reststop/exportList?from=0100&to=9999&format=JSON&styles=false out: build/hts.json headers: User-Agent: Meridian Protocol (open-analytics; research@meridian.online) - 07fetch_release_afterop: http_fetch@1
Fetch the revision name again after the download. The load step fails the run if the two differ, so a schedule that changed mid-fetch is never published under one revision's name.
producesbuild/release_after.jsonwithurl: https://hts.usitc.gov/reststop/currentRelease out: build/release_after.json headers: User-Agent: Meridian Protocol (open-analytics; research@meridian.online) - 08outlinesql: ../hts/models/outline.sql
Read USITC's rows in order, with every code prefix's text, by running the HTS Protocol's own outline file.
readsbuild/hts.jsonmodel../hts/models/outline.sql - 09loadsql: models/load.sql
Build the ladder from chapter to commodity, with its text, the line columns and the release. Fails the run on a revision that changed mid-fetch, on a Schedule B year that is not USITC's HS edition, and where USITC names fewer than 99% of the headings or subheadings Schedule B implies.
readsbuild/econcord_date.jsonbuild/release_after.jsonbuild/release_before.jsonbuild/sbsdf/econcord.txtbuild/schedule_b_index.htmlmodelmodels/load.sql - 10packagesql: models/package.sql
Materialise the terminal table (adds the search corpus).
producesschedule_b_outmodelmodels/package.sql - 11export_schedule_bop: parquet_export@1
Write the unstamped Parquet to build/stage/schedule_b.parquet, which the describe, coverage, contract, gate and stamp steps read. order_by code (the primary key) is a total order → reproducible bytes.
readsschedule_b_outproducesbuild/stage/schedule_b.parquetwithinput: schedule_b_out dest: build/stage/schedule_b.parquet compression: zstd order_by: code - 12describeop: datapackage_describe@1
Describe the staged Parquet into build/stage/described.json — finetype types every column; descriptor.overrides.json overlays the curated identity, licenses, sources, per-field prose, primaryKey, the code-list declaration and the NAICS join. Hard-fails on drift.
readsbuild/stage/schedule_b.parquetdescriptor.overrides.jsonnominations.finetype.jsonproducesbuild/stage/described.jsonwithparquet: build/stage/schedule_b.parquet overrides: descriptor.overrides.json nominations: nominations.finetype.json out: build/stage/described.json - 13derive_join_coverageop: uv@1
Run the NAICS join the descriptor declares against this build's Parquet and the published census/naics_2022 file, and write what it measures — the rows in scope and the rows that find a partner — as the join's coverage into build/stage/measured.json, a copy of the staged descriptor.
reads../../datasets/edgar/datapackage.json../../datasets/edgar_gleif/datapackage.json../../datasets/gleif/datapackage.json../../datasets/hts/datapackage.json../../datasets/naics/datapackage.jsonbuild/stage/described.jsonbuild/stage/schedule_b.parquetproducesbuild/stage/measured.jsonwithscript: ../edgar_gleif/scripts/crosswalk_join.py sha256: d16e99e888ea74206d930214d8ccf8b9ad377327a6f9026b1487a70e611eab26 deps: [duckdb==2.0.0] args: [--write, --package, schedule_b, --datasets-dir, ../../datasets, --descriptor, schedule_b=build/stage/described.json, --out, build/stage/measured.json, --local, schedule_b=build/stage/schedule_b.parquet] reads: [build/stage/schedule_b.parquet, build/stage/described.json, ../../datasets/edgar/datapackage.json, ../../datasets/edgar_gleif/datapackage.json, ../../datasets/gleif/datapackage.json, ../../datasets/hts/datapackage.json, ../../datasets/naics/datapackage.json] produces: [build/stage/measured.json] - 14check_code_listop: uv@1
Hold the built Parquet to the shared code-list contract: refuse a file with a parent that is missing or not one level up, a depth that is not its level's position, or a code that is repeated or empty.
readsbuild/stage/schedule_b.parquetproducesbuild/code_list_check.jsonwithscript: ../../scripts/check_code_list.py sha256: a6d4fd3c42aca53591f7f94bfef625f9c4f503b2d4331b64bea8103db88f7102 deps: [duckdb==2.0.0] args: [--parquet, build/stage/schedule_b.parquet, --levels, chapter,heading,subheading,commodity, --json, build/code_list_check.json] reads: [build/stage/schedule_b.parquet] produces: [build/code_list_check.json] - 15validatefinetype gateop: finetype_validate@1
Gate the built Parquet against the self-derived finetype contract. CHECK-ONLY: types every column and checks it against schema.finetype.json — a pass-through today, a fail-closed drift tripwire for future runs. Writes nothing; the Parquet stays byte-identical.
readsbuild/stage/schedule_b.parquetschema.finetype.jsonwithparquet: build/stage/schedule_b.parquet schema: schema.finetype.json - 16stamp_descriptorop: uv@1
Read the staged Parquet and the measured descriptor, and write the published pair: build/schedule_b.parquet, with the finished descriptor in its own footer so the file describes itself from its URL alone, and datapackage.json, with the stamped file's size and hash. When the published file already holds the same data pages, the descriptor keeps its earlier created date.
readsbuild/stage/measured.jsonbuild/stage/schedule_b.parquetproducesbuild/schedule_b.parquetdatapackage.jsonwithscript: ../../scripts/stamp_descriptor.py sha256: a9356ac8d5412d94a940ebeef1e2508eaed6b77f1d923ada88edbacccf9d9b81 deps: [duckdb==2.0.0] args: [stamp, --descriptor, build/stage/measured.json, --parquet, build/stage/schedule_b.parquet, --out, build/schedule_b.parquet, --out-descriptor, datapackage.json, --carry-created] reads: [build/stage/measured.json, build/stage/schedule_b.parquet] produces: [build/schedule_b.parquet, datapackage.json] - 17publishop: ducklake_write@1
Write the table into the open lake as one snapshot: the catalogue, the data file and the table's readable Parquet copy at openlake.meridian.online. It reads the stamped table and its descriptor, and any other file the Protocol uploads beside them, so it runs last. Without the three R2 credentials set, a run publishes into build/lake/ on your machine instead.
readsbuild/schedule_b.parquetdatapackage.jsonwithpublic_base: https://openlake.meridian.online/ catalog: catalog/open.ducklake data_path: lake/ runs: runs/ manifest: manifest.json local_catalog: build/lake/open.ducklake tables: [[object Object]] credential: type: r2 bucket: meridian-open env: key_id: R2_ACCESS_KEY_ID secret: R2_SECRET_ACCESS_KEY account_id: R2_ACCOUNT_ID