HTS — Harmonized Tariff Schedule of the United States
12 stepsmanifest declares duckdb >=2.0b4/1This Protocol builds the dataset Meridian maintains from the U.S. International Trade Commission’s public files. Meridian holds the usitc namespace until the U.S. International Trade Commission claims it.
The U.S. International Trade Commission (USITC) publishes the Harmonized Tariff Schedule of the United States as a JSON export of its current revision. This Protocol downloads that export, names the revision before and after the download, and builds Meridian's usitc.hts dataset from it: one row per chapter, heading, subheading, tariff line and statistical line, each with its parent, and each line with its duty rates, units and footnotes as USITC prints them.
producesusitc.htsHTS — Harmonized Tariff Schedule of the United StatesChecked against a finetype schema
Once the Parquet is built, this Protocol’s validate step checks it against schema.finetype.json, a finetype schema kept in the repository: each column’s type, its length bounds, numeric ranges, closed lists of values and finetype labels. The step writes nothing, so the Parquet is unchanged, and a run whose data no longer fits the schema stops there.
The flow
Every step runs in order. Each step is one of three kinds: an op step runs an operator built into arc, a sql step runs a DuckDB query from the repository, and a command step runs a shell command, which arc runs as written without knowing what it reads or writes.
Operations used: Fetch from the web, Read a JSON file, Cross join tables, Join tables on a key, Append tables, Change a column's type, Add a column by condition, Add a calculated column, Combine columns into one, Fill in missing values, Parse nested values, Match with a regular expression, Choose and rename columns, Clean up text, Deduplicate rows, Split a list into rows, Filter rows, Number or rank rows, Summarise by group, Calculate across rows, Read a CSV or delimited file, Handle errors and retries, Write a columnar file, Describe a dataset, Run a script, Check data quality, Write to a database, Upload to cloud storage.
- 01fetch_release_beforeop: http_fetch@1
Fetch the name of USITC's current HTS revision, before the schedule is downloaded.
producesbuild/release_before.jsonwithurl: https://hts.usitc.gov/reststop/currentRelease out: build/release_before.json headers: User-Agent: Meridian Protocol (open-analytics; research@meridian.online) - 02fetch_htsop: http_fetch@1
Fetch USITC's HTS export as JSON (about 12.6 MB): the current revision, which is the only one it serves.
producesbuild/hts.jsonwithurl: https://hts.usitc.gov/reststop/exportList?from=0100&to=9999&format=JSON&styles=false out: build/hts.json headers: User-Agent: Meridian Protocol (open-analytics; research@meridian.online) - 03fetch_release_afterop: http_fetch@1
Fetch the revision name again after the download. The load step fails the run if the two differ, so a schedule that changed mid-fetch is never published under one revision's name.
producesbuild/release_after.jsonwithurl: https://hts.usitc.gov/reststop/currentRelease out: build/release_after.json headers: User-Agent: Meridian Protocol (open-analytics; research@meridian.online) - 04outlinesql: models/outline.sql
Read USITC's rows in order, with each row's chain of indents and every prefix of its code. The Schedule B Protocol runs this same file.
readsbuild/hts.jsonmodelmodels/outline.sql - 05loadsql: models/load.sql
Build the ladder from chapter to statistical line, with each row's parent, text, rates and revision. Fails the run if the two revision names differ, or if a chapter has no title in chapters.csv.
readsbuild/release_after.jsonbuild/release_before.jsonchapters.csvmodelmodels/load.sql - 06packagesql: models/package.sql
Materialise the terminal table (adds the search corpus).
produceshts_outmodelmodels/package.sql - 07export_htsop: parquet_export@1
Write the terminal Parquet. order_by code (the primary key) is a total order → reproducible bytes.
readshts_outproducesbuild/hts.parquetwithinput: hts_out dest: build/hts.parquet compression: zstd order_by: code - 08describeop: datapackage_describe@1
Emit datapackage.json — finetype types every column; descriptor.overrides.json overlays the curated identity, licenses, sources, per-field prose, primaryKey and the code-list declaration. Hard-fails on drift.
readsbuild/hts.parquetdescriptor.overrides.jsonnominations.finetype.jsonproducesdatapackage.jsonwithparquet: build/hts.parquet overrides: descriptor.overrides.json nominations: nominations.finetype.json out: datapackage.json - 09check_code_listop: uv@1
Hold the built Parquet to the shared code-list contract: refuse a file with a parent that is missing or not one level up, a depth that is not its level's position, or a code that is repeated or empty.
readsbuild/hts.parquetproducesbuild/code_list_check.jsonwithscript: ../../scripts/check_code_list.py sha256: a6d4fd3c42aca53591f7f94bfef625f9c4f503b2d4331b64bea8103db88f7102 deps: [duckdb==2.0.0] args: [--parquet, build/hts.parquet, --levels, chapter,heading,subheading,tariff_line,statistical_line, --json, build/code_list_check.json] reads: [build/hts.parquet] produces: [build/code_list_check.json] - 10validatefinetype gateop: finetype_validate@1
Gate the built Parquet against the self-derived finetype contract. CHECK-ONLY: types every column and checks it against schema.finetype.json — a pass-through today, a fail-closed drift tripwire for future runs. Writes nothing; the Parquet stays byte-identical.
readsbuild/hts.parquetschema.finetype.jsonwithparquet: build/hts.parquet schema: schema.finetype.json - 11stamp_descriptorop: uv@1
Write the finished datapackage.json into the Parquet's own footer, so the published file describes itself from its URL alone, then re-measure the stamped file and write its new size and hash back into datapackage.json. The data pages are compared byte for byte before and after; a stamp that moved one is refused and the original is left in place.
readsbuild/hts.parquetdatapackage.jsonwithscript: ../../scripts/stamp_descriptor.py sha256: a9356ac8d5412d94a940ebeef1e2508eaed6b77f1d923ada88edbacccf9d9b81 deps: [duckdb==2.0.0] args: [stamp, --descriptor, datapackage.json, --parquet, build/hts.parquet] reads: [datapackage.json, build/hts.parquet] produces: [datapackage.json, build/hts.parquet] - 12publishop: ducklake_write@1
Write the table into the open lake as one snapshot: the catalogue, the data file and the table's readable Parquet copy at openlake.meridian.online. It reads the stamped table and its descriptor, and any other file the Protocol uploads beside them, so it runs last. Without the three R2 credentials set, a run publishes into build/lake/ on your machine instead.
readsbuild/hts.parquetdatapackage.jsonwithpublic_base: https://openlake.meridian.online/ catalog: catalog/open.ducklake data_path: lake/ runs: runs/ manifest: manifest.json local_catalog: build/lake/open.ducklake tables: [[object Object]] credential: type: r2 bucket: meridian-open env: key_id: R2_ACCESS_KEY_ID secret: R2_SECRET_ACCESS_KEY account_id: R2_ACCOUNT_ID