Protocols / SEC EDGAR — Company Tickers

7 stepsduckdb >=1.2b4/1

This Protocol builds the dataset Meridian maintains from the U.S. Securities and Exchange Commission’s public files. Meridian holds the sec namespace until the U.S. Securities and Exchange Commission claims it.

The reproducible definition of how the sec.company_tickers_exchange Dataset is ingested from the SEC's official company_tickers_exchange.json. Fetch (with the SEC-required User-Agent and a conditional-re-fetch freshness contract), flatten the columnar JSON to the four-field schema, materialise, export a reproducible Parquet, then describe and gate it. Clean information retrieval — the entity-resolution join lives in the separate crosswalk Protocol.

producessec.company_tickers_exchangeSEC EDGAR — Company Tickers

Gated against a finetype schema contract

This Protocol’s output is checked against a self-derived finetype schema (schema.finetype.json) by its finetype_validate@1 step (validate). The gate runs in check-only mode: it types every column and verifies the observable envelope — type, length bounds, numeric ranges, closed enum domains and semantic labels. A pass-through on today’s run; a fail-closed drift tripwire for future ones. It writes nothing, so the built Parquet stays byte-identical.

The flow

Every step runs in order. op: steps are typed, versioned operators; sql: steps are DuckDB models; a command step is opaque by design.

01fetch_edgarop: http_fetch@1
Fetch the SEC's company_tickers_exchange.json. Carries the content-freshness contract (ETag / Last-Modified sidecar → an unchanged upstream is a 304, not a re-download).
producesbuild/edgar.json
with
```
url: https://www.sec.gov/files/company_tickers_exchange.json
out: build/edgar.json
headers:
  User-Agent: Meridian Protocol (open-analytics; research@meridian.online)
```
02loadsql: models/load.sql
Flatten the columnar {fields, data} JSON to the four-field schema (cik, ticker, exchange, company_name).
readsbuild/edgar.json
model
```
models/load.sql
```
03packagesql: models/package.sql
Materialise the terminal table (adds the search corpus).
producesedgar_out
model
```
models/package.sql
```
04export_edgarop: parquet_export@1
Write the terminal Parquet the publish seam consumes. The total order_by makes the bytes reproducible.
readsedgar_outproducesbuild/edgar.parquet
with
```
input: edgar_out
dest: build/edgar.parquet
compression: zstd
order_by: company_name, cik, ticker
```
05describeop: datapackage_describe@1
Emit datapackage.json — finetype types every column; descriptor.overrides.json overlays the curated identity, licenses, sources, per-field prose and primaryKey. Hard-fails if a curated key names a column absent from the Parquet.
readsbuild/edgar.parquetdescriptor.overrides.jsonnominations.finetype.jsonproducesdatapackage.json
with
```
parquet: build/edgar.parquet
overrides: descriptor.overrides.json
nominations: nominations.finetype.json
out: datapackage.json
```
06validatefinetype gateop: finetype_validate@1
Gate the built Parquet against the self-derived finetype contract. CHECK-ONLY: types every column and checks it against schema.finetype.json — a pass-through today, a fail-closed drift tripwire for future runs. Writes nothing; the Parquet stays byte-identical.
readsbuild/edgar.parquetschema.finetype.json
with
```
parquet: build/edgar.parquet
schema: schema.finetype.json
```
07stamp_descriptorcommand
Write the finished datapackage.json into the Parquet's own footer, so the published file describes itself from its URL alone, then re-measure the stamped file and write its new size and hash back into datapackage.json. The data pages are compared byte for byte before and after; a stamp that moved one is refused and the original is left in place.
command
```
python3 ../../scripts/stamp_descriptor.py stamp --descriptor datapackage.json --parquet build/edgar.parquet
```