Protocols / EDGAR ↔ GLEIF — Company-to-LEI Crosswalk

23 stepsduckdb >=1.2b4/1

The reproducible definition of how the crosswalk.edgar_gleif Dataset is made. It fetches the published EDGAR and GLEIF Datasets, builds the SEC entity universe, pulls two authoritative fund-LEI sources (GLEIF SEC registrations and four quarters of SEC Form N-CEN), name-resolves the operating-company tail with a frozen Splink model, tiers deterministic and probabilistic edges together, then exports, gates and describes one point-in-time crosswalk. Every step is a typed operator or a SQL model — no opaque steps.

producescrosswalk.edgar_gleifEDGAR ↔ GLEIF — Company-to-LEI Crosswalk

Gated against a finetype schema contract

This Protocol’s output is checked against a self-derived finetype schema (schema.finetype.json) by its finetype_validate@1 step (validate). The gate runs in check-only mode: it types every column and verifies the observable envelope — type, length bounds, numeric ranges, closed enum domains and semantic labels. A pass-through on today’s run; a fail-closed drift tripwire for future ones. It writes nothing, so the built Parquet stays byte-identical.

The flow

Every step runs in order. op: steps are typed, versioned operators; sql: steps are DuckDB models; a command step is opaque by design.

01fetch_edgarop: http_fetch@1
Fetch the published source.edgar Parquet (the crosswalk left side).
producesbuild/edgar.parquet
with
```
url: https://openlake.meridian.online/edgar.parquet
out: build/edgar.parquet
```
02fetch_gleifop: http_fetch@1
Fetch the published source.gleif Parquet.
producesbuild/gleif.parquet
with
```
url: https://openlake.meridian.online/gleif.parquet
out: build/gleif.parquet
```

03fetch_cik_lookupop: http_fetch@1

Fetch the raw SEC cik-lookup-data list (the SEC entity universe).

producesbuild/cik_lookup.txt

with

url: https://www.sec.gov/Archives/edgar/cik-lookup-data.txt
out: build/cik_lookup.txt
headers:
  User-Agent: Meridian Protocol (open-analytics; research@meridian.online)

04build_sec_entitiessql: models/sec_entities.sql
Parse the cik-lookup list and ticker-join it in DuckDB into the SEC entity table.
readsbuild/cik_lookup.txtbuild/edgar.parquet
model
```
models/sec_entities.sql
```
05export_sec_entitiesop: parquet_export@1
Export the typed SEC-entities Parquet that load + resolve read.
readssec_entitiesproducesbuild/sec_entities.parquet
with
```
input: sec_entities
dest: build/sec_entities.parquet
```
06fetch_gleif_sec_registrationsop: gleif_ra_fetch@1
Fetch GLEIF entities registered at SEC EDGAR (RA000665) — the deterministic CIK/series↔LEI backbone for funds. Cursor-paged via dlt.
producesbuild/gleif_ra_sec.csv
with
```
ra: RA000665
out: build/gleif_ra_sec.csv
```

07fetch_ncen_2026q2op: http_fetch@1

Fetch SEC Form N-CEN 2026 Q2 (the annual fund filing — most direct fund-LEI authority).

producesbuild/ncen_zips/2026q2.zip

with

url: https://www.sec.gov/files/dera/data/form-n-cen-data-sets/2026q2_ncen.zip
out: build/ncen_zips/2026q2.zip
headers:
  User-Agent: Meridian Protocol (open-analytics; research@meridian.online)

08extract_ncen_2026q2op: archive_extract@1
Extract REGISTRANT.tsv (registrant CIK↔LEI) + FUND_REPORTED_INFO.tsv (series↔LEI) from the 2026 Q2 zip.
readsbuild/ncen_zips/2026q2.zipproducesbuild/ncen/2026q2/fund_reported_info.tsvbuild/ncen/2026q2/registrant.tsv
with
```
archive: build/ncen_zips/2026q2.zip
members: [REGISTRANT.tsv, FUND_REPORTED_INFO.tsv]
dest: build/ncen/2026q2
```

09fetch_ncen_2026q1op: http_fetch@1

Fetch SEC Form N-CEN 2026 Q1 (quarters are pinned for reproducibility).

producesbuild/ncen_zips/2026q1.zip

with

url: https://www.sec.gov/files/dera/data/form-n-cen-data-sets/2026q1_ncen.zip
out: build/ncen_zips/2026q1.zip
headers:
  User-Agent: Meridian Protocol (open-analytics; research@meridian.online)

10extract_ncen_2026q1op: archive_extract@1
Extract REGISTRANT.tsv + FUND_REPORTED_INFO.tsv from the 2026 Q1 zip.
readsbuild/ncen_zips/2026q1.zipproducesbuild/ncen/2026q1/fund_reported_info.tsvbuild/ncen/2026q1/registrant.tsv
with
```
archive: build/ncen_zips/2026q1.zip
members: [REGISTRANT.tsv, FUND_REPORTED_INFO.tsv]
dest: build/ncen/2026q1
```

11fetch_ncen_2025q4op: http_fetch@1

Fetch SEC Form N-CEN 2025 Q4.

producesbuild/ncen_zips/2025q4.zip

with

url: https://www.sec.gov/files/dera/data/form-n-cen-data-sets/2025q4_ncen.zip
out: build/ncen_zips/2025q4.zip
headers:
  User-Agent: Meridian Protocol (open-analytics; research@meridian.online)

12extract_ncen_2025q4op: archive_extract@1
Extract REGISTRANT.tsv + FUND_REPORTED_INFO.tsv from the 2025 Q4 zip.
readsbuild/ncen_zips/2025q4.zipproducesbuild/ncen/2025q4/fund_reported_info.tsvbuild/ncen/2025q4/registrant.tsv
with
```
archive: build/ncen_zips/2025q4.zip
members: [REGISTRANT.tsv, FUND_REPORTED_INFO.tsv]
dest: build/ncen/2025q4
```

13fetch_ncen_2025q3op: http_fetch@1

Fetch SEC Form N-CEN 2025 Q3.

producesbuild/ncen_zips/2025q3.zip

with

url: https://www.sec.gov/files/dera/data/form-n-cen-data-sets/2025q3_ncen.zip
out: build/ncen_zips/2025q3.zip
headers:
  User-Agent: Meridian Protocol (open-analytics; research@meridian.online)

14extract_ncen_2025q3op: archive_extract@1
Extract REGISTRANT.tsv + FUND_REPORTED_INFO.tsv from the 2025 Q3 zip.
readsbuild/ncen_zips/2025q3.zipproducesbuild/ncen/2025q3/fund_reported_info.tsvbuild/ncen/2025q3/registrant.tsv
with
```
archive: build/ncen_zips/2025q3.zip
members: [REGISTRANT.tsv, FUND_REPORTED_INFO.tsv]
dest: build/ncen/2025q3
```
15loadsql: models/load.sql
Load + normalise the fetched sources (SEC entities, GLEIF, RA registrations, four N-CEN quarters) into typed tables.
readsbuild/gleif.parquetbuild/gleif_ra_sec.csvbuild/ncen/2025q3/fund_reported_info.tsvbuild/ncen/2025q3/registrant.tsvbuild/ncen/2025q4/fund_reported_info.tsvbuild/ncen/2025q4/registrant.tsvbuild/ncen/2026q1/fund_reported_info.tsvbuild/ncen/2026q1/registrant.tsvbuild/ncen/2026q2/fund_reported_info.tsvbuild/ncen/2026q2/registrant.tsvbuild/sec_entities.parquet
model
```
models/load.sql
```
16resolveop: splink_resolve@1
Probabilistic Fellegi-Sunter / Splink name match for the operating-company tail (no official CIK↔LEI). Frozen model, SEED=42, precision-first.
readsbuild/gleif.parquetbuild/sec_entities.parquetproducesbuild/resolved.parquet
with
```
edgar: build/sec_entities.parquet
gleif: build/gleif.parquet
out: build/resolved.parquet
```
retry capped at 1 attempt; timeout 14400s (~4h full-corpus job).
17tiersql: models/tier.sql
Union the deterministic (authoritative) edges with the Splink (confirmed / candidate) edges into one tiered edge table.
readsbuild/resolved.parquetproducescrosswalk_edges
model
```
models/tier.sql
```
18packagesql: models/package.sql
Materialise the terminal edge table (adds the as_of stamp + search corpus).
readscrosswalk_edgesproducesedgar_gleif_out
model
```
models/package.sql
```
19export_edgar_gleifop: parquet_export@1
Write the terminal Parquet the publish seam consumes. The total order_by makes the bytes reproducible.
readsedgar_gleif_outproducesbuild/edgar_gleif.parquet
with
```
input: edgar_gleif_out
dest: build/edgar_gleif.parquet
compression: zstd
row_group_size: 50000
order_by: company_name, key_type, key, lei
```
20describeop: datapackage_describe@1
Emit datapackage.json from the built Parquet — finetype types every column; descriptor.overrides.json overlays the curated identity, licenses, per-field prose, primaryKey and foreignKeys. Hard-fails on curated-vs-Parquet drift.
readsbuild/edgar_gleif.parquetdescriptor.overrides.jsonproducesdatapackage.json
with
```
parquet: build/edgar_gleif.parquet
overrides: descriptor.overrides.json
nominations: nominations.finetype.json
out: datapackage.json
```

21derive_join_coverageop: uv@1

Run each join the descriptor declares (x-joins) against the Parquets this build just wrote, and write what it measures — the rows in scope and the rows that find a partner — into datapackage.json as the join's coverage, with the opening sentence of its cardinality derived from it. The script reads the columns and references from the descriptor rather than knowing anything about EDGAR or GLEIF, and it measures this build's own files, not the previously published ones.

with

script: scripts/crosswalk_join.py
sha256: 1c4f7d542853b9714420250056a6e45f285251ce561c0237798d6b264f5d6367
deps: [duckdb==1.5.5]
args: [--write, --package, edgar_gleif, --datasets-dir, ../../datasets, --local, edgar_gleif=build/edgar_gleif.parquet, --local, edgar=build/edgar.parquet, --local, gleif=build/gleif.parquet]
reads: [build/edgar_gleif.parquet, build/edgar.parquet, build/gleif.parquet, datapackage.json, ../../datasets/edgar/datapackage.json, ../../datasets/gleif/datapackage.json, ../../datasets/naics/datapackage.json]
produces: [datapackage.json]

22validatefinetype gateop: finetype_validate@1
Gate the built Parquet against the self-derived finetype contract. CHECK-ONLY: types every column and checks it against schema.finetype.json — a pass-through today, a fail-closed drift tripwire for future runs. Writes nothing; the Parquet stays byte-identical.
readsbuild/edgar_gleif.parquetschema.finetype.json
with
```
parquet: build/edgar_gleif.parquet
schema: schema.finetype.json
```
23stamp_descriptorop: uv@1
Write the finished datapackage.json into the Parquet's own footer, so the published file describes itself from its URL alone, then re-measure the stamped file and write its new size and hash back into datapackage.json. The data pages are compared byte for byte before and after; a stamp that moved one is refused and the original is left in place.
with
```
script: ../../scripts/stamp_descriptor.py
sha256: b926a77bed9956951814bc2e50d372a6ab7be8b05764d0977f42e3ca5158c72a
deps: [duckdb==1.5.5]
args: [stamp, --descriptor, datapackage.json, --parquet, build/edgar_gleif.parquet]
reads: [datapackage.json, build/edgar_gleif.parquet]
produces: [build/edgar_gleif.parquet, datapackage.json]
```