Protocols / GLEIF — Legal Entity Identifiers

7 stepsduckdb >=1.2b4/1

This Protocol builds the dataset Meridian maintains from GLEIF’s public files. Meridian holds the gleif namespace until GLEIF claims it.

The reproducible definition of how the gleif.lei Dataset is ingested from the GLEIF golden copy. The fetch is an opaque command by design — the golden copy is republished daily and its URL is resolved at run time from a JSON API (not a stable, pinnable link), and it is a ~470 MB zip, so forcing it into a typed link-discovery operator would misrepresent the work. Everything downstream of the raw CSV is first-class config: normalise to the published ten-field schema, materialise, export a reproducible Parquet, describe and gate.

producesgleif.leiGLEIF — Legal Entity Identifiers

finetype gate — pending

This Protocol has no finetype gate step yet. Its schema.finetype.json is derived from — and committed on — the first successful full-scale run, the same way as the other Datasets; the gate step is added to the manifest then, so it is not shown as a promise the pipeline does not yet keep.

The flow

Every step runs in order. op: steps are typed, versioned operators; sql: steps are DuckDB models; a command step is opaque by design.

01fetch_gleifcommand
Resolve the current full-file CSV URL from the GLEIF publishes JSON API, download (~470 MB zip), unzip. Opaque command by design — a daily-rotating, non-pinnable URL that is not an HTML href.
producesgleif_raw
command
```
bash scripts/fetch_gleif.sh build/gleif_src
```
precondition: skip if build/gleif_src was modified within the last 24h.
02loadsql: models/load.sql
Normalise to the published ten-field schema (dates sliced from the ISO timestamps).
readsgleif_raw
model
```
models/load.sql
```
03packagesql: models/package.sql
Materialise the terminal table (adds the search corpus).
producesgleif_out
model
```
models/package.sql
```
04export_gleifop: parquet_export@1
Write the terminal Parquet. The total order_by (lei is unique) makes the bytes reproducible for a given golden-copy snapshot.
readsgleif_outproducesbuild/gleif.parquet
with
```
input: gleif_out
dest: build/gleif.parquet
compression: zstd
order_by: country, legal_name, lei
```
05describeop: datapackage_describe@1
Emit datapackage.json — finetype types every column; descriptor.overrides.json overlays the curated identity, licenses, sources, per-field prose and primaryKey. Hard-fails on drift.
readsbuild/gleif.parquetdescriptor.overrides.jsonnominations.finetype.jsonproducesdatapackage.json
with
```
parquet: build/gleif.parquet
overrides: descriptor.overrides.json
nominations: nominations.finetype.json
out: datapackage.json
```
06stamp_versioncommand
Record which finetype release produced the descriptor: write the version that the finetype on PATH reports into datapackage.json as x-finetype-version. It refuses to restamp content that another release already stamped and that describe has not regenerated, rather than attribute it to an engine that did not produce it.
command
```
python3 ../../scripts/stamp_finetype_version.py --descriptor datapackage.json
```
07stamp_descriptorcommand
Write the finished datapackage.json into the Parquet's own footer, so the published file describes itself from its URL alone, then re-measure the stamped file and write its new size and hash back into datapackage.json. The data pages are compared byte for byte before and after; a stamp that moved one is refused and the original is left in place.
command
```
python3 ../../scripts/stamp_descriptor.py stamp --descriptor datapackage.json --parquet build/gleif.parquet
```