Protocols / GLEIF — Legal Entity Identifiers

7 stepsduckdb >=1.2b4/1

The reproducible definition of how the source.gleif Dataset is ingested from the GLEIF golden copy. The fetch is an opaque command by design — the golden copy is republished daily and its URL is resolved at run time from a JSON API (not a stable, pinnable link), and it is a ~470 MB zip, so forcing it into a typed link-discovery operator would misrepresent the work. Everything downstream of the raw CSV is first-class config: normalise to the published ten-field schema, materialise, export a reproducible Parquet, describe and gate.

producessource.gleifGLEIF — Legal Entity Identifiers

finetype gate — pending

This Protocol has no finetype gate step yet. Its schema.finetype.json is derived from — and committed on — the first successful full-scale run, the same way as the other Datasets; the gate step is added to the manifest then, so it is not shown as a promise the pipeline does not yet keep.

The flow

Every step runs in order. op: steps are typed, versioned operators; sql: steps are DuckDB models; a command step is opaque by design.

01fetch_gleifcommand
Resolve the current full-file CSV URL from the GLEIF publishes JSON API, download (~470 MB zip), unzip. Opaque command by design — a daily-rotating, non-pinnable URL that is not an HTML href.
producesgleif_raw
command
```
bash scripts/fetch_gleif.sh build/gleif_src
```
precondition: skip if build/gleif_src was modified within the last 24h.
02loadsql: models/load.sql
Normalise to the published ten-field schema (dates sliced from the ISO timestamps).
readsgleif_raw
model
```
models/load.sql
```
03packagesql: models/package.sql
Materialise the terminal table (adds the search corpus).
producesgleif_out
model
```
models/package.sql
```
04export_gleifop: parquet_export@1
Write the terminal Parquet. The total order_by (lei is unique) makes the bytes reproducible for a given golden-copy snapshot.
readsgleif_outproducesbuild/gleif.parquet
with
```
input: gleif_out
dest: build/gleif.parquet
compression: zstd
order_by: country, legal_name, lei
```
05describeop: datapackage_describe@1
Emit datapackage.json — finetype types every column; descriptor.overrides.json overlays the curated identity, licenses, sources, per-field prose and primaryKey. Hard-fails on drift.
readsbuild/gleif.parquetdescriptor.overrides.jsonproducesdatapackage.json
with
```
parquet: build/gleif.parquet
overrides: descriptor.overrides.json
nominations: nominations.finetype.json
out: datapackage.json
```
06stamp_versioncommand
Record which finetype release produced the descriptor: write the version that the finetype on PATH reports into datapackage.json as x-finetype-version. It refuses to restamp content that another release already stamped and that describe has not regenerated, rather than attribute it to an engine that did not produce it.
command
```
python3 ../../scripts/stamp_finetype_version.py --descriptor datapackage.json
```
07stamp_descriptorcommand
Write the finished datapackage.json into the Parquet's own footer, so the published file describes itself from its URL alone, then re-measure the stamped file and write its new size and hash back into datapackage.json. The data pages are compared byte for byte before and after; a stamp that moved one is refused and the original is left in place.
command
```
python3 ../../scripts/stamp_descriptor.py stamp --descriptor datapackage.json --parquet build/gleif.parquet
```