Protocols / GLEIF — Legal Entity Identifiers

6 stepsduckdb >=1.2

The reproducible definition of how the source.gleif Dataset is ingested from the GLEIF golden copy. The fetch is an opaque command by design — the golden copy is republished daily and its URL is resolved at run time from a JSON API (not a stable, pinnable link), and it is a ~470 MB zip, so forcing it into a typed link-discovery operator would misrepresent the work. Everything downstream of the raw CSV is first-class config: normalise to the published ten-field schema, materialise, export a reproducible Parquet, describe and gate.

producessource.gleifGLEIF — Legal Entity Identifiers

Gated against a finetype schema contract

This Protocol’s output is checked against a self-derived finetype schema (schema.finetype.json) by its finetype_validate@1 step (validate). The gate runs in check-only mode: it types every column and verifies the observable envelope — type, length bounds, numeric ranges, closed enum domains and semantic labels. A pass-through on today’s run; a fail-closed drift tripwire for future ones. It writes nothing, so the built Parquet stays byte-identical.

Note: this Protocol’s schema.finetype.json is not checked in yet — it is derived and committed on the first successful run, the same way as the other Datasets.

The flow

Every step runs in order. op: steps are typed, versioned operators; sql: steps are DuckDB models; a command step is opaque by design.

01fetch_gleifcommand
Resolve the current full-file CSV URL from the GLEIF publishes JSON API, download (~470 MB zip), unzip. Opaque command by design — a daily-rotating, non-pinnable URL that is not an HTML href.
producesgleif_raw
command
```
bash scripts/fetch_gleif.sh build/gleif_src
```
precondition: skip if build/gleif_src was modified within the last 24h.
02loadsql: models/load.sql
Normalise to the published ten-field schema (dates sliced from the ISO timestamps).
readsgleif_raw
model
```
models/load.sql
```
03packagesql: models/package.sql
Materialise the terminal table (adds the search corpus).
producesgleif_out
model
```
models/package.sql
```
04export_gleifop: parquet_export@1
Write the terminal Parquet. The total order_by (lei is unique) makes the bytes reproducible for a given golden-copy snapshot.
readsgleif_outproducesbuild/gleif.parquet
with
```
input: gleif_out
dest: build/gleif.parquet
compression: zstd
order_by: country, legal_name, lei
```
05describeop: datapackage_describe@1
Emit datapackage.json — finetype types every column; descriptor.overrides.json overlays the curated identity, licenses, sources, per-field prose and primaryKey. Hard-fails on drift.
readsbuild/gleif.parquetdescriptor.overrides.jsonproducesdatapackage.json
with
```
parquet: build/gleif.parquet
overrides: descriptor.overrides.json
out: datapackage.json
```
06validatefinetype gateop: finetype_validate@1
Gate the built Parquet against the self-derived finetype contract. CHECK-ONLY, fail-closed drift tripwire. Note: this Protocol’s schema.finetype.json is not checked in yet — it is derived + committed on the first successful run.
readsbuild/gleif.parquetschema.finetype.json
with
```
parquet: build/gleif.parquet
schema: schema.finetype.json
```