Protocols / GLEIF — Legal Entity Identifiers

5 stepsduckdb >=1.2b4/1

The reproducible definition of how the source.gleif Dataset is ingested from the GLEIF golden copy. The fetch is an opaque command by design — the golden copy is republished daily and its URL is resolved at run time from a JSON API (not a stable, pinnable link), and it is a ~470 MB zip, so forcing it into a typed link-discovery operator would misrepresent the work. Everything downstream of the raw CSV is first-class config: normalise to the published ten-field schema, materialise, export a reproducible Parquet, describe and gate.

producessource.gleifGLEIF — Legal Entity Identifiers

finetype gate — pending

This Protocol has no finetype gate step yet. Its schema.finetype.json is derived from — and committed on — the first successful full-scale run, the same way as the other Datasets; the gate step is added to the manifest then, so it is not shown as a promise the pipeline does not yet keep.

The flow

Every step runs in order. op: steps are typed, versioned operators; sql: steps are DuckDB models; a command step is opaque by design.

  1. 01fetch_gleifcommand

    Resolve the current full-file CSV URL from the GLEIF publishes JSON API, download (~470 MB zip), unzip. Opaque command by design — a daily-rotating, non-pinnable URL that is not an HTML href.

    producesgleif_raw
    command
    bash scripts/fetch_gleif.sh build/gleif_src

    precondition: skip if build/gleif_src was modified within the last 24h.

  2. 02loadsql: models/load.sql

    Normalise to the published ten-field schema (dates sliced from the ISO timestamps).

    readsgleif_raw
    model
    models/load.sql
  3. 03packagesql: models/package.sql

    Materialise the terminal table (adds the search corpus).

    producesgleif_out
    model
    models/package.sql
  4. 04export_gleifop: parquet_export@1

    Write the terminal Parquet. The total order_by (lei is unique) makes the bytes reproducible for a given golden-copy snapshot.

    readsgleif_outproducesbuild/gleif.parquet
    with
    input: gleif_out
dest: build/gleif.parquet
compression: zstd
order_by: country, legal_name, lei
  5. 05describeop: datapackage_describe@1

    Emit datapackage.json — finetype types every column; descriptor.overrides.json overlays the curated identity, licenses, sources, per-field prose and primaryKey. Hard-fails on drift.

    readsbuild/gleif.parquetdescriptor.overrides.jsonproducesdatapackage.json
    with
    parquet: build/gleif.parquet
overrides: descriptor.overrides.json
out: datapackage.json