Publish / describe · All operations

Describe a dataset

Emit metadata: data dictionary, schema file, lineage, data package descriptor.

Status

In arcform

Not built yet

arcform does not do this yet. Its place on the roadmap

arcform has datapackage_describe for a Frictionless Data Package descriptor.

Shape of the result
the table sent out, with nothing new to look at
Level
needs an arcform operator

Variants

Describe a table or model in place

Attach comments, glossary terms, schema descriptions or metadata fields to a table, column or model.

What other tools call it

Meridian's classification; an independent blind rating agreed on 86.2% of 312 rows (exact class, version 2).

Alteryx Designer (3)
  • Connect JSON Builderread 2026-09-27

    Build the JSON metadata document for the Alteryx Connect API in a metadata loader

  • Connect Outputread 2026-09-27

    Send harvested metadata to the Alteryx Connect API in a metadata loader

  • ODBC Loaderread 2026-09-27

    Internal tool used by Connect metadata loaders over ODBC sources

Apache Hop (2)
  • Build documentationread 2026-09-27

    Generate documentation for the pipelines and workflows of a project

  • Database impact inputread 2026-09-27

    Emit the database tables and fields each pipeline reads or writes, as lineage rows

Bruin (6)
  • Dashboard assets (family)read 2026-09-27

    15 BI tool types (tableau, looker, metabase, powerbi, superset, …): no-op assets kept for lineage

  • Glossaryread 2026-09-27

    Shared entities and attributes that asset columns extend to inherit name, type and description

  • Source assets (family)read 2026-09-27

    14 platform types (bq.source, pg.source, …): no-op assets documenting existing tables for lineage

  • quicksight.dashboardread 2026-09-27

    Represent a QuickSight dashboard; no-op, for lineage

  • tableau.dashboardread 2026-09-27

    Represent a Tableau dashboard; no-op, for lineage

  • tableau.worksheetread 2026-09-27

    Represent a Tableau worksheet; no-op, for lineage

DuckDB (1)
  • COMMENT ONread 2026-09-27

    Attach a comment to a table, column, view or other catalog entry

EasyMorph (1)
  • Catalog commandread 2026-09-27

    List, create, move, rename or delete Data Catalog assets, collections, fields and metrics

OpenRefine (1)
  • Export Wikidata schemaread 2026-09-27

    Write the column-to-statement schema as a JSON file for reuse on other projects

Pentaho Data Integration (2)
RapidMiner Studio (1)
  • Scan Processesread 2026-09-27

    Parse the stored processes under a repository folder and list every operator they use

Yggdrasil Decision Forests (2)
  • ModelMetadataread 2026-09-27

    Owner, creation date, uid, framework and custom fields stored inside a model

  • metadata / set_metadataread 2026-09-27

    Read or replace the model's owner, creation date, uid, framework and custom fields

Background

Metadata is data that defines and describes the characteristics of other data, and it often makes that data easier to retrieve, use or manage.[1] A data dictionary typically names and describes the tables and fields of a database, with details such as the type and length of each data element.[2] In the Data Package standard, a dataset's metadata is stored in a descriptor, a file named datapackage.json when the package is a directory of files.[3]

When to use it

Database users and application developers can benefit from an authoritative data dictionary document that catalogues the organisation, contents and conventions of one or more databases.[2] There is no universal standard for the level of detail in a data dictionary document.[2] Decide who will read the description before choosing its form: a person reading a data dictionary, or a program reading a schema file, wants different things. (Meridian's view)

A passive data dictionary is updated manually and independently of any changes to the structure of the database it describes.[2] A description written by hand beside the data goes stale quietly, so generate it from the data whenever the data is rebuilt. (Meridian's view)

In Table Schema, the Data Package standard's format for declaring a schema for tabular data, a field with no type property is of the type any, and a consumer must not infer its type from its values.[4] By default, Table Schema requires the fields in a data source to match the fields in the schema exactly, in number and in order.[4] Regenerate the schema when columns are added or reordered, or loosen the match on purpose, rather than letting the two drift apart. (Meridian's view)

The method

The National Information Standards Organization (NISO) distinguishes three types of metadata: descriptive, structural and administrative.[1] NISO's descriptive metadata is typically used for discovery and identification, with elements such as title, authors, subjects, keywords and publisher.[1]

In DuckDB, DESCRIBE can be used as a subquery, so a table's description can be saved as a table of its own.[5]

A Data Package descriptor must have a resources property and should have name, id, licenses and profile properties.[3] A Data Package descriptor carries general metadata such as the package's title, licence and publisher, and lists the package's resources with their locations and, possibly, their schemas.[3] Table Schema is a language- and implementation-agnostic way to declare a schema for tabular data.[4] In Table Schema, each field descriptor must have a name, and may also give the field a title, a description, a type and a format.[4]

In DuckDB, the COMMENT ON statement adds metadata to catalog entries such as tables, columns and views, following PostgreSQL's syntax.[6] In DuckDB, comments are read back from the comment column of metadata functions such as duckdb_tables() and duckdb_columns(), and a comment set to NULL is removed.[6] In DuckDB, COMMENT ON cannot comment on schemas or databases.[6]

Sources

  1. [1]Metadata - Wikipediasecondary source
  2. [2]Data dictionary - Wikipediasecondary source
  3. [3]Data Package | Data Package Standardstandard
  4. [4]Table Schema | Data Package Standardstandard
  5. [5]Describe – DuckDBdocumentation
  6. [6]COMMENT ON Statement – DuckDBdocumentation

Used in

15 Protocols use this operation: