Join / union · All operations

Append tables with a source column

A variant of Append tables

Stack the rows of several tables and add a column that records which table each row came from.

Status

In arcform

Runs as a SQL step

arcform carries it as a sql: step today.

The DuckDB step

Append tables with a column naming the source

SQL
CREATE TEMP TABLE t1 AS SELECT * FROM (VALUES (1, 'a'), (2, 'b')) t(id, v);
CREATE TEMP TABLE t2 AS SELECT * FROM (VALUES (2, 'b'), (3, 'c')) t(id, v);
COPY t1 TO '<your_scratch_dir>/tag_one.csv' (HEADER);
COPY t2 TO '<your_scratch_dir>/tag_two.csv' (HEADER);
-- tables: a literal for each; the row (2, b) is in both and stays twice
CREATE TEMP TABLE from_tables AS
SELECT 't1' AS source, * FROM t1 UNION ALL BY NAME SELECT 't2' AS source, * FROM t2;
-- files: the reader adds the path of the file each row came from (filename = true); only its name is kept
CREATE TEMP TABLE from_files AS
SELECT parse_filename(filename) AS source, id, v
FROM read_csv(['<your_scratch_dir>/tag_one.csv', '<your_scratch_dir>/tag_two.csv'], filename = true);
SELECT * FROM from_tables ORDER BY source, id;
SELECT * FROM from_files ORDER BY source, id;
SELECT CASE WHEN ((SELECT list((source, id) ORDER BY source, id) = [('t1', 1), ('t1', 2), ('t2', 2), ('t2', 3)] FROM from_tables)
  AND (SELECT list((source, id, v) ORDER BY source, id) = [('tag_one.csv', 1, 'a'), ('tag_one.csv', 2, 'b'), ('tag_two.csv', 2, 'b'), ('tag_two.csv', 3, 'c')] FROM from_files)
  AND (SELECT list(column_name) FROM (DESCRIBE from_tables)) = ['source', 'id', 'v']) THEN 'as expected' ELSE error('each of the 4 rows should name the table or file it came from, the row found in both tables should appear once for each, and the source column should come first') END AS verdict;

Replace <your_scratch_dir> with a directory of your own that you can write to before running the statement.

  • The statement writes tag_one.csv and tag_two.csv itself, in <your_scratch_dir>.
  • It needs nothing installed beyond DuckDB itself.
  • The statement checks its own result and raises an error when it is wrong: “each of the 4 rows should name the table or file it came from, the row found in both tables should appear once for each, and the source column should come first”.
  • It passed on DuckDB v2.0.0 (Cyanoptera) d54cbbe3df.

What other tools call it

Meridian's classification; an independent blind rating agreed on 95.7% of 324 rows (variant, version 3).

dbt-utils (1)
  • union_relationsread 2026-09-27

    UNION ALL a list of relations, aligning columns by name, null-filling gaps, adding a source column

Power Query M (1)
  • Table.FromPartitionsread 2026-09-27

    Append partitioned tables into one, adding a column that carries each partition's value

Background

In DuckDB, the filename argument of a file reader adds an extra filename column showing which file each row came from.[1] The dbt-utils macro union_relations combines relations with union all and adds a column, _dbt_source_relation, giving each record's source.[2]

When to use it

Add a source column whenever the tables being stacked do not already say where each row came from, such as monthly extracts with no month field. (Meridian's view) DuckDB's filename argument also accepts a string to use as the added column's name, which avoids a clash when the data already has a filename column.[1] In dbt-utils, union_relations takes a source_column_name argument to rename the source column, and passing None omits the column.[2]

A source column makes rows from different inputs differ, so if one copy of each row is wanted across all inputs, drop duplicates before adding it. (Meridian's view) Store a short, stable label rather than a full path where the path holds machine-specific folders, so the column means the same thing on every run. (Meridian's view)

The method

Hive partitioning splits a table into files in folders, where the value of each partition key is determined by the folder's name.[3] In DuckDB, reading Hive-partitioned files with hive_partitioning set to true includes the partition columns read from the directory structure.[3]

Sources

  1. [1]Reading Multiple Files – DuckDBdocumentation
  2. [2]README.mddocumentation
  3. [3]Hive Partitioning – DuckDBdocumentation