Join / union · All operations
Append tables with a source column
A variant of Append tables
Stack the rows of several tables and add a column that records which table each row came from.
Status
- In arcform
Runs as a SQL step
arcform carries it as a
sql:step today.
The DuckDB step
Append tables with a column naming the source
CREATE TEMP TABLE t1 AS SELECT * FROM (VALUES (1, 'a'), (2, 'b')) t(id, v);
CREATE TEMP TABLE t2 AS SELECT * FROM (VALUES (2, 'b'), (3, 'c')) t(id, v);
COPY t1 TO '<your_scratch_dir>/tag_one.csv' (HEADER);
COPY t2 TO '<your_scratch_dir>/tag_two.csv' (HEADER);
-- tables: a literal for each; the row (2, b) is in both and stays twice
CREATE TEMP TABLE from_tables AS
SELECT 't1' AS source, * FROM t1 UNION ALL BY NAME SELECT 't2' AS source, * FROM t2;
-- files: the reader adds the path of the file each row came from (filename = true); only its name is kept
CREATE TEMP TABLE from_files AS
SELECT parse_filename(filename) AS source, id, v
FROM read_csv(['<your_scratch_dir>/tag_one.csv', '<your_scratch_dir>/tag_two.csv'], filename = true);
SELECT * FROM from_tables ORDER BY source, id;
SELECT * FROM from_files ORDER BY source, id;
SELECT CASE WHEN ((SELECT list((source, id) ORDER BY source, id) = [('t1', 1), ('t1', 2), ('t2', 2), ('t2', 3)] FROM from_tables)
AND (SELECT list((source, id, v) ORDER BY source, id) = [('tag_one.csv', 1, 'a'), ('tag_one.csv', 2, 'b'), ('tag_two.csv', 2, 'b'), ('tag_two.csv', 3, 'c')] FROM from_files)
AND (SELECT list(column_name) FROM (DESCRIBE from_tables)) = ['source', 'id', 'v']) THEN 'as expected' ELSE error('each of the 4 rows should name the table or file it came from, the row found in both tables should appear once for each, and the source column should come first') END AS verdict;Replace <your_scratch_dir> with a directory of your own that you can write to before running the statement.
- The statement writes
tag_one.csvandtag_two.csvitself, in<your_scratch_dir>. - It needs nothing installed beyond DuckDB itself.
- The statement checks its own result and raises an error when it is wrong: “each of the 4 rows should name the table or file it came from, the row found in both tables should appear once for each, and the source column should come first”.
- It passed on DuckDB
v2.0.0 (Cyanoptera) d54cbbe3df.
What other tools call it
Meridian's classification; an independent blind rating agreed on 95.7% of 324 rows (variant, version 3).
dbt-utils (1)
union_relationsread 2026-09-27UNION ALL a list of relations, aligning columns by name, null-filling gaps, adding a source column
Power Query M (1)
Table.FromPartitionsread 2026-09-27Append partitioned tables into one, adding a column that carries each partition's value
Background
In DuckDB, the filename argument of a file reader adds an extra filename column showing which file each row came from.[1] The dbt-utils macro union_relations combines relations with union all and adds a column, _dbt_source_relation, giving each record's source.[2]
When to use it
Add a source column whenever the tables being stacked do not already say where each row came from, such as monthly extracts with no month field. (Meridian's view) DuckDB's filename argument also accepts a string to use as the added column's name, which avoids a clash when the data already has a filename column.[1] In dbt-utils, union_relations takes a source_column_name argument to rename the source column, and passing None omits the column.[2]
A source column makes rows from different inputs differ, so if one copy of each row is wanted across all inputs, drop duplicates before adding it. (Meridian's view) Store a short, stable label rather than a full path where the path holds machine-specific folders, so the column means the same thing on every run. (Meridian's view)
The method
Hive partitioning splits a table into files in folders, where the value of each partition key is determined by the folder's name.[3] In DuckDB, reading Hive-partitioned files with hive_partitioning set to true includes the partition columns read from the directory structure.[3]
Sources
- [1]Reading Multiple Files – DuckDBdocumentation
- [2]README.mddocumentation
- [3]Hive Partitioning – DuckDBdocumentation