Shape / columns · All operations
Pack columns into a nested value
A variant of Parse nested values
Combine several columns of each row into one struct, list, JSON, XML or Avro column.
Status
- In arcform
Runs as a SQL step
arcform carries it as a
sql:step today, with theavroextension.
The DuckDB step
Pack several columns of each row into one struct, list, JSON, XML and Avro column
CREATE TEMP TABLE items AS SELECT * FROM (VALUES
(1, 'Alice Ng', 120.00, 'north', 'widget'), (2, 'Tom & <Jerry>', NULL, NULL, 'gizmo')
) t(id, customer, amount, region, product);
-- each text is escaped (& first, then < and >) before it is put between tags; a missing value gives an empty tag
CREATE MACRO esc(x) AS replace(replace(replace(x, '&', '&'), '<', '<'), '>', '>');
CREATE TEMP TABLE result AS
SELECT id,
{'customer': customer, 'amount': amount} AS as_struct,
[region, product] AS as_list,
to_json({'customer': customer, 'amount': amount}) AS as_json,
'<row><customer>' || coalesce(esc(customer), '') || '</customer><amount>' || coalesce(amount::VARCHAR, '') || '</amount></row>' AS as_xml
FROM items;
SELECT * FROM result ORDER BY id;
DESCRIBE result;
SELECT CASE WHEN ((SELECT list(column_name) FROM (DESCRIBE result)) = ['id', 'as_struct', 'as_list', 'as_json', 'as_xml'] AND (SELECT as_struct.customer || '/' || as_struct.amount FROM result WHERE id = 1) = 'Alice Ng/120.00' AND (SELECT as_list FROM result WHERE id = 2) = [NULL, 'gizmo'] AND (SELECT as_json::VARCHAR FROM result WHERE id = 2) = '{"customer":"Tom & <Jerry>","amount":null}' AND (SELECT as_xml FROM result WHERE id = 2) = '<row><customer>Tom & <Jerry></customer><amount></amount></row>' AND (SELECT as_xml FROM result WHERE id = 1) = '<row><customer>Alice Ng</customer><amount>120.00</amount></row>') THEN 'as expected' ELSE error('customer and amount were not packed into one struct, one JSON and one escaped XML column, or region and product into one list') END AS verdict;
-- Avro: the packed struct is written to an Avro file and read back as one column
COPY (SELECT id, as_struct FROM result) TO '<your_scratch_dir>/packed.avro' (FORMAT avro);
SELECT CASE WHEN ((SELECT count(*) FROM read_avro('<your_scratch_dir>/packed.avro') WHERE as_struct.customer = 'Tom & <Jerry>' AND as_struct.amount IS NULL) = 1 AND (SELECT count(*) FROM read_avro('<your_scratch_dir>/packed.avro')) = 2) THEN 'as expected' ELSE error('the packed struct did not come back from the Avro file as one column') END AS verdict;Replace <your_scratch_dir> with a directory of your own that you can write to before running the statement.
- The statement writes
packed.avroitself, in<your_scratch_dir>. - It needs the
avroextension. - The statement checks its own result and raises an error when it is wrong: “customer and amount were not packed into one struct, one JSON and one escaped XML column, or region and product into one list; then the packed struct did not come back from the Avro file as one column”.
- It passed on DuckDB
v2.0.0 (Cyanoptera) d54cbbe3df.
What other tools call it
Meridian's classification; an independent blind rating agreed on 95.7% of 324 rows (variant, version 3).
Alteryx Designer (1)
JSON Buildread 2026-09-27Rebuild JSON text from the name and value rows of JSON Parse
Apache DataFusion (1)
Struct Functionsread 2026-09-273 functions, 1 of them aliases: named_struct, row, struct
Apache Hop (2)
Add XMLread 2026-09-27Encode several fields of a row as an XML fragment in a new string column
Avro encoderead 2026-09-27Encode chosen fields into a single Avro record column
Apache NiFi (2)
AttributesToCSVread 2026-09-27Serialise FlowFile attributes as a CSV line, written to an attribute or to content
AttributesToJSONread 2026-09-27Serialise FlowFile attributes as a JSON object, written to an attribute or to content
Duckle (1)
Merge Objectsread 2026-09-25merge JSON objects
EasyMorph (1)
Construct JSONread 2026-09-27Build a JSON object from table values
Ibis (2)
Arrayread 2026-09-27Build an array value from a list of expressions
StructColumnread 2026-09-27Build a struct column from named expressions
KNIME Analytics Platform (6)
Column to XMLread 2026-09-27Build one XML element per row from column values, with custom name and attributes
Columns to Collectionread 2026-09-27Collect several columns into one list or set column
Columns to JSONread 2026-09-27Build one JSON object per row from selected columns
JSON Column Combinerread 2026-09-27Combine several JSON columns into one JSON column
Nominal Probability Distribution Creatorread 2026-09-27Pack probability columns, or one-hot a string column, into a probability-distribution cell
XML Column Combinerread 2026-09-27Merge several XML columns into one XML column
Pentaho Data Integration (1)
Add XMLread 2026-09-27Encode several fields into an XML fragment
Polars (5)
DataFrame.to_structread 2026-09-27Pack the columns of each row into one struct value, giving a Series
concat_arrread 2026-09-27Join several columns row by row into one fixed-size array column
concat_listread 2026-09-27Join several columns row by row into one list column
listread 2026-09-27Collect several columns into a list column, one element per expression
structread 2026-09-27Pack several columns into one struct column
Power Query M (1)
Table.CombineColumnsToRecordread 2026-09-27Merge several columns into one record-valued column keyed by the source column names
RapidMiner Studio (1)
Data To JSONread 2026-09-27Build nested JSON documents from the rows of a table
Talend (5)
tWriteAvroFieldsread 2026-09-27Serialise rows into Avro binaries for components that need serialised input
tWriteDynamicFieldsread 2026-09-27Combine input columns into one output column of Dynamic type
tWriteJSONFieldread 2026-09-27Build a JSON field from the columns of the incoming rows
tWriteXMLFieldread 2026-09-27Build an XML structure from the input columns and write it into a field
tWriteXMLFieldsread 2026-09-27Serialise the columns of each row into one XML string or byte array column
Trifacta (2)
Nest Transformread 2026-09-27Build an Object or Array column from the names and values of several columns
Nest columnsread 2026-09-27Convert columns into an Object or Array column
VisiData (1)
contract-source-colsread 2026-09-23From Columns Sheet, contract selected columns into one grouped column
Background
In DuckDB, a STRUCT is typically used to nest multiple columns into a single column, and the nested column can be of any type.[1] In DuckDB, struct_pack(name := expr, ...) or the notation {'name': expr, ...} builds a STRUCT from expressions.[1] A JavaScript Object Notation (JSON) object is a pair of curly brackets around zero or more name/value pairs, each name being a string.[2]
When to use it
A DuckDB LIST holds values that must all have the same underlying type, while each entry of a STRUCT may have a different type.[3][4] Pack into a struct when the columns differ in type and their names matter; pack into a list only when the columns hold the same kind of thing, such as several phone numbers. (Meridian's view) Every row of a DuckDB STRUCT column must have the same keys, and the names of the entries are part of the schema.[1]
The method
DuckDB builds a LIST with list_value(expr, ...) or the equivalent bracket notation [expr, ...].[3] DuckDB's json_object creates a JSON object from key and value pairs given as arguments, and json_array creates a JSON array from its arguments.[5] A DuckDB TUPLE is a struct whose entries have no names and are referenced by position, created with the row function.[1]
Compared with the alternatives
To build a list from the values of a table column across rows, DuckDB uses the list aggregate function.[3] DuckDB's json_group_array and json_group_object return a JSON array or object built from all the values in an aggregation.[6] Pack when each row should keep its own nested value; aggregate when many rows should collapse into one. (Meridian's view)
Sources
- [1]Struct Data Type – DuckDBdocumentation
- [2]RFC 8259: The JavaScript Object Notation (JSON) Data Interchange Formatstandard
- [3]List Type – DuckDBdocumentation
- [4]Data Types – DuckDBdocumentation
- [5]Creating JSON – DuckDBdocumentation
- [6]JSON Processing Functions – DuckDBdocumentation