Control / script · All operations

Transform each row with code

A variant of Run a script

Run a user function or snippet on every row, value or column to compute or change fields.

Status

In arcform

Not built yet

arcform does not do this yet; it would live in how a Protocol runs. Its place on the roadmap

What other tools call it

Meridian's classification; an independent blind rating agreed on 95.7% of 324 rows (variant, version 3).

Apache Hop (2)
  • JavaScriptread 2026-09-27

    Run JavaScript over each row to compute or change fields

  • Scriptread 2026-09-27

    Run a script in JavaScript, Groovy, Python or another JSR-223 language over rows

Apache NiFi (1)
  • ScriptedTransformRecordread 2026-09-27

    Run a user script against each record to transform it, drop it or fork more records

Apache SeaTunnel (2)
  • DynamicCompileread 2026-09-27

    Process rows with user Java, Groovy or Scala source compiled at run time

  • Pythonread 2026-09-27

    Next docs, not in 2.3.13: run a user Python function per row and append the fields it returns

AWS Glue (1)
  • Mapread 2026-09-27

    Apply a user function to every record

Duckle (3)
  • JavaScript UDFread 2026-09-25

    per-row user JavaScript transform function

  • Python UDFread 2026-09-25

    transform rows with a user Python function

  • WebAssembly UDFread 2026-09-25

    per-row transform by a user WebAssembly module

KNIME Analytics Platform (2)
  • Java Snippetread 2026-09-27

    Run user-written Java per row to create or replace columns and flow variables

  • Java Snippet (simple)read 2026-09-27

    Run a short user-written Java expression per row into one new column

OpenRefine (2)
  • Clojureread 2026-09-27

    Write an expression as Clojure code; fewer variables are available than in GREL

  • Jythonread 2026-09-27

    Write an expression as Python code run on the bundled Jython, with Python and Java libraries

Pentaho Data Integration (2)
  • Modified Java Script Valueread 2026-09-27

    Run JavaScript programs over rows

  • Scriptread 2026-09-27

    Compute values with Ruby, Python, Groovy, JavaScript or other scripting languages

Polars (5)
  • DataFrame.map_columnsread 2026-09-27

    Apply an eager user function to chosen columns

  • DataFrame.map_rowsread 2026-09-27

    Run a user function on every row passed as a tuple

  • Expr.map_batchesread 2026-09-27

    Run a user Python function on a whole column passed as a Series

  • Expr.map_elementsread 2026-09-27

    Run a user Python function on every element of a column

  • map_batchesread 2026-09-27

    Run a user function over several columns passed as whole Series

Talend (2)
  • tDotNETRowread 2026-09-27

    Pass rows to and from methods of .NET classes or custom DLL files

  • tJavaRowread 2026-09-27

    Run custom Java code on each row of the flow

Trifacta (1)
VisiData (1)
  • addcol-shellread 2026-09-23

    Add columns with stdout and stderr of a shell command run per row, $column as variables

Background

In many programming languages, map is a higher-order function that applies a given function to each element of a collection, such as a list or set.[1] In Apache Hop, the code typed into the JavaScript transform's script area is executed once for each input row.[2] In Polars, map_elements maps a custom function to each element of a column.[3]

When to use it

In Polars, map_elements is much slower than the native expressions API, and is meant only for logic that cannot be implemented otherwise.[3] In Polars, the native expression engine runs in Rust while a user's Python function runs in Python.[4] In Polars, native expressions can be parallelised, which a Python function typically cannot, and can be logically optimised, which a Python function cannot.[4] Write the change as a built-in expression first, and keep row code for the logic no expression can say. (Meridian's view)

In Apache Hop, integers of more than about 15 to 16 significant digits can change when they pass through the JavaScript transform, because of JavaScript's internal floating-point number representation.[2] Carry long identifiers, such as account or card numbers, through row code as text rather than as numbers. (Meridian's view)

In Polars, map_elements tries to infer the output data type by calling the function with dummy data unless return_dtype is set, and setting it is recommended whenever possible.[3] In Polars, an expensive function can be given an @lru_cache decorator so that it is not called more than once for a given input.[3]

The method

Map is often called apply-to-all when considered in functional form.[1] The concept of a map is not limited to lists: it works for sequential containers, tree-like containers, and abstract containers such as futures and promises.[1]

Polars' DataFrame.map_rows passes each row to the user's function as a tuple of values.[4] Because the function given to DataFrame.map_rows may drop, rearrange, transform or add columns, Polars cannot track column names through it, and map_elements is the way to keep them.[4] In Polars, using a Python function forces the DataFrame to be materialised in memory.[4]

Apache Hop's JavaScript transform is not an input transform, so it requires an input stream from the pipeline.[2] Apache Hop's JavaScript transform allows multiple scripts in a single transform instance.[2]

Compared with the alternatives

Conceptually, a user-defined function used in a SELECT statement over a table is evaluated once per row.[5] Row code and a registered function can do the same per-row work; the difference is where the logic lives, in one pipeline step or under a name that every query can call. (Meridian's view)

Sources

  1. [1]Map (higher-order function) - Wikipediasecondary source
  2. [2]JavaScript | Apache Hopdocumentation
  3. [3]polars.Expr.map_elements — Polars documentationdocumentation
  4. [4]polars.DataFrame.map_rows — Polars documentationdocumentation
  5. [5]User-defined function - Wikipediasecondary source