Shape / columns · All operations
Mask or anonymise values
A variant of Hash or mask a column
Replace values with masked, shuffled, random or fake ones that hide the originals.
Status
- In arcform
Runs as a SQL step
arcform carries it as a
sql:step today.
The DuckDB step
Mask, shuffle, randomise and fake values
CREATE TEMP TABLE people AS SELECT * FROM (VALUES
(1, 'Alice Ng', '4111111111111111'), (2, 'Bob Okafor', '5500005555555559'), (3, 'Carla Diaz', '1234'),
(4, 'Dev Patel', '12'), (5, 'Eve Larsen', NULL), (6, 'Frank Moore', '378282246310005'),
(7, 'Alice Ng', '12345')
) t(id, name, card);
-- masked keeps the last four characters of a longer value and hides all of a short one. Shuffled puts the rows in the order of a
-- hash of the id and gives each row the name of the next row in that order (the last takes the first), so no row keeps its own name.
-- Random replaces the name with a number; fake picks a stand-in from a list by a hash of the name, so a name that occurs twice
-- gets the same stand-in.
CREATE TEMP TABLE result AS
WITH ranked AS (SELECT id, name, row_number() OVER (ORDER BY hash(id)) AS pos, count(*) OVER () AS n FROM people),
moved AS (SELECT a.id, b.id AS shuffled_from, b.name AS shuffled_name FROM ranked a JOIN ranked b ON b.pos = a.pos % a.n + 1)
SELECT p.id, p.name,
CASE WHEN length(card) > 4 THEN repeat('*', length(card) - 4) || right(card, 4) ELSE repeat('*', length(card)) END AS card_masked,
m.shuffled_from, m.shuffled_name,
'user_' || lpad((random() * 99999)::INTEGER::VARCHAR, 5, '0') AS random_name,
['Pat Example', 'Sam Sample', 'Kim Placeholder'][(hash(p.name) % 3 + 1)::BIGINT] AS fake_name
FROM people p JOIN moved m USING (id);
-- random_name differs on every run, so it is checked below and not printed
SELECT * EXCLUDE (random_name) FROM result ORDER BY id;
SELECT CASE WHEN ((SELECT list(card_masked ORDER BY id) FROM result) = ['************1111', '************5559', '****', '**', NULL, '***********0005', '*2345'] AND (SELECT count(*) FROM result WHERE shuffled_from = id) = 0 AND (SELECT count(DISTINCT shuffled_from) FROM result) = 7 AND (SELECT list(name ORDER BY name, id) FROM result) = (SELECT list(shuffled_name ORDER BY shuffled_name, id) FROM result) AND (SELECT count(*) FROM result r JOIN people p ON p.id = r.shuffled_from WHERE p.name <> r.shuffled_name) = 0 AND (SELECT count(*) FROM result WHERE regexp_matches(random_name, '^user_[0-9]{5}$') AND random_name <> name) = 7 AND (SELECT count(*) FROM result WHERE fake_name = name) = 0 AND (SELECT count(*) FROM result WHERE fake_name IN ('Pat Example', 'Sam Sample', 'Kim Placeholder')) = 7 AND (SELECT count(DISTINCT fake_name) FROM result WHERE name = 'Alice Ng') = 1) THEN 'as expected' ELSE error('the cards were not masked to their last four, the names not moved to other rows, randomised and faked, or the two Alice Ng rows did not get the same stand-in') END AS verdict;- The statement builds its own data.
- It needs nothing installed beyond DuckDB itself.
- The statement checks its own result and raises an error when it is wrong: “the cards were not masked to their last four, the names not moved to other rows, randomised and faked, or the two Alice Ng rows did not get the same stand-in”.
- It passed on DuckDB
v2.0.0 (Cyanoptera) d54cbbe3df.
What other tools call it
Meridian's classification; an independent blind rating agreed on 95.7% of 324 rows (variant, version 3).
Bruin (1)
Column masking (ingestr assets)read 2026-09-27Apply a masking rule such as hash to a column during ingestion
Duckle (1)
Mask / Anonymizeread 2026-09-25mask a column in place: salted hash, partial, null or constant
EasyMorph (1)
Anonymize/Hashread 2026-09-27Replace values with anonymised ones while keeping the column's cardinality
RapidMiner Studio (1)
Obfuscateread 2026-09-27Replace nominal values and column names with random strings, saving the mapping
Talend (3)
tDataMaskingread 2026-09-27Replace column values with realistic substitutes using masking functions, one row out per row in
tDataShufflingread 2026-09-27Swap the values of chosen columns between rows, optionally within partition groups
tPatternMaskingread 2026-09-27Mask values that follow a pattern with random values drawn from given ranges or sets
VisiData (1)
setcol-fakeread 2026-09-23Replace current column for selected rows with generated fake values of a chosen type
Background
Masking systematically removes a field or replaces it with a value that does not preserve its analytic utility, such as asterisks or a random pseudonym.[1] Masked data must stay usable for valid test cycles, and must also look real and appear consistent.[2] Masking is more often applied to data outside a production system, such as data needed for application development and testing.[2]
When to use it
Masked data must stay meaningful to application logic, so a substituted card number may still need to pass a checksum test of the Luhn algorithm.[2] The Luhn algorithm, created by the IBM scientist Hans Peter Luhn, is a check digit formula that most credit card numbers use to tell valid numbers from mistyped ones.[3] Masked values may need to be consistent across databases, which needs repeatable masking, where the same input always yields the same output.[2] Deterministic masking replaces a value with the same value wherever it occurs, so that Lynne always becomes Denise.[2]
A masked data set can give identities away through its pattern, as when ten records in one high earning bracket match ten senior managers known to earn that much.[2] Mask the record, not the field: check what the masked columns reveal together, and keep each column a later test or join needs in a form that still passes its checks. (Meridian's view)
The method
Substitution replaces a value with another authentic-looking value, such as a real-life surname drawn from a lookup file.[2] Shuffling derives its substitutes from the same column being masked, moving the values randomly within it.[2] Masking out hides part of a value, as when a call centre operator sees only the last four digits of a credit card number.[2] Presidio's mask operator replaces a chosen number of a value's characters with a masking character, counting from either the start or the end.[4]
Static data masking is applied to a copy of the data at rest, often after the copy is cut down to the subset a round of testing needs.[2] Dynamic data masking happens at run time and on demand, so no second data source is needed to store the masked data.[2]
Compared with the alternatives
Article 4(5) of the EU General Data Protection Regulation (GDPR) defines pseudonymisation as processing personal data so that it can no longer be attributed to a specific person without additional information kept separately.[5] Statistical data obfuscation, such as differential privacy, perturbs the data at random while preserving some of its statistical properties, as an alternative to static data masking.[2]
Sources
- [1]De-Identifying Government Datasets: Techniques and Governancestandard
- [2]Data masking - Wikipediasecondary source
- [3]Luhn algorithm - Wikipediasecondary source
- [4]Home - Presidiodocumentation
- [5]Pseudonymization - Wikipediasecondary source