Shape / columns · All operations

Hash or mask a column

Hash, checksum, mask, encrypt, anonymise.

Status

In arcform

Runs as a SQL step

arcform carries it as a sql: step today.

Shape of the result
one row out per row in, with columns added or changed
Level
one clause of SQL

The DuckDB step

Hash and mask

It reads the orders table the proofs share. Create it first:

SQL
CREATE TABLE orders AS SELECT * FROM (VALUES
  (1,  'Alice Ng',      'north', DATE '2026-01-03', 120.00, 'widget', 'a|b',   '{"channel":"web","items":[1,2]}'),
  (2,  'alice ng',      'north', DATE '2026-01-05',  80.50, 'gadget', 'b',     '{"channel":"shop","items":[3]}'),
  (3,  'Bob  Okafor',   'south', DATE '2026-01-05', 300.00, 'widget', 'a|c',   '{"channel":"web","items":[4,5,6]}'),
  (4,  'Carla Díaz',    'east',  DATE '2026-01-09',  NULL,  'gizmo',  NULL,    '{"channel":"web","items":[]}'),
  (5,  'Bob Okafor',    'south', DATE '2026-01-12',  45.25, 'gadget', 'c',     '{"channel":"phone","items":[7]}'),
  (6,  'Dev Patel',     'west',  DATE '2026-01-12', 510.00, 'widget', 'a',     '{"channel":"web","items":[8,9]}'),
  (7,  'Carla Diaz',    'east',  DATE '2026-01-20',  99.99, 'gizmo',  'b|c',   '{"channel":"shop","items":[10]}'),
  (8,  'Dev Patel',     'west',  DATE '2026-02-02', 510.00, 'widget', 'a',     '{"channel":"web","items":[8,9]}'),
  (9,  'Eve Larsen',    NULL,    DATE '2026-02-03',  15.00, 'gadget', NULL,    '{"channel":"web","items":[11]}'),
  (10, 'Alice Ng',      'north', DATE '2026-02-10', 220.00, 'gizmo',  'a|b|c', '{"channel":"phone","items":[12,13]}')
) t(order_id, customer, region, order_date, amount, product, tags, meta);

Then run the statement:

SQL
CREATE TEMP TABLE result AS
SELECT customer, sha256(customer) AS digest, left(customer, 1) || repeat('*', length(customer) - 1) AS masked FROM orders;
SELECT * FROM result LIMIT 3;
SELECT CASE WHEN ((SELECT count(DISTINCT digest) = count(DISTINCT customer) AND min(length(digest)) = 64 AND max(masked) FILTER (WHERE customer = 'Dev Patel') = 'D********' FROM result)) THEN 'as expected' ELSE error('the digest is not one per name at 64 characters, or the mask is wrong') END AS verdict;
  • It needs nothing installed beyond DuckDB itself.
  • The statement checks its own result and raises an error when it is wrong: “the digest is not one per name at 64 characters, or the mask is wrong”.
  • It passed on DuckDB v2.0.0 (Cyanoptera) d54cbbe3df.

Variants

Hash a value

Replace or extend a value with its digest under an algorithm such as MD5 or SHA-256.

Sign or verify content

Sign content with a key, or check a signature or MAC against it.

Restore masked values

Recover the original values from a reversible masking.

What other tools call it

Meridian's classification; an independent blind rating agreed on 86.2% of 312 rows (exact class, version 2).

Apache DataFusion (1)
  • Hashing Functionsread 2026-09-27

    6 functions: md5, sha256, sha512, sha224, digest, …

Apache Hop (3)
Apache NiFi (8)
  • CryptographicHashContentread 2026-09-27

    Hash the content with a chosen algorithm and write the digest to an attribute

  • DecryptContentAgeread 2026-09-27

    Decrypt content encrypted under the age-encryption.org v1 specification

  • DecryptContentPGPread 2026-09-27

    Decrypt the contents of OpenPGP messages

  • EncryptContentAgeread 2026-09-27

    Encrypt content under the age-encryption.org v1 specification

  • EncryptContentPGPread 2026-09-27

    Encrypt content with OpenPGP

  • SignContentPGPread 2026-09-27

    Sign content with an OpenPGP private key

  • VerifyContentMACread 2026-09-27

    Compute a message authentication code over content and compare it with a supplied MAC

  • VerifyContentPGPread 2026-09-27

    Verify OpenPGP signatures with public keys

Apache SeaTunnel (1)
  • Encryptread 2026-09-27

    Next docs, not in 2.3.13: encrypt or decrypt named fields with AES-GCM or AES-CBC

AWS Glue (4)
  • CryptographicHashread 2026-09-27

    Hash the values of a column with a chosen algorithm

  • Decryptread 2026-09-27

    Decrypt column values that were encrypted under an AWS KMS key

  • Detect PIIread 2026-09-27

    Detect personally identifiable information per row or per column, then mask, remove or report it

  • Encryptread 2026-09-27

    Encrypt column values with an AWS KMS key

Bruin (2)
dbt-utils (1)
Duckle (4)
  • Hashread 2026-09-25

    hash a column with md5, sha1 or sha256

  • Mask / Anonymizeread 2026-09-25

    mask a column in place: salted hash, partial, null or constant

  • PII Redactread 2026-09-25

    redact email, phone, SSN and card numbers by regex

  • Row Hash (fingerprint)read 2026-09-25

    hash N columns into one fingerprint column

EasyMorph (1)
  • Anonymize/Hashread 2026-09-27

    Replace values with anonymised ones while keeping the column's cardinality

Ibis (3)
  • Hashread 2026-09-27

    Integer hash of a value

  • HashBytesread 2026-09-27

    Hash a string or binary value to bytes by a named algorithm (md5, sha1, sha256, sha512, cityHash64 and others)

  • HexDigestread 2026-09-27

    Hex-string digest of a string or binary value by md5, sha1, sha256 or sha512

KNIME Analytics Platform (1)
OpenRefine (1)
Pentaho Data Integration (1)
  • Add a Checksumread 2026-09-27

    Add a checksum column computed over each input row

Polars (2)
  • DataFrame.hash_rowsread 2026-09-27

    Hash each row across all its columns into one integer column

  • Expr.hashread 2026-09-27

    Hash every value of a column to an integer, with optional seeds

RapidMiner Studio (2)
  • De-Obfuscateread 2026-09-27

    Restore original values and column names using the obfuscation map file

  • Obfuscateread 2026-09-27

    Replace nominal values and column names with random strings, saving the mapping

SQLMesh (1)
  • @GENERATE_SURROGATE_KEYread 2026-09-27

    Hash the concatenated text of several columns into a surrogate key, MD5 by default

Talend (8)
  • tAddCRCRowread 2026-09-27

    Add a CRC checksum column computed from one or more columns, as a surrogate key

  • tDataDecryptread 2026-09-27

    Decrypt column values that tDataEncrypt encrypted

  • tDataEncryptread 2026-09-27

    Encrypt column values into cipher text with a password and cryptographic file or a secret key

  • tDataMaskingread 2026-09-27

    Replace column values with realistic substitutes using masking functions, one row out per row in

  • tDataShufflingread 2026-09-27

    Swap the values of chosen columns between rows, optionally within partition groups

  • tDataUnmaskingread 2026-09-27

    Recover original values that tDataMasking masked with a reversible FF1 method and password

  • tPatternMaskingread 2026-09-27

    Mask values that follow a pattern with random values drawn from given ranges or sets

  • tPatternUnmaskingread 2026-09-27

    Recover original values that tPatternMasking masked with a reversible FF1 method and password

VisiData (1)
  • setcol-fakeread 2026-09-23

    Replace current column for selected rows with generated fake values of a chosen type

Background

Data masking modifies sensitive data so that it is of little or no value to unauthorised intruders while staying usable by software or authorised staff.[1] A cryptographic hash function maps a binary string of any length to one of fixed size, and is quick to calculate yet hard to reverse.[2] Data masking can also be called anonymisation or tokenisation, depending on the context.[1]

When to use it

Hashing turns a value such as an email address or a phone number into a hash in a consistent way, so the same input always creates the same hash.[3] Companies often hash identifying data they are uncomfortable writing down or sharing but still want to store for matching against later.[3] Hashes of values drawn from a small set, such as phone numbers, email addresses or Social Security Numbers, can be reversed by trying every possible value.[3][2]

A salt is random data fed into a hash together with the input, and it defends against precomputed tables and dictionaries of hashed values.[4] Presidio's hash operator generates a random salt for each entity unless one is supplied, and a consistent salt is needed to keep referential integrity across records.[5] Decide before hashing whether the hashed column must still join to other tables, because that decides whether its salt is fixed or random. (Meridian's view)

Collisions against MD5 can be calculated within seconds, which makes MD5 unsuitable for most uses where a cryptographic hash is required.[2] Keep MD5 for spotting accidental differences between copies, and reach for SHA-256 or a keyed hash wherever someone might try to forge a match. (Meridian's view)

The method

A cryptographic hash function maps a binary string of any length to a binary string of a fixed size of n bits.[2] Even a small change in the input gives a very different hash, which makes hashing an efficient and consistent way to check whether two copies of data match.[2] MD5 was designed by Ronald Rivest in 1991 to replace an earlier hash function, MD4, and produces a digest of 128 bits.[2] Secure Hash Algorithm 2 (SHA-2) is a set of cryptographic hash functions designed by the United States National Security Agency and first published in 2001.[2] SHA-256 produces a digest of 32 bytes and SHA-512 one of 64 bytes.[2] In DuckDB, md5 returns the MD5 hash of a string as a VARCHAR, and sha256 returns the SHA-256 hash as a VARCHAR.[6]

A hash-based message authentication code (HMAC) combines a cryptographic hash function with a secret key, and can verify both the integrity and the authenticity of a message.[7] The HMAC construction was first defined and analysed in a 1996 paper by Mihir Bellare, Ran Canetti and Hugo Krawczyk.[7]

In Presidio, a deanonymizer reverts an anonymisation only where the operation is reversible, as encryption is.[5] Pseudonymised data can be restored to its original state with additional information that allows individuals to be re-identified, whereas anonymisation is meant to prevent re-identification.[8]

Compared with the alternatives

Cryptographic hash values are sometimes called fingerprints, checksums or message digests, even though each of those terms stands for a more general function with different properties.[2]

Sources

  1. [1]Data masking - Wikipediasecondary source
  2. [2]Cryptographic hash function - Wikipediasecondary source
  3. [3]No, hashing still doesn't make your data anonymoussecondary source
  4. [4]Salt (cryptography) - Wikipediasecondary source
  5. [5]Home - Presidiodocumentation
  6. [6]Utility Functions – DuckDBdocumentation
  7. [7]HMAC - Wikipediasecondary source
  8. [8]Pseudonymization - Wikipediasecondary source

Types it works with

FineType's registry gives 12 types a personal-data flag.

Type
Credit Card Number
Full Address
Indian PAN
US Social Security Number
Email Address
Email with Display Name
First Name
Full Name
Last Name
Password
Phone Number (E.164)
Phone Number

Used in

One Protocol uses this operation: