# Hash or mask a column Shape / columns hash, checksum, mask, encrypt, anonymise Page: ## Status Runs as a SQL step. arcform carries it as a `sql:` step today. ## The DuckDB step hash and mask It reads the `orders` table the proofs share. Create it first: ```sql CREATE TABLE orders AS SELECT * FROM (VALUES (1, 'Alice Ng', 'north', DATE '2026-01-03', 120.00, 'widget', 'a|b', '{"channel":"web","items":[1,2]}'), (2, 'alice ng', 'north', DATE '2026-01-05', 80.50, 'gadget', 'b', '{"channel":"shop","items":[3]}'), (3, 'Bob Okafor', 'south', DATE '2026-01-05', 300.00, 'widget', 'a|c', '{"channel":"web","items":[4,5,6]}'), (4, 'Carla Díaz', 'east', DATE '2026-01-09', NULL, 'gizmo', NULL, '{"channel":"web","items":[]}'), (5, 'Bob Okafor', 'south', DATE '2026-01-12', 45.25, 'gadget', 'c', '{"channel":"phone","items":[7]}'), (6, 'Dev Patel', 'west', DATE '2026-01-12', 510.00, 'widget', 'a', '{"channel":"web","items":[8,9]}'), (7, 'Carla Diaz', 'east', DATE '2026-01-20', 99.99, 'gizmo', 'b|c', '{"channel":"shop","items":[10]}'), (8, 'Dev Patel', 'west', DATE '2026-02-02', 510.00, 'widget', 'a', '{"channel":"web","items":[8,9]}'), (9, 'Eve Larsen', NULL, DATE '2026-02-03', 15.00, 'gadget', NULL, '{"channel":"web","items":[11]}'), (10, 'Alice Ng', 'north', DATE '2026-02-10', 220.00, 'gizmo', 'a|b|c', '{"channel":"phone","items":[12,13]}') ) t(order_id, customer, region, order_date, amount, product, tags, meta); ``` Then run the statement: ```sql CREATE TEMP TABLE result AS SELECT customer, sha256(customer) AS digest, left(customer, 1) || repeat('*', length(customer) - 1) AS masked FROM orders; SELECT * FROM result LIMIT 3; SELECT CASE WHEN ((SELECT count(DISTINCT digest) = count(DISTINCT customer) AND min(length(digest)) = 64 AND max(masked) FILTER (WHERE customer = 'Dev Patel') = 'D********' FROM result)) THEN 'as expected' ELSE error('the digest is not one per name at 64 characters, or the mask is wrong') END AS verdict; ``` It needs nothing installed beyond DuckDB itself. ## Variants - [Encrypt or decrypt values](https://meridian.online/operations/hash-or-mask-a-column/encrypt-or-decrypt-values): Encrypt values under a key so they can be decrypted later, or decrypt them back. - Hash a value: Replace or extend a value with its digest under an algorithm such as MD5 or SHA-256. - [Mask or anonymise values](https://meridian.online/operations/hash-or-mask-a-column/mask-or-anonymise-values): Replace values with masked, shuffled, random or fake ones that hide the originals. - [Find and redact personal data](https://meridian.online/operations/hash-or-mask-a-column/find-and-redact-personal-data): Detect personal data such as emails and card numbers in text, then redact or report it. - [Hash several columns into one key](https://meridian.online/operations/hash-or-mask-a-column/hash-columns-into-a-key): Combine several columns into one checksum or surrogate-key column. - Sign or verify content: Sign content with a key, or check a signature or MAC against it. - Restore masked values: Recover the original values from a reversible masking. ## What other tools call it Meridian's classification; an independent blind rating agreed on 86.2% of 312 rows (exact class, version 2). - Bruin: [`Column masking (ingestr assets)`](https://getbruin.com/docs/bruin/assets/columns.html) (read 2026-09-27) - Bruin: [`bruin.generate_surrogate_key(fields)`](https://getbruin.com/docs/bruin/assets/templating/macros.html) (read 2026-09-27) - Apache DataFusion: [`Hashing Functions`](https://datafusion.apache.org/user-guide/sql/scalar_functions.html) (read 2026-09-27) - dbt-utils: [`generate_surrogate_key`](https://raw.githubusercontent.com/dbt-labs/dbt-utils/main/README.md) (read 2026-09-27) - Duckle: [`Hash`](https://github.com/slothflowlabs/duckle/blob/f85f8960edfb6efbf4cb89d58a7b32e16df0e867/website/docs/components.html) (read 2026-09-25) - Duckle: [`Mask / Anonymize`](https://github.com/slothflowlabs/duckle/blob/f85f8960edfb6efbf4cb89d58a7b32e16df0e867/frontend/src/workflow-ui/palette-data.ts) (read 2026-09-25) - Duckle: [`PII Redact`](https://github.com/slothflowlabs/duckle/blob/f85f8960edfb6efbf4cb89d58a7b32e16df0e867/website/docs/components.html) (read 2026-09-25) - Duckle: [`Row Hash (fingerprint)`](https://github.com/slothflowlabs/duckle/blob/f85f8960edfb6efbf4cb89d58a7b32e16df0e867/website/docs/components.html) (read 2026-09-25) - EasyMorph: [`Anonymize/Hash`](https://help.easymorph.com/doku.php?id=transformations:anonymizehash) (read 2026-09-27) - AWS Glue: [`CryptographicHash`](https://docs.aws.amazon.com/glue/latest/dg/aws-glue-api-pyspark-transforms-CryptographicHash.html) (read 2026-09-27) - AWS Glue: [`Decrypt`](https://docs.aws.amazon.com/glue/latest/dg/aws-glue-api-pyspark-transforms-Decrypt.html) (read 2026-09-27) - AWS Glue: [`Detect PII`](https://docs.aws.amazon.com/glue/latest/dg/detect-PII.html) (read 2026-09-27) - AWS Glue: [`Encrypt`](https://docs.aws.amazon.com/glue/latest/dg/aws-glue-api-pyspark-transforms-Encrypt.html) (read 2026-09-27) - Apache Hop: [`Add a checksum`](https://hop.apache.org/manual/latest/pipeline/transforms/addchecksum.html) (read 2026-09-27) - Apache Hop: [`PGP decrypt stream`](https://hop.apache.org/manual/latest/pipeline/transforms/pgpdecryptstream.html) (read 2026-09-27) - Apache Hop: [`PGP encrypt stream`](https://hop.apache.org/manual/latest/pipeline/transforms/pgpencryptstream.html) (read 2026-09-27) - Ibis: [`Hash`](https://github.com/ibis-project/ibis/blob/05d2b293344e86cb41c1a426071734f7e270335a/ibis/expr/operations/generic.py) (read 2026-09-27) - Ibis: [`HashBytes`](https://github.com/ibis-project/ibis/blob/05d2b293344e86cb41c1a426071734f7e270335a/ibis/expr/operations/generic.py) (read 2026-09-27) - Ibis: [`HexDigest`](https://github.com/ibis-project/ibis/blob/05d2b293344e86cb41c1a426071734f7e270335a/ibis/expr/operations/generic.py) (read 2026-09-27) - KNIME Analytics Platform: [`KNIME Presidio Extension (extension)`](https://api.hub.knime.com/extensions/org.knime.python.features.presidio) (read 2026-09-27) - Apache NiFi: [`CryptographicHashContent`](https://nifi.apache.org/components/org.apache.nifi.processors.standard.CryptographicHashContent/) (read 2026-09-27) - Apache NiFi: [`DecryptContentAge`](https://nifi.apache.org/components/org.apache.nifi.processors.cipher.DecryptContentAge/) (read 2026-09-27) - Apache NiFi: [`DecryptContentPGP`](https://nifi.apache.org/components/org.apache.nifi.processors.pgp.DecryptContentPGP/) (read 2026-09-27) - Apache NiFi: [`EncryptContentAge`](https://nifi.apache.org/components/org.apache.nifi.processors.cipher.EncryptContentAge/) (read 2026-09-27) - Apache NiFi: [`EncryptContentPGP`](https://nifi.apache.org/components/org.apache.nifi.processors.pgp.EncryptContentPGP/) (read 2026-09-27) - Apache NiFi: [`SignContentPGP`](https://nifi.apache.org/components/org.apache.nifi.processors.pgp.SignContentPGP/) (read 2026-09-27) - Apache NiFi: [`VerifyContentMAC`](https://nifi.apache.org/components/org.apache.nifi.processors.cipher.VerifyContentMAC/) (read 2026-09-27) - Apache NiFi: [`VerifyContentPGP`](https://nifi.apache.org/components/org.apache.nifi.processors.pgp.VerifyContentPGP/) (read 2026-09-27) - OpenRefine: [`Encoding and hashing`](https://openrefine.org/docs/manual/grelfunctions#encoding-and-hashing) (read 2026-09-27) - Pentaho Data Integration: [`Add a Checksum`](https://docs.pentaho.com/pdia-data-integration/9.3-data-integration/pdi-transformation-steps-reference-overview) (read 2026-09-27) - Polars: [`DataFrame.hash_rows`](https://docs.pola.rs/api/python/stable/reference/dataframe/computation.html) (read 2026-09-27) - Polars: [`Expr.hash`](https://docs.pola.rs/api/python/stable/reference/expressions/computation.html) (read 2026-09-27) - RapidMiner Studio: [`De-Obfuscate`](https://docs.rapidminer.com/latest/studio/operators/utility/data_anonymization/de_obfuscate.html) (read 2026-09-27) - RapidMiner Studio: [`Obfuscate`](https://docs.rapidminer.com/latest/studio/operators/utility/data_anonymization/obfuscate.html) (read 2026-09-27) - Apache SeaTunnel: [`Encrypt`](https://seatunnel.apache.org/docs/transforms/encrypt) (read 2026-09-27) - SQLMesh: [`@GENERATE_SURROGATE_KEY`](https://github.com/SQLMesh/sqlmesh/blob/23b237cea7218cc8f338b89c14f0957394a0003b/docs/concepts/macros/sqlmesh_macros.md) (read 2026-09-27) - Talend: [`tAddCRCRow`](https://help.qlik.com/talend/en-US/components/8.0/identification/identification-component) (read 2026-09-27) - Talend: [`tDataDecrypt`](https://help.qlik.com/talend/en-US/components/8.0/data-privacy/data-privacy-component) (read 2026-09-27) - Talend: [`tDataEncrypt`](https://help.qlik.com/talend/en-US/components/8.0/data-privacy/data-privacy-component) (read 2026-09-27) - Talend: [`tDataMasking`](https://help.qlik.com/talend/en-US/components/8.0/data-privacy/data-privacy-component) (read 2026-09-27) - Talend: [`tDataShuffling`](https://help.qlik.com/talend/en-US/components/8.0/data-privacy/data-privacy-component) (read 2026-09-27) - Talend: [`tDataUnmasking`](https://help.qlik.com/talend/en-US/components/8.0/data-privacy/data-privacy-component) (read 2026-09-27) - Talend: [`tPatternMasking`](https://help.qlik.com/talend/en-US/components/8.0/data-privacy/data-privacy-component) (read 2026-09-27) - Talend: [`tPatternUnmasking`](https://help.qlik.com/talend/en-US/components/8.0/data-privacy/data-privacy-component) (read 2026-09-27) - VisiData: [`setcol-fake`](https://github.com/saulpw/visidata/blob/7609212ec2febafa9caba7183cbebfaf49627909/visidata/features/setcol_fake.py#L65) (read 2026-09-23)