Shape / columns · All operations

Transliterate or remove accents

A variant of Clean up text

Convert text to a standard character set, such as removing accents, replacing curly quotes or writing another script in Latin letters.

Status

In arcform

Runs as a SQL step

arcform carries it as a sql: step today.

The DuckDB step

Remove accents, straighten curly quotes and write Russian and Greek in Latin letters (a simplified BGN/PCGN scheme)

SQL
-- Scheme: ASCII only, no diacritics. Russian: the 33 modern letters, with ё as yo, й as y, and the hard and soft signs ъ and ь dropped.
-- Greek: the 24 letters, final ς, the accented vowels and ΐ and ΰ. Text is put in composed form (nfc_normalize) first. A capital letter gives a capital initial (Ж is Zh, Щ is Shch, Ψ is Ps).
CREATE TEMP TABLE script_letters(k, v) AS VALUES
  ('а', 'a'), ('б', 'b'), ('в', 'v'), ('г', 'g'), ('д', 'd'), ('е', 'e'),
  ('ё', 'yo'), ('ж', 'zh'), ('з', 'z'), ('и', 'i'), ('й', 'y'), ('к', 'k'),
  ('л', 'l'), ('м', 'm'), ('н', 'n'), ('о', 'o'), ('п', 'p'), ('р', 'r'),
  ('с', 's'), ('т', 't'), ('у', 'u'), ('ф', 'f'), ('х', 'kh'), ('ц', 'ts'),
  ('ч', 'ch'), ('ш', 'sh'), ('щ', 'shch'), ('ъ', ''), ('ы', 'y'), ('ь', ''),
  ('э', 'e'), ('ю', 'yu'), ('я', 'ya'),
  ('α', 'a'), ('β', 'v'), ('γ', 'g'), ('δ', 'd'), ('ε', 'e'), ('ζ', 'z'),
  ('η', 'i'), ('θ', 'th'), ('ι', 'i'), ('κ', 'k'), ('λ', 'l'), ('μ', 'm'),
  ('ν', 'n'), ('ξ', 'x'), ('ο', 'o'), ('π', 'p'), ('ρ', 'r'), ('σ', 's'),
  ('ς', 's'), ('τ', 't'), ('υ', 'y'), ('φ', 'f'), ('χ', 'kh'), ('ψ', 'ps'),
  ('ω', 'o'), ('ά', 'a'), ('έ', 'e'), ('ή', 'i'), ('ί', 'i'), ('ό', 'o'),
  ('ύ', 'y'), ('ώ', 'o'), ('ϊ', 'i'), ('ϋ', 'y'), ('ΐ', 'i'), ('ΰ', 'y');
CREATE TEMP TABLE charmap AS
SELECT map_from_entries(list((k, v))) AS m FROM (
  SELECT k, v FROM script_letters
  UNION ALL SELECT upper(k), upper(v[1]) || v[2:] FROM script_letters WHERE k NOT IN ('ς', 'ΐ', 'ΰ')
  UNION ALL SELECT * FROM (VALUES ('Ø', 'O'), ('ø', 'o'), ('Ł', 'L'), ('ł', 'l'), ('ß', 'ss'), ('Æ', 'AE'), ('æ', 'ae'), ('Œ', 'OE'), ('œ', 'oe'),
    ('‘', chr(39)), ('’', chr(39)), ('“', '"'), ('”', '"')) t(k, v));
CREATE TEMP TABLE raw AS SELECT * FROM (VALUES
  (1, 'Café “Zoë’s” Müller'),
  (2, 'Øre Łódź Straße Æon œuvre'),
  (3, 'Москва Жуков Щёлково Юрий'),
  (4, 'Ольга Эрмитаж Объект'),
  (5, 'Ελλάδα Αθήνα Βόλος Ελλάς'),
  (6, 'ΣΟΦΙΑ ΨΥΧΗ'),
  (7, 'ταΐζω'),
  (8, 'Юри' || chr(1080) || chr(774))  -- й written as и and a combining breve
) t(id, s);
CREATE TEMP TABLE result AS
SELECT id, strip_accents(array_to_string(list_transform(string_split(nfc_normalize(s), ''), lambda ch: coalesce(m[ch], ch)), '')) AS plain
FROM raw, charmap;
SELECT * FROM result ORDER BY id;
SELECT CASE WHEN ((SELECT list(plain ORDER BY id) FROM result) = ['Cafe "Zoe''s" Muller', 'Ore Lodz Strasse AEon oeuvre', 'Moskva Zhukov Shchyolkovo Yuriy', 'Olga Ermitazh Obekt', 'Ellada Athina Volos Ellas', 'SOFIA PsYKhI', 'taizo', 'Yuriy']) THEN 'as expected' ELSE error('accents, curly quotes and the Russian and Greek letters were not all written as Latin letters by the scheme') END AS verdict;
  • The statement builds its own data.
  • It needs nothing installed beyond DuckDB itself.
  • The statement checks its own result and raises an error when it is wrong: “accents, curly quotes and the Russian and Greek letters were not all written as Latin letters by the scheme”.
  • It passed on DuckDB v2.0.0 (Cyanoptera) d54cbbe3df.

What other tools call it

Meridian's classification; an independent blind rating agreed on 95.7% of 324 rows (variant, version 3).

OpenRefine (1)
Talend (3)
Trifacta (1)

Background

Transliteration is the attempt to represent the text of one language in the writing system of another.[1] In DuckDB, strip_accents removes only combining diacritical marks, leaving distinct letters such as ø, ł and ß unchanged.[2]

When to use it

Reach for it when the same name arrives with and without its accents, or in another script, and the values must match. (Meridian's view) OpenRefine's fingerprint normalises extended western characters to ASCII to reproduce the mistakes made when such characters are typed on an ASCII-only keyboard.[3] In OpenRefine's fingerprint, that normalisation can also lead to false positives: gödel and godél both become godel, though they are likely different names.[3] Use accent-free text as a matching key, and keep the original spelling for anything a person will read. (Meridian's view)

Curly quotes are not ASCII, though they are valid UTF-8, and OpenRefine has a transform that replaces them with the straight double quote.[4]

The Russian Российская Республика can be transliterated as Rossiyskaya Respublika or as Rossijskaja Respublika.[1] The BGN/PCGN system for Russian was developed by the United States Board on Geographic Names (BGN) and the Permanent Committee on Geographical Names for British Official Use (PCGN).[5] BGN/PCGN Russian needs only the basic letters and punctuation of English-language keyboards, with no diacritics.[5] Record which scheme a column was transliterated with, and use the same one for every column it will be matched against. (Meridian's view)

The method

In Unicode, character decomposition splits a precomposed character into its base letter and combining marks, and character composition is the reverse.[6] Unicode normalisation is not injective: the angstrom sign and the Swedish letter Å both decompose to the letter A followed by a combining ring.[6]

Compared with the alternatives

PostgreSQL's to_ascii converts a string to ASCII mainly by dropping accents, and only from the LATIN1, LATIN2, LATIN9 and WIN1250 encodings.[7] In DuckDB, a column declared with COLLATE NOACCENT matches hännes when compared with hannes, without the stored value changing.[8] A collation suits a comparison that should ignore accents; transliteration suits output that must contain only ASCII or only Latin letters. (Meridian's view)