# Transliterate or remove accents

Shape / columns

A variant of [Clean up text](https://meridian.online/operations/clean-up-text).

Convert text to a standard character set, such as removing accents, replacing curly quotes or writing another script in Latin letters.

Page: <https://meridian.online/operations/clean-up-text/transliterate-or-remove-accents>

## Status

Runs as a SQL step.

arcform carries it as a `sql:` step today.

## The DuckDB step

remove accents, straighten curly quotes and write Russian and Greek in Latin letters (a simplified BGN/PCGN scheme)

```sql
-- Scheme: ASCII only, no diacritics. Russian: the 33 modern letters, with ё as yo, й as y, and the hard and soft signs ъ and ь dropped.
-- Greek: the 24 letters, final ς, the accented vowels and ΐ and ΰ. Text is put in composed form (nfc_normalize) first. A capital letter gives a capital initial (Ж is Zh, Щ is Shch, Ψ is Ps).
CREATE TEMP TABLE script_letters(k, v) AS VALUES
  ('а', 'a'), ('б', 'b'), ('в', 'v'), ('г', 'g'), ('д', 'd'), ('е', 'e'),
  ('ё', 'yo'), ('ж', 'zh'), ('з', 'z'), ('и', 'i'), ('й', 'y'), ('к', 'k'),
  ('л', 'l'), ('м', 'm'), ('н', 'n'), ('о', 'o'), ('п', 'p'), ('р', 'r'),
  ('с', 's'), ('т', 't'), ('у', 'u'), ('ф', 'f'), ('х', 'kh'), ('ц', 'ts'),
  ('ч', 'ch'), ('ш', 'sh'), ('щ', 'shch'), ('ъ', ''), ('ы', 'y'), ('ь', ''),
  ('э', 'e'), ('ю', 'yu'), ('я', 'ya'),
  ('α', 'a'), ('β', 'v'), ('γ', 'g'), ('δ', 'd'), ('ε', 'e'), ('ζ', 'z'),
  ('η', 'i'), ('θ', 'th'), ('ι', 'i'), ('κ', 'k'), ('λ', 'l'), ('μ', 'm'),
  ('ν', 'n'), ('ξ', 'x'), ('ο', 'o'), ('π', 'p'), ('ρ', 'r'), ('σ', 's'),
  ('ς', 's'), ('τ', 't'), ('υ', 'y'), ('φ', 'f'), ('χ', 'kh'), ('ψ', 'ps'),
  ('ω', 'o'), ('ά', 'a'), ('έ', 'e'), ('ή', 'i'), ('ί', 'i'), ('ό', 'o'),
  ('ύ', 'y'), ('ώ', 'o'), ('ϊ', 'i'), ('ϋ', 'y'), ('ΐ', 'i'), ('ΰ', 'y');
CREATE TEMP TABLE charmap AS
SELECT map_from_entries(list((k, v))) AS m FROM (
  SELECT k, v FROM script_letters
  UNION ALL SELECT upper(k), upper(v[1]) || v[2:] FROM script_letters WHERE k NOT IN ('ς', 'ΐ', 'ΰ')
  UNION ALL SELECT * FROM (VALUES ('Ø', 'O'), ('ø', 'o'), ('Ł', 'L'), ('ł', 'l'), ('ß', 'ss'), ('Æ', 'AE'), ('æ', 'ae'), ('Œ', 'OE'), ('œ', 'oe'),
    ('‘', chr(39)), ('’', chr(39)), ('“', '"'), ('”', '"')) t(k, v));
CREATE TEMP TABLE raw AS SELECT * FROM (VALUES
  (1, 'Café “Zoë’s” Müller'),
  (2, 'Øre Łódź Straße Æon œuvre'),
  (3, 'Москва Жуков Щёлково Юрий'),
  (4, 'Ольга Эрмитаж Объект'),
  (5, 'Ελλάδα Αθήνα Βόλος Ελλάς'),
  (6, 'ΣΟΦΙΑ ΨΥΧΗ'),
  (7, 'ταΐζω'),
  (8, 'Юри' || chr(1080) || chr(774))  -- й written as и and a combining breve
) t(id, s);
CREATE TEMP TABLE result AS
SELECT id, strip_accents(array_to_string(list_transform(string_split(nfc_normalize(s), ''), lambda ch: coalesce(m[ch], ch)), '')) AS plain
FROM raw, charmap;
SELECT * FROM result ORDER BY id;
SELECT CASE WHEN ((SELECT list(plain ORDER BY id) FROM result) = ['Cafe "Zoe''s" Muller', 'Ore Lodz Strasse AEon oeuvre', 'Moskva Zhukov Shchyolkovo Yuriy', 'Olga Ermitazh Obekt', 'Ellada Athina Volos Ellas', 'SOFIA PsYKhI', 'taizo', 'Yuriy']) THEN 'as expected' ELSE error('accents, curly quotes and the Russian and Greek letters were not all written as Latin letters by the scheme') END AS verdict;
```

The statement builds its own data.

It needs nothing installed beyond DuckDB itself.

## What other tools call it

Meridian's classification; an independent blind rating agreed on 95.7% of 324 rows (variant, version 3).

- OpenRefine: [`Replace smart quotes with ASCII`](https://openrefine.org/docs/manual/cellediting#replace-smart-quotes-with-ascii) (read 2026-09-27)
- Talend: [`tJapaneseNumberNormalize`](https://help.qlik.com/talend/en-US/components/8.0/text-standardization/text-standardization-component) (read 2026-09-27)
- Talend: [`tJapaneseTransliterate`](https://help.qlik.com/talend/en-US/components/8.0/text-standardization/text-standardization-component) (read 2026-09-27)
- Talend: [`tTransliterate`](https://help.qlik.com/talend/en-US/components/8.0/text-standardization/text-standardization-component) (read 2026-09-27)
- Trifacta: [`Remove accents in text`](https://help.alteryx.com/aac/en/trifacta-classic/reference/transformation-reference.html) (read 2026-09-27)