Locale Code
technology.code.locale_codeLanguage tag in the IETF BCP 47 form of RFC 5646: a language subtag, an optional script subtag and an optional region subtag (e.g., en, en-US, zh-Hant-TW, es-419). The language subtag is drawn from ISO 639, the script subtag from ISO 15924, and the region subtag from ISO 3166-1 or the UN M.49 numeric codes; RFC 5646 gives es-419 as Spanish appropriate to the UN-defined Latin America and Caribbean region. One extended language subtag may follow the language (zh-yue); that slot is checked for shape only, with membership left to the substance guard. An underscore separator (pt_BR), the POSIX and Java locale form, is accepted with a note. Variants, extensions, private-use and grandfathered tags are not accepted.
Worked samples
DuckDB 1.5.5 · finetype 0.6.64
| Value | As VARCHAR | country | language |
|---|---|---|---|
| en | en | en | |
| en-US | en-US | us | en |
| fr-FR | fr-FR | fr | fr |
| zh-Hans-CN | zh-Hans-CN | cn | zh |
| de-AT | de-AT | at | de |
Shape
^[a-zA-Z]{2,3}(?:[-_][a-zA-Z]{3})?(?:[-_][a-zA-Z]{4})?(?:[-_](?:[a-zA-Z]{2}|[0-9]{3}))?$| Value | Length | Shape |
|---|---|---|
| en | 2 | matches |
| en-US | 5 | matches |
| fr-FR | 5 | matches |
| zh-Hans-CN | 10 | matches |
| de-AT | 5 | matches |
| e | 1 | rejected: one character short |
Background
Origin
RFC 5646 was edited by A. Phillips and M. Davis.[1] RFC 5646 replaces RFC 4646, which obsoleted RFC 3066, which in turn replaced RFC 1766.[1] RFC 5646, in combination with RFC 4647, comprises BCP 47.[1]
Under RFC 5646, a language tag is a sequence of one or more subtags separated by a hyphen.[1] The namespace of language tags and their subtags is administered by IANA, and the Language Subtag Registry it maintains is the source for valid subtags.[1] RFC 5646 says language tags form the basis for most implementations of locale identifiers, pointing to Unicode's CLDR as an example.[1]
Unicode Technical Standard #35 defines Unicode locale identifiers, which inherit the design and the repertoire of subtags from BCP 47.[2] UTS #35 allows the underscore as a field separator for backwards compatibility, though its canonical form uses the hyphen.[2] UTS #35 notes that some people recommend distinguishing language codes from locale codes by hyphen versus underscore, but says pragmatically the two must be treated as equivalent on input.[2]
Use
HTML uses language tags in the lang attribute and XML in the xml:lang attribute.[3] HTTP uses language tags in the Accept-Language and Content-Language headers.[3] The W3C advises keeping a language tag as short as possible, adding region or script subtags only where they add useful distinguishing information.[3]
The GNU C Library's locale names mostly follow XPG syntax: language, then optionally an underscore and territory, a dot and codeset, and an at sign and modifier.[4] The GNU C Library manual gives de_AT.iso885915@euro as a German-language locale for Austria using ISO-8859-15 with the Euro as the currency symbol.[4] Java's Locale.toString joins its fields with underscores, giving strings such as de_DE and zh_CN_#Hans, and is documented as being for debugging.[5] Java's Locale documentation says to use toLanguageTag() to represent a Locale as a string for interchange purposes.[5] A column of locale codes may therefore mix BCP 47 tags with POSIX or Java-style strings, so look at the separators before deciding which convention a source follows. (Meridian's view)
RFC 5646 says language tags and their subtags are case insensitive, so mn-Cyrl-MN and MN-cYRL-mn mean the same thing.[1] RFC 5646 recommends writing language subtags in lowercase, script subtags in title case and two-letter region subtags in uppercase.[1] RFC 5646 warns that uppercasing the region subtag 'in' with Turkish locale rules produces a dotted capital I, and asks implementers to use a locale-neutral casing operation.[1] Compare locale codes case-insensitively, normalise the separator to a hyphen, and do any case folding with a locale-neutral function. (Meridian's view)
Java's Locale documentation notes that a well-formed tag is not necessarily valid, giving xx-YY as an example whose subtags are not registered in the IANA Language Subtag Registry.[5] A pattern check only shows that a value is shaped like a tag; to know it names a real language and region, look its subtags up in the IANA registry. (Meridian's view)
Sources
- [1]RFC 5646: Tags for Identifying Languages | RFC Editorstandard
- [2]Unicode Locale Data Markup Language (LDML)standard
- [3]Language tags in HTML and XMLsecondary source
- [4]Locale Names (The GNU C Library)documentation
- [5]Locale (Java SE 21 & JDK 21)documentation
Detection notes
v1 migration: Was code.locale_code. IETF BCP 47 format. Common codes: en, en-US, fr, de, zh-Hans, zh-Hant. Decompose extracts language and country. bcp47 alias added — locale_code already covers BCP 47 format.
Use it
$ finetype infer -i "en" --mode column
→ technology.code.locale_code
SELECT ft_infer('en');
-- → 'technology.code.locale_code'CAST({col} AS VARCHAR)Schema
https://meridian.online/schemas/technology.code.locale_code · Frictionless type string
finetype taxonomy technology.code.locale_code -o json-schema