Control / exec · All operations

Change a file's encoding

A variant of Run an external command

Rewrite a file in another character encoding, line-ending convention or representation.

Status

In arcform

Not built yet

arcform does not do this yet; it would live in how a Protocol runs. Its place on the roadmap

What other tools call it

Meridian's classification; an independent blind rating agreed on 95.7% of 324 rows (variant, version 3).

Apache Hop (2)
Pentaho Data Integration (1)
Talend (2)
  • tChangeFileEncodingread 2026-09-27

    Rewrite a file in another character encoding

  • tHConvertFileread 2026-09-27

    Convert a file from one representation to another with Talend Data Mapper structures, as a Spark batch

Background

Apache Hop's Change File Encoding transform changes a text file from one encoding to another.[1] Character encoding is a convention of using a numeric value to represent each character of a writing script.[2] Apache Hop's Convert file between Windows and Unix action converts files between the Windows and the Unix conventions for ending lines, in either direction.[3]

When to use it

Mojibake is the garbled or gibberish text that results from text being decoded using an unintended character encoding.[4] Mojibake is often seen with text that has been tagged with a wrong encoding, or not tagged at all and moved between computers with different default encodings.[4] If the encoding is not specified, software decides it by other means, typically configuration or charset detection heuristics, both of which are prone to mis-prediction.[4] Some encodings, such as UTF-8, are easy to detect, while many are hard to distinguish, such as EUC-JP and Shift-JIS.[4] Failed rendering of glyphs because of missing fonts or glyphs is a different issue from mojibake.[4] Apache Hop's Change File Encoding transform takes the source encoding of the text files as a setting.[1]

Files composed on Windows and on Unix-like operating systems can conflict, so that a file composed on one cannot be properly formatted or interpreted by the other, such as a Unix shell script written in Windows Notepad.[5]

Talend's tChangeFileEncoding component transforms the character encoding of a given file and generates a new file with the transformed encoding.[6] Pentaho Data Integration's Change file encoding step changes a file's encoding and creates a new file.[7] Keep the original file until the converted one has been read back and checked. (Meridian's view)

The method

The numerical values that make up a character encoding are known as code points.[2] Early character encodings could represent only a subset of the characters used in languages, and encodings representing more characters were created over time, such as ASCII, ISO/IEC 8859, and the Unicode encodings UTF-8 and UTF-16.[2] UTF-8 is the most popular character encoding on the World Wide Web, used in 99.0% of surveyed web sites as of September 2026.[2] For Unicode text, one solution to a wrongly assumed encoding is a byte order mark, but many parsers do not tolerate one in source code or other machine-readable text.[4]

A newline is a control character or sequence of control characters that signifies the end of a line of text and the start of a new one.[5] MS-DOS adopted CP/M's carriage return plus line feed (CR+LF) in order to be compatible, and Microsoft's later Windows operating system inherited the convention.[5] Unix followed the Multics practice of using a line feed (LF) alone as its newline, and later Unix-like systems followed Unix.[5] Special-purpose programs for converting files between newline conventions include unix2dos and dos2unix.[5]

Sources

  1. [1]Change file encoding | Apache Hopdocumentation
  2. [2]Character encoding - Wikipediasecondary source
  3. [3]Convert file between Windows and Unix | Apache Hopdocumentation
  4. [4]Mojibake - Wikipediasecondary source
  5. [5]Newline - Wikipediasecondary source
  6. [6]Formatting component | Talend Components for Jobs Helpdocumentation
  7. [7]PDI transformation steps | 9.3 Data Integration | Pentahodocumentation