MeridianMERIDIAN

Plain Text

representation.text.plain_text

Unstructured text content (sentences, paragraphs, descriptions). Broad categorization.

Worked samples

DuckDB 1.5.5 · finetype 0.6.64

ValueAs VARCHARchar_countword_count
This is a plain text descriptionThis is a plain text description326
A longer paragraph with multiple sentencesA longer paragraph with multiple sentences426
Single wordSingle word112
Scope
broad_characters

Background

Origin

FineType defines representation.text.plain_text itself, as unstructured text content such as sentences, paragraphs and descriptions.[1] FineType's registry notes call plain_text a very broad category.[1] FineType's registry notes say that telling plain text apart from other text types, such as sentence or word, requires statistical analysis.[1] FineType's registry gives plain_text the broad type VARCHAR.[1]

The Unicode Standard defines plain text as a pure sequence of character codes.[2] The Unicode Standard contrasts plain text with styled text, also known as rich text, which adds information such as a language identifier, font size, colour or hypertext links.[2] Wikipedia describes plain text as a loose term for unformatted text.[3]

Use

The Data Package Standard's Table Schema defines a string field type for fields that contain sequences of characters.[4] DuckDB stores strings in its VARCHAR type, and lists TEXT and STRING among the type's alternative names.[5] DuckDB encodes VARCHAR data internally as UTF-8.[5] DuckDB's documentation says a maximum length given as VARCHAR(n) has no effect and is only provided for compatibility.[5] DuckDB's documentation suggests a CHECK constraint when a string's length must be restricted.[5]

Wikipedia notes that, in most file systems, text files have no indicator of their character encoding.[3] Confirm the character encoding of a free-text column before you load it, rather than trusting a reader's default. (Meridian's view) Expect line breaks, tabs and long values inside free-text cells, and check that your CSV or spreadsheet reader keeps quoted multi-line cells whole. (Meridian's view) Treat plain text as the label of last resort: first ask whether the column is really something narrower, such as names, codes or a small set of categories, before leaving it as free text. (Meridian's view)

Where you'll meet it

The US Census Bureau's 2022 NAICS descriptions file has three columns: Code, Title and Description.[6] The Description column in the 2022 NAICS descriptions file holds prose such as "This industry comprises establishments primarily engaged in growing soybeans and/or producing soybean seeds."[6] Many entries in the Description column of the 2022 NAICS descriptions file run to several paragraphs separated by line breaks.[6] Some rows in the 2022 NAICS descriptions file carry the literal text NULL in the Description column.[6]

Sources

  1. [1]finetype/labels/definitions_representation.yaml at main · meridian-online/finetypeFineType
  2. [2]Chapter 2 – Unicode 16.0.0standard
  3. [3]Plain textsecondary source
  4. [4]Table Schema | Data Package Standardstandard
  5. [5]Text Typesdocumentation
  6. [6]2022_NAICS_Descriptions.xlsxdataset

Detection notes

v1 migration: Was text.text marked broad_characters. Very broad category. Distinguishing from other text types (sentence, word, etc.) requires statistical analysis. Generated on request only: a catch-all leaf whose synthetic values have no shape, kept for the training set's label coverage.

In Meridian's open data

Use it

CLI · DuckDB
$ finetype infer -i "This is a plain text description" --mode column
→ representation.text.plain_text

SELECT ft_infer('This is a plain text description');
-- → 'representation.text.plain_text'
Cast expression
CAST({col} AS VARCHAR)

Schema

https://meridian.online/schemas/representation.text.plain_text · Frictionless type string

finetype taxonomy representation.text.plain_text -o json-schema

Aliases

textdescriptioncontent

Type Registry