Model / text · All operations

Split text into words

Tokenise, stem, n-grams, stop words, TF-IDF.

Status

In arcform

Runs as a SQL step

arcform carries it as a sql: step today.

Shape of the result
more rows, each traceable to one input row
Level
several clauses of SQL, or a window

The DuckDB step

Tokens and bigrams with counts

It reads the orders table the proofs share. Create it first:

SQL
CREATE TABLE orders AS SELECT * FROM (VALUES
  (1,  'Alice Ng',      'north', DATE '2026-01-03', 120.00, 'widget', 'a|b',   '{"channel":"web","items":[1,2]}'),
  (2,  'alice ng',      'north', DATE '2026-01-05',  80.50, 'gadget', 'b',     '{"channel":"shop","items":[3]}'),
  (3,  'Bob  Okafor',   'south', DATE '2026-01-05', 300.00, 'widget', 'a|c',   '{"channel":"web","items":[4,5,6]}'),
  (4,  'Carla Díaz',    'east',  DATE '2026-01-09',  NULL,  'gizmo',  NULL,    '{"channel":"web","items":[]}'),
  (5,  'Bob Okafor',    'south', DATE '2026-01-12',  45.25, 'gadget', 'c',     '{"channel":"phone","items":[7]}'),
  (6,  'Dev Patel',     'west',  DATE '2026-01-12', 510.00, 'widget', 'a',     '{"channel":"web","items":[8,9]}'),
  (7,  'Carla Diaz',    'east',  DATE '2026-01-20',  99.99, 'gizmo',  'b|c',   '{"channel":"shop","items":[10]}'),
  (8,  'Dev Patel',     'west',  DATE '2026-02-02', 510.00, 'widget', 'a',     '{"channel":"web","items":[8,9]}'),
  (9,  'Eve Larsen',    NULL,    DATE '2026-02-03',  15.00, 'gadget', NULL,    '{"channel":"web","items":[11]}'),
  (10, 'Alice Ng',      'north', DATE '2026-02-10', 220.00, 'gizmo',  'a|b|c', '{"channel":"phone","items":[12,13]}')
) t(order_id, customer, region, order_date, amount, product, tags, meta);

Then run the statement:

SQL
CREATE TEMP TABLE result AS
WITH t AS (SELECT order_id, regexp_split_to_array(lower(strip_accents(customer || ' ' || product)), '\s+') AS toks FROM orders)
SELECT tok, count(*) AS n FROM (SELECT unnest(toks) AS tok FROM t) GROUP BY tok;
SELECT * FROM result ORDER BY n DESC, tok LIMIT 5;
SELECT list_transform(range(1, len(toks)), lambda i: toks[i] || ' ' || toks[i + 1]) AS bigrams
FROM (SELECT ['data', 'becomes', 'a', 'view'] AS toks);
SELECT CASE WHEN ((SELECT max(n) FILTER (WHERE tok = 'widget') = 4 AND max(n) FILTER (WHERE tok = 'okafor') = 2 AND count(*) FILTER (WHERE tok = '') = 0 FROM result) AND list_transform(range(1, 4), lambda i: ['data', 'becomes', 'a', 'view'][i] || ' ' || ['data', 'becomes', 'a', 'view'][i + 1]) = ['data becomes', 'becomes a', 'a view']) THEN 'as expected' ELSE error('the token counts or the bigrams are wrong') END AS verdict;
  • It needs nothing installed beyond DuckDB itself.
  • The statement checks its own result and raises an error when it is wrong: “the token counts or the bigrams are wrong”.
  • It passed on DuckDB v2.0.0 (Cyanoptera) d54cbbe3df.

Variants

Convert text to and from documents

Wrap table text as document objects for a text pipeline, or turn documents back into rows.

Remove stop words and unwanted tokens

Drop tokens that are stop words or fail a test on content, length or tag.

Build n-grams from tokens

Turn tokens into runs of consecutive terms or characters.

Split text into terms

Break each text value into word tokens, one row per token, under rules for separators, case and language.

What other tools call it

Meridian's classification; an independent blind rating agreed on 86.2% of 312 rows (exact class, version 2).

Alteryx Designer (2)
  • Part-of-Speech Taggerread 2026-09-27

    Tag words in text as nouns, verbs, adjectives and other parts of speech

  • Text Pre-processingread 2026-09-27

    Lemmatise text and filter digits, punctuation and stop words

Apache Hop (2)
  • Stanford NLPread 2026-09-27

    Analyse text with Stanford CoreNLP: base word forms, parts of speech, named entities

  • Text chunkerread 2026-09-27

    Split text into chunks for embedding without breaking words or sentences

Duckle (1)
  • Text Chunkerread 2026-09-25

    split long text into overlapping character-window chunks

KNIME Analytics Platform (2)
RapidMiner Studio (38)
subtoken (3)
  • create_fts_indexread 2026-09-27

    Pragma that builds a full-text index over chosen columns, with stemmer, stopwords and accent options

  • stemread 2026-09-27

    Reduce a word to its base form with a named stemmer

  • tokenizeread 2026-09-27

    Macro, created per index, that splits a string into terms under the index's rules

Talend (3)
  • tChunkingread 2026-09-27

    Split the text of a column into smaller chunks, by fixed size or by token count

  • tJapaneseTokenizeread 2026-09-27

    Split Japanese text into tokens by adding spaces at word boundaries, using the IPADIC dictionary

  • tStemread 2026-09-27

    Reduce the variant forms of words in a column to a common stem

Background

Word segmentation is the problem of dividing a string of written language into its component words.[1] In English and many other languages written in the Latin alphabet, the space is a good approximation of a word divider.[1] Chinese, Japanese, Thai, Lao and Vietnamese do not delimit words in writing, so word segmentation in those languages is not trivial.[1]

When to use it

A rule-based program that performs lexical tokenisation, converting text into meaningful pieces called tokens, is called a tokeniser, lexer or scanner.[2] The bag-of-words model represents a text as an unordered collection of its words, disregarding word order but keeping how many times each word occurs.[3] A bag-of-words representation removes all word order, so the texts man bites dog and dog bites man are represented the same.[3] Split text into words when the question is which words a text uses and how often; keep the text whole when its meaning hangs on the order of the words. (Meridian's view)

Even in languages that put spaces between words, tokenisation has edge cases such as contractions, hyphenated words, emoticons and Uniform Resource Identifiers (URIs), which for some purposes may count as single tokens.[2] A classic example is New York-based, which a naive tokeniser may break at the space even though the better break is arguably at the hyphen.[2] Before trusting a word count, look at the tokens a rule produced from a sample of the real text, especially names, hyphenated words and web addresses. (Meridian's view)

Stop words are words filtered out before or after processing text because they are deemed to have little semantic value, or are otherwise insignificant for the task at hand.[4] No single universal list of stop words is used by all natural language processing tools, and not all tools use such a list.[4] Information retrieval systems have tended over time to move from large stop lists of 200–300 terms to very small ones of 7–12 terms, and then to no stop list at all.[4] For search engines that drop common short function words such as the, is and on, stop words can cause problems when searching for phrases that include them, as in names such as The Who.[4] Choose a stop list for the question at hand, and check what it would remove from the names and phrases in the data before dropping anything. (Meridian's view)

The method

A tokeniser often relies on simple heuristics, such as separating tokens at whitespace or punctuation and keeping each contiguous run of letters, or of digits, as one token.[2] Punctuation and whitespace may or may not be included in the resulting list of tokens.[2] Tokens are often defined by regular expressions.[2] RapidMiner's Tokenize operator splits at every non-letter character by default, so that each word becomes a single token, and also accepts a set of split characters or a regular expression.[5] In DuckDB, the full-text search extension's create_fts_index pragma by default converts text to lower case, strips accents and ignores every non-alphabetic character.[6] The DuckDB full-text search index removes a predefined list of 571 English stop words by default, and accepts a table of other stop words or none.[6]

An n-gram is a sequence of n adjacent symbols in a particular order, where the symbols may be letters, syllables or, more rarely, whole words.[7] An n-gram of size 1 is called a unigram and one of size 2 a bigram, and when the items are words, n-grams may also be called shingles.[7] In natural language processing, n-grams let bag-of-words models capture information such as word order, which the traditional bag of words cannot.[7]

Hans Peter Luhn, one of the pioneers of information retrieval, is credited with coining the phrase stop word and with using the concept in his Key Word in Context automatic indexing process.[4]

Compared with the alternatives

Lexical tokenisation differs from the tokenisation in large language models (LLMs): it is usually based on a lexical grammar, whereas LLM tokenisers are usually probability-based and also convert tokens into numbers.[2] Count words with a word tokeniser rather than with a language model's tokens, since the two are made for different purposes. (Meridian's view)

Sources

  1. [1]Text segmentation - Wikipediasecondary source
  2. [2]Lexical analysis - Wikipediasecondary source
  3. [3]Bag-of-words model - Wikipediasecondary source
  4. [4]Stop word - Wikipediasecondary source
  5. [5]Tokenize - Rapidminer Documentationdocumentation
  6. [6]Full-Text Search Extension – DuckDBdocumentation
  7. [7]N-gram - Wikipediasecondary source