# Split text into words Model / text tokenise, stem, n-grams, stop words, TF-IDF Page: ## Status Runs as a SQL step. arcform carries it as a `sql:` step today. ## The DuckDB step tokens and bigrams with counts It reads the `orders` table the proofs share. Create it first: ```sql CREATE TABLE orders AS SELECT * FROM (VALUES (1, 'Alice Ng', 'north', DATE '2026-01-03', 120.00, 'widget', 'a|b', '{"channel":"web","items":[1,2]}'), (2, 'alice ng', 'north', DATE '2026-01-05', 80.50, 'gadget', 'b', '{"channel":"shop","items":[3]}'), (3, 'Bob Okafor', 'south', DATE '2026-01-05', 300.00, 'widget', 'a|c', '{"channel":"web","items":[4,5,6]}'), (4, 'Carla Díaz', 'east', DATE '2026-01-09', NULL, 'gizmo', NULL, '{"channel":"web","items":[]}'), (5, 'Bob Okafor', 'south', DATE '2026-01-12', 45.25, 'gadget', 'c', '{"channel":"phone","items":[7]}'), (6, 'Dev Patel', 'west', DATE '2026-01-12', 510.00, 'widget', 'a', '{"channel":"web","items":[8,9]}'), (7, 'Carla Diaz', 'east', DATE '2026-01-20', 99.99, 'gizmo', 'b|c', '{"channel":"shop","items":[10]}'), (8, 'Dev Patel', 'west', DATE '2026-02-02', 510.00, 'widget', 'a', '{"channel":"web","items":[8,9]}'), (9, 'Eve Larsen', NULL, DATE '2026-02-03', 15.00, 'gadget', NULL, '{"channel":"web","items":[11]}'), (10, 'Alice Ng', 'north', DATE '2026-02-10', 220.00, 'gizmo', 'a|b|c', '{"channel":"phone","items":[12,13]}') ) t(order_id, customer, region, order_date, amount, product, tags, meta); ``` Then run the statement: ```sql CREATE TEMP TABLE result AS WITH t AS (SELECT order_id, regexp_split_to_array(lower(strip_accents(customer || ' ' || product)), '\s+') AS toks FROM orders) SELECT tok, count(*) AS n FROM (SELECT unnest(toks) AS tok FROM t) GROUP BY tok; SELECT * FROM result ORDER BY n DESC, tok LIMIT 5; SELECT list_transform(range(1, len(toks)), lambda i: toks[i] || ' ' || toks[i + 1]) AS bigrams FROM (SELECT ['data', 'becomes', 'a', 'view'] AS toks); SELECT CASE WHEN ((SELECT max(n) FILTER (WHERE tok = 'widget') = 4 AND max(n) FILTER (WHERE tok = 'okafor') = 2 AND count(*) FILTER (WHERE tok = '') = 0 FROM result) AND list_transform(range(1, 4), lambda i: ['data', 'becomes', 'a', 'view'][i] || ' ' || ['data', 'becomes', 'a', 'view'][i + 1]) = ['data becomes', 'becomes a', 'a view']) THEN 'as expected' ELSE error('the token counts or the bigrams are wrong') END AS verdict; ``` It needs nothing installed beyond DuckDB itself. ## Variants - [Split text into chunks](https://meridian.online/operations/split-text-into-words/split-text-into-chunks): Cut long text into segments by size, window or boundary pattern, one row per chunk. - Convert text to and from documents: Wrap table text as document objects for a text pipeline, or turn documents back into rows. - Remove stop words and unwanted tokens: Drop tokens that are stop words or fail a test on content, length or tag. - Build n-grams from tokens: Turn tokens into runs of consecutive terms or characters. - [Tag parts of speech](https://meridian.online/operations/split-text-into-words/tag-parts-of-speech): Label each word as a noun, verb, adjective or other part of speech. - [Reduce words to their stems](https://meridian.online/operations/split-text-into-words/reduce-words-to-their-stems): Map each word to its stem or base form, one value out per value in. - [Count and weight terms per document](https://meridian.online/operations/split-text-into-words/weight-terms-per-document): Build a document-term table of counts or TF-IDF weights from tokenised text. - Split text into terms: Break each text value into word tokens, one row per token, under rules for separators, case and language. ## What other tools call it Meridian's classification; an independent blind rating agreed on 86.2% of 312 rows (exact class, version 2). - Alteryx Designer: [`Part-of-Speech Tagger`](https://help.alteryx.com/current/en/designer/tools/alteryx-intelligence-suite/text-mining/part-of-speech-tagger.html) (read 2026-09-27) - Alteryx Designer: [`Text Pre-processing`](https://help.alteryx.com/current/en/designer/tools/alteryx-intelligence-suite/text-mining/text-pre-processing.html) (read 2026-09-27) - Duckle: [`Text Chunker`](https://github.com/slothflowlabs/duckle/blob/f85f8960edfb6efbf4cb89d58a7b32e16df0e867/website/docs/components.html) (read 2026-09-25) - Apache Hop: [`Stanford NLP`](https://hop.apache.org/manual/latest/pipeline/transforms/stanfordnlp.html) (read 2026-09-27) - Apache Hop: [`Text chunker`](https://hop.apache.org/manual/next/pipeline/transforms/textchunker.html) (read 2026-09-27) - KNIME Analytics Platform: [`KNIME Textprocessing (extension)`](https://api.hub.knime.com/extensions/org.knime.features.ext.textprocessing) (read 2026-09-27) - KNIME Analytics Platform: [`KNIME Textprocessing Turkish Language Pack (extension)`](https://api.hub.knime.com/extensions/org.knime.features.ext.textprocessing.language.turkish) (read 2026-09-27) - RapidMiner Studio: [`Cut Document`](https://docs.rapidminer.com/latest/studio/operators/extensions/Text%20Processing/transformation/cut_document.html) (read 2026-09-27) - RapidMiner Studio: [`Data to Documents`](https://docs.rapidminer.com/latest/studio/operators/extensions/Text%20Processing/data_to_documents.html) (read 2026-09-27) - RapidMiner Studio: [`Documents to Data`](https://docs.rapidminer.com/latest/studio/operators/extensions/Text%20Processing/documents_to_data.html) (read 2026-09-27) - RapidMiner Studio: [`Extract Document`](https://docs.rapidminer.com/latest/studio/operators/extensions/Text%20Processing/extract_document.html) (read 2026-09-27) - RapidMiner Studio: [`Filter Stopwords (Arabic)`](https://docs.rapidminer.com/latest/studio/operators/extensions/Text%20Processing/filtering/filter_stopwords_arabic.html) (read 2026-09-27) - RapidMiner Studio: [`Filter Stopwords (Czech)`](https://docs.rapidminer.com/latest/studio/operators/extensions/Text%20Processing/filtering/filter_stopwords_czech.html) (read 2026-09-27) - RapidMiner Studio: [`Filter Stopwords (Dictionary)`](https://docs.rapidminer.com/latest/studio/operators/extensions/Text%20Processing/filtering/filter_stopwords_dictionary.html) (read 2026-09-27) - RapidMiner Studio: [`Filter Stopwords (English)`](https://docs.rapidminer.com/latest/studio/operators/extensions/Text%20Processing/filtering/filter_stopwords_english.html) (read 2026-09-27) - RapidMiner Studio: [`Filter Stopwords (French)`](https://docs.rapidminer.com/latest/studio/operators/extensions/Text%20Processing/filtering/filter_stopwords_french.html) (read 2026-09-27) - RapidMiner Studio: [`Filter Stopwords (German)`](https://docs.rapidminer.com/latest/studio/operators/extensions/Text%20Processing/filtering/filter_stopwords_german.html) (read 2026-09-27) - RapidMiner Studio: [`Filter Tokens (by Content)`](https://docs.rapidminer.com/latest/studio/operators/extensions/Text%20Processing/filtering/filter_tokens_by_content.html) (read 2026-09-27) - RapidMiner Studio: [`Filter Tokens (by Length)`](https://docs.rapidminer.com/latest/studio/operators/extensions/Text%20Processing/filtering/filter_by_length.html) (read 2026-09-27) - RapidMiner Studio: [`Filter Tokens (by POS Ratios)`](https://docs.rapidminer.com/latest/studio/operators/extensions/Text%20Processing/filtering/filter_tokens_by_pos_ratios.html) (read 2026-09-27) - RapidMiner Studio: [`Filter Tokens (by POS Tags)`](https://docs.rapidminer.com/latest/studio/operators/extensions/Text%20Processing/filtering/filter_tokens_by_pos.html) (read 2026-09-27) - RapidMiner Studio: [`Filter Tokens (by Region)`](https://docs.rapidminer.com/latest/studio/operators/extensions/Text%20Processing/filtering/filter_tokens_by_regions.html) (read 2026-09-27) - RapidMiner Studio: [`Filter Tokens Using ExampleSet`](https://docs.rapidminer.com/latest/studio/operators/extensions/Operator%20Toolbox/text_processing/filter_tokens_using_exampleset.html) (read 2026-09-27) - RapidMiner Studio: [`Generate TFIDF`](https://docs.rapidminer.com/latest/studio/operators/blending/attributes/generation/generate_tfidf.html) (read 2026-09-27) - RapidMiner Studio: [`Generate n-Grams (Characters)`](https://docs.rapidminer.com/latest/studio/operators/extensions/Text%20Processing/transformation/generate_n_grams_characters.html) (read 2026-09-27) - RapidMiner Studio: [`Generate n-Grams (Terms)`](https://docs.rapidminer.com/latest/studio/operators/extensions/Text%20Processing/transformation/generate_n_grams_terms.html) (read 2026-09-27) - RapidMiner Studio: [`Process Documents`](https://docs.rapidminer.com/latest/studio/operators/extensions/Text%20Processing/process_documents.html) (read 2026-09-27) - RapidMiner Studio: [`Process Documents from Data`](https://docs.rapidminer.com/latest/studio/operators/extensions/Text%20Processing/process_document_from_data.html) (read 2026-09-27) - RapidMiner Studio: [`Process Documents from Files`](https://docs.rapidminer.com/latest/studio/operators/extensions/Text%20Processing/process_document_from_file.html) (read 2026-09-27) - RapidMiner Studio: [`Process Documents from Mail Store`](https://docs.rapidminer.com/latest/studio/operators/extensions/Text%20Processing/process_mail_documents.html) (read 2026-09-27) - RapidMiner Studio: [`Split Document into Collection`](https://docs.rapidminer.com/latest/studio/operators/extensions/Operator%20Toolbox/text_processing/split_document_into_collection.html) (read 2026-09-27) - RapidMiner Studio: [`Split File by Content`](https://docs.rapidminer.com/latest/studio/operators/extensions/Text%20Processing/utility/files/split_file_by_content.html) (read 2026-09-27) - RapidMiner Studio: [`Split File by Point`](https://docs.rapidminer.com/latest/studio/operators/extensions/Text%20Processing/utility/files/split_file_by_point.html) (read 2026-09-27) - RapidMiner Studio: [`Stem (Arabic)`](https://docs.rapidminer.com/latest/studio/operators/extensions/Text%20Processing/stemming/stem_arabic.html) (read 2026-09-27) - RapidMiner Studio: [`Stem (Arabic, Light)`](https://docs.rapidminer.com/latest/studio/operators/extensions/Text%20Processing/stemming/stem_arabic_light.html) (read 2026-09-27) - RapidMiner Studio: [`Stem (Dictionary)`](https://docs.rapidminer.com/latest/studio/operators/extensions/Text%20Processing/stemming/stem_dictionary.html) (read 2026-09-27) - RapidMiner Studio: [`Stem (German)`](https://docs.rapidminer.com/latest/studio/operators/extensions/Text%20Processing/stemming/stem_german.html) (read 2026-09-27) - RapidMiner Studio: [`Stem (Lovins)`](https://docs.rapidminer.com/latest/studio/operators/extensions/Text%20Processing/stemming/stem_lovins.html) (read 2026-09-27) - RapidMiner Studio: [`Stem (Porter)`](https://docs.rapidminer.com/latest/studio/operators/extensions/Text%20Processing/stemming/stem_porter.html) (read 2026-09-27) - RapidMiner Studio: [`Stem (Snowball)`](https://docs.rapidminer.com/latest/studio/operators/extensions/Text%20Processing/stemming/stem_snowball.html) (read 2026-09-27) - RapidMiner Studio: [`Stem Tokens Using ExampleSet`](https://docs.rapidminer.com/latest/studio/operators/extensions/Operator%20Toolbox/text_processing/stem_tokens_using_exampleset.html) (read 2026-09-27) - RapidMiner Studio: [`Text Vectorization`](https://docs.rapidminer.com/latest/studio/operators/blending/attributes/generation/text_vectorization.html) (read 2026-09-27) - RapidMiner Studio: [`Tokenize`](https://docs.rapidminer.com/latest/studio/operators/extensions/Text%20Processing/tokenization/tokenize.html) (read 2026-09-27) - RapidMiner Studio: [`Window Document`](https://docs.rapidminer.com/latest/studio/operators/extensions/Text%20Processing/transformation/window_document.html) (read 2026-09-27) - RapidMiner Studio: [`WordList to Data`](https://docs.rapidminer.com/latest/studio/operators/extensions/Text%20Processing/utility/wordlist_to_data.html) (read 2026-09-27) - subtoken: [`create_fts_index`](https://duckdb.org/docs/current/core_extensions/full_text_search.html) (read 2026-09-27) - subtoken: [`stem`](https://duckdb.org/docs/current/core_extensions/full_text_search.html) (read 2026-09-27) - subtoken: [`tokenize`](https://duckdb.org/docs/current/core_extensions/full_text_search.html) (read 2026-09-27) - Talend: [`tChunking`](https://help.qlik.com/talend/en-US/components/8.0/ai/ai-component) (read 2026-09-27) - Talend: [`tJapaneseTokenize`](https://help.qlik.com/talend/en-US/components/8.0/text-standardization/text-standardization-component) (read 2026-09-27) - Talend: [`tStem`](https://help.qlik.com/talend/en-US/components/8.0/text-standardization/text-standardization-component) (read 2026-09-27)