Model / text · All operations

Reduce words to their stems

A variant of Split text into words

Map each word to its stem or base form, one value out per value in.

Status

In arcform

Runs as a SQL step

arcform carries it as a sql: step today, with the fts extension.

The DuckDB step

Reduce each word to its stem and to its base form, one value out per value in

SQL
INSTALL fts; LOAD fts;
CREATE TEMP TABLE words AS SELECT * FROM (VALUES (1, 'Running'), (2, 'jumps'), (3, 'Studies'), (4, 'happily'), (5, 'Mice'), (6, 'ran'), (7, 'Better'), (8, 'cats'), (9, 'relational'), (10, NULL), (11, '')) t(id, word);
-- the lexicon is the reader's to supply: a table of inflected forms and their base forms, read from a file in practice; a word it lacks keeps its own form
CREATE TEMP TABLE lexicon AS SELECT * FROM (VALUES ('mice', 'mouse'), ('ran', 'run'), ('better', 'good'), ('studies', 'study'), ('cats', 'cat'), ('running', 'run'), ('jumps', 'jump')) t(form, base);
-- the stem is a scalar function of the word (Snowball English); the base form is a lookup of the lower-cased word
CREATE TEMP TABLE result AS
SELECT w.id, w.word, stem(lower(w.word), 'english') AS stem, coalesce(l.base, lower(w.word)) AS base_form
FROM words w LEFT JOIN lexicon l ON l.form = lower(w.word);
SELECT * FROM result ORDER BY id;
-- the stems are those of the Snowball English stemmer (checked against NLTK on these words); 'mice', 'ran' and 'better' do not stem to their
-- base forms, which is why the lexicon is the second column
SELECT CASE WHEN ((SELECT count(*) = 11 AND count(*) = (SELECT count(*) FROM words) FROM result)
  AND (SELECT list(stem ORDER BY id) = ['run', 'jump', 'studi', 'happili', 'mice', 'ran', 'better', 'cat', 'relat', NULL, ''] FROM result)
  AND (SELECT list(base_form ORDER BY id) = ['run', 'jump', 'study', 'happily', 'mouse', 'run', 'good', 'cat', 'relational', NULL, ''] FROM result)) THEN 'as expected' ELSE error('the 11 words should give 11 stems and 11 base forms, with NULL staying NULL and the empty string staying empty') END AS verdict;
  • The statement builds its own data.
  • It needs the fts extension.
  • The statement checks its own result and raises an error when it is wrong: “the 11 words should give 11 stems and 11 base forms, with NULL staying NULL and the empty string staying empty”.
  • It passed on DuckDB v2.0.0 (Cyanoptera) d54cbbe3df.

What other tools call it

Meridian's classification; an independent blind rating agreed on 95.7% of 324 rows (variant, version 3).

Alteryx Designer (1)
RapidMiner Studio (8)
subtoken (1)
  • stemread 2026-09-27

    Reduce a word to its base form with a named stemmer

Talend (1)
  • tStemread 2026-09-27

    Reduce the variant forms of words in a column to a common stem

Background

Stemming reduces inflected, or sometimes derived, words to their word stem, base or root form.[1] A stem need not be a valid root; it is usually enough that related words map to the same stem.[1] The Porter algorithm, for example, reduces argue, argued, argues, arguing and argus to the stem argu.[1]

When to use it

Many search engines treat words with the same stem as synonyms, a kind of query expansion called conflation.[1] Google Search adopted word stemming in 2003, and before that a search for fish would not have returned fishing.[1] Stemming is used as a task in pre-processing texts before text mining analyses.[1] Stem when the goal is to match or count the forms of a word together, and keep the original word beside its stem so the output can still be read. (Meridian's view)

Overstemming maps two words that should be kept apart to the same stem, understemming leaves apart two words that should share a stem, and reducing one error can increase the other.[1] The widely used Porter stemmer stems universal, university and universe to univers, a case of overstemming.[1] A user searching for marketing will not be satisfied by most documents that mention markets but not marketing.[1] Suffix-stripping stemmers are sometimes regarded as crude because they perform poorly on exceptional relations such as ran and run.[1] The lemma of a word is its dictionary form, and the lemma of better is good, a link that stemming misses because it requires a dictionary look-up.[2]

Snowball is a small string-processing language for creating stemming algorithms, together with a collection of stemming algorithms written in it.[3] The Snowball stemmers aim to conflate words with the same meaning rather than all words with a common linguistic root, so awe and awful do not share a stem.[3] A Snowball stem is often a word itself but not always, because text search systems, the intended field of use, do not require one.[3] Where the output has to be a real word, such as a label a reader sees, map words through a table of base forms rather than a stemmer. (Meridian's view)

The method

A simple stemmer looks up the inflected form in a lookup table, which is fast and handles exceptions, but cannot handle new or unfamiliar words missing from the table.[1] Lemmatisation, a more complex approach, first determines the part of speech of a word, then applies different normalisation rules for each part of speech.[1] A hybrid stemmer may keep a small table of frequent exceptions, such as ran to run, and apply suffix stripping or lemmatisation to every word not in it.[1] In DuckDB, the full-text search extension adds a stem function that takes a string and the name of a stemmer, such as porter, english, german or none.[4]

The first published stemmer was written by Julie Beth Lovins in 1968.[1] The original paper on the Porter stemming algorithm was written in 1979 in the Computer Laboratory, Cambridge, as part of a larger information retrieval project.[5] The Porter stemmer was published in the July 1980 issue of the journal Program, and became the de facto standard algorithm for English stemming.[1] In October 2001, a Perl script advertised on the web as an implementation of the Porter algorithm was found to stem 14 percent of words in a large sample vocabulary incorrectly.[6] Many implementations of the Porter algorithm contained subtle flaws, so Martin Porter released an official free implementation around 2000 and then built Snowball, a framework for writing stemming algorithms.[1] Snowball's English stemmer is a new algorithm derived from the Porter stemmer after twenty years of its use, and could be called the Porter2 stemmer to tell the two apart.[7]

Compared with the alternatives

A stemmer operates on a single word without knowledge of its context, so unlike lemmatisation it cannot tell apart words whose meaning depends on their part of speech.[2] Stemmers are typically easier to implement and run faster than lemmatisation.[2] In information retrieval systems, stemming improves recall compared with lemmatisation, but reduces precision.[2] The word meeting can be the base form of a noun or a form of the verb to meet, and lemmatisation tries to select the right lemma from the context.[2]

Sources

  1. [1]Stemming - Wikipediasecondary source
  2. [2]Lemmatization - Wikipediasecondary source
  3. [3]Snowballdocumentation
  4. [4]Full-Text Search Extension – DuckDBdocumentation
  5. [5]Porter Stemming Algorithmdocumentation
  6. [6]Snowball: A language for stemming algorithms - Snowballdocumentation
  7. [7]The English (Porter2) stemming algorithm - Snowballdocumentation