# Reduce words to their stems

Model / text

A variant of [Split text into words](https://meridian.online/operations/split-text-into-words).

Map each word to its stem or base form, one value out per value in.

Page: <https://meridian.online/operations/split-text-into-words/reduce-words-to-their-stems>

## Status

Runs as a SQL step.

arcform carries it as a `sql:` step today, with the `fts` extension.

## The DuckDB step

reduce each word to its stem and to its base form, one value out per value in

```sql
INSTALL fts; LOAD fts;
CREATE TEMP TABLE words AS SELECT * FROM (VALUES (1, 'Running'), (2, 'jumps'), (3, 'Studies'), (4, 'happily'), (5, 'Mice'), (6, 'ran'), (7, 'Better'), (8, 'cats'), (9, 'relational'), (10, NULL), (11, '')) t(id, word);
-- the lexicon is the reader's to supply: a table of inflected forms and their base forms, read from a file in practice; a word it lacks keeps its own form
CREATE TEMP TABLE lexicon AS SELECT * FROM (VALUES ('mice', 'mouse'), ('ran', 'run'), ('better', 'good'), ('studies', 'study'), ('cats', 'cat'), ('running', 'run'), ('jumps', 'jump')) t(form, base);
-- the stem is a scalar function of the word (Snowball English); the base form is a lookup of the lower-cased word
CREATE TEMP TABLE result AS
SELECT w.id, w.word, stem(lower(w.word), 'english') AS stem, coalesce(l.base, lower(w.word)) AS base_form
FROM words w LEFT JOIN lexicon l ON l.form = lower(w.word);
SELECT * FROM result ORDER BY id;
-- the stems are those of the Snowball English stemmer (checked against NLTK on these words); 'mice', 'ran' and 'better' do not stem to their
-- base forms, which is why the lexicon is the second column
SELECT CASE WHEN ((SELECT count(*) = 11 AND count(*) = (SELECT count(*) FROM words) FROM result)
  AND (SELECT list(stem ORDER BY id) = ['run', 'jump', 'studi', 'happili', 'mice', 'ran', 'better', 'cat', 'relat', NULL, ''] FROM result)
  AND (SELECT list(base_form ORDER BY id) = ['run', 'jump', 'study', 'happily', 'mouse', 'run', 'good', 'cat', 'relational', NULL, ''] FROM result)) THEN 'as expected' ELSE error('the 11 words should give 11 stems and 11 base forms, with NULL staying NULL and the empty string staying empty') END AS verdict;
```

The statement builds its own data.

It needs the `fts` extension.

## What other tools call it

Meridian's classification; an independent blind rating agreed on 95.7% of 324 rows (variant, version 3).

- Alteryx Designer: [`Text Pre-processing`](https://help.alteryx.com/current/en/designer/tools/alteryx-intelligence-suite/text-mining/text-pre-processing.html) (read 2026-09-27)
- RapidMiner Studio: [`Stem (Arabic)`](https://docs.rapidminer.com/latest/studio/operators/extensions/Text%20Processing/stemming/stem_arabic.html) (read 2026-09-27)
- RapidMiner Studio: [`Stem (Arabic, Light)`](https://docs.rapidminer.com/latest/studio/operators/extensions/Text%20Processing/stemming/stem_arabic_light.html) (read 2026-09-27)
- RapidMiner Studio: [`Stem (Dictionary)`](https://docs.rapidminer.com/latest/studio/operators/extensions/Text%20Processing/stemming/stem_dictionary.html) (read 2026-09-27)
- RapidMiner Studio: [`Stem (German)`](https://docs.rapidminer.com/latest/studio/operators/extensions/Text%20Processing/stemming/stem_german.html) (read 2026-09-27)
- RapidMiner Studio: [`Stem (Lovins)`](https://docs.rapidminer.com/latest/studio/operators/extensions/Text%20Processing/stemming/stem_lovins.html) (read 2026-09-27)
- RapidMiner Studio: [`Stem (Porter)`](https://docs.rapidminer.com/latest/studio/operators/extensions/Text%20Processing/stemming/stem_porter.html) (read 2026-09-27)
- RapidMiner Studio: [`Stem (Snowball)`](https://docs.rapidminer.com/latest/studio/operators/extensions/Text%20Processing/stemming/stem_snowball.html) (read 2026-09-27)
- RapidMiner Studio: [`Stem Tokens Using ExampleSet`](https://docs.rapidminer.com/latest/studio/operators/extensions/Operator%20Toolbox/text_processing/stem_tokens_using_exampleset.html) (read 2026-09-27)
- subtoken: [`stem`](https://duckdb.org/docs/current/core_extensions/full_text_search.html) (read 2026-09-27)
- Talend: [`tStem`](https://help.qlik.com/talend/en-US/components/8.0/text-standardization/text-standardization-component) (read 2026-09-27)