Model / text · All operations

Count and weight terms per document

A variant of Split text into words

Build a document-term table of counts or TF-IDF weights from tokenised text.

Status

In arcform

Runs as a SQL step

arcform carries it as a sql: step today.

The DuckDB step

Count and weight terms per document with TF-IDF

SQL
CREATE TEMP TABLE docs AS SELECT * FROM (VALUES (1, 'the cat sat on the mat'), (2, 'the dog sat on the log'), (3, 'the cat chased the dog')) t(doc, body);
CREATE TEMP TABLE tokens AS SELECT doc, unnest(string_split(lower(body), ' ')) AS term FROM docs;
-- counts: one row per document and term
CREATE TEMP TABLE counts AS SELECT doc, term, count(*) AS n FROM tokens GROUP BY doc, term;
-- TF-IDF: the term's share of its document times ln(documents / documents holding the term)
CREATE TEMP TABLE weights AS
SELECT c.doc, c.term, c.n,
       c.n::DOUBLE / sum(c.n) OVER (PARTITION BY c.doc) * ln((SELECT count(*) FROM docs)::DOUBLE / count(*) OVER (PARTITION BY c.term)) AS tfidf
FROM counts c;
SELECT * FROM weights ORDER BY doc, term;
-- the document-term table in wide form: one row per document and one column per term
CREATE TEMP TABLE wide AS PIVOT (SELECT doc, term, n FROM counts) ON term USING sum(n) GROUP BY doc ORDER BY doc;
-- 'the' is in every document, so its weight is 0; 'chased' is only in document 3 and weighs most there
SELECT CASE WHEN ((SELECT count(*) = 14 FROM weights)
  AND (SELECT list(n ORDER BY term) = [1, 1, 1, 1, 2] AND list(term ORDER BY term) = ['cat', 'mat', 'on', 'sat', 'the'] FROM weights WHERE doc = 1)
  AND (SELECT max(tfidf) = 0 FROM weights WHERE term = 'the')
  AND (SELECT abs(tfidf - 0.067577518) < 1e-9 FROM weights WHERE doc = 1 AND term = 'cat')
  AND (SELECT abs(tfidf - 0.183102048) < 1e-9 FROM weights WHERE doc = 1 AND term = 'mat')
  AND (SELECT abs(tfidf - 0.081093022) < 1e-9 FROM weights WHERE doc = 3 AND term = 'cat')
  AND (SELECT abs(tfidf - 0.219722458) < 1e-9 FROM weights WHERE doc = 3 AND term = 'chased')
  AND (SELECT arg_max(term, tfidf) = 'chased' FROM weights WHERE doc = 3)
  AND (SELECT count(*) = 3 FROM wide) AND (SELECT count(*) = 9 FROM (DESCRIBE wide))
  AND (SELECT "the" = 2 AND "chased" = 1 AND "cat" = 1 AND "mat" IS NULL FROM wide WHERE doc = 3)) THEN 'as expected' ELSE error('the three documents should give 14 term counts, a TF-IDF of 0 for the in every document and a weight of 0.2197 for chased in document 3') END AS verdict;
  • The statement builds its own data.
  • It needs nothing installed beyond DuckDB itself.
  • The statement checks its own result and raises an error when it is wrong: “the three documents should give 14 term counts, a TF-IDF of 0 for the in every document and a weight of 0.2197 for chased in document 3”.
  • It passed on DuckDB v2.0.0 (Cyanoptera) d54cbbe3df.

What other tools call it

Meridian's classification; an independent blind rating agreed on 95.7% of 324 rows (variant, version 3).

RapidMiner Studio (7)

Background

Tf–idf (term frequency–inverse document frequency) measures how important a word is to a document in a collection, adjusted for words common in general.[1] Tf–idf is the product of two statistics, term frequency and inverse document frequency.[1]

When to use it

Variations of the tf–idf weighting scheme were often used by search engines as a central tool in scoring and ranking a document's relevance to a user query.[1]

Tf–idf did not prove more effective in all cases than plain term frequency: applied to citations, it gave no improvement over a simple citation count with no inverse document frequency component.[1] There are various ways of determining the exact values of term frequency and inverse document frequency.[1] RapidMiner's Process Documents operator can ignore words that appear in more, or fewer, than a given percentage or number of documents when it builds its word list.[2]

The method

Term frequency is the relative frequency of a term within a document: its raw count divided by the total number of terms in that document.[1] Other definitions of term frequency include the raw count itself, a Boolean 1 or 0 for whether the term occurs, a logarithmically scaled count, and an augmented frequency that prevents a bias towards longer documents.[1] Inverse document frequency is the logarithm of the total number of documents divided by the number of documents that contain the term.[1] A high tf–idf weight comes from a high term frequency in the given document and a low document frequency across the collection, so the weights tend to filter out common terms.[1] As a term appears in more documents, the ratio inside the logarithm approaches 1, bringing inverse document frequency and tf–idf closer to 0.[1] RapidMiner's Generate TFIDF operator takes either binary occurrences, which it normalises into term frequencies, or term frequencies already calculated.[3]

Karen Spärck Jones introduced inverse document frequency, as term specificity, in a 1972 paper.[1] Although inverse document frequency has worked well as a heuristic, its theoretical foundations were troublesome for at least three decades afterwards.[1]

Compared with the alternatives

Tf–idf is a refinement of the simple bag-of-words model, allowing the weight of words to depend on the rest of the corpus.[1] One of the simplest ranking functions sums the tf–idf of each query term, and many more sophisticated ranking functions are variants of that simple model.[1] DuckDB's full-text search extension searches its index with a match_bm25 macro, whose k and b parameters are those of the Okapi BM25 retrieval model.[4]

Sources

  1. [1]tf–idf - Wikipediasecondary source
  2. [2]Process Documents - Rapidminer Documentationdocumentation
  3. [3]Generate TFIDF - Rapidminer Documentationdocumentation
  4. [4]Full-Text Search Extension – DuckDBdocumentation