Classify text

Status

Variants

Train a text classifier

Train a model from labelled text that assigns new text to categories.

Classify text without training

Score each text against a list of categories you name, with no labelled training data.

What other tools call it

Meridian's classification; an independent blind rating agreed on 86.2% of 312 rows (exact class, version 2).

Background

When to use it

Text classification techniques have been applied to spam filtering, email routing, language identification, genre classification, readability assessment and sentiment analysis.^[1] Reach for it when every text in a column needs one answer from a short list you can name, such as a topic, a priority or a sentiment. (Meridian's view)

In Alteryx Designer, the Text Classification tool trains its model from training data that must have a column with text and a column with the text's label.^[4] Naive Bayes classifiers have worked quite well in many real-world situations, famously document classification and spam filtering, and need only a small amount of training data.^[5] The effectiveness of a model at a zero-shot task seems to scale with model size, larger models generally doing better.^[6] In Hugging Face Transformers, the zero-shot pipeline lets the candidate classes be chosen at runtime, which usually makes it slower but much more flexible than a text-classification pipeline.^[7] Where you hold labelled examples and the categories will not change, a trained classifier is the steadier choice; where the categories are still moving, a zero-shot classifier lets you redraw them without relabelling. (Meridian's view)

In scikit-learn's example on the 20 newsgroups dataset, the word caltech became one of the top predictive features for atheism, through metadata such as the email addresses of earlier senders.^[8] In scikit-learn's 20 newsgroups example, the classifier trained with message headers, footers and quotes gave over-optimistic scores, and without them the problem was less accurate but more representative.^[8] Strip anything that identifies where a text came from rather than what it says, such as signatures, headers and quoted replies, before training, and judge the model on texts it has not seen. (Meridian's view)

Naive Bayes is known as a decent classifier but a bad estimator, so the probability outputs of scikit-learn's naive Bayes models are not to be taken too seriously.^[5] In the Hugging Face Transformers zero-shot pipeline, the label scores for each text sum to 1 by default, and with its multi_label option set to true each label is scored independently.^[7] Read a classifier's score as a ranking of the categories rather than a calibrated probability, and where a text can belong to several categories at once, score the labels independently rather than as shares of one total. (Meridian's view)

The method

Machine learning's advantages over the knowledge engineering approach, in which domain experts define a classifier by hand, are very good effectiveness, savings in expert manpower and straightforward portability to different domains.^[2]

Alteryx Designer's Text Classification tool converts raw text into numerical data with term frequency-inverse document frequency (TF-IDF) vectorisation.^[4] In Alteryx Designer, the multinomial naive Bayes model predicts the probability that a piece of text belongs to a label, and assumes that all features are independent of one another.^[4] In scikit-learn's multinomial naive Bayes, smoothing priors account for features absent from the training samples and prevent zero probabilities, and a prior of 1 is called Laplace smoothing.^[5] Alteryx Designer also offers Linear SVC, a model of the support vector machine class that is effective in high-dimensional space such as text but may be slow on a large training dataset.^[4]

Wenpeng Yin, Jamaal Hay and Dan Roth framed zero-shot text classification as textual entailment in 2019, benchmarking it on labels for topic, emotion and situation.^[3]

Compared with the alternatives

Supervised document classification learns from an external mechanism, such as human feedback, that gives the correct class, while unsupervised document classification, also called document clustering, uses no external information.^[1] Cluster when you do not yet know what the categories are, and classify once you can name them. (Meridian's view)

Sources

1. [1]Document classification - Wikipediasecondary source
2. [2][cs/0110053] Machine Learning in Automated Text Categorizationsecondary source
3. [3][1909.00161] Benchmarking Zero-shot Text Classification: Datasets, Evaluation and Entailment Approachsecondary source
4. [4]Text Classificationdocumentation
5. [5]1.9. Naive Bayes — scikit-learn 1.9.1 documentationdocumentation
6. [6]What is Zero-Shot Classification? - Hugging Facedocumentation
7. [7]Pipelinesdocumentation
8. [8]Classification of text documents using sparse features — scikit-learn 1.9.1 documentationdocumentation