Tag parts of speech

Status

What other tools call it

Meridian's classification; an independent blind rating agreed on 95.7% of 324 rows (variant, version 3).

Background

When to use it

POS tagging is harder than looking each word up in a list of words and their parts of speech, because some words can represent more than one part of speech.^[1] The word dogs is usually a plural noun, but is a verb in the sentence The sailor dogs the hatch.^[1] Lemmatisation, which finds the dictionary form of a word, depends on correctly identifying its intended part of speech and meaning.^[2] Alteryx's Part-of-Speech Tagger tool uses the part-of-speech capabilities of the spaCy package, and puts its tagging accuracy for English at about 97%.^[3]

In POS tagging by computer, it is typical to distinguish from 50 to 150 separate parts of speech for English.^[1] The most popular tag set for American English is probably the Penn tag set, developed in the Penn Treebank project.^[1] Petrov and colleagues proposed a universal tag set with 12 categories, and whether a small set of broad tags or a larger set of precise ones is preferable depends on the purpose.^[1] Automatic tagging is easier on smaller tag sets.^[1] The Universal Dependencies POS tags mark core part-of-speech categories, such as NOUN for noun, VERB for verb, ADJ for adjective and X for other.^[4] Choose the tag set by what the next step needs: a filter that keeps only nouns needs broad tags, and a study of tense needs fine ones. (Meridian's view)

Tagging words in agglutinative languages such as the Inuit languages may be virtually impossible.^[1] Alteryx's Part-of-Speech Tagger supports English, French, German, Italian, Portuguese and Spanish, but its output tags are only available in English.^[3]

The method

The Brown Corpus, the first major corpus of English for computer analysis, was developed at Brown University by Henry Kučera and W. Nelson Francis in the mid-1960s.^[1] A first tagging of the Brown Corpus, by a program by Greene and Rubin, got about 70% correct, and its results were corrected by hand until the tagging was nearly perfect by the late 1970s.^[1] In the mid-1980s, researchers in Europe began to use hidden Markov models (HMMs) to disambiguate parts of speech, counting cases and making a table of the probabilities of sequences.^[1] An HMM tagger can decide that can in the can is far more likely to be a noun than a verb or a modal.^[1] In 1987, Steven DeRose and Kenneth W. Church independently developed dynamic programming algorithms for tagging, similar to the Viterbi algorithm, which both achieved accuracy of over 95%.^[1]

Eugene Charniak pointed out in 1997 that merely assigning the most common tag to each known word, and proper noun to every unknown one, will approach 90% accuracy.^[1] E. Brill's tagger, one of the first and most widely used English POS taggers, uses rule-based algorithms.^[1] Measure a tagger on a sample of your own text, and compare it with the simple rule of giving each word its most common tag, rather than trusting a benchmark figure. (Meridian's view)

Compared with the alternatives

Lexical tokenisation converts text into tokens belonging to categories, which for a natural language include nouns, verbs, adjectives and punctuation.^[5] Read a tagger's output as one more column per word: splitting the text decides the rows, and the tag only labels them. (Meridian's view)

Sources

1. [1]Part-of-speech tagging - Wikipediasecondary source
2. [2]Lemmatization - Wikipediasecondary source
3. [3]Part-of-Speech Taggerdocumentation
4. [4]Universal POS tagsstandard
5. [5]Lexical analysis - Wikipediasecondary source