Resolve entities

Status

Variants

Match records with a trained model

Score candidate pairs with a trained matching model and give each group of matching records one identifier.

Reconcile values against an entity service

Match each value to an entity in an external reconciliation service, then accept, review or discard the matches.

What other tools call it

Meridian's classification; an independent blind rating agreed on 86.2% of 312 rows (exact class, version 2).

Background

When to use it

Deterministic record linkage is a good option when the entities are identified by a common identifier, or by several identifiers of relatively high quality.^[1] In deterministic linkage, a small decrease in data quality or a small increase in the complexity of the data can result in a very large increase in the number of rules needed.^[1] Where two tables share a clean key, join on it; reach for a matching model when there is no shared key and the hand-written rules keep multiplying. (Meridian's view)

Record linkage is highly sensitive to the quality of the data being linked, so the data sets should ideally undergo a data quality assessment first.^[1] Standardise names, dates and addresses before matching, and bring every column that tells two entities apart, not only the name. (Meridian's view)

Interactive record linkage systems have been proposed because linkage errors propagate into the linked data and its analysis.^[1]

In AWS Glue, FindIncrementalMatches matches an incremental set of records against an existing, pre-matched set and assigns a unique identifier to each group of matching records.^[3] In AWS Glue, the match_id column that FindMatches adds is an arbitrary identifier, and records with the same match_id have been identified as matching each other.^[4] Treat a match identifier as a label for a group of records, not as a key with a meaning of its own. (Meridian's view)

In OpenRefine, reconciliation is semi-automated, and human judgement is required to review and approve the results.^[2] OpenRefine reconciles through string searching by default, so typos, whitespace and extraneous characters affect the results.^[2]

The method

The idea of record linkage goes back to Halbert L. Dunn's 1946 article Record Linkage, published in the American Journal of Public Health.^[1] Howard B. Newcombe laid the probabilistic foundations of modern record linkage theory in a 1959 article in Science, giving greater weight to agreements on less frequent values.^[1] Ivan Fellegi and Alan Sunter formalised Howard B. Newcombe's ideas in 1969, in A Theory for Record Linkage.^[1] The Fellegi–Sunter rule compares a likelihood ratio with an upper and a lower threshold, and sends the pairs that fall between them to clerical review.^[1]

In the Fellegi–Sunter model, the m-probability is the probability that an identifier agrees when two records match, and the u-probability is the probability that it agrees by chance when they do not.^[1] In probabilistic record linkage, agreement on a rare value, such as an uncommon surname, counts for more than agreement on a common one.^[1] Multiplying the likelihood ratios of separate identifiers assumes the comparisons are conditionally independent given whether the records match, the assumption a naive Bayes classifier makes.^[1] The independence assumption may be inaccurate when identifiers are associated, for example when first and last names are correlated with one another.^[1] The m- and u-probabilities can be estimated from known matches or from the data being linked, often using the expectation–maximisation (EM) algorithm.^[1]

Comparing every record in one file with every record in another quickly becomes computationally infeasible.^[1] Blocking reduces the cost by comparing only pairs of records that agree on one or more selected identifiers, called blocking keys.^[1] Under blocking, true matches that disagree on a blocking key are never compared.^[1] Record linkage systems often use several blocking passes with different keys, so that a match missed by one pass can be found by another.^[1] Running names through a phonetic algorithm such as Soundex or metaphone can help match names that are slightly different.^[1]

Splink is a Python package for probabilistic record linkage that can deduplicate and link records from datasets without unique identifiers.^[5] Splink's core linkage algorithm is based on the Fellegi–Sunter model of record linkage, with various customisations to improve accuracy.^[5] Splink predicts which rows link together and clusters those links to produce an estimated person ID.^[5] Splink can train its models without training data, using an unsupervised approach.^[5] Splink recommends estimating the u-probabilities directly from random record comparisons, because on a sufficiently large dataset two random records will almost certainly not match.^[6] Splink recommends EM training for the m-probabilities, and notes that EM is not guaranteed to converge to a global minimum, only a local one.^[6] In Splink, the match weight of two records is the sum of a prior weight and one weight for each compared column, which follows from the Fellegi–Sunter independence assumption.^[7]

When labelled training data is available, supervised methods such as random forests can be used in record linkage, and some studies have reported higher accuracy for them.^[1] In AWS Glue, the FindMatches transform is taught by labelling example records to show which rows match, and the labelled records must be present in the source dataset.^[4] To reconcile against an external dataset in OpenRefine, that dataset must offer a web service that conforms to the Reconciliation Service API standards.^[2] OpenRefine shows a confident match as a single dark blue link, and otherwise lists one or more candidates with their reconciliation scores for a person to choose between.^[2]

Compared with the alternatives

Deterministic or rules-based record linkage generates links from the number of identifiers that match, while probabilistic linkage weighs the evidence from several identifiers.^[1] Record linkage and linked data are two separate approaches, although both involve identifying matching entities across different data sets.^[1]

Sources

1. [1]Record linkage - Wikipediasecondary source
2. [2]Reconciling | OpenRefinedocumentation
3. [3]FindIncrementalMatches class - AWS Gluedocumentation
4. [4]Record matching with AWS Lake Formation FindMatches - AWS Gluedocumentation
5. [5]Splinkdocumentation
6. [6]Training rationale - Splinkdocumentation
7. [7]The Fellegi-Sunter Model - Splinkdocumentation

Used in