P(Correct | ‘Smith’ = Name, ‘Will’ = Surname) = Very low.Indeed, what is the reasoning process that leads us to believe that Name=‘Smith’/Surname=‘Will’ is likely to be a data entry mistake? Because - based on our knowledge of the world - we have learnt that: Naive Bayes mimics the way how humans think (for some simple issues). In this setting, you notice that it is not unusual to find rows in which a person’s first and last name are switched (the error rate is around 3%), as happens in the third row of the following table: Of course, first name and last name are fundamental features for record linkage. In this c ase, the only viable approach is probabilistic record linkage: a set of methods that enable to quantify the probability that two or more rows of a dataset are referred to the same individual.

Moreover, deduplication is made hard by the fact that unique IDs (such as Social Security Number) are scarcely present. The dataset suffers from a strong duplication issue, because data come from different sources. Imagine that you have a huge dataset (more than 20 million rows) containing personal data about some people. The problem: mixing up first name and last name The hope is that this application may serve as inspiration for other comparable case studies. This article comes from a very specific real-life problem that arose on a real-life dataset, and was addressed with some success through the described methodology.

Sven computer real name