The register shows three service recipients. In fact it is one person: Olena Petrenko, entered three times. Once correctly, once with an extra dot after the first name, and once with a different Latin spelling, because that is how another office does it. The example is illustrative, but anyone who has worked with a large register will recognise it.
Where duplicates come from
A duplicate rarely appears out of carelessness. More often it is a consequence of how the process is built.
- Typos at entry. An extra space, a dot, swapped letters. To a person it is the same record, to the system a new one.
- Transliteration. One surname can be written in Latin script in several ways. Petrenko, Petrenco, Petrenka: each version is formally different.
- A new document. The person changed their passport, took a new surname after marriage or got an ID card. Nobody found the old record, so a new one was created.
- Several entry points. An online application, a service centre, an import from a neighbouring system. Each channel creates records its own way, and nobody checks whether the person already exists.
What they distort
The most obvious one: the count. If 10% of a register are duplicates, a coverage report is inflated by roughly the same amount, and that figure is then compared with the plan.
It gets worse when money depends on the record. Two records of one person can receive two payments. Or the opposite: the payment is tied to the old record, while the person now comes through the new one and hears that nothing is due.
It is like a phone book where one person is saved as "Olena", "Olena work" and "O. Petrenko". Calling is inconvenient, but you can live with it. (The analogy has limits: your phone does not send money to each contact.)
How to find likely duplicates
Comparing records by name alone is not enough: different people share names. Comparing by document number alone is not enough either: documents change. So the search uses a key built from several fields:
- Surname and first name after normalisation: no spaces or dots, one transliteration rule.
- Date of birth.
- Document number without the series and separators.
- A tax ID or another stable identifier, if there is one.
An exact key match finds the obvious duplicates. For the rest, fuzzy matching is used: an algorithm scores how similar two records are and suggests pairs with a high score.
It suggests, it does not decide. I would not merge records automatically: wrongly merging two different people is far harder to undo than finding a duplicate. High-score pairs are reviewed by someone who knows the process.
The key decision: which record is the master
Finding a duplicate is easy. Deciding what to do with it is harder. That takes rules, not a "merge" button:
- Which record remains the master record, and on what principle: the oldest, the most complete, the one confirmed by a document.
- What happens to the history of payments and services linked to the other records.
- Who approves a merge when money depends on it.
- How other systems that hold references to the old record learn about the change.
This is a data governance question, not only a technical one. Without a data owner, duplicate cleaning turns into an argument about whose record is more correct.
Why the fix belongs at the source
The most common mistake is removing duplicates in the report. Every month an analyst writes yet another cleaning rule, the report becomes correct, and the register stays the same. Next month it all repeats, and the rule gets a little more complicated.
Checking at entry is much cheaper. When an operator enters a new person, the form searches for similar records using the same key and shows them: "A similar record already exists. Is this the same person?" That check works once and for good, not every month.
What is better not to do
A few approaches look sensible but usually make things worse:
- Deleting duplicates. A deleted record takes its history with it: applications, decisions, payments. It is better to mark the record as merged and keep a reference to the master.
- Trusting one field. "If the tax ID matches, it is the same person" works until zeros or someone else's number get into the field. A key from several fields is more robust.
- Cleaning once a year as a big project. After the cleanup, duplicates pile up again at the same rate if the source has not changed.
Each of these gives a quick result in the report. And each postpones the real problem: a process that creates new duplicates every day.
Check it on your own data
Take your main register and count two numbers: the number of records and the number of unique people by a multi-field key. If the second is noticeably smaller, you have found not only a data quality problem but also a place where coverage reports say more than there really is.
Do you know how many people in your register have more than one record?