Businesses, often in heavily regulated industries, spend a huge amount of time and effort in sanctions screening, anti-money laundering and due diligence - all involving KYC checks for individuals, across many different data sources from an article online to daily data extracts. With all this effort though, many still rely too heavily on names and addresses being written consistently across all platforms. This is almost never the case.
So let's take a moment to really identify the problem. If a customer name appears when checking against a sanctions list, the system should identify it. That sounds totally reasonable. However, real-world data is never perfect. Names may be misspelled, abbreviated, certain characters may have changed or been used incorrectly when transcribing from other alphabets or even a keyboard missing an ñ key. As a result, relying on exact matches can miss potential matches:
- John Smith vs Jon Smith
- Jonathan vs Jonathon
- Shaun vs Sean
- Pena vs Peña
- O'Connor vs Oconnor
- And many more
So we're looking into a possible solution, one that can help your team focus their time where it matters. Here I will show you that it's possible to treat exact, partial and weak matches differently, bucket them up so that you can spend your time verifying the partial matches - while the exact and weak are treated accordingly. How are we going to do it? With Levenshtein distance. It's a type of fuzzy matching technique, and it's an incredibly good tool to know about.
The Business Problem We Need To Solve
A common task for compliance professionals is likely screening new customers against a sanctions list. Imagine the sanctions list contains the name "Anne Wight", while the customer database contains "Ann Wight".
Most people would immediately recognise that these names appear very similar. A human would likely review the result further. Unfortunately though, to a computer trying to find exact matches they are in no way related and will be overlooked.
If your workload is low, and you have under 100 names then you can hopefully handle this volume. However, time management becomes more significant when screening thousands or millions of names against another list of thousands or millions of names. Small differences in spelling, spacing, and punctuation can prevent genuine matches from being identified. The result is increased operational risk and a greater reliance on manual review processes.
But why exactly does exact matching fail?
Computers are extremely literal. When performing an exact comparison, every character must match perfectly. How do they check if letters are exactly the same? By using numbers. As far as the computer is concerned, the letter 'a' is not the letter 'a', it's in fact the number 97. 'b' is 98, 'c' is 99, and so on.
So computers store text as long chains of numbers. For example, this is what the computer sees when you type 'hello world':
104-101-108-108-111-32-119-111-114-108-100
h - e - l - l - o - [space] - w - o - r - l - d
So when the computer is comparing names for exact matches, it turns the name into a long list of numbers. If any of those numbers are different, it flags it as two completely different names. For example, the names "Jonathan Pennell" and "Jonathon Pennell" differ by only a single character (for the computer, that's the difference between 97 and 111 - clearly distinctly different). To us though, we may hardly notice the difference, but to a computer performing an exact comparison they are completely different.
j - o - n - a - t - h - o - n
106-111-110-97-116-104-111-110
106-111-110-97-116-104-97-110
j - o - n - a - t - h - a - n
That's the underlying root cause for many common data issues, such as additional spaces, missing hyphens, alternative spellings and abbreviations. It can get more complicated though, as names with non-ASCII characters introduce a world of complexity for computers that many people just aren't aware of - for example, the letter 'a' and the equivalent character in the Cyrillic alphabet are almost identical to you or me - but they're completely different as far as the computer is concerned.
Exact matching works well when data quality from source (e.g. a customer service agent or web form) all the way to end reporting database is perfect. Unfortunately, with so many potential ways of breaking, data quality is never perfect - the larger the dataset, the more likely you'll find issues with the data quality.
Now, it's entirely possible to pay an external vendor a lot of money to tackle this problem for you, and I can imagine a lot of companies have gone down this route. However, it's entirely possible to do this in-house, on your own hardware, with sensitive information never having left your network.
What is Levenshtein Distance anyway?
So, how do we tell a computer if one name is similar to another? Well, if we use the Feynman technique for a moment we can discover some fairly simple rules:
- Similarity decreases as the number of different characters increases, assuming there are the same number of characters e.g. Jonathan vs Jonathon
- Similarity decreases if we need to add more characters to get them to match
- Similarity decreases if we need to remove more characters to get them to match
Which means that, if we count the number of differences, and number of characters to add or remove in order to achieve the same string, we then have a number we can use to show how similar the two strings are. This is Levenshtein distance, it's a method that calculates the minimum number of character insertions, deletions and substitutions required to transform one string into another.
For example, let's look at changing "Jon" to "John". Jon requires the insertion of a single character 'h', after which they are identical. The Levenshtein distance is then one.
The smaller the distance, the more similar the strings are likely to be. This means we can get an actual quantitative number from the comparison, instead of a qualitative answer. What can we do with numbers? Run code and formulas on them!
So rather than producing a simple yes-or-no answer as a computer would, or a 'sort of' answer that a human would, fuzzy matching Levenshtein distance provides a measure of similarity that can be used in Excel, or better yet, Python!
How to use Levenshtein Distance in Python
Whatever your use case may be, implementing Levenshtein distance in Python is straightforward and can be done the fastest using external libraries.
Let's look at a few examples:
# Below we have an exact match between two names >>> import Levenshtein >>> score = Levenshtein.distance("Jonathan", "Jonathan") >>> print(score) 0 # Below we have two sets of names that are similar >>> import Levenshtein >>> score = Levenshtein.distance("Jonathan", "Jonathon") >>> print(score) 1 >>> score = Levenshtein.distance("Ben", "Ken") >>> print(score) 1 # Below we have two names that are quite different >>> score = Levenshtein.distance("Jonathan", "Melissa") >>> print(score) 9
So, here we can see that 'Ben' and 'Ken' have the same Levenshtein distance as 'Jonathan' and 'Jonathon'. Is that right? Should we trust this result? Is there a way to improve the accuracy? If we switch to Levenshtein ratio instead, we can get a value between 0 and 1 that represents the similarity - let's have a look at how that changes things:
# Below we have an exact match between two names >>> import Levenshtein >>> score = Levenshtein.ratio("Jonathan", "Jonathan") >>> print(score) 1.0 # Below we have two sets of names that are similar >>> import Levenshtein >>> score = Levenshtein.ratio("Jonathan", "Jonathon") >>> print(score) 0.94 >>> score = Levenshtein.ratio("Ben", "Ken") >>> print(score) 0.667 # Below we have two names that are quite different >>> score = Levenshtein.ratio("Jonathan", "Melissa") >>> print(score) 0.0
Much better! Now we can see that 'Jonathan' and 'Jonathon' are much more similar than 'Ben' and 'Ken'. We can now set a threshold for similarity, and only return matches that are above that threshold. This will help us to reduce the number of false positives, and focus our attention on the matches that are most likely to be correct.
Let's group these into categories based on their similarity scores: exact matches, close matches, and distant matches.
# Let's imagine 'targets' has the names we need to check, # and 'customers' are our customers. >>> import Levenshtein >>> for target in targets: >>> for customer in customers: >>> score = Levenshtein.ratio(target, customer) >>> if score == 1: >>> high_risk.append((target, customer, score)) >>> elif score > 0.85: >>> medium_risk.append((target, customer, score)) >>> elif score > 0.70: >>> low_risk.append((target, customer, score))
We now have lists of matches grouped by their similarity scores. These can be prioritized accordingly by you and your team!
Conclusion
So, we've gone over a technique that is in large part, free to use and learn with freely available common open-source Python libraries, and can save you a lot of time in data cleaning and matching tasks. There are plenty of other techniques to learn about, check out our other articles, or get in touch for more information about our training courses.