I want to identify and remove duplicate records from our database...


1) Situation...

Poor quality data is costing our company dearly. Not just in terms of wasted mailings and postage but also in misdirected marketing effort through incorrect and missing information.

At an absolute minimum, we estimate 2.5% - 5% of our UK and European customer database contains duplicate records - but we just don't know.  Assuming 80% of our business comes from just 20% of our customers, those customers will by default have had more communications with our company and therefore there is a strong likelihood that most of our duplicates fall in this group.  If we upset or lose this 2.5% and they are regular purchasers, we could be looking at losing 10% of our income - each year!  

We have previously tried to identify duplicate records using a mix of in-house skills and brought-in software, but this has only been partially successful.

The reasons for this are mainly due to:

  • a lack of data entry standardisation

  • poor address quality

  • sales and telemarketing staff very often misspelling company details and people's names

  • the capture of data from hand written coupons and web enquiries often being incomplete or inaccurate

Our software and matching algorithms simply don't work - being rather limited, inflexible, difficult to use and often identifying duplicates that are actually quite separate customers.

Worse though is that when duplicates have been identified, data from the multiple records has not been consolidated before a customer record has been removed - hence we lose valuable sales and marketing information including multiple telephone numbers, email addresses, profile data, etc. Additionally, and this really does hurt, is that customers incorrectly identified as duplicates have also been removed from our database!

To add to the de-duplication issue, our IT department are in the process of implementing a new CRM system and therefore it is a good opportunity for us to de-duplicate and clean the data prior to it being loaded. If we don't do this now, there is is a good chance it will never get cleaned.

What we require therefore is either the specialist software, training and support to help us undertake the de-duplication work ourselves, or for a specialist company who really understand the worth of accurate data to a business and its sales and marketing targeting, rather than having purely an IT focus, to do the work for us.


 

2) Solution approach...

3) Real benefits...

4) Prevention better than cure...