Are your predictive models still up to date?

28 March 2019

Article written by Jurriaan Nagelkerke, Principal Consultant

Are your predictive models still up to date?
Predictive models are a powerful and common tool to gain more knowledge and value from data, that isn’t always obvious at first glance. For example, predictive modelling can be used to select customers who offer a good chance at a sale or cross selling, or to identify those customers who are likely to cancel their subscription and/or leave. A campaign can be more effective and save a company money when this campaign is only aimed at customers who, according to the model, are likely to take their business elsewhere. Moreover, you do not flood the same customer with campaigns. However, the power of a predictive model depends on the quality. That is why it is important to have regular checks and see if your model is still up to date.

Many predictive models are custom made

A good predictive model is usually made to fit. They are built by knowledgeable analysts or by an external contractor who has specialized in predictive techniques. A few weeks of collecting data and data preparation, combining, analyzing and tweaking it, results in a good model which can be used for selective processes. Depending on the type of campaign the customer or prospect base will be scored on a weekly, monthly or quarterly basis. That way, those customers can be selected who offer the best chance at sales, to make an appropriate offer in the campaign.

Using an outdated predictive model is not without risks

In our customer relations, we have experienced many successful cases of predictive modelling. Created by us and subsequently handed over, or sometimes made by the customers own analysts. We often see that customers keep using their predictive models over and over again without any questions asked. Simply said: as long as the model generates output that can be used for a campaign nobody checks the model. Yet as time goes by, there are many internal and external influences on your model. Which as a result no longer works optimally and doesn’t automatically select the right target group anymore. This is a real shame! By adapting the model, retraining the model or adding new data sources to the predictive model, the return would possibly have been better.
Secondly, using an outdated model is not without risks. An outdated predictive model can result in selecting customers with data that is no longer allowed because of new privacy regulations. Or a campaign that targets the wrong customers because of changes in the data landscape.

Optimise your campaign and check your predictive model

Part of professionalizing and optimizing campaigns are ongoing quality checks. This can be done in part by monitoring the predictive power of the used model. Checking multiple fields to see if the model still performs like it is supposed to, is just as important. But is that not the task of the analysts who have created the model, or who are responsible for its management? In our experience analysts are predominantly working on developing new insights and models. Because that is what their employer asks from them, but also because it gives them more energy. Once management of the model has been transferred to the department, it becomes difficult to assess whether or not the quality of the model is still sufficient. Checking periodically if everything still works like it’s supposed to, is usually not a priority. This becomes even worse when the person or company that created the model, is no longer part of the organization.

An example on the effect of prolonged use of a predictive model

An example to explain the consequences of prolonged use of a predictive model. By using the same model to search the same database of prospects for too long, the model will keep selecting the same prospect each time. But these prospects probably received the same offer multiple times. Yet is it highly unlikely that they will accept the same offer this time. They probably will choose to opt out for any future communication. Another risk is that one of the predictive indicators in the model will contain different values after a longer period of use, than it did while we were creating the model. An example could be a product for which the name has been changed. While having a ‘Plus subscription’ would lead to a strong increase in response from customers, it is no longer a predictive indicator when this particular subscription has been renamed into ‘Premium subscription’. Name changes or other changes occur more often than suspected and can make an unexpected mess of predictive models. What should also be discussed: is it allowed to keep using certain predictive indicators? Take the contact behavior of the customer over a longer period of time for example. Does it comply with internal regulations to use this data for this particular goal?

Don’t miss it: the periodic check of your predictive model

To assess the risk that the model no longer works like it’s supposed to, we from Cmotions have developed the Check for Predictive Models (currently only available in Dutch). Just like getting your car serviced at least once a year, to be sure that everything is safe, it is wise to have your predictive model checked regularly. Does your model still make the best possible prediction with the available data and within applicable legislation? And does your model still fit the purpose of your current campaign?

Online MOT: the check for your predictive model

From our broad experience with building and implementing predictive models, we developed an Online Check based on 10 questions for management. The questionnaire is aimed at four different fields. The first is an indication for which of this 4 fields your predictive model possibly is at risk. You will know directly if there is any risk. Is there an increased risk for your predictive model, then we have an extensive roadmap for a (self)scan in each of the four fields. This scan looks more closely at things like variables, monitoring, predictions, legislation, etc. These questions are to be answered by the analyst, because of his specific knowledge of all data. The results will be discussed with all parties involved during an expert session. This meeting will make it clear whether or not any changes are necessary, and which changes that would be. Our aim is to close this session with a well-earned certificate to identify that the model is futureproof.

Are you in doubt about the predictive model that is being used by your organization? Then do the Online Check (only in Dutch) and you will know within 5 minutes if your model is at risk, or if it’s perfectly safe to keep using it.

Do the Online Check (only in Dutch)>

More information about the Online Check (only in Dutch).