Beyond the Buzz: recommendation engines

From useful suggestions to harmful amplification

Dear internet, what should I watch? What should I read? What should I buy? What music should I listen to? And lo and behold: the internet has a response just for you. We’ve got recommendation engines to thank for that, for better or worse. This specific application of data science can greatly improve your customer experience. Unfortunately, the technology can also amplify unhinged conspiracy theories and lead to radicalisation.

Luckily, most recommendation engines are actually relatively harmless customer-facing applications of predictive analytics. Based on user data and machine learning, the AI engine tries to predict what you would like to consume next.

Two applications

On a practical level, there are two major applications:

• Selling more products or services: the recommendation engine determines the ‘next best offer/action/price’ for an individual customer based on online (and offline) customer data. In this way, it can improve customer experiences while boosting sales.

• Optimising and personalising content: the engine makes personalised content suggestions based on a user’s history and profile, for example, on news websites, blogs, video platforms, social media, etc.

How do you know if a recommendation engine works?

Recommendation engines are one of the most popular data science applications for good reason: their impact is undeniable, and a glance at your own behaviour on Netflix, YouTube, Amazon and the likes only confirms this. But how do you know if it works?

The most common approach is via A/B testing. In most cases, half of the visitors see content that is recommended by the engine and the other half don’t. Then, you compare how both groups reacted to the content they saw. If the recommendation engine works, there should be a significantly higher conversion rate in the study group.

Misconceptions, hurdles and controversies

Although recommendation engines are well-known, there are still a few pitfalls, myths, and misunderstandings to heed for any business interested in implementing them.

• In many cases, you’ll need to collect offline customer data as well. For example, many older people still physically visit their health insurance providers, or their banks, or ... To get a comprehensive picture of your customers, you’ll need to collect this data as well–which is often challenging in itself due to legacy systems and the use of lock-in software.

• You’re never sure who’s actually behind the screen. Many couples and families share their Netflix accounts. But we also like to ask our friends or family members for their opinions before buying anything online. This has a big impact on our final decision, and the recommendation engine has no way of measuring its influence. As a result, many businesses overestimate the impact of recommendation engines when it comes to actual sales increases.

• Often, recommendation engines keep us from expanding our horizons. What we get is what an algorithm has calculated will keep you hooked for as long as possible. In this way, recommendation engines often appeal to our baser instincts and amplify our worst biases, boosting conspiracy theories and leading to increasing radicalisation. In a consumer context, this means that recommendation engines can also steer users away from things they might be interested in.

Recommendation engines can’t account for the impact people close to us have on our purchase decisions.