Recommendation engines

Evolution of recommendation engines and personalized marketing

Recommendation engines emerged in the early 1990s as a way to filter and suggest information based on user preferences. Initially developed in academic settings, these systems have since become essential tools in digital marketing and e-commerce.

1. Early collaborative filtering

One of the first breakthroughs came from the GroupLens Research Project at the University of Minnesota in 1994. Researchers like Paul Resnick and John Riedl developed collaborative filtering systems that could recommend news articles based on user ratings.

Read more about rating systems:  https://en.wikipedia.org/wiki/Rating_system

2. Commercial adoption

In 1995, the Firefly project brought collaborative filtering to a commercial audience. Led by MIT researcher Pattie Maes, it allowed users to receive music recommendations based on similar tastes. This marked the beginning of recommendation systems being used for marketing purposes.

Read more about Firefly:  https://en.wikipedia.org/wiki/Firefly_(website)

3. Amazon and scalable personalization

In the late 1990s, Amazon revolutionized the field by introducing item-to-item collaborative filtering. This approach, developed by Greg Linden and his team, allowed the platform to provide scalable and real-time product recommendations, personalizing the shopping experience for millions of users.

Read more about Amazon's recommendation algorithm:  https://www.amazon.science/the-history-of-amazons-recommendation-algorithm

4. From recommendations to personalized marketing

Today, recommendation engines are at the core of personalized marketing strategies. By analyzing user behavior, purchase history, and preferences, businesses deliver tailored content, emails, and product suggestions. This evolution has transformed how brands connect with consumers, making marketing more relevant and data-driven.

A. Numeric

A "numeric top 10" of the most sold products might be a list of the top 10 products ranked by the number of units sold, with the product that sold the most at the top of the list and the product that sold the 10th most at the bottom of the list. Similarly, a "numeric top 5" of the most watched movies might be a list of the top 5 movies ranked by the number of views, with the movie that was viewed the most at the top of the list and the movie that was viewed the 5th most at the bottom of the list.

At a digital summit, everyone is shown exactly the same list. So you don't really need an algorithm for this. After all, you just calculate which products sold the best. 

Numeric tops can be useful for various applications, such as identifying popular products, highlighting trends or patterns, or informing recommendations based on the most popular items. But they also have a lot of disadvantages:

1. Everyone gets the same list
2. Does not account individual user preferences
3. Good selling products sell even more
4. The niche gets little opportunity

A recommendation engine offers a good solution to eliminate these disadvantages

A recommendation engine, also known as a recommender system, is an AI algorithm that suggests items to users based on their interests and preferences. These items could be anything from products on an e-commerce website to movies on a streaming service.

The recommendation engine works by analyzing data on the user's behavior and interactions with the system. This data might include the items they have viewed or purchased in the past, as well as any ratings or reviews they have provided.

Using this data, the recommendation engine builds a model of the user's preferences and interests. It then uses this model to suggest new items that the user is likely to be interested in. The suggestions may be based on factors such as similarity to previously viewed items, popularity among similar users, or other relevant attributes.

For example, let's say you are browsing an online bookstore and have purchased several science fiction novels in the past. A recommendation engine might use this information to suggest new science fiction novels that you haven't read yet, based on your past behavior and the behavior of similar users.

Recommendation engines can be extremely useful for both users and businesses. Users benefit from personalized recommendations that help them discover new and interesting items, while businesses benefit from increased engagement and sales.

There are several different types of recommendation engines, including collaborative filtering, content-based filtering, and hybridapproaches that combine both methods. Each type has its own strengths and weaknesses, and the choice of which to use will depend on the specific use case and available data.

Collaborative filtering and content-based filtering are two different approaches to building recommendation engines.

B. Collaborative filtering

Collaborative filtering is based on the idea that people who share similar interests or behavior in the past are likely to share similar interests or behavior in the future. Collaborative filtering uses data on users' interactions with the system, such as their ratings, reviews, or purchase history, to find similarities between users and recommend items that similar users have enjoyed in the past.

For example, let's say you are browsing a streaming service and have watched several action movies in the past. A collaborative filtering recommendation engine might use this information to recommend other action movies that have been enjoyed by other users with similar viewing histories.

Collaborative filtering using customer data

Collaborative filtering is based on the idea that people who liked similar things in the past will like similar things in the future. To make this work, you track what users do: what they buy, watch, rate, or click on.

For example, if Alice and Bob both watched the same three shows, and Alice watched a fourth show that Bob hasn’t seen yet, the system might recommend that fourth show to Bob — simply because Alice and Bob have similar taste.

To make these comparisons, the system looks at large tables of user behavior and finds people or items that are often linked together. It doesn’t need to understand what the items are — just how people interact with them.

C. Content-based filtering

Content-based filtering, on the other hand, focuses on the attributes of items themselves to make recommendations. It uses data on the content or characteristics of items, such as their genre, plot, or style, to recommend similar items to users.

For example, let's say you are browsing an online bookstore and have purchased several romance novels in the past. A content-based filtering recommendation engine might use this information to recommend other romance novels with similar themes, settings, or writing styles.

Collaborative filtering and content-based filtering both have their strengths and weaknesses, and the choice of which to use will depend on the specific use case and available data. Hybrid approaches that combine both methods can also be effective in certain situations.

Content-Based filtering using product data

Content-based filtering doesn’t look at other users — it looks at the product itself. Each item is described using features such as genre, actors, keywords, brand, or technical specs.

For example, if you watch many science fiction movies that include space and robots, the system will look for other movies with the same features and recommend those — even if no one else has watched them yet.

When product descriptions are in text (like a movie plot or a product summary), the system can analyze the words used and compare them with other descriptions. The more words or categories two items have in common, the more similar they are.

Using TF-IDF for Content-Based Filtering

TF-IDF stands for "Term Frequency – Inverse Document Frequency". It's a way to understand which words are important in a product description or document.

In content-based filtering, TF-IDF helps convert product descriptions into numbers, so the system can compare them. For example, if two movies both mention "space", "future", and "robot" often, they are likely to be similar.

TF (Term Frequency) counts how often a word appears in a description. IDF (Inverse Document Frequency) reduces the weight of common words like "the" or "and", and increases the weight of rare but meaningful words.

By combining TF and IDF, the system creates a list of weighted keywords for each product. Then it compares these lists to find items that are similar in meaning, even if the exact words differ.

This is useful for finding related products, articles, or media — based purely on their content, without needing user behavior.

 D. Hybrid filtering

Hybrid filtering is a combination of collaborative and content based filtering.

Advantages of a recommendation engine

1. Everyone gets customized suggestions
2. Takes into account individual user preferences
3. People learning to know items they otherwise never get to know
4. The niche gets much more opportunities