Content Recommendations That Work: Coding the System

Introduction

We've already discussed everything you need to know about the content recommendation systems in the previous article, now it's time to talk about how it works in practice. If you haven't read the previous part of the series about the Content Recommendations That Work, be sure to check it out!

Before we continue, let's make a quick recap of the previous article and what to expect from this part of the series. Previously, we've covered the basics of content recommendation systems, including what they are and why they matter. We also discussed the different types of recommendation systems, their pros, and cons, and saw some of the platforms that utilize these systems.

As you might remember, we've dedicated the whole section talking about the Recommendation Pipeline System (RPS) and its benefits. Now that we understand the theory behind content recommendation systems, let's dive into how to actually implement it in code. It's going to be a basic and straightforward implementation.

Note. We'll use TypeScript, since its syntax is very similar to many other programming languages. I cannot use plain JavaScript since it doesn't have interfaces and types that we need to use in a proper recommendation system. The idea is just to show you an API of RPS, it will be your job to integrate it with your application and database.

Recap of RPS Components

Since we aren't concerned about the application structure and programming languages involved, we are not going to cover TypeScript installation. Instead, the focus of this article is to provide you with a basic understanding of how the RPS works and how to implement it in your code.

We already know that RPS consists of five main components:

1. The data stream is the object that carries the information;
2. A filter is a handler that processes the data;
3. An item is the element to be recommended, like a blog post, movie, or book;
4. An excluded item is the element that should not be recommended;
5. A fetched item is the element that has been retrieved from the database or other type of storage;

The final API

Let's start from seeing the final result of our implementation to have a more in-depth understanding of what we're aiming for.

const recommendations = new RecommendationSystem(6)
    .pipe(new ExclusionFilter([1, 2, 3]))
    .pipe(new LikedItemsFilter())
    .get()

I would simply create an API endpoint and put this logic in it. Then, when a user visits some product page, I would make an AJAX request to get recommendations for this current page.

In this code, we are initializing a new instance of the RecommendationSystem class and passing the number of recommendations we want to collect. In our example, we aim to collect six items. We are piping two filters to our recommendation system: the ExclusionFilter and the LikedItemsFilter.

As we discussed earlier in the previous article, you can have as many filters as you want, each serving a specific purpose. These 2 filters are just examples from my personal use case where I excluded items 1, 2, and 3 from recommendations and applied a LikedItemsFilter to recommend items based on the user's previous likes.

Just to give you an alternative code, here is the same Recommendation system instantiation in one of my Laravel projects:

$recommendations = (new RecommendationSystem(3))
    ->pipe(new LikedPostsFilter())
    ->pipe(new PostSeriesFilter($post))
    ->pipe(new PostTagsFilter($post))
    ->get();

First, I exclude all the posts that have been already liked. Second, I add posts belonging to the same series as the current post. Finally, I include posts having similar tags as the current post.

As a result, when a particular user is reading a post about Vue.js, for example. They will receive recommendations for other posts that are part of the same series or have similar tags, while also excluding any posts they have already liked. Very handy, isn't it?

If the second filter adds 3 posts (the amount I need) belonging to the same series as the current one, all the next filters in the pipeline will be ignored. Because the system has already fulfilled the maximum number of recommendations, which is 3 in this case.

The level of customization and automation that a recommendation system can achieve is truly mind-blowing! It's wonderful to think that we, as programmers, have the power to create such personalized solutions that can meet the unique needs of every user. It's moments like these that remind us why we love what we do and why we are so passionate about programming.

The DataStream object

Before we continue talking about the implementation of the RPS, let's see how the Data stream looks like and how it works. The Data stream is a regular object that represents the source of data for the recommendation system. This object serves as a context for the filters and the system itself to operate on. It's like a water stream that carries molecules of information traveling through the filters and pipes. This is what it looks like:

class DataStream {
    public constructor(
        public items: Item[],
        public exclude: number[],
        public remaining: number,
        public readonly limit: number,
    ) {
    }

    public needsMoreContent(): boolean {
        this.remaining = this.limit - this.items.length
        return this.remaining > 0
    }
}

It's just an object with 4 properties that holds information about the data used for recommendations. The needsMoreContent methods are optional, but it simplifies the code and makes it more readable. Since this article is a detailed explanation of this type of recommendation system, here is the clarification of each property of the DataStream object:

• The items property refers to the array or collection of items that will be recommended to the user. It could be an array of posts, products, movies, or anything else.
• The exclude property is an array of item IDs that don't want to show the user anymore, therefore should be excluded from the recommendations. It also contains IDs of items that we already added to items property. If we didn't have this property, we would end up with duplicate items in the recommendations.
• The remaining property indicates how many more recommendations the system can provide before reaching its limit. It's also dynamic and changes throughout of our pipeline based on the filters that we apply.
• The limit property is the maximum number of recommendations that the system can provide, which has been set to 3 in our example. If your programming language allows you to make this property a readonly flag, it's a good practice to do so, as it ensures that the value of this property cannot be changed accidentally or intentionally.
• The needsMoreContent method updates the remaining property based on the current number of items in the items array and checks if there is still space for more recommendations in the items array, by comparing the remaining property with 0.

The RecommendationSystem object

The RecommendationSystem object is the main heart of the whole API, and it uses the DataStream class to store and process the data. The purpose of this object is to glue everything together and provide a simple, user-friendly API for generating recommendations.

As you saw previously, the RecommendationSystem object has only 2 public methods, which are pipe and get. Let's take a look at them:

class RecommendationSystem {
    private stream: DataStream

    public constructor(limit: number) {
        this.stream = new DataStream([], [], limit, limit)
    }

    public pipe(filter: Filter): this {
        if (this.stream.needsMoreContent()) {
            this.stream = filter.process(this.stream)
        }

        return this
    }

    public get(): Item[] {
        if (this.stream.needsMoreContent()) {
            const randomItems = this.getRandomItems()
            return this.stream.items.concat(randomItems)
        }

        return this.stream.items.slice(0, this.stream.limit)
    }

    private getRandomItems(): Item[] {
        const result = new Database()
            .getItems()
            .exclude(this.stream.exclude)
            .limit(this.stream.remaining)
            .get()

        const remainingItems = this.stream.remaining - result.length

        if (remainingItems === 0) {
            return result
        }

        const additionalItems = new Database()
            .getItems()
            .exclude(result.map(item => item.id))
            .limit(remainingItems)
            .get()

        result.push(...additionalItems)

        return result
    }
}

Let's talk shortly about each of these methods:

• The pipe method takes a filter object as an input and processes it with the DataStream. If the DataStream needs more content, it uses the Filter to generate recommendations and updates its state accordingly.
• The get method returns the final result. However, if the DataStream still requires more content, it generates some random recommendations and adds them to the final result.
• The private method getRandomItems is responsible for generating random recommendations when the DataStream still requires more content. If you have lots of content in your database, it's less likely that this method will be called frequently. It is not significant, you can define it differently based on the needs of your project. However, in some of my projects it matters a lot, that's why I recommend having it.

Overall, these methods are used to manipulate the DataStream and generate recommendations. Nothing like the rocket science here, just simple yet effective logic that does the job efficiently.

The Filter object

The Filter objects are used as arguments to a pipe method. The pipe method executes the filter. That's what Filter object does, it processes the data and manipulating the data stream.

Every Filter has to implement a process method that takes a DataStream as input and returns a new, modified DataStream. Here is the interface:

interface Filter {
    process(stream: DataStream): DataStream
}

Filters could be complicated or simple, depending on your requirements. Here is the example of a simple Filter object that excludes given items from the final recommendation list:

class ExclusionFilter implements Filter {
    public constructor(private readonly excludeIds: number[]) {
    }

    public process(stream: DataStream): DataStream {
        stream.exclude.push(...this.excludeIds)
        return stream
    }
}

Filters are just a way to hide the complexity of data processing and deal with the Database. SQL queries and other Database related operations can be encapsulated within filters to make the code easier to read and maintain.

Here is an example of a more complex Filter object, that adds items that have the same tags that the item being watched by the user:

class ItemTagsFilter implements Filter {
    public constructor(private item: Item) {
    }

    public process(stream: DataStream): DataStream {
        stream.exclude.push(this.item.id)

        const tags: Tag[] = this.item
            .getTags()
            .select(['id'])
            .get()

        for (const tag of tags) {
            const items = this.getTagItems(tag, stream)

            stream.exclude.push(...items.map(item => item.id))
            stream.items.push(...items)

            if (!stream.needsMoreContent()) {
                break
            }
        }

        return stream
    }

    private getTagItems(tag: Tag, stream: DataStream): Item[] {
        return tag.getItems()
            .exclude(stream.exclude)
            .limit(stream.remaining)
            .get()
    }
}

It doesn't need much explanation, it's easy to see that it fetches items from the Database, based on tags that the current item has. Then, it just pushes fetched items to the Data stream until we have reached our the content limit or there are no more items left to add.

Summary

Let's digest what we have learned here and look at the whole picture again. I want to show the RPS graph from the previous article, which demonstrates the entire workflow.

First, we instantiate the RecommendationSystem object with a new instance of the DataStream. We apply some filters using the pipe method of the DataStream object. In our scheme, we are using “Rated items” and “Meta related” filters. In the end, we call the 'get' method, which checks if we have enough content to return to the user.

If you still find this system complicated, or have some additional questions, don't worry! It's normal to have questions and concerns when dealing with complex systems. Leave a comment down below, and I'll be happy to answer them and provide further clarifications.

Conclusion

In conclusion, building a recommendation system can be complex, but implementing it with the right set of tools and methodologies will help us achieve our goal of delivering accurate and personalized recommendations to our users. In this article, we have learned about the implementation details of the Recommendation Pipeline System (RPS).

I don't know your application and programming language you are using, that's why the code snippet provided may not be directly applicable to your project. However, the general concepts and strategies presented here can be applied to any programming language or project. It doesn't matter if you use Kotlin, PHP, Python, or Go, the principles behind building a recommendation system remain the same.

RSP is not perfect, I would be glad to hear what you think and maybe introduce some changes to this article if needed.

In the next article of this series, we will cover a Content-based Recommendation System, which is a very popular approach to building recommendation systems used by many companies. It's going to be an impressive article with lots of valuable information. Stay tuned and keep learning!

If you need a reference project where this system is implemented, I've built a simple Laravel project that you can find on GitHub. I did it just for the demonstration purposes and some references.

This is a page where all the posts are listed:

The next page is a single post page that has recommendations based on post tags and currently viewed post: