Searching in a huge amount of data

In this section, we are going to develop a new feature for our URL shortener. This feature will need to search for data in a huge table, which, as you can guess, could delay the response of the system. Let's see the feature we are going to add and the solutions that we are going to implement.

The huge blacklist problem

Now, imagine that our URL shortener is very successful. We want to offer a premium service for our users, We will have a new feature, when a user clicks one of our short URLs http://shor.ty/4324, we guarantee that the final website http://www.blog.com/category/page/1 is safe for the user.

In order to achieve a secure service like this, we are going to call a web service from an Internet provider that has over 1 million blacklisted URLs in a table. So when we transform a short URL into the original one, we can check the blacklist table to see if the URL is secure or not and act upon it. The process is described in the following figure:

...