List examples with redis-cli
Since Lists in Redis are linked lists, there are commands used to insert data into the head and tail of a List. The command LPUSH inserts data at the beginning of a List (left push), and the command RPUSH inserts data at the end of a List (right push):
The command LLEN returns the length of a List. The command LINDEX returns the element in a given index (indices are zero-based). Elements in a List are always accessed from left to right, which means that index 0 is the first element, index 1 is the second element, and so on. It is possible to use negative indices to access the tail of the List, in which -1 is the last element, -2 is penultimate element, and so on. LINDEX does not modify a List:
The command LRANGE returns an array with all elements from a given index range, including the elements in both the start and end indices. As we mentioned previously, indices are zero-based and can be positive or negative. See the following example:
The command LPOP removes and returns the first element of a List. The command RPOP removes and returns the last element of a List. Unlike LINDEX, both LPOP and RPOP modify the List:
Implementing a generic Queue System
The following implementation is going to use JavaScript prototypes, and it is going to be similar to a class-based solution seen in many programming languages.
Create a file called queue.js in the chapter 1 folder with the following code:
- Create a function called Queue, which receives a queue name and the Redis client object as parameters.
- Save queueName as a property.
- Save redisClient as a property.
- Set the property queueKey to the proper Redis key name, based on the function parameter.
- Set the property timeout to zero, which means that when List commands are executed, they will have no timeout.
We need to implement three methods to perform queue operations: size, push, and pop.
The first method we are going to create is size:
- Create the Queue method size, which expects a callback as an argument.
- Execute LLEN on the queue key name and pass the callback as an argument. This is necessary because the Redis client is asynchronous.
The implementation of the push method is as follows:
- Create the Queue method push that expects one argument. This argument can be anything that can be represented as a string.
- Execute LPUSH by passing the queue key name and the data argument.
As this is a generic queue system and Redis lists only store bytes, we assume that all of the data that is sent to the queue can be transformed into a JavaScript string. If you want to make it more generic, you can use JSON serialization and store the serialized string. The previous example used LPUSH because we were implementing a queue, and by definition, items are inserted at the front of the queue and removed from the end of the queue. A helpful way to remember this is FIFO (First In, First Out)—we went from left to right.
The implementation of the pop method is as follows:
- Create the Queue method pop, which expects a callback as an argument.
- Execute BRPOP, passing the queue key name, the queue timeout property, and the callback as arguments.
As we mentioned earlier, elements are inserted at the front of the queue and removed from the end of the queue, which is why BRPOP was used (if RPUSH was used, then BLPOP would be necessary).
The command BRPOP removes the last element of a Redis List. If the List is empty, it waits until there is something to remove. BRPOP is a blocking version of RPOP. However, RPOP is not ideal. If the List is empty, we would need to implement some kind of polling by ourselves to make sure that items are handled as soon as they are added to the queue. It is better to take advantage of BRPOP and not worry about empty lists.
A concrete producer/consumer implementation is shown next. Different log messages are pushed into the "logs" queue by the producer and then popped by the consumer in another terminal window.
The complete Queue code, saved as queue.js, is as follows:
- This is required to expose Queue to different modules. This explicit export is specific to Node.js, and it is necessary in order to run require("./queue").
Create a file called producer-worker.js in the chapter 1 folder, which is going to add log events to a queue named "logs", and save the following:
- Require the module queue, which we've already created and saved as queue.js.
- Create an instance of the function Queue defined in the queue.js file.
- Create a loop that runs five times.
- Push some logs into the logs queue.
- Print the number of logs created.
Execute the producer file to push logs into the queue:
Save the following code in a file called consumer-worker.js:
- Require the queue module (this is the queue.js file).
- Create a Queue instance named logs and pass the Redis client to it.
- Create the function logMessages.
- Retrieve an element from the queue instance using the pop method. If the List is empty, this function waits until a new element is added. The timeout is zero and it uses a blocking command, BRPOP, internally.
- Display a message retrieved from the queue.
- Display the queue size after popping a message from the queue.
- Call the function (recursively) to repeat the process over and over again. This function runs forever.
- Call logMessages to initialize the queue consumption.
This queue system is completed. Now run the file consumer-worker.js and watch the elements being popped in the same order in which they were added by producer-worker.js:
This file will run indefinitely. More messages can be added to the queue by executing producer-worker.js again in a different terminal, and the consumer will continue reading from the queue as soon as new items are added.
The example shown in this section is not reliable enough to deploy to production. If anything goes wrong with the callbacks that pop from the queue, items may be popped but not properly handled. There is no such thing as a retry or any way to track failures.
A good way of solving the reliability problem is to use an additional queue. Each element that is popped from the queue goes to this additional queue. You must remove the item from this extra queue only if everything has worked correctly. You can monitor this extra queue for stuck elements in order to retry them or create failure alerts. The command RPOPLPUSH is very suitable for this situation, because it does a RPOP in a queue, then does a LPUSH in a different queue, and finally returns the element, all in a single step—it is an atomic command.