Transferring data using lock-free queues

In the C++ threading for multi-threaded low latency applications section, we hinted that one possible application of having multiple threads is to set up a pipelined system. Here, one component thread performs part of the processing and forwards the results to the next stage of the pipeline for further processing. We will be using such a design in our electronic trading system, but there’ll be more on that later.

Communicating between threads and processes

There are a lot of options when it comes to transferring data between processes and/or threads. Inter-Process Communication (IPC), such as mutexes, semaphores, signals, memory-mapped files, and shared memory, can be used for these purposes. It also gets tricky when there is concurrent access to shared data and the important requirement is to avoid data corruption. Another important requirement is to make sure that the reader and writer have consistent views of the shared data...