The CAP Theorem in practice: The consistency vs. availability trade-off in distributed databases

When you choose a database you are making a design decision. One of the best frameworks for understanding what this means in practice is the CAP Theorem.

What is the CAP Theorem?

The CAP Theorem, developed by computer scientist Eric Brewer in the late nineties, states that databases can only ever fulfil two out of three elements:

• Consistency - that reads are always up to date, which means any client making a request to the database will get the same view of data.
• Availability - database requests always receive a response (when valid).
• Partition tolerance - that a network fault doesn’t prevent messaging between nodes.

In the context of distributed (NoSQL) databases, this means there is always going to be a trade-off between consistency and availability. This is because distributed systems are always necessarily partition tolerant (ie. it simply wouldn’t be a distributed database if it wasn’t partition tolerant.)

Read next: Different types of NoSQL databases and when to use them

How do you use the CAP Theorem when making database decisions?

Although the CAP Theorem can feel quite abstract, it has practical, real-world consequences. From both a technical and business perspective the trade-offs will lead you to some very important questions. There are no right answers. Ultimately it will be all about the context in which your database is operating, the needs of the business, and the expectations and needs of users.

You will have to consider things like:

• Is it important to avoid throwing up errors in the client?
• Or are we willing to sacrifice the visible user experience to ensure consistency?
• Is consistency an actual important part of the user’s experience
• Or can we actually do what we want with a relational database and avoid the need for partition tolerance altogether?

As you can see, these are ultimately user experience questions. To properly understand those, you need to be sensitive to the overall goals of the project, and, as said above, the context in which your database solution is operating. (Eg. Is it powering an internal analytics dashboard? Or is it supporting a widely used external-facing website or application?)

And, as the final bullet point highlights, it’s always worth considering whether the consistency v availability trade-off should matter at all. Avoid the temptation to think a complex database solution will always be better when a simple, more traditional solution will do the job. Of course, it’s important to note that systems that aren’t partition tolerant are a single point of failure in a system. That introduces the potential for unreliability.

Prioritizing consistency in a distributed database

It’s possible to get into a lot of technical detail when talking about consistency and availability, but at a really fundamental level the principle is straightforward: you need consistency (or what is called a CP database) if the data in the database must always be up to date and aligned, even in the instance of a network failure (eg. the partitioned nodes are unable to communicate with one another for whatever reason).

Particular use cases where you would prioritize consistency is when you need multiple clients to have the same view of the data. For example, where you’re dealing with financial information, personal information, using a database that gives you consistency and confidence that data you are looking at is up to date in a situation where the network is unreliable or fails.

Examples of CP databases

MongoDB

• Learning MongoDB 4 [Video]
• MongoDB 4 Quick Start Guide
• MongoDB, Express, Angular, and Node.js Fundamentals

Redis

• Build Complex Express Sites with Redis and Socket.io [Video]
• Learning Redis

HBase

• Learn by Example : HBase - The Hadoop Database [Video]
• HBase Design Patterns

Prioritizing availability in a distributed database

Availability is essential when data accumulation is a priority. Think here of things like behavioral data or user preferences. In scenarios like these, you will want to capture as much information as possible about what a user or customer is doing, but it isn’t critical that the database is constantly up to date. It simply just needs to be accessible and available even when network connections aren’t working.

The growing demand for offline application use is also one reason why you might use a NoSQL database that prioritizes availability over consistency.

Examples of AP databases

Cassandra

• Learn Apache Cassandra in Just 2 Hours [Video]
• Mastering Apache Cassandra 3.x - Third Edition

DynamoDB

• Managed NoSQL Database In The Cloud - Amazon AWS DynamoDB [Video]
• Hands-On Amazon DynamoDB for Developers [Video]

Limitations and criticisms of CAP Theorem

It’s worth noting that the CAP Theorem can pose problems. As with most things, in truth, things are a little more complicated. Even Eric Brewer is circumspect about the theorem, especially as what we expect from distributed databases.

Back in 2012, twelve years after he first put his theorem into the world, he wrote that:

“Although designers still need to choose between consistency and availability when partitions are present, there is an incredible range of flexibility for handling partitions and recovering from them. The modern CAP goal should be to maximize combinations of consistency and availability that make sense for the specific application. Such an approach incorporates plans for operation during a partition and for recovery afterward, thus helping designers think about CAP beyond its historically perceived limitations.”

So, this means we must think about the trade-off between consistency and availability as a balancing act, rather than a binary design decision.

Elsewhere, there have been more robust criticisms of CAP Theorem. Software engineer Martin Kleppmann, for example, pleaded Please stop calling databases CP or AP in 2015. In a blog post he argues that CAP Theorem only works if you adhere to specific definitions of consistency, availability, and partition tolerance.

“If your use of words matches the precise definitions of the proof, then the CAP theorem applies to you," he writes. “But if you’re using some other notion of consistency or availability, you can’t expect the CAP theorem to still apply.”

The consequences of this are much like those described in Brewer’s piece from 2012. You need to take a nuanced approach to database trade-offs in which you think them through on your own terms and up against your own needs.

The PACELC Theorem

One of the developments of this line of argument is an extension to the CAP Theorem: the PACELC Theorem. This moves beyond thinking about consistency and availability and instead places an emphasis on the trade-off between consistency and latency.

The PACELC Theorem builds on the CAP Theorem (the ‘PAC’) and adds an else (the ‘E’). What this means is that while you need to choose between availability and consistency if communication between partitions has failed in a distributed system, even if things are running properly and there are no network issues, there is still going to be a trade-off between consistency and latency (the ‘LC’).

Conclusion: Learn to align context with technical specs

Although the CAP Theorem might seem somewhat outdated, it is valuable in providing a way to think about database architecture design. It not only forces engineers and architects to ask questions about what they want from the technologies they use, but it also forces them to think carefully about the requirements of a given project. What are the business goals? What are user expectations?

The PACELC Theorem builds on CAP in an effective way. However, the most important thing about these frameworks is how they help you to think about your problems.

Of course the CAP Theorem has limitations. Because it abstracts a problem it is necessarily going to lack nuance. There are going to be things it simplifies. It’s important, as Kleppmann reminds us - to be mindful of these nuances. But at the same time, we shouldn’t let an obsession with nuance and detail allow us to miss the bigger picture.