Scaling out versus scaling up
So you know it's time to add more muscle to your cluster, but how do you know whether to scale up or out?
If you're not familiar with the difference, scaling up refers to converting existing infrastructure into better or more robust hardware (or instance types in cloud environments). This can mean adding storage capacity, increasing memory, moving to newer machines with more cores, and so on.
Scaling out simply means adding more machines that roughly match the specifications of the existing machines. Since Cassandra scales linearly with its peer-to-peer architecture, scaling out is often more desirable.
Note
In general, it is better to replace physical hardware components incrementally rather than all at one time. This is because in large systems, failures tend to occur after hardware ages to a certain point, which is statistically likely to happen simultaneously for some subset of your nodes.
For example, purchasing a large amount of drives from a single source...