This section of the chapter will cover the scenario where a number of OSDs may have gone offline in a short period of time, leaving some objects with no valid replica copies. It's important to note that there is a difference between an object that has no remaining copies and an object that has a remaining copy, but it is known that another copy has had more recent writes. The latter is normally seen when running the cluster with min_size set to 1.
To demonstrate how to recover an object that has an out-of-date copy of data, let's perform a series of steps to break the cluster:
- First, let's set min_size to 1; hopefully by the end of this example, you will see why you don't ever want to do this in real life:
sudo ceph osd pool set rbd min_size 1
- Create a test object that we will make later make Ceph believe is lost:
sudo rados -p rbd put lost_object logo.png...
