Once you have installed and configured your Ceph storage cluster, the next task is performing storage provisioning. Storage provisioning is the process of assigning storage space or capacity to physical or virtual servers, either in the form of block, file, or object storage. A typical computer system or server comes with a limited local storage capacity that might not be enough for your data storage needs.

Storage solutions such as Ceph provide virtually unlimited storage capacity to these servers, making them capable of storing all your data and making sure that you do not run out of space. Using a dedicated storage system instead of local storage gives you the much-needed flexibility in terms of scalability, reliability, and performance.

Ceph can provision storage capacity in a unified way, which includes block, filesystem, and object storage. The following diagram...

Any regular Linux host (RHEL or Debian-based) can act as a Ceph client. The client interacts with the Ceph storage cluster over the network to store or retrieve user data. Ceph RBD support has been added to the Linux mainline kernel, starting with 2.6.34 and later versions.

As we did earlier, we will set up a Ceph client machine using Vagrant and VirtualBox. We will use the same Vagrantfile that we cloned in the last chapter. Vagrant will then launch a CentOS 7.3 virtual machine that we will configure as a Ceph client:

1. From the directory where we cloned the Ceph-Cookbook-Second-Edition GitHub repository, launch the client virtual machine using Vagrant:

         $ vagrant status...

Up to now, we have configured Ceph client, and now we will demonstrate creating a Ceph Block Device from the client-node1 machine.

1. Create a RADOS Block Device named rbd1 of size 10240 MB:

        # rbd create rbd1 --size 10240 --name client.rbd

2. There are multiple options that you can use to list RBD images:

        ## The default pool to store block device images is "rbd",
           you can also specify the pool name with the rbd 
           command using -p option:
        # rbd ls --name client.rbd
        # rbd ls -p rbd --name client.rbd
        # rbd list --name client.rbd

3. Check the details of the RBD image:

        # rbd --image rbd1 info --name client.rbd

Now that we have created a block device on a Ceph cluster, in order to use this block device, we need to map it to the client machine. To do this, execute the following commands from the client-node1 machine.

1. Map the block device to the client-node1:

        # rbd map --image rbd1 --name client.rbd

2. With Ceph Jewel the new default format for RBD images is 2 and Ceph Jewel default configuration includes the following default Ceph Block Device features:
   • layering: layering support
   • exclusive-lock: exclusive locking support
   • object-map: object map support (requires exclusive-lock)
   • deep-flatten: snapshot flatten...

Ceph supports thin provisioned block devices, which means that the physical storage space will not get occupied until you begin storing data on the block device. The Ceph Block Device is very flexible; you can increase or decrease the size of an RBD on the fly from the Ceph storage end. However, the underlying filesystem should support resizing. Advance filesystems such as XFS, Btrfs, EXT, ZFS, and others support filesystem resizing to a certain extent. Please follow filesystem specific documentation to know more about resizing.

XFS does not currently support shrinking, Btrfs, and ext4 do support shrinking but should be done with caution!

To increase or decrease the Ceph RBD image size, use...

Ceph extends full support to snapshots, which are point-in-time, read-only copies of an RBD image. You can preserve the state of a Ceph RBD image by creating snapshots and restoring the snapshot to get the original data.

If you take a snapshot of an RBD image while I/O is in progress to the image the snapshot may be inconsistent. If this occurs you will be required to clone the snapshot to a new image for it to be mountable. When taking snapshots it is recommended to cease I/O from the application to the image before taking the snapshot. This can be done by customizing the application to issue a freeze before a snapshot or can manually be done using the fsfreeze command (man page for fsfreeze details this command further).

Ceph supports a very nice feature for creating Copy-On-Write (COW) clones from RBD snapshots. This is also known as snapshot layering in Ceph. Layering allows clients to create multiple instant clones of Ceph RBD. This feature is extremely useful for cloud and virtualization platforms such as OpenStack, CloudStack, Qemu/KVM, and so on. These platforms usually protect Ceph RBD images containing an OS/VM image in the form of a snapshot. Later, this snapshot is cloned multiple times to spawn new virtual machines / instances. Snapshots are read-only, but COW clones are fully writable; this feature of Ceph provides a greater level of flexibility and is extremely useful in cloud platforms. In the later chapters, we will discover more on COW clones for spawning OpenStack instances:

Every cloned image (child image) stores references of its parent snapshot to read...

RBD mirroring is an asynchronous replication of RBD images between multiple Ceph clusters. RBD mirroring validates a point-in-time consistent replica of any change to an RBD image, including snapshots, clones, read and write IOPS and block device resizing. RBD mirroring can run in an active+active setup or an active+passive setup. RBD mirroring utilizes the RBD journaling and exclusive lock features which enables the RBD image to record all changes to the image in order of which they occur. These features validate that a crash-consistent copy of the remote image is available locally. Before mirroring can be enabled on a Ceph cluster the journaling feature must be enabled on the RBD image.

The daemon responsible for ensuring point-in-time consistency from one Ceph cluster to the other is the rbd-mirror daemon. Depending on your...

RBD mirroring is configured by enabling it on a pool basis in a primary and secondary Ceph cluster. There are two modes that can be configured with RBD mirroring depending on what level of data you choose to mirror. Note that the enabled RBD mirroring configuration must be the same per pool on primary and secondary clusters.

• Pool mode: Any image in a pool with journaling enabled is mirrored to the secondary cluster.
• Image mode: Only specifically chosen images with mirroring enabled will be mirrored to the secondary cluster. This requires the image to have mirroring enabled for it.

Before any data can be mirrored from the Ceph cluster to the backup...

Image mirroring can be used when you choose to only want to mirror a specific subset of images and not an entire pool. The next recipe we will enable mirroring on a single image in the data pool and not mirror the other two images in the pool. This recipe requires you to have completed step 1 - step 9 in Disaster recovery replication using RBD mirroring recipe and have rbd-mirror running on backup site:

1. Create three images in the ceph cluster as we did in the previous recipe:

       # rbd create image-1 --size 1024 --pool data 
         --image-feature exclusive-lock,journaling

2. Enable image mirroring on the data pool on the ceph and backup clusters:

        # rbd mirror pool enable...

Two-way mirroring requires a rbd-mirror daemon running on both clusters, the primary and the secondary. With two way mirroring it is possible to mirror data or images from the primary site to a secondary site, and the secondary site can mirror data or images back to the primary site.

We will not be demoing this configuration in this book at it is very similar to one-way configuration, but we will highlight the changes needed for two way replication at the pool level, these steps are covered in the one-way replication recipe.

1. Both clients must have the rbd-mirror installed and running:

        # yum install rbd-mirror
        # systemctl enable ceph-rbd-mirror.target
        #...

The following recipes will show how to fail over to the mirrored data on the backup cluster after the primary cluster ceph has encountered a disaster and how to failback once the ceph cluster has recovered. There are two methods for failover when dealing with a disaster:

• Orderly: Failover after an orderly shutdown. This would be a proper shutdown of the cluster and demotion and promotion of the image.
• Non-orderly: Failover after a non-orderly shutdown. This would be a complete loss of the primary cluster. In this case, the failback would require a resynchronizing of the image.

1. How to properly failover after an orderly shutdown:
   • Stop all client's that are writing to the primary...