Ceph's RBD block service is heavily utilized by cloud platforms like OpenStack and other virtualization solutions. Effective caching in memory on the client side can significantly improve the performance that users experience:
[client.yourclientname]
rbd_cache = true
rbd_cache_size = 67108864
rbd_cache_max_dirty = 50331648
rbd_cache_target_dirty = 33554432
These values of 64MB, 48MB, and 32MB are double the default numerical values as of Luminous.
Settings must be selected so that
rbd_cache_size > rbd_cache_max_dirty > rbd_cache_target_dirty.
rbd_cache_size > rbd_cache_max_dirty > rbd_cache_target_dirty.
Some references suggest larger values, but consider that these cache buffers are allocated per attached volume. Consider a dense hypervisor node hosting 100 smaller VM instances, each mapping an RBD volume for its boot drive. With these values, the hypervisor node will dedicate more than 6 GB of RAM just for caching...