Filesystem and data storage

RethinkDB supports major used filesystems such as NTFS, EXT and so on. RethinkDB also supports direct I/O filesystems for efficiency and performance, but it is not enabled by default.

About direct I/O

File is stored on disk and when it's been requested by any program, the operating system first puts it into the main memory for faster reads. The operating system can read directly from disk too, but that would slow down the response time because of heavy-cost I/O operation. Hence, the operating system first puts it into the main memory for operation. This is called buffer cache.

Databases generally manage data caching at the application and they do not need the operating system to cache it for them. In such cases, the process of buffering at two places (main memory and application cache) becomes an overhead since data is first moved to the main memory and then the application cache.

This double buffering of data results in more CPU consumption and load on the memory too.

Direct I/O is a filesystem for those applications that want to avoid the buffering at the main memory and directly read files from disk. When direct I/O is used, data is transferred directly to the application buffer instead of the memory buffer, as shown in the following diagram:

Direct I/O can be used in two ways:

• Mounting the filesystem using direct I/O (options vary from OS to OS)
• Opening the file using the O_DIRECT option specified in the open() system call

Direct I/O provides great efficiency and performance by reducing CPU consumption and the overhead of managing two buffers.

Data storage

RethinkDB uses a custom-built storage engine inspired by the Binary tree file system by Oracle (BTRFS). There is not enough information available on the RethinkDB custom filesystem right now, but we have found the following promises by it:

• Fully concurrent garbage compactor
• Low CPU overhead
• Efficient multi-core operation
• SSD optimization
• Power failure recovery
• Data consistency in case of failure
• MVCC supports

Due to these features, RethinkDB can handle large amounts of data in very little memory storage.