RocksDB's data files are usually generated in an appending way. File system may choose buffer the write until the dirty pages hit a threshold and write out all of those pages all together. This can create a burst of write I/O and cause the online I/Os to wait too long and cause long query latency. Rather, you can ask RocksDB to periodically hint OS to write out outstanding dirty pages by setting for SST files and options.wal_bytes_per_sync for WAL files. Underlying it calls sync_file_range() on Linux every time a file is appended for such size. The most recent pages are not included in the range sync.

Rate Limiter

You can control the total rate RocksDB writes to data files through options.rate_limiter, in order to reserve enough I/O bandwidth to online queries. See for details.

When appending a file, RocksDB has internal buffering of files before writing to the file system, unless an explicit fsync is needed. The max size of this buffer can be controlled by options.writable_file_max_buffer_size. Tuning this parameter is more critical in [[Direct IO] mode or to a file system without page cache. With non-direct I/O mode, enlarging this buffer only reduces number of write() system calls and is unlikely to change the I/O behavior, so unless this is what you want, it may be desirable to keep the default value 0 to save the memory.

File Deletion

While opening an SST file for reads, users can decide whether RocksDB will call fadvise with FADV_RANDOM, by setting options.advise_random_on_open = true (default). If the value is false, no fadvise will be called while opening a file. Setting the option to be true usually works well if the dominating queries are either Get() or iterating a very short range, because read-ahead is not helpful in these cases anyway. Otherwise, can usually improve performance to hint the file system to do underlying read-ahead.

Unfortunately, there isn't a good setting for mixed workload. There is an ongoing project to address this, by doing read-ahead for iterators inside RocksDB.

Compaction inputs

Compaction inputs are special. They are long sequential reads, so applying the same fadvise hint as user reads is usually not optimal. Also, usually, compaction input files are often going to be deleted soon, though there is no guarantee. RocksDB provides multiple ways to deal with that:

fadvise hint

Use a different file descriptor for compaction inputs

If options.new_table_reader_for_compaction_inputs = true, RocksDB will use different file descriptors for compaction inputs. This can avoid the mixture of fadvise setting for normal data files and compaction inputs. The limitation of the setting is that, RocksDB does not just create a new file descriptor, but read index, filter and other meta blocks again and store them in memory, so that it takes extra I/O and use more memory.

readahead for compaction inputs

You can do its own readahead following options.compaction_readahead_size if it is not 0. options.new_table_reader_for_compaction_inputs is automatically switched to true if the option is set. This setting can allow users to keep to NONE.

It is critical to set the option if Direct IO is on or the file system doesn't support readahead.

options.allow_mmap_reads and options.allow_mmap_writes make RocksDB mmap the whole data file while doing read or write, respectively. The benefit of the approach is to reduce the file system calls doing pread() and write(), and in many cases, reduce the memory copying too. can usually significantly improve performance if the DB is run on ramfs. They can be used on file systems backed by block device too. However, based on our previous experience, file systems aren't usually doing a perfect job maintaining this kind of memory mapping, and some times cause slow queries. In this case, we advise you try out this option only when necessary and with cautious.