ROCKSDB 704: RocksDB Details

ROCKSDB 605 followed a write through the WAL and memtable, then followed flushed SST files through compaction. Under a light write load, flush and compaction keep up in the background.

Under sustained pressure, writes can arrive faster than the background work can drain them. This chapter looks at flow control, compaction picking, compaction parallelism, and Region-aware output files.

Flow Control

Write pressure appears in three forms: immutable memtables waiting to flush, L0 files waiting to compact, and pending compaction bytes waiting to move through the deeper levels.

If RocksDB stalls a write only after it reaches the engine, Raftstore and apply may already be waiting behind it. TiKV's storage scheduler therefore applies flow control before submitting the write to Raftstore:

client write
    |
    v
storage scheduler (flow control)
    |
    v
Raftstore -> apply -> RocksDB

With flow control enabled, TiKV takes over this backpressure role instead of relying on RocksDB's write stalls. For memtable and L0 pressure, it estimates a sustainable write rate and delays writes that exceed it. The delay is charged by bytes, so a larger write consumes more of the available rate.

Pending compaction bytes use a different response. Below a soft limit, TiKV accepts writes normally. Between the soft and hard limits, it increasingly rejects new writes with a busy response. At the hard limit, every new write is rejected until compaction catches up.

The Compaction Picker

Once compaction is needed, RocksDB must choose a set of SST files to rewrite. Its compaction picker first chooses a level under pressure and a seed file in that level.

The picker includes every file in the next level whose key range overlaps the seed. It can also add adjacent files from the input level, but only when that does not require more next-level files and stays within the compaction size limit. The result does more useful work without widening the expensive part of the compaction.

If any required file is already part of another compaction, RocksDB cannot run this selection. The picker looks for a different independent set instead.

Compaction Parallelism

RocksDB has a limited background-job budget shared by flush and compaction. Within that budget, independent compactions can run at the same time because they use different files and key ranges.

One large compaction can also be divided by key range into subcompactions. The subcompactions read and write separate parts of one logical compaction in parallel.

TiKV derives both the background-job and subcompaction limits from the CPU capacity available to the process. They are resource budgets, not one fixed concurrency setting.

Region-Aware Output Files

Compaction creates new SST files as it rewrites its input. RocksDB normally cuts those files near a target size. TiKV adds a compaction guard that can cut at a Region boundary instead.

The guard waits until the current output file has reached a minimum size before using a boundary. That keeps small Regions from producing many tiny SST files, while making output files approximately align with Regions.

That alignment helps later Region-level work. When an entire SST belongs to a destroyed Region, RAFTSTORE 703 can remove the file directly instead of deleting its keys one by one.

ROCKSDB 705 continues with Titan, which changes the LSM path for large values.