RAFTSTORE 703: Region Worker

The Region worker is a Raftstore component that runs background jobs for Regions. It has three basic responsibilities:

• hand snapshot-generation work to the snapshot generator;
• apply a received snapshot; and
• remove data for a destroyed Region.

Snapshot Generation

The Region worker forwards snapshot-generation work to the snapshot-generator worker. Snapshot generation was covered in RAFT 603.

Destroying a Region

When a replica is removed, its Region stops owning its key range on that TiKV. Its physical data in RocksDB still needs to disappear. TiKV does not necessarily delete it immediately: an in-flight read may still hold a RocksDB snapshot over that range.

The Region worker records the obsolete range with RocksDB's current sequence number. It later compares that number with the oldest sequence number held by an active RocksDB snapshot:

replica is removed
        |
        v
record its range and current sequence number
        |
        v
wait until no older read snapshot remains
        |
        v
remove the physical data

Once the oldest active snapshot is newer than the recorded sequence number, no earlier read can still depend on the removed data. The worker can clean the range without breaking that read's view.

Some SST files lie wholly inside the obsolete range. TiKV can delete those files directly, which avoids scanning their keys.

For an SST file that crosses a Region boundary, TiKV deletes the obsolete range by keys.

Applying a Snapshot

Applying a snapshot replaces a Region's local contents with its state at a particular Raft log position. This requires more than loading the snapshot's SST files: TiKV must also remove stale data and prevent unfinished cleanup from deleting the new data.

Before applying the snapshot, Raftstore checks whether its key range conflicts with another Region or a snapshot already being accepted. Conflicts block the ordinary apply path. Once accepted, the snapshot's range is reserved so that another Region cannot claim the same space during initialization.

But reserving the range does not mean the underlying storage is empty. A peer that previously owned those keys may have been destroyed while its physical data is still waiting for background deletion.

The Region worker therefore handles overlapping cleanup before ingesting the snapshot:

1. Take overlapping deletion tasks out of the pending queue.
2. Clear the stale data.
3. Ingest the snapshot's SST files.

This order solves two problems. First, ingestion does not remove old keys that are absent from the snapshot. If local storage contains {a, b} and the snapshot contains {a, c}, loading the SST files alone can leave the obsolete b behind. Second, an old deletion task must not run after ingestion and erase the newly restored data.

Cleanup must also preserve existing readers. An older RocksDB snapshot may still need to read the old data. TiKV therefore removes whole SST files only when the reader check says it is safe. Otherwise, it uses range or key deletions at a newer sequence number, so older readers retain their view. Meanwhile, the peer does not apply normal Raft writes during snapshot application, and the Region worker serializes cleanup and ingestion.

Finally, ingestion may need to wait for RocksDB. If the relevant column families have too many L0 files, adding more SST files could push the engine toward a write stall. The Region worker queues the snapshot application and retries later, allowing compaction to reduce the pressure first. The flow-control section in ROCKSDB 704 explains this storage pressure in more detail.