Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

RAFT 603: Raft Snapshot

As discussed in TiKV 201, a Region is the smallest unit of data movement between TiKV stores. In RAFT 602, we saw that moving a replica requires adding a peer on one store and removing it from another. This chapter looks at how the Region's data reaches the new peer.

The general problem has two parts:

  1. Bulk copy: copy the existing state to the new peer.
  2. Incremental catch-up: apply the changes that happen during or after the copy.

The second part is exactly what Raft's log provides. The leader sends the new peer the log entries after the copied state, and the peer applies them in order.

Why the Raft Log Is Not Enough

One possible way to copy a Region would be to replay its entire Raft log from the first entry to the latest one. In practice, this is too slow. More importantly, TiKV does not keep Raft log entries forever.

Raft log entries are stored in the Raft Engine. Each peer applies committed entries to its own local KV Engine, which is RocksDB. The log establishes the order of changes and drives the state transitions. Once the changes have been applied locally, the old log entries are no longer needed by that state machine. They are still needed by replicas that have fallen behind, but after the replicas have caught up, keeping every old entry would only consume disk space.

TiKV therefore periodically truncates old Raft log entries. The decision to truncate is coordinated through Raft, while the actual deletion is performed in the local Raft Engine after the operation is applied.

Now consider a follower whose log is behind the truncation point:

leader:   old entries removed | 81 | 82 | 83 | ...
follower: applied through 25

The follower needs entries 26 through 80, but the leader no longer has them. Incremental log replication cannot fill this gap. The leader must send a Raft snapshot.

A Snapshot Represents a State

Think of RocksDB as a state machine. Raft log entries are the ordered events that move the state machine forward. If replicas apply the same entries in the same order, they reach the same state.

Each Raft log position therefore corresponds to a particular state:

log index 80  ->  Region state after applying entries through 80
log index 81  ->  Region state after applying entry 81

A Raft snapshot captures the Region state at one Raft log position. A snapshot at index 80 contains all the Region's key-value data after entry 80 has been applied. Once a follower restores that snapshot, it can continue with ordinary log replication from entry 81.

follower before snapshot:  applied through 25
                                     |
                         install snapshot at 80
                                     v
follower after snapshot:   applied through 80
                            catch up from entry 81

This is the meaning of a Raft snapshot: it replaces a missing prefix of the log with the complete state produced by that prefix.

It is different from the RocksDB snapshot used for reads in ROCKSDB 403 and RAFTSTORE 502. A RocksDB snapshot provides a local read position. A Raft snapshot transfers Region state from one replica to another.

Generating a Snapshot

To construct a snapshot, TiKV must read all key-value data belonging to the Region and package it into data files. The data must correspond to a specific Raft index.

The local KV Engine records the peer's applied Raft index together with its data. When TiKV obtains a RocksDB snapshot for the Region, that applied index identifies the Raft position represented by the data. TiKV scans the Region through that RocksDB snapshot and writes the result into SST files.

An SST file is a sorted data file that RocksDB can add directly to its storage. The details of how RocksDB writes and organizes SST files are covered in ROCKSDB 605: RocksDB LSM Tree. For this chapter, it is enough to think of the files as a bulk representation of the Region's key-value state.

A snapshot contains both data and metadata:

snapshot
  Region metadata
  snapshot index and term
  Region key-value data in SST files

The snapshot index tells the receiver how much Raft history the data already includes. The snapshot term is the term of the log entry at that index. Together, the index and term identify the Raft position represented by the snapshot. The Region metadata identifies the key range and peer configuration that the snapshot belongs to.

Snapshot generation can be large and expensive. TiKV performs it in a separate snapshot worker pool so that a long Region scan does not occupy the normal Raftstore processing path.

How raft-rs Requests a Snapshot

When a follower rejects an AppendEntries request because its log is too far behind, the leader learns that ordinary replication cannot repair the gap. raft-rs then decides that the follower needs a snapshot.

raft-rs does not build the snapshot itself. Raftstore provides it with a storage adapter called PeerStorage. When raft-rs asks PeerStorage for a snapshot, PeerStorage starts or checks the background generation job.

Snapshot generation may not be finished when the request arrives. In that case, PeerStorage reports that the snapshot is temporarily unavailable. Raftstore tries again during a later Raft heartbeat, normally every two seconds, and can also check immediately when the background job finishes.

The interaction can be summarized as:

raft-rs decides a snapshot is needed
              |
              v
raft-rs requests one from PeerStorage
              |
              v
PeerStorage starts a snapshot worker job
              |
              v
snapshot worker scans RocksDB and builds SST files
              |
              v
PeerStorage returns the ready snapshot

The snapshot data files can be much larger than ordinary Raft messages. TiKV sends a snapshot through its dedicated Snapshot gRPC streaming API, rather than the normal Raft or BatchRaft APIs. This is a separate API endpoint and stream on the same TiKV gRPC server, not a separate port. The first snapshot chunk carries the Raft message and snapshot metadata; later chunks carry the SST data. This keeps bulk file transfer off the normal Raft message streams.

Receiving a Snapshot

The receiving peer must still clean up old data, install the SST files, and restore the snapshot's local state before ordinary log replication can continue. This is local storage work rather than a Raft protocol decision. We will look at it in more detail in RAFTSTORE 703: Region Worker.