RAFTSTORE 901: Peer Lifecycle and Crash Recovery

Prerequisites

This chapter builds on the following chapters:

• RAFT 602: Replica Movement introduced creating and removing replicas.
• RAFT 603: Raft Snapshot and RAFTSTORE 703: Region Worker followed snapshot transfer and application.
• RAFTSTORE 604: Region Split and RAFTSTORE 801: Region Merge covered the normal split and merge paths.

This chapter looks at what those paths leave behind when they are interrupted. A Raftstore PeerFsm is in memory, so a restart discards it. TiKV instead records enough durable peer state to rebuild the peer or resume the unfinished operation.

Durable Peer Records

One peer has persistent state in two engines:

The Raft engine recovers the Raft protocol. The KV-engine records recover the state machine and which Region belongs on this store.

When an applied Raft entry changes user data, TiKV writes the data changes and the new RaftApplyState.applied_index in one KV-engine batch. After a restart, the index and the visible data therefore agree: TiKV neither reapplies a completed entry nor skips one whose data was missing.

RegionLocalState is the durable lifecycle marker:

Tombstone retains the removed Region's peer identity and epoch. An old Raft message therefore cannot create that removed peer again.

Restart Begins From These Records

During startup, Raftstore scans RegionLocalState and reconstructs the current in-memory peers from it:

• A Normal peer becomes a new PeerFsm.
• An Applying peer recovers the Raft state required by its snapshot, then reschedules snapshot application.
• A Tombstone peer is not recreated; TiKV clears its stale Raft metadata, while its physical data follows the cleanup path from RAFTSTORE 703.
• A Merging peer becomes a PeerFsm with its saved merge state, so it can resume the merge decisions from RAFTSTORE 801.

The durable records are thus the source of truth. The in-memory structures are rebuilt from them.

Peer Creation

Consider the replica movement from RAFT 602. Region 10 initially has three peers:

Store 1: Peer 101 (leader)
Store 2: Peer 102
Store 3: Peer 103

PD decides to add Peer 104 on Store 4. When each existing peer applies the configuration change, it adds Peer 104 to the peer list in RegionLocalState. This updates the Region metadata, but Peer 104 has not yet been physically created on Store 4.

The actual creation of Peer 104 is triggered by a Raft message addressed to it, but not just any Raft message. Only certain messages are considered initial messages and can create a new peer.

Initial messages include a special heartbeat sent by the leader to an unknown follower, as well as vote requests sent by candidates. These messages carry the metadata required for peer creation, including the Region's key range and epoch.

Other Raft messages are considered non-initial and are ignored if they are addressed to a peer that does not yet exist.

When Store 4 receives an initial message for Peer 104, it creates Peer 104 in memory. At this point, Peer 104 is uninitialized: Store 4 knows that the peer should exist and which Region it belongs to, but it has not received the Region's key-value data.

This creation is still entirely in memory. Store 4 has not persisted any state for Peer 104. If TiKV restarts now, the in-memory peer disappears, and a later initial message must trigger its creation again. Peer 104 must wait for the leader to send a Raft snapshot before it can serve.

Peer Snapshot Application

RAFTSTORE 703: Region Worker covered the snapshot application process in general. Here, we focus on the persistence points that allow the process to survive a restart.

Before the peer can become ready to serve, the snapshot must update state in both the KV engine and the Raft engine. TiKV cannot write to the two engines atomically, so it needs a durable handoff between them.

After raft-rs accepts an incoming snapshot, the Peer FSM obtains it through Raft Ready and submits a write task to the store writer. For an ordinary data replica, the store writer first writes three records in the same KV-engine batch:

• RegionLocalState contains the complete Region metadata from the snapshot and sets the peer state to Applying.
• RaftApplyState advances the applied index and log-truncation boundary to the snapshot position.
• snapshot_raft_state_key stores a recovery copy of RaftLocalState, with its commit index and last index set to the snapshot index.

The store writer then writes the Raft state to the Raft engine. After both writes complete, it notifies the Peer FSM, which sends the snapshot application task to the Region worker.

Peer FSM: submit write task to store writer
                    |
                    v
Store writer: write Applying state, apply state,
              and snapshot Raft-state copy to KV engine
                    |
                    v
Store writer: write snapshot Raft state to Raft engine
                    |
                    v
Peer FSM: receive persistence-completion notification;
          submit snapshot application task to Region worker
                    |
                    v
Region worker: finish data cleanup and SST ingestion
                    |
                    v
Region worker: write Normal state and remove
               the recovery copy in KV engine

The first KV-engine write is the persistence point for snapshot application. If TiKV crashes before the Raft-engine write, startup finds the Applying state and the saved RaftLocalState. It can repair the Raft state if necessary and resume snapshot application.

After applying the snapshot data, the Region worker atomically changes RegionLocalState to Normal and removes snapshot_raft_state_key in the KV engine. It then notifies the Peer FSM, which completes snapshot processing. Peer 104 is now initialized and can operate normally as a replica.

Peer Deletion

1. Configuration-Change Deletion

When a peer applies a configuration change that removes itself, the Apply FSM persists a tombstone RegionLocalState in the KV engine, then notifies the Peer FSM through ExecResult::ChangePeer. The Peer FSM updates its in-memory metadata and starts destruction: it deletes the apply state while preserving the tombstone, then clears RaftLocalState and the Raft logs.

Persisting the tombstone is the durable deletion point. If TiKV crashes before cleanup finishes, startup recognizes the tombstone and continues cleanup.

2. Orphaned Peer GC

A peer may never apply the entry that removes it, for example, because it was isolated or had not finished initialization. It may discover its removal through normal Raft traffic:

• It contacts other replicas and finds that it is no longer in the Region's peer list.
• It receives a Raft message addressed to a newer peer ID for the same Region on the same store. The local peer has been superseded, so TiKV initiates its destruction to make way for the newer peer.

Losing contact with the leader does not by itself mean that a peer has been removed. It may still be a valid member waiting to reconnect. However, if all the replicas it knows have been replaced, it may have no way to learn of its removal through normal Raft traffic. This is why, after a prolonged absence of a leader, TiKV also checks with PD.

Once removal is confirmed, the Peer FSM runs the destruction routine, persisting a tombstone before clearing the remaining state.

3. What Is Persisted After Deletion?

The tombstone retains the Region information available locally:

TiKV uses the epoch to reject outdated messages. When the tombstone still includes the local peer, its ID provides an additional check: messages addressed to the same or a smaller peer ID are rejected.

This is essential for an uninitialized peer, which has no useful epoch. Peer IDs increase across generations, so its saved identity alone can prevent delayed messages from recreating the deleted peer.

Race Between Snapshot Application and Peer Deletion

The Peer FSM handles snapshot persistence and peer destruction serially. The result depends on which one begins first:

• Peer destruction begins first. The Peer FSM marks the peer for removal. Any pending or subsequently received snapshot will not proceed.

• Snapshot persistence begins first. When a peer-destroy message arrives, the Peer FSM delays destruction and sets a flag to destroy the peer after snapshot application ends.

  Once scheduled, the Region worker's apply task runs concurrently with the Peer FSM. There is a time window after scheduling but before the task starts running. Canceling within this window allows peer destruction to proceed immediately.

  If the task has started running, TiKV still requests cancellation. Destruction must then wait for the task to stop safely or finish normally.

The wait is necessary because the Region worker writes RegionLocalState to Normal after applying the snapshot. Deletion cannot persist a tombstone while the task may still overwrite it.

Peer Split

At the peer-lifecycle level, a Region split creates a new peer alongside the existing one: R0 becomes R0 and R1. When the split entry is applied, the Apply FSM writes the following records to the KV engine in one batch:

• Updated RegionLocalState for R0 and new RegionLocalState for R1, both in Normal state.
• Updated RaftApplyState for R0, recording that the split entry has been applied.
• Initial RaftApplyState for R1.

Once that batch is persisted, the Apply FSM sends the split result to the Peer FSM, which creates R1's in-memory peer.

R1's key-value data is already present in the shared KV engine. The split changes which Region owns each key range. R0 and R1 then continue as separate Region peers.

Peer Split Crash Recovery

The KV-engine batch is the persistence point for the split. The crash point determines what startup sees:

The router, range index, and read delegates are rebuilt from the persisted state. A crash after the KV batch but before those in-memory updates does not lose R1.

Race Between Peer Split and Peer Creation

Suppose a store is being added to R0 as Peer 3. While Peer 3 is still waiting for its snapshot, R0 splits into R0 and R1. R1's peer on this store is Peer 1003.

There are now two ways to initialize Peer 1003:

1. Through the parent. Peer 3 receives a pre-split snapshot, then applies the split entry and creates Peer 1003. The child's data comes from the parent's snapshot.
2. On its own. A Raft message for Peer 1003 arrives first, creating an uninitialized peer that waits for its own snapshot.

Creating the empty child first does not decide which path wins. TiKV must coordinate both snapshot application and creation of the child peer.

The two snapshots cannot install overlapping data concurrently. The parent's pre-split snapshot includes the child's range, so it overlaps with the child's own snapshot. TiKV checks both existing Region ranges and snapshots already accepted for processing. pending_snapshot_regions records the latter, preventing both snapshots from passing the overlap checks together.

An empty child can be replaced by split, but an initialized child cannot be replaced by an empty one. If split reaches child creation first, Raft-message creation gives up. If Raft-message creation gets there first, split can replace its uninitialized peer:

Split first:   create initialized Peer 1003
Message first: create empty Peer 1003 -> split replaces it

pending_create_peers records the expected peer ID and whether split has claimed its creation. Together with the StoreMeta lock, it prevents a delayed Raft-message creation from overwriting the initialized child.

The replaced empty peer may still be handling a snapshot message. Before accepting it, the peer checks that its Region metadata still matches StoreMeta. After replacement, that check fails, so the old peer rejects the snapshot. This checks whether the peer handling the message is still current, rather than comparing the snapshot against metadata on disk.

Finally, split can replace only the empty peer with the ID it expects. If PD has already removed Peer 1003 and added another peer with a larger ID on the same store, the delayed split must leave that newer peer alone.

Race Between Peer Split and Peer Deletion

Suppose a Raft message has already created an empty Peer 1003. Peer 3 is about to apply the split and replace that empty peer with an initialized child. Meanwhile, TiKV learns that PD has removed Peer 1003 and starts destroying it.

The same empty peer is now being replaced and deleted concurrently. TiKV must prevent two problems: split recreating a peer after deletion, and deletion erasing the state that split has just created.

pending_create_peers coordinates these operations through a shared lock:

• Destruction gets there first. It holds the lock until Peer 1003's tombstone is persisted. When split checks the child afterward, it finds the tombstone and skips creating it. The deleted peer stays deleted.
• Split gets there first. It records that it will replace the empty Peer 1003. Destruction sees this and skips persistent cleanup, leaving split free to create the initialized child. That child can later discover that it has been removed and destroy itself through the peer garbage-collection mechanism described earlier.

Merge Flow

RAFTSTORE 801 covered the normal merge flow. A merge transfers the source Region's key range to the target Region, then removes the source.

The source first prepares to be absorbed. When it applies PrepareMerge, it persists a Merging RegionLocalState. This records the target Region, min_index, and the index of the PrepareMerge entry. These identify the source logs that may be needed to bring a lagging source peer up to the merge point. The source remains frozen while waiting for the merge to complete or roll back.

The target completes the transfer on each store. Before applying CommitMerge, the target waits for the source peer on the same store to catch up and stop its apply work. This ensures that the source's data is ready and its Apply FSM will no longer modify it.

The target then persists both Region-state changes in one KV-engine batch:

target: Normal, with its range expanded to include the source
source: Tombstone, recording the merge target

This batch makes the transfer durable. The source peer can then be removed from memory.

A lagging merge can leave one source peer behind. That source must ask: does the local target still need me? Suppose the merge completes on two stores, but the third store falls behind. Through communication with the merged replicas, its source peer learns which target absorbed the source and the target's epoch at that merge point.

• The local target has not moved past that point. The source waits: the target may still need it when applying CommitMerge.
• The local target's epoch has advanced past that point. The merge target may, for example, have gone through another split and advanced beyond the recorded epoch. This shows that the target has moved on, so the leftover source can be destroyed without waiting for the original merge. TiKV checks this against the recorded merge target even if the local target no longer overlaps the source.

Race Between Merge and Snapshot

A lagging target peer may catch up through a snapshot instead of replaying CommitMerge. The question is whether that snapshot already includes the merge:

• Before the merge: The target may still need to execute CommitMerge, so the source must remain.
• After the merge: The snapshot already includes the source's data, so the source can be removed. TiKV identifies this case by checking that the snapshot's Region version is greater than the recorded target version.

For the same target peer, removing the source and committing to snapshot recovery must happen atomically. Otherwise, a crash could leave the source deleted but the target still recovering from its old log, where CommitMerge needs that source.

atomic_snap_regions is an in-memory map that tracks which source peers must be removed as part of a target's snapshot application, and whether those peers are ready. TiKV uses it to coordinate writing these records in one KV-engine batch:

source: Tombstone
target: Applying, with snapshot recovery metadata

The snapshot data is applied afterward. If TiKV crashes in between, the persisted Applying state tells startup to resume the snapshot.

A replacement target peer does not have the old target peer's pending CommitMerge to replay. TiKV recognizes the replacement by its larger peer ID. Once its snapshot is confirmed to be post-merge, the source can be destroyed before the target's snapshot metadata is persisted: there is no risk of that replacement restarting and executing the old peer's CommitMerge.

Merge History

A Region can receive several merges and later merge into another Region itself:

A merges into B -> B merges into C

Before B disappears, TiKV must ensure that its peers will not miss the earlier merge from A.

Before proposing PrepareMerge for B -> C, B's leader checks every peer's replication and reported commit progress. If the remaining log interval contains the earlier CommitMerge for A -> B, TiKV postpones the new merge. The same check blocks configuration changes, splits, and other administrative commands that change the Region's epoch or log structure.

Every peer must have replicated the earlier merge and reported it committed, but it need not have applied it yet. Raft's apply order provides the remaining guarantee:

Apply CommitMerge:  absorb A into B
        |
        v
Apply PrepareMerge: prepare B to merge into C

A slow peer can therefore finish these steps in order. It will not prepare to remove B before applying the earlier merge into B.

TiKV does not persist a complete chain of past merges, such as A -> B -> C. Its merge state records only the current source-to-target relationship. This works because the checks above ensure that every peer has the earlier merge committed in its log before the next merge begins. Raft's apply order then ensures that each peer processes those merges in sequence, without needing a separate record of the full merge history.

Across peer creation, deletion, snapshots, splits, and merges, the recurring question is: if execution stops here, what will the next message or a restart be allowed to do?

Tombstones prevent delayed messages from recreating deleted peers. Atomic metadata batches make splits and merge transitions recoverable. In-memory locks and flags keep concurrent operations from undoing each other's work. Together, these mechanisms let each store recover from its persisted state, even when it falls behind or stops halfway through a transition.