Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

RAFT 601: Leader Transfer

Besides a leader change triggered passively by failure or an election timeout, Raft also supports an active leader transfer.

Why is it needed? First, for load balancing: a cluster may have hundreds of thousands of Regions, and their leaders should be distributed evenly. Second, during a rolling restart, PD can evict leaders before restarting a TiKV store to reduce the availability impact.

Leader transfer is initiated by PD. PD sends the current Raft leader a candidate set: the peers that are acceptable new leaders.

TiKV chooses one candidate from that set, usually the peer whose Raft log is the most caught up. A more current peer can complete the transfer with less waiting. Sometimes PD sends a set containing only one peer. In that case, PD has fixed the target rather than leaving TiKV a choice.

Raftstore Level

TiKV first performs a handshake. The leader sends a MsgTransferLeader message to the target follower. The follower performs checks, then replies with the same message type as an acknowledgement.

The follower rejects the request when, for example:

  • its term does not match the leader's term;
  • it is not a voting peer;
  • it has a pending snapshot;
  • its disk usage is unacceptable.

The acknowledgement contains the follower's applied index. The leader compares it with its own last index and requires the gap to stay below a configured threshold, 128 by default. This ensures that the target is reasonably close to being fully applied. Without an acknowledgement, the transfer cannot proceed.

There is a more complicated path for in-memory pessimistic locks. A leader transfer may ask for a second acknowledgement to transfer that lock state between peers. TXN 803 introduces that case. For now, consider the no-lock path.

If everything looks good, TiKV calls RawNode::transfer_leader(), which steps a local MsgTransferLeader into raft-rs.

raft-rs Level

The leader stops accepting new proposals. This is why the target should already be reasonably close: otherwise, the interval during which the Region cannot accept new writes becomes longer.

If the target has not fully caught up, the leader sends it more MsgAppend messages. Once the target's matched index reaches the leader's last index, the leader sends MsgTimeoutNow to trigger an immediate election on the target.

leader stops proposals
        |
        v
target catches up through MsgAppend
        |
        v
leader sends MsgTimeoutNow
        |
        v
target starts an election
        |
        v
target becomes leader

Because the target's log is fully caught up, it should be able to secure enough votes when a quorum is available. The election still follows the normal Raft safety rules. Leader transfer only triggers the election more directly.