Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

RAFTSTORE 802: Raft Hibernation

A TiKV store can host hundreds of thousands of Region peers, most of them idle. Even when a peer is idle, it has periodic Raft ticks to process, which drive heartbeats and election timeouts. At that scale, this background work can consume significant Raftstore CPU.

Raft hibernation pauses those normal ticks for a quiet Raft group. This chapter asks when a group can sleep, what wakes it for new work, and how a peer later detects a failed leader.

When a Peer Can Hibernate

A Raft tick advances a peer's logical clock. A Region can sleep only after it becomes quiet.

Follower. A follower can stop its own ticks after it has caught up with a valid leader and has no pending read or snapshot work. It does not need a view of the whole group: as long as the leader keeps ticking, its next heartbeat wakes the follower again.

Leader. When the leader stops ticking, the group's regular heartbeats stop too. Before it can hibernate:

  • Every available follower must be at the leader's latest log.
  • The caught-up peers must form a quorum.
  • The leader must have applied its latest log and have no pending write, read, or leader-transfer work.

The leader must also obtain hibernation approval from a quorum through a poll. The first passing check starts that polling round. At the next election-timeout interval, it confirms the conditions and responses again. With the current defaults, this is about ten seconds later. A command in between keeps the group awake.

Agreeing to Hibernate

The polling leader asks the other peers:

leader
  |
  | MsgHibernateRequest
  v
followers
  |
  | MsgHibernateResponse
  v
leader

Each follower responds only if it meets its local conditions.

The leader waits until every peer it still considers available has agreed. If some peers are already known to be down, their responses may be omitted, but the peers that agreed must still form a quorum. Once the checks pass, the leader stops its normal base tick.

Quick Election After Wake-Up

TiKV uses skipped ticks as a quick-election mechanism. Before a follower stops scheduling ticks, it records several that it would otherwise have processed. With the current election and heartbeat settings, it records six.

When the follower wakes, Raftstore replays that count into raft-rs. If the leader has failed, the follower can reach an election timeout sooner than it would after a full reset. TiKV still leaves the leader enough time to send a heartbeat, so a healthy leader does not trigger an unnecessary election.

Waking for Work

A new proposal, a ReadIndex request, or a relevant incoming Raft message wakes the peer and registers its normal base tick again.

hibernated peer
      |
      | proposal, ReadIndex, or Raft message
      v
resume normal Raft ticks

The peer returns to the normal active state, processes the work, and may hibernate again after the group becomes stable.

Detecting a Failed Leader

A Region can remain idle while its leader fails. Detecting that failure can take several minutes, but hibernation applies only while the group has no traffic. If a request arrives, it wakes the peer promptly.

Hibernated peers retain a much slower stale check, which runs every five minutes by default. During its stale check, a hibernated leader sends a Raft heartbeat. Receiving that heartbeat prevents the followers from progressing toward an election.

TiKV calls the normal hibernation state Idle. PreChaos is a one-check grace state after a follower fails to hear from its leader. Chaos means the leader may be gone and the peer is about to restore normal ticks.

If a follower no longer hears from the leader, it moves through two states before restoring the normal Raft tick:

Idle
  |
  | first stale check
  v
PreChaos
  |
  | next stale check without a leader heartbeat
  v
Chaos
  |
  | next stale check
  v
resume normal Raft ticks and election detection

PreChaos is only a warning state. The leader's and follower's stale-check timers are not synchronized, so a healthy follower may briefly enter it before the leader's heartbeat arrives. That heartbeat returns the follower to Ordered, the normal stable state from which it can hibernate again.

If the leader is truly gone, no heartbeat resets the follower. The follower eventually reaches Chaos, resumes ordinary Raft ticks, and can start an election after its remaining election timeout passes.

This path deliberately detects failure more slowly than a continuously ticking Raft group. The tradeoff is worthwhile for idle Regions: TiKV preserves Raft safety and can recover when a quorum remains available, while avoiding constant tick work across the store.