RAFTSTORE 702: Slow Score

A crashed TiKV store has a clear recovery path. Raft elects new leaders, and PD replaces a replica if the store stays down.

A store can also be alive but slow. A disk problem can leave it able to send heartbeats and run Raft, but make every Region leader on that store slower than its peers. Those Regions still serve requests through their leaders, so the slow store can hurt latency without ever crashing.

TiKV uses a slow score to recognize this half-failed state. It measures whether the store's disks are repeatedly making too little progress, and PD moves leaders away before the store becomes unavailable.

Detecting a Slow Disk

The checks follow the store's disk layout. A TiKV store can keep its Raft logs and KV data on one disk or on separate disks.

For the Raft disk, TiKV sends an inspection request every 100 ms by default. The request enters the Raftstore path and completes when its batch has appended its Raft logs. Slow score measures the time spent handling the inspection in Raftstore and writing that batch's Raft logs to disk.

For a separate KV disk, a dedicated probe worker periodically checks that disk as well. If the Raft and KV engines use the same mount path, TiKV skips the KV probe: the Raft inspection already covers that disk.

An inspection times out when the next check begins before the previous one has finished. TiKV keeps a separate score for each inspected disk path and reports the larger one as the store's slow score.

Raising and Lowering the Score

The score starts at 1. Every 30 inspection ticks, TiKV updates it from the timeouts in that round. With the default 100 ms interval, a round lasts about three seconds.

If a round contains timeouts, the score increases multiplicatively. The timeout ratio determines the increase, up to a doubling of the current score. At the default 10% threshold, three timeouts in a 30-check round double the score.

If a round has no timeouts, the score falls linearly. A healthy store takes at least five minutes to recover from 100 to 1.

This asymmetry is deliberate: recurring stalls are recognized quickly, but a brief healthy period does not immediately erase evidence of a bad disk. The inspection interval, ticks per round, timeout-ratio threshold, and recovery time are the algorithm's parameters.

Evicting Leaders

TiKV includes its slow score in its periodic store heartbeat to PD. When the slow-store scheduler finds one slow TiKV whose score has reached 100, it schedules leader transfers away from that store.

The replicas and their data stay where they are. Requests then go to leaders on healthy stores while the slow TiKV recovers.