RAFT 602: Replica Movement

Moving Replicas

Leader transfer moves only the leader role and involves no data movement. This chapter covers moving replicas.

Replica movement is needed when TiKV stores are added or removed during cluster scale-out or scale-in. PD also uses it to balance data between stores. For example, PD may add a replica to one store and remove another replica from a different store while keeping the usual replication factor of three.

PD has the global view. It decides which peer to add or remove and on which store. The current Region leader receives the request and checks that the request is still valid, that the requested peer change makes sense, and that another change is not already in progress.

A Region's configuration is the set of peers that belong to its Raft group, including which peers can vote. Adding or removing a peer is a configuration change, or conf change.

It is easier to start with raft-rs.

raft-rs Level

raft-rs exposes two important APIs for a conf change:

propose_conf_change
apply_conf_change

propose_conf_change is similar to proposing a normal Raft log entry. It creates a special entry that describes a peer change. Raft replicates and commits that entry through the normal log path.

propose conf change
        |
        v
replicate to the Raft group
        |
        v
commit the entry
        |
        v
Raftstore's apply system applies the entry

After TiKV has made the peer change durable, it calls apply_conf_change. That makes the change effective in raft-rs's in-memory configuration. The change may add or remove a peer, and therefore change which peers can vote and how Raft calculates a quorum.

committed conf-change entry
        |
        v
TiKV makes the peer change durable
        |
        v
apply_conf_change
        |
        v
raft-rs updates its in-memory configuration

TiKV Level

TiKV records a Region's current key range, peer list, and version information as its Region metadata. The peer list describes the Region's configuration. Its configuration version increases whenever that peer list changes.

TiKV mirrors the raft-rs protocol from proposal through application.

First, the leader's peer event loop calls propose_conf_change. This creates the Raft entry described above. Raft then replicates and commits it in the normal way.

After the entry commits, Raftstore's apply system updates the Region metadata: it changes the peer list, advances the configuration version, and persists the new metadata.

The apply system then reports completion to the Region peer. The peer calls apply_conf_change to update raft-rs's in-memory configuration. It also updates the local runtime information used to route Raft messages and track peers.

The exact persistent records and internal messages that pass work between these components are implementation details. RAFTSTORE 901: Peer Lifecycle and Crash Recovery introduces them later. The important order here is: first persist the Region metadata; then update the in-memory Raft configuration.

peer event loop
  propose_conf_change
        |
        v
Raft commits the entry
        |
        v
Raftstore apply system
  update and persist Region metadata
  advance configuration version
        |
        v
Region peer event loop
  apply_conf_change
  update runtime state

Adding a Replica

The previous process happens on existing replicas. A newly added replica does not exist on its target store yet.

TiKV normally adds it as a learner first. After that peer change applies, the leader begins tracking how far the learner has caught up and sends it Raft messages.

When the target store receives an initial Raft message but has no peer for that Region, TiKV creates an uninitialized peer: a local placeholder that knows the Region exists but does not have its data yet.

The new peer then catches up through log replication or, more commonly, a Raft snapshot. The next chapter introduces that data-transfer path. Once it has caught up, PD can promote the learner to a voting peer.

Removing a Replica

Each existing replica applies the removal configuration change locally. If a peer finds that the change removes itself, it records durable removal state, stops its local event loops, and removes its Region data and Raft logs. The durable record prevents an old Raft message from recreating the removed peer.

A typical replica movement therefore looks like this:

add learner on the target store
        |
        v
initialize and catch up the new peer
        |
        v
promote the learner to voter
        |
        v
remove the old peer from the source store

This peer lifecycle is important and involved. Later chapters revisit its details through Raft snapshots, Region split, Region merge, and restart recovery.