TiFlash stays disconnected for minutes after PD leader transfer due to proxy pd_client NO_RETRY

[JaySon-Huang]

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

1. Deploy a TiDB cluster with PD (multi-member) and TiFlash.

2. Confirm TiFlash stores are healthy / connected on the PD side.

3. Manually transfer PD leader, for example:

   # From any PD endpoint, transfer leadership to the target PD member
   curl -X POST http://<pd-addr>:2379/pd/api/v1/leader/transfer/<new-pd-name>
   # or via pd-ctl equivalent

4. Observe PD store status for TiFlash and TiFlash proxy (tiflash_tikv) logs around the transfer.

Observed incident timeline (2026-07-25, op gogame cluster; PD leader transfer from pd-10.107.16.121-2379 to pd-10.107.15.193-2379):

Time	Event
12:15:14.173	Manual PD leader transfer: POST /pd/api/v1/leader/transfer/pd-10.107.15.193-2379 (from 10.107.15.136)
12:15:14.173	Old leader pd-10.107.16.121-2379 starts leadership transfer to pd-10.107.15.193-2379
12:15:14.175	New leader 2d1b65bafc63a56b (10.107.15.193) elected at term 7
12:15:14.361	New PD leader ready to serve
12:15:14.882	TiFlash 10.107.15.196 compute PD client: switch leader from http://10.107.16.121:2379 to http://10.107.15.193:2379 (source=pingcap.pd)
12:15:14.901	TiFlash 10.107.15.194 proxy: first report min resolved_ts failed with Grpc(... UNAVAILABLE, "not leader")
12:15:15.757	TiFlash 10.107.15.194 compute PD client: switch leader ... to the new leader
12:15:19 ~ 12:15:22	store heartbeat failed starts with the same not leader error
12:15:15 ~ 12:18:37	TiFlash stores stay disconnected on PD for ~3.5 minutes; proxy keeps failing store_heartbeat / report_min_resolved_ts against the old leader
12:18:37.755	TiFlash 10.107.15.196 proxy PD client reconnects successfully (update pd client: prev_leader=10.107.16.121 → leader=10.107.15.193)
12:18:53.670	TiFlash 10.107.15.194 proxy PD client reconnects successfully the same way

2. What did you expect to see? (Required)

After PD leader transfer:

• TiFlash proxy PD client should reconnect to the new PD leader promptly (similar to TiKV).
• store_heartbeat / report_min_resolved_ts should succeed against the new leader within a short window (seconds).
• PD should not report TiFlash stores as disconnected for minutes.

3. What did you see instead (Required)

• PD shows TiFlash stores as disconnected for ~3.5 minutes after leader transfer.

• TiFlash compute C++ PD client switches leader quickly (switch leader ..., source=pingcap.pd).

• TiFlash proxy (Rust pd_client) keeps talking to the old leader and repeatedly logs:

  report min resolved_ts failed  err=Grpc(RpcFailure(RpcStatus { code: 14-UNAVAILABLE, message: "not leader", ... }))
  store heartbeat failed         err=Grpc(RpcFailure(RpcStatus { code: 14-UNAVAILABLE, message: "not leader", ... }))
  

Root cause (code):

In contrib/tiflash-proxy/components/pd_client/src/client.rs, periodic RPCs use NO_RETRY = 1:

const LEADER_CHANGE_RETRY: usize = 10;
// periodic request like store_heartbeat, we don't need to retry.
const NO_RETRY: usize = 1;

Both store_heartbeat and report_min_resolved_ts call:

self.pd_client.request(req, executor, NO_RETRY).execute()

Even though gRPC not leader is retryable (Error::Grpc), with NO_RETRY the request returns immediately and never reaches reconnect_if_needed() → reconnect(true). Upper layers in raftstore only log the error and do not force reconnect.

Unlike TiKV, TiFlash is a Learner and typically does not send region heartbeats as leader, so it cannot use the region-heartbeat stream-disconnect path that forces TiKV to reconnect quickly. Recovery then depends on slower side paths (periodic update_loop with default update_interval = 10m, and/or TSO stream close → reconnect(true)), which matched the ~3.5 minute delay in this incident.

4. What is your TiFlash version? (Required)

• v8.5.4
• Component under investigation: contrib/tiflash-proxy PD client (components/pd_client)