Repository navigation
fix: keep idle scan_reader connections alive and harden startup and shutdown - #1066
Conversation
|
@suxb201 could you take a look when you have time? We've been running a build of this branch in production: The PR adds no reconnects: a real connection loss still stops redis-shake as before. It only keeps idle connections alive with PINGs, makes dial and keepalive configurable, and fixes the double close of the dump queue on shutdown. The CLA is signed. CI is waiting for approval to run, since this is my first PR here. |
|
Hi @suxb201, friendly ping. Could you take a look when you have time? We have been running this branch in production for over a week: scan_reader + ksn, The diff looks bigger than it is: about 70% of it is new tests. The code change is CI is waiting for workflow approval. @suxb201 您好,想跟进一下这个 PR,有空时麻烦看一下。 这个分支已经在我们的生产环境运行了一周多:scan_reader + ksn,每个分片一个 这个 PR 看起来比实际改动大:约 70% 是新增测试。代码改动约 200 行,涉及四个文件, CI 正在等待 workflow 审批。 |
…hutdown Co-authored-by: suxb201 <suxb201@gmail.com>
|
Thanks for the PR and the detailed write-up. The three problems are real and the direction is right (heartbeats, no reconnect). The tests also helped the review: removing each fix in turn made the matching test fail, so they do reproduce the production failures. Two things I'd like to change before merging, both about size rather than behaviour:
Below is what I'd like to merge: same values as yours (dial timeout 10 s, keepalive 30 s, PING every 10 s on both connections), 2 files, +52/−4. Differences from your branch:
Diff against v4 (a9b0a74)diff --git a/internal/client/redis.go b/internal/client/redis.go
index f73b0d7..39affd1 100644
--- a/internal/client/redis.go
+++ b/internal/client/redis.go
@@ -35,9 +35,9 @@ func NewRedisClient(ctx context.Context, address string, username string, passwo
var conn net.Conn
var dialer = &net.Dialer{
Timeout: 5 * time.Minute,
- KeepAlive: 5 * time.Minute,
+ KeepAlive: 30 * time.Second, // NAT and VPN gateways drop flows that stay silent for minutes
}
- ctxWithDeadline, cancel := context.WithTimeout(ctx, 1*time.Second)
+ ctxWithDeadline, cancel := context.WithTimeout(ctx, 10*time.Second)
defer cancel()
var err error
if Tls {
diff --git a/internal/reader/scan_standalone_reader.go b/internal/reader/scan_standalone_reader.go
index 50f5426..4f96895 100644
--- a/internal/reader/scan_standalone_reader.go
+++ b/internal/reader/scan_standalone_reader.go
@@ -9,6 +9,7 @@ import (
"strconv"
"strings"
"sync"
+ "time"
"RedisShake/internal/client"
"RedisShake/internal/client/proto"
@@ -39,9 +40,13 @@ type dbKey struct {
key string
}
+// dumpPing in needDumpQueue asks dump() to ping its connection.
+type dumpPing struct{}
+
type needRestoreItem struct {
dbId int
key string
+ ping bool // the reply to read is PONG, not DUMP of a key
}
type scanStandaloneReader struct {
@@ -53,6 +58,7 @@ type scanStandaloneReader struct {
needRestoreChan chan *needRestoreItem
dumpClient *client.Redis
subWG sync.WaitGroup
+ scanWG sync.WaitGroup
isValkey bool
stat struct {
@@ -79,6 +85,9 @@ func NewScanStandaloneReader(ctx context.Context, opts *ScanReaderOptions) Reade
func (r *scanStandaloneReader) StartRead(ctx context.Context) []chan *entry.Entry {
r.ctx = ctx
+ if r.opts.Scan {
+ r.scanWG.Add(1)
+ }
if r.opts.KSN {
r.subWG.Add(1)
go r.subscribe()
@@ -118,19 +127,43 @@ func (r *scanStandaloneReader) subscribe() {
// wait
r.subWG.Done()
+ // PING keeps the subscription alive through gateways and proxies that drop silent flows.
+ // Each PONG also makes dump() ping its connection, which idles while no keys change.
+ go func() {
+ ticker := time.NewTicker(10 * time.Second)
+ defer ticker.Stop()
+ for {
+ select {
+ case <-r.ctx.Done():
+ c.Close() // unblocks Receive below
+ return
+ case <-ticker.C:
+ c.Send("PING")
+ }
+ }
+ }()
+
regex := regexp.MustCompile(`\d+`)
for {
select {
case <-r.ctx.Done():
log.Infof("[%s] scanStandaloneReader subscribe finished.", r.stat.Name)
+ r.scanWG.Wait() // scan keeps putting keys until it sees ctx.Done
r.needDumpQueue.Close()
return
default:
resp, err := c.Receive()
if err != nil {
+ if r.ctx.Err() != nil {
+ continue // the connection was closed above on shutdown
+ }
log.Panicf("%v", err)
}
respSlice := resp.([]interface{})
+ if len(respSlice) == 2 { // reply to PING: ["pong", ""]
+ r.needDumpQueue.Put(dumpPing{})
+ continue
+ }
key := respSlice[3].(string)
dbId := regex.FindString(respSlice[2].(string))
dbIdInt, err := strconv.Atoi(dbId)
@@ -152,6 +185,7 @@ func (r *scanStandaloneReader) subscribe() {
}
func (r *scanStandaloneReader) scan() {
+ defer r.scanWG.Done()
c := client.NewRedisClient(r.ctx, r.opts.Address, r.opts.Username, r.opts.Password, r.opts.Tls, r.opts.TlsConfig, r.opts.PreferReplica)
defer c.Close()
dbs := r.dbs
@@ -177,7 +211,9 @@ func (r *scanStandaloneReader) scan() {
select {
case <-r.ctx.Done():
log.Infof("[%s] scanStandaloneReader scan finished.", r.stat.Name)
- r.needDumpQueue.Close()
+ if !r.opts.KSN {
+ r.needDumpQueue.Close()
+ }
return
default:
}
@@ -216,6 +252,11 @@ func (r *scanStandaloneReader) dump() {
}
for item := range r.needDumpQueue.Ch {
+ if _, ok := item.(dumpPing); ok {
+ r.dumpClient.Send("PING")
+ r.needRestoreChan <- &needRestoreItem{ping: true}
+ continue
+ }
r.stat.NeedUpdateCount = int64(r.needDumpQueue.Len())
dbId := item.(dbKey).db
key := item.(dbKey).key
@@ -229,7 +270,7 @@ func (r *scanStandaloneReader) dump() {
if len(r.opts.SkipUnknownType) > 0 {
r.dumpClient.Send("TYPE", key)
}
- r.needRestoreChan <- &needRestoreItem{dbId, key}
+ r.needRestoreChan <- &needRestoreItem{dbId: dbId, key: key}
}
close(r.needRestoreChan)
log.Infof("[%s] scanStandaloneReader dump finished.", r.stat.Name)
@@ -242,6 +283,13 @@ func (r *scanStandaloneReader) dump() {
func (r *scanStandaloneReader) restore() {
nowDbId := 0
for item := range r.needRestoreChan {
+ if item.ping {
+ reply, err := r.dumpClient.Receive()
+ if err != nil || reply != "PONG" {
+ log.Panicf("scanStandaloneReader ping failed. reply=[%v], err=[%v]", reply, err)
+ }
+ continue
+ }
dbId := item.dbId
key := item.key
if nowDbId != dbId {Could you update the branch to this (or equivalent)? Then it's good to go from my side. |
fb32bb3 to
9051f6b
Compare
|
I've updated the branch to the version from my comment above ( Authorship is kept: the commit is authored by @korchak-aleksandr with me as co-author, and the commit message is yours. CI is running. @korchak-aleksandr please look at the diff; if it misses something you had (e.g. the retry, or the config) tell me and I'll adjust. |
|
CI is green on 9051f6b: all 9 jobs pass (redis 2.8, 3.0, 4.0, 5.0, 6.0, 7.0, 8.0; valkey 8.0, 9.0), so the change also passes the existing black-box regression suite without the new Go tests. |
|
@suxb201 One thing I'd still like is test coverage: |
|
Sounds good, and a separate PR is fine. Two notes so the case proves what we expect:
Your SET/DEL sequence works as is. |
The source runs with `timeout 15` and stays quiet for 20 seconds after the scan. The subscription must keep its idle time under 12 seconds, and a key written afterwards must still reach the target, which needs the DUMP connection. Both checks fail on v4 before #1066. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Problem
We run
scan_readerwithksn = trueto keep a target continuously in sync with a large source (~60M keys per shard). The source is reached through SNAT and an IPsec tunnel. redis-shake keeps restarting, and every restart means a full rescan that takes hours.idle350–950 s inCLIENT LIST). A gateway along the path drops the flow, and redis-shake exits withread: connection reset by peerinsubscribe(). After the scan, the DUMP connection idles the same way, and Redis also closes it oncetimeoutpasses.dial failed … lookup … i/o timeout.ksn = true, both the scan and the subscription close the dump queue:panic: close of closed channel.Changes
advanced.dial_timeout_sec(default 10, was 1). A failed dial is retried up to 5 times with exponential backoff. Only the initial connect is retried; nothing has been read yet, so no data can be lost.advanced.tcp_keepalive_sec(default 30, was 5 minutes).pongreplies are told apart from key events. Message parsing is moved toparseKeyEvent, which also no longer indexes blindly into the reply.restorereads thePONGin pipeline order.UniqueQueue.Closeis idempotent, andPutafterClosedoes nothing.These changes add no reconnects. A real connection loss still stops redis-shake, as before: reconnecting would silently lose the keyspace events missed in between. The goal is to stop spurious exits on idle connections, not to hide real ones.
Tests
Unit tests: dial retry and cancellation,
parseKeyEvent, andUniqueQueue(double close, put after close, concurrent put and close).Integration tests in
internal/reader. They startredis-serverand are skipped when it is not inPATH:timeout 1closing the idle DUMP connection;With the heartbeats disabled, the first two reproduce the production failures:
connection reset by peerinsubscribe()andEOFinrestore(). The last one reproducesclose of closed channelwithout the queue fix.go test -race ./...passes. The black-boxcases/scan.pypasses.🤖 Generated with Claude Code