Skip to content

fix: keep idle scan_reader connections alive and harden startup and shutdown - #1066

Merged
suxb201 merged 1 commit into
tair-opensource:v4from
korchak-aleksandr:fix/connection-stability
Oct 28, 2026
Merged

suxb201 merged 1 commit into
tair-opensource:v4from
korchak-aleksandr:fix/connection-stability

Conversation

@korchak-aleksandr

Copy link
Copy Markdown
Contributor

Problem

We run scan_reader with ksn = true to keep a target continuously in sync with a large source (~60M keys per shard). The source is reached through SNAT and an IPsec tunnel. redis-shake keeps restarting, and every restart means a full rescan that takes hours.

  • Silent connections get reset. When keyspace notifications are rare, the subscription carries no traffic for many minutes (we saw idle 350–950 s in CLIENT LIST). A gateway along the path drops the flow, and redis-shake exits with read: connection reset by peer in subscribe(). After the scan, the DUMP connection idles the same way, and Redis also closes it once timeout passes.
  • Dial fails on startup. A dial, including the DNS lookup, was limited to 1 second with no retry. A slow resolver made the pod crash-loop with dial failed … lookup … i/o timeout.
  • Shutdown during the scan panics. With ksn = true, both the scan and the subscription close the dump queue: panic: close of closed channel.

Changes

  • advanced.dial_timeout_sec (default 10, was 1). A failed dial is retried up to 5 times with exponential backoff. Only the initial connect is retried; nothing has been read yet, so no data can be lost.
  • advanced.tcp_keepalive_sec (default 30, was 5 minutes).
  • The ksn subscription is pinged every 10 s, and pong replies are told apart from key events. Message parsing is moved to parseKeyEvent, which also no longer indexes blindly into the reply.
  • The DUMP connection is pinged every 10 s. restore reads the PONG in pipeline order.
  • UniqueQueue.Close is idempotent, and Put after Close does nothing.

These changes add no reconnects. A real connection loss still stops redis-shake, as before: reconnecting would silently lose the keyspace events missed in between. The goal is to stop spurious exits on idle connections, not to hide real ones.

Tests

  • Unit tests: dial retry and cancellation, parseKeyEvent, and UniqueQueue (double close, put after close, concurrent put and close).

  • Integration tests in internal/reader. They start redis-server and are skipped when it is not in PATH:

    • a proxy that resets connections idle for 1 s, as a NAT or VPN gateway does;
    • Redis timeout 1 closing the idle DUMP connection;
    • DEL forwarded while PING replies arrive on the subscription;
    • shutdown requested in the middle of the scan.

    With the heartbeats disabled, the first two reproduce the production failures: connection reset by peer in subscribe() and EOF in restore(). The last one reproduces close of closed channel without the queue fix.

  • go test -race ./... passes. The black-box cases/scan.py passes.

🤖 Generated with Claude Code

@CLAassistant

CLAassistant commented Oct 11, 2026 •

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

@korchak-aleksandr

Copy link
Copy Markdown
Contributor Author

@suxb201 could you take a look when you have time?

We've been running a build of this branch in production: scan_reader with ksn = true, about 60M keys per shard, with the source reached through SNAT and an IPsec tunnel. Before this change, pods kept restarting on connection reset by peer in subscribe() and on 1-second dial timeouts. Every restart meant a full rescan that took hours. With the patch, those restarts have stopped, and the sync has been stable since.

The PR adds no reconnects: a real connection loss still stops redis-shake as before. It only keeps idle connections alive with PINGs, makes dial and keepalive configurable, and fixes the double close of the dump queue on shutdown.

The CLA is signed. CI is waiting for approval to run, since this is my first PR here.

@korchak-aleksandr

Copy link
Copy Markdown
Contributor Author

Hi @suxb201, friendly ping. Could you take a look when you have time?

We have been running this branch in production for over a week: scan_reader + ksn,
one redis-shake per shard, source behind NAT/IPsec. Zero pod restarts since the
deploy; before it, the idle connections were dropped and every drop meant a full
rescan. I think the fix is useful for anyone running scan_reader with ksn across
NAT, SLB or VPN.

The diff looks bigger than it is: about 70% of it is new tests. The code change is
about 200 lines in four files and only touches the connection handling.

CI is waiting for workflow approval.


@suxb201 您好,想跟进一下这个 PR,有空时麻烦看一下。

这个分支已经在我们的生产环境运行了一周多:scan_reader + ksn,每个分片一个
redis-shake,源端在 NAT/IPsec 后面。部署后 pod 零重启;之前空闲连接会被断开,
每次断开都要重新全量扫描。我认为这个修复对所有跨 NAT、SLB 或 VPN 使用
scan_reader + ksn 的用户都有帮助。

这个 PR 看起来比实际改动大:约 70% 是新增测试。代码改动约 200 行,涉及四个文件,
只改动了连接处理部分。

CI 正在等待 workflow 审批。

…hutdown

Co-authored-by: suxb201 <suxb201@gmail.com>
@suxb201

suxb201 commented Oct 26, 2026

Copy link
Copy Markdown
Member

Thanks for the PR and the detailed write-up. The three problems are real and the direction is right (heartbeats, no reconnect). The tests also helped the review: removing each fix in turn made the matching test fail, so they do reproduce the production failures.

Two things I'd like to change before merging, both about size rather than behaviour:

  1. Tests. internal/ only has pure unit tests; behaviour is tested from tests/cases/ with pybbt. Starting redis-server and a TCP proxy from go test is a pattern I'd rather not introduce, and the fix is small enough that I'm fine merging it without new Go tests.
  2. Config. We try to keep shake.toml small. dial_timeout_sec / tcp_keepalive_sec, the constant fallbacks and the package variables used by the tests are more surface than the problem needs. 10 s and 30 s are fine as fixed values.

Below is what I'd like to merge: same values as yours (dial timeout 10 s, keepalive 30 s, PING every 10 s on both connections), 2 files, +52/−4. Differences from your branch:

  • No dial retry. redis-shake fails fast and lets the supervisor restart it; a failed dial at startup loses no progress, and 10 s already covers the slow resolver.
  • Subscription heartbeat is a goroutine that PINGs every 10 s and, on ctx cancel, closes the connection so Receive returns at once (today subscribe can only exit on the next keyspace event). PONG is told apart by len(resp) == 2; the parsing code stays as is.
  • DUMP heartbeat has no ticker of its own: each PONG on the subscription puts a dumpPing{} marker into needDumpQueue; dump() sends PING and restore() reads PONG in pipeline order, as in your version. The DUMP connection only idles with ksn, and the marker only exists there; UniqueQueue dedups it.
  • Double close is fixed at the call sites instead of in UniqueQueue: with ksn, scan() no longer closes the queue on ctx cancel (same rule as its normal exit), and subscribe() waits for scan() to return before closing, so there is one closer and no Put after Close.
Diff against v4 (a9b0a74)
diff --git a/internal/client/redis.go b/internal/client/redis.go
index f73b0d7..39affd1 100644
--- a/internal/client/redis.go
+++ b/internal/client/redis.go
@@ -35,9 +35,9 @@ func NewRedisClient(ctx context.Context, address string, username string, passwo
 	var conn net.Conn
 	var dialer = &net.Dialer{
 		Timeout:   5 * time.Minute,
-		KeepAlive: 5 * time.Minute,
+		KeepAlive: 30 * time.Second, // NAT and VPN gateways drop flows that stay silent for minutes
 	}
-	ctxWithDeadline, cancel := context.WithTimeout(ctx, 1*time.Second)
+	ctxWithDeadline, cancel := context.WithTimeout(ctx, 10*time.Second)
 	defer cancel()
 	var err error
 	if Tls {
diff --git a/internal/reader/scan_standalone_reader.go b/internal/reader/scan_standalone_reader.go
index 50f5426..4f96895 100644
--- a/internal/reader/scan_standalone_reader.go
+++ b/internal/reader/scan_standalone_reader.go
@@ -9,6 +9,7 @@ import (
 	"strconv"
 	"strings"
 	"sync"
+	"time"
 
 	"RedisShake/internal/client"
 	"RedisShake/internal/client/proto"
@@ -39,9 +40,13 @@ type dbKey struct {
 	key string
 }
 
+// dumpPing in needDumpQueue asks dump() to ping its connection.
+type dumpPing struct{}
+
 type needRestoreItem struct {
 	dbId int
 	key  string
+	ping bool // the reply to read is PONG, not DUMP of a key
 }
 
 type scanStandaloneReader struct {
@@ -53,6 +58,7 @@ type scanStandaloneReader struct {
 	needRestoreChan chan *needRestoreItem
 	dumpClient      *client.Redis
 	subWG           sync.WaitGroup
+	scanWG          sync.WaitGroup
 	isValkey        bool
 
 	stat struct {
@@ -79,6 +85,9 @@ func NewScanStandaloneReader(ctx context.Context, opts *ScanReaderOptions) Reade
 
 func (r *scanStandaloneReader) StartRead(ctx context.Context) []chan *entry.Entry {
 	r.ctx = ctx
+	if r.opts.Scan {
+		r.scanWG.Add(1)
+	}
 	if r.opts.KSN {
 		r.subWG.Add(1)
 		go r.subscribe()
@@ -118,19 +127,43 @@ func (r *scanStandaloneReader) subscribe() {
 	// wait
 	r.subWG.Done()
 
+	// PING keeps the subscription alive through gateways and proxies that drop silent flows.
+	// Each PONG also makes dump() ping its connection, which idles while no keys change.
+	go func() {
+		ticker := time.NewTicker(10 * time.Second)
+		defer ticker.Stop()
+		for {
+			select {
+			case <-r.ctx.Done():
+				c.Close() // unblocks Receive below
+				return
+			case <-ticker.C:
+				c.Send("PING")
+			}
+		}
+	}()
+
 	regex := regexp.MustCompile(`\d+`)
 	for {
 		select {
 		case <-r.ctx.Done():
 			log.Infof("[%s] scanStandaloneReader subscribe finished.", r.stat.Name)
+			r.scanWG.Wait() // scan keeps putting keys until it sees ctx.Done
 			r.needDumpQueue.Close()
 			return
 		default:
 			resp, err := c.Receive()
 			if err != nil {
+				if r.ctx.Err() != nil {
+					continue // the connection was closed above on shutdown
+				}
 				log.Panicf("%v", err)
 			}
 			respSlice := resp.([]interface{})
+			if len(respSlice) == 2 { // reply to PING: ["pong", ""]
+				r.needDumpQueue.Put(dumpPing{})
+				continue
+			}
 			key := respSlice[3].(string)
 			dbId := regex.FindString(respSlice[2].(string))
 			dbIdInt, err := strconv.Atoi(dbId)
@@ -152,6 +185,7 @@ func (r *scanStandaloneReader) subscribe() {
 }
 
 func (r *scanStandaloneReader) scan() {
+	defer r.scanWG.Done()
 	c := client.NewRedisClient(r.ctx, r.opts.Address, r.opts.Username, r.opts.Password, r.opts.Tls, r.opts.TlsConfig, r.opts.PreferReplica)
 	defer c.Close()
 	dbs := r.dbs
@@ -177,7 +211,9 @@ func (r *scanStandaloneReader) scan() {
 			select {
 			case <-r.ctx.Done():
 				log.Infof("[%s] scanStandaloneReader scan finished.", r.stat.Name)
-				r.needDumpQueue.Close()
+				if !r.opts.KSN {
+					r.needDumpQueue.Close()
+				}
 				return
 			default:
 			}
@@ -216,6 +252,11 @@ func (r *scanStandaloneReader) dump() {
 	}
 
 	for item := range r.needDumpQueue.Ch {
+		if _, ok := item.(dumpPing); ok {
+			r.dumpClient.Send("PING")
+			r.needRestoreChan <- &needRestoreItem{ping: true}
+			continue
+		}
 		r.stat.NeedUpdateCount = int64(r.needDumpQueue.Len())
 		dbId := item.(dbKey).db
 		key := item.(dbKey).key
@@ -229,7 +270,7 @@ func (r *scanStandaloneReader) dump() {
 		if len(r.opts.SkipUnknownType) > 0 {
 			r.dumpClient.Send("TYPE", key)
 		}
-		r.needRestoreChan <- &needRestoreItem{dbId, key}
+		r.needRestoreChan <- &needRestoreItem{dbId: dbId, key: key}
 	}
 	close(r.needRestoreChan)
 	log.Infof("[%s] scanStandaloneReader dump finished.", r.stat.Name)
@@ -242,6 +283,13 @@ func (r *scanStandaloneReader) dump() {
 func (r *scanStandaloneReader) restore() {
 	nowDbId := 0
 	for item := range r.needRestoreChan {
+		if item.ping {
+			reply, err := r.dumpClient.Receive()
+			if err != nil || reply != "PONG" {
+				log.Panicf("scanStandaloneReader ping failed. reply=[%v], err=[%v]", reply, err)
+			}
+			continue
+		}
 		dbId := item.dbId
 		key := item.key
 		if nowDbId != dbId {

Could you update the branch to this (or equivalent)? Then it's good to go from my side.

@suxb201
suxb201 force-pushed the fix/connection-stability branch from fb32bb3 to 9051f6b Compare October 26, 2026 06:42
@suxb201

suxb201 commented Oct 26, 2026

Copy link
Copy Markdown
Member

I've updated the branch to the version from my comment above (fb32bb3 → 9051f6b, one commit, 2 files, +52/−4). The original commits are still in the PR timeline if you want them for reference.

Authorship is kept: the commit is authored by @korchak-aleksandr with me as co-author, and the commit message is yours. CI is running. @korchak-aleksandr please look at the diff; if it misses something you had (e.g. the retry, or the config) tell me and I'll adjust.

@suxb201

suxb201 commented Oct 26, 2026

Copy link
Copy Markdown
Member

CI is green on 9051f6b: all 9 jobs pass (redis 2.8, 3.0, 4.0, 5.0, 6.0, 7.0, 8.0; valkey 8.0, 9.0), so the change also passes the existing black-box regression suite without the new Go tests.

@korchak-aleksandr

korchak-aleksandr commented Oct 26, 2026 •

Copy link
Copy Markdown
Contributor Author

@suxb201
Thanks for the review and for reworking the branch! The diff looks good to me.

One thing I'd still like is test coverage: ksn isn't covered in tests/cases/ at all.
I can add a pybbt case in a separate PR so this one isn't blocked: start the source with
--timeout 15 --notify-keyspace-events EA, run scan_reader with ksn = true, stay idle past
the timeout, then check that new keys and DELs still reach the target. Would that work for you?

@suxb201

suxb201 commented Oct 27, 2026

Copy link
Copy Markdown
Member

Sounds good, and a separate PR is fine. Two notes so the case proves what we expect:

  1. --timeout 15 covers the DUMP half only. Redis never times out subscribed clients (CLIENT_PUBSUB is skipped in the clients cron since 2.8), so a timeout on the source can't exercise the subscription heartbeat. For the DUMP half the timing is: let the scan finish, stay idle past 15 s, then write a key. With the heartbeat the DUMP connection is kept alive and the key is delivered; without it, the first DUMP after idle hits the closed connection, so the case should fail on v4. Please run it against v4 first, as you did with the Go tests.

  2. To cover the subscription half in the same case, keep the source quiet and poll CLIENT LIST for the connection with psub=1; its idle should stay under ~12 s, since the PING goes out every 10 s. Without the heartbeat it climbs with the poll window. Filter on psub= rather than CLIENT LIST TYPE pubsub: TYPE only exists since 5.0 and CI still runs 2.8, 3.0 and 4.0.

Your SET/DEL sequence works as is.

@korchak-aleksandr

Copy link
Copy Markdown
Contributor Author

@suxb201 Thanks, that's very helpful. I'll make sure the case fails on v4 first.
Nothing is missing from my side, so feel free to merge #1066; the test will follow in a separate PR.

@suxb201
suxb201 merged commit 1e083dc into tair-opensource:v4 Oct 28, 2026
10 checks passed
suxb201 pushed a commit that referenced this pull request Oct 30, 2026
The source runs with `timeout 15` and stays quiet for 20 seconds after
the scan. The subscription must keep its idle time under 12 seconds, and
a key written afterwards must still reach the target, which needs the
DUMP connection. Both checks fail on v4 before #1066.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants