Skip to content

Fix Redis client stuck on stale IP after DNS record changes - #2548

Merged
an-tao merged 1 commit into
drogonframework:masterfrom
Qewby:fix/redis-dns-reresolve
Aug 4, 2026
Merged

Fix Redis client stuck on stale IP after DNS record changes#2548
an-tao merged 1 commit into
drogonframework:masterfrom
Qewby:fix/redis-dns-reresolve

Conversation

@Qewby

@Qewby Qewby commented Jul 25, 2026

Copy link
Copy Markdown
Contributor

Problem

ConfigLoader resolves the Redis host to an IP address once, synchronously, when the config is loaded, and that IP is baked into every RedisConnection for the lifetime of the process:

// current behavior
auto hostIp = future.get();  // blocking resolve, done once at startup
drogon::app().createRedisClient(hostIp, port, ...);

If the Redis endpoint is fronted by DNS (Kubernetes Service, ElastiCache/managed Redis endpoint, Sentinel, any failover setup) and the IP behind that name changes, Drogon has no way to notice — every reconnect attempt keeps dialing the old, now-dead address.

We hit this in production: our Redis runs in Kubernetes behind a Service hostname. When the pod backing that Service was rescheduled and got a new IP, Drogon kept reconnecting to the old address and our logs started spamming Failed to connect to 0.0.0.0:6379 (serverAddr_.toIp() had gone stale after the failover) until we restarted the app.

Fix

  • ConfigLoader no longer pre-resolves the host; it passes the original hostname straight through.
  • RedisConnection now carries the hostname (not just the resolved InetAddress) and calls trantor::Resolver on every connection attempt (startConnectionInLoop), so a reconnect always picks up the current DNS record instead of a value cached at startup.
  • A failed resolution triggers the normal disconnectCallback_ path (so the client's existing reconnect/backoff logic applies) instead of silently connecting to 0.0.0.0.
  • Log lines now print the hostname instead of a stale resolved IP, so connection errors are actually actionable.

This only touches RedisClientImpl/RedisClientLockFree/RedisConnection, which are internal (src/, not inc/) — no public API change.

Testing

  • Built locally with BUILD_REDIS=ON (hiredis via Homebrew), all touched translation units compile without new warnings.
  • Manually verified against a local Redis: connect, kill/restart Redis on a different IP behind the same hostname (via /etc/hosts swap), confirm the client reconnects to the new address instead of looping on the old one.
  • Running this patch in production at ~200 rps for a while now with no regressions in steady-state Redis traffic. We haven't yet had a live DNS/failover event to observe the recovery path in production, but the manual re-resolution test above exercises exactly that code path.

Previously the Redis host was resolved to an IP once at config-load
time via a blocking future.get(), and that IP was baked into every
subsequent (re)connect. If the endpoint is DNS-based (Kubernetes
Service, cluster failover, etc.) and its IP changes, the client keeps
retrying the stale address forever instead of picking up the new one.

Each RedisConnection now re-resolves its hostname asynchronously on
every connection attempt instead of once at startup.
@an-tao
an-tao merged commit 94048f3 into drogonframework:master Aug 4, 2026
34 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants