Fix Redis client stuck on stale IP after DNS record changes - #2548
Merged
Conversation
Previously the Redis host was resolved to an IP once at config-load time via a blocking future.get(), and that IP was baked into every subsequent (re)connect. If the endpoint is DNS-based (Kubernetes Service, cluster failover, etc.) and its IP changes, the client keeps retrying the stale address forever instead of picking up the new one. Each RedisConnection now re-resolves its hostname asynchronously on every connection attempt instead of once at startup.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
ConfigLoaderresolves the Redishostto an IP address once, synchronously, when the config is loaded, and that IP is baked into everyRedisConnectionfor the lifetime of the process:If the Redis endpoint is fronted by DNS (Kubernetes
Service, ElastiCache/managed Redis endpoint, Sentinel, any failover setup) and the IP behind that name changes, Drogon has no way to notice — every reconnect attempt keeps dialing the old, now-dead address.We hit this in production: our Redis runs in Kubernetes behind a
Servicehostname. When the pod backing that Service was rescheduled and got a new IP, Drogon kept reconnecting to the old address and our logs started spammingFailed to connect to 0.0.0.0:6379(serverAddr_.toIp()had gone stale after the failover) until we restarted the app.Fix
ConfigLoaderno longer pre-resolves the host; it passes the original hostname straight through.RedisConnectionnow carries the hostname (not just the resolvedInetAddress) and callstrantor::Resolveron every connection attempt (startConnectionInLoop), so a reconnect always picks up the current DNS record instead of a value cached at startup.disconnectCallback_path (so the client's existing reconnect/backoff logic applies) instead of silently connecting to0.0.0.0.This only touches
RedisClientImpl/RedisClientLockFree/RedisConnection, which are internal (src/, notinc/) — no public API change.Testing
BUILD_REDIS=ON(hiredis via Homebrew), all touched translation units compile without new warnings./etc/hostsswap), confirm the client reconnects to the new address instead of looping on the old one.