Repository navigation
fix(backend): synchronize stick-tables across a reload with a peers section - #378
Merged
Merged
Conversation
…ection Stick-tables live in the HAProxy worker's memory, and HAProxy does not carry them across a reload unless they belong to a peers section. Without one, every "Apply configuration" — which reloads through the master socket — started the new worker with empty tables, so an attacker's ban was lifted whenever an admin changed any config, and rate counters reset with it. Generate a `peers gp_stick_tables` section with a single local peer, name it on every `st_ddos_vhost_*` and `st_ban_vhost_*` table, and add the matching `localpeer localpeer` global directive. The directive is not optional: HAProxy discards a peers section that does not name the local peer, and a stick-table whose section was discarded stops synchronizing without reporting an error. Both are emitted only when a policy actually enables DDoS protection. A restarted or recreated HAProxy container still starts empty — there is no predecessor to synchronize from — so the docs say that rather than claiming the state is lost on any reload.
4 of 5 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Found while testing the beta.7 -> beta.8 upgrade: an attacker's ban was lifted whenever an admin clicked Apply configuration.
The bug
Bans and rate-limit counters are HAProxy stick-tables, i.e. process memory. HAProxy only carries them across a reload if they belong to a
peerssection, and the generated config had none.Apply configurationreloads through the master socket (config_apply._reload_haproxy), which forks a new worker with empty tables — so any config change, however unrelated, dropped every ban and reset every counter.Measured on a test stack (rate limit 3/10s, ban threshold 5, ban duration 3600s): a ban at
gpc0=6was reported byGET /security/banned-ipsand enforced with429; after one Apply configuration that only added an unrelated vhost, the list was empty, the entry was back togpc0=0, and traffic returned200.The fix
haproxy.cfg.j2now emits, only when a policy actually enables DDoS protection:peers gp_stick_tableswith a single local peer (127.0.0.1:10001, loopback inside the container, not published);peers gp_stick_tableson everyst_ddos_vhost_*andst_ban_vhost_*line;localpeer localpeeringlobal.The
localpeerdirective is not optional: HAProxy silently drops a peers section that does not name the local peer, and a stick-table left pointing at a dropped section stops synchronizing without any error. Verified with the pinnedhaproxy:3.0.29-alpine/ backend image3.0.29-1~bpo12+1.Verification
Unit: generator and renderer tests assert the peers wiring, that the section is declared before the first table that uses it, that two tables (rate limiting and auto-ban) reference it, and that nothing is emitted when DDoS protection is off. The existing
haproxy -cvalidation tests cover the rendered config. Full backend suite green (880 passed, 6 skipped).End to end on the test host with a locally built backend image, real policy and traffic through HAProxy:
gpc0=6,banned=true,429) -> Apply configuration -> stillgpc0=6, stillbanned=trueinGET /security/banned-ipsand inshow table st_ban_vhost_1on the admin socket, still429on the wire;http_req_rate(10000)=2survives the reload, and also survives a reload that changes the table definition (rate_limit_window_seconds10 -> 20), where the entry comes back ashttp_req_rate(20000)=2;haproxy -c -fclean on the generated config.Two cases still start empty, and are the reason the docs changed rather than being deleted:
docs/architecture.md(DDoS Protection & Auto-Ban) and the release guide's "What an upgrade does not carry over" bullet now say exactly this instead of claiming the state is dropped on any reload.