Skip to content

fix(backend): synchronize stick-tables across a reload with a peers section - #378

Merged
bihius merged 1 commit into
mainfrom
fix/stick-table-peers
Oct 14, 2026
Merged

bihius merged 1 commit into
mainfrom
fix/stick-table-peers

Conversation

@bihius

@bihius bihius commented Oct 14, 2026

Copy link
Copy Markdown
Owner

Found while testing the beta.7 -> beta.8 upgrade: an attacker's ban was lifted whenever an admin clicked Apply configuration.

The bug

Bans and rate-limit counters are HAProxy stick-tables, i.e. process memory. HAProxy only carries them across a reload if they belong to a peers section, and the generated config had none. Apply configuration reloads through the master socket (config_apply._reload_haproxy), which forks a new worker with empty tables — so any config change, however unrelated, dropped every ban and reset every counter.

Measured on a test stack (rate limit 3/10s, ban threshold 5, ban duration 3600s): a ban at gpc0=6 was reported by GET /security/banned-ips and enforced with 429; after one Apply configuration that only added an unrelated vhost, the list was empty, the entry was back to gpc0=0, and traffic returned 200.

The fix

haproxy.cfg.j2 now emits, only when a policy actually enables DDoS protection:

  • peers gp_stick_tables with a single local peer (127.0.0.1:10001, loopback inside the container, not published);
  • peers gp_stick_tables on every st_ddos_vhost_* and st_ban_vhost_* line;
  • localpeer localpeer in global.

The localpeer directive is not optional: HAProxy silently drops a peers section that does not name the local peer, and a stick-table left pointing at a dropped section stops synchronizing without any error. Verified with the pinned haproxy:3.0.29-alpine / backend image 3.0.29-1~bpo12+1.

Verification

Unit: generator and renderer tests assert the peers wiring, that the section is declared before the first table that uses it, that two tables (rate limiting and auto-ban) reference it, and that nothing is emitted when DDoS protection is off. The existing haproxy -c validation tests cover the rendered config. Full backend suite green (880 passed, 6 skipped).

End to end on the test host with a locally built backend image, real policy and traffic through HAProxy:

  • ban created (gpc0=6, banned=true, 429) -> Apply configuration -> still gpc0=6, still banned=true in GET /security/banned-ips and in show table st_ban_vhost_1 on the admin socket, still 429 on the wire;
  • rate counters too: http_req_rate(10000)=2 survives the reload, and also survives a reload that changes the table definition (rate_limit_window_seconds 10 -> 20), where the entry comes back as http_req_rate(20000)=2;
  • haproxy -c -f clean on the generated config.

Two cases still start empty, and are the reason the docs changed rather than being deleted:

  • a restarted or recreated HAProxy container has no predecessor to sync from;
  • the first reload that introduces the peers section — the Apply right after upgrading — also starts empty, because the outgoing worker is the one that serves the synchronization and it does not know the section yet. Every reload after that carries the tables over.

docs/architecture.md (DDoS Protection & Auto-Ban) and the release guide's "What an upgrade does not carry over" bullet now say exactly this instead of claiming the state is dropped on any reload.

…ection

Stick-tables live in the HAProxy worker's memory, and HAProxy does not carry
them across a reload unless they belong to a peers section. Without one,
every "Apply configuration" — which reloads through the master socket —
started the new worker with empty tables, so an attacker's ban was lifted
whenever an admin changed any config, and rate counters reset with it.

Generate a `peers gp_stick_tables` section with a single local peer, name it
on every `st_ddos_vhost_*` and `st_ban_vhost_*` table, and add the matching
`localpeer localpeer` global directive. The directive is not optional:
HAProxy discards a peers section that does not name the local peer, and a
stick-table whose section was discarded stops synchronizing without
reporting an error. Both are emitted only when a policy actually enables
DDoS protection.

A restarted or recreated HAProxy container still starts empty — there is no
predecessor to synchronize from — so the docs say that rather than claiming
the state is lost on any reload.
Copilot AI balanced review requested due to automatic review settings October 14, 2026 19:29

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@bihius
bihius merged commit 31d8bf8 into main Oct 14, 2026
6 checks passed
@bihius
bihius deleted the fix/stick-table-peers branch October 14, 2026 19:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants