Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
30 changes: 21 additions & 9 deletions docs/architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -265,15 +265,27 @@ counter: each rate or connection violation increments it, and once it passes
inactivity. ACME HTTP-01 challenge paths are exempt from both, so a flood of
challenges cannot lock a vhost's own certificate renewal out.

**The state is volatile.** Stick-tables live in the HAProxy worker's memory;
nothing writes them to a file or to PostgreSQL. A fresh worker — `Apply
configuration`, which reloads through the master socket, or a
restarted/recreated HAProxy container — starts with empty tables, so every
ban and every rate counter is dropped. `GET /security/banned-ips` reads them
back through the HAProxy admin Runtime API rather than the database, and
therefore also reports an empty list after a reload; the dashboard's banned-IP
count is the one overview metric that can fail independently of PostgreSQL.
A source that was banned before the reload starts from zero afterwards.
**Surviving a reload.** Stick-tables live in the HAProxy worker's memory;
nothing writes them to a file or to PostgreSQL. The generated config therefore
declares a `peers` section (`gp_stick_tables`, one local peer) and names it on
every stick-table, together with a `localpeer localpeer` global directive that
marks which entry in that section is this process. On a master-worker reload —
`Apply configuration` reloads through the master socket — the outgoing worker
is still running while the new one starts, so the new worker pulls the table
contents from it and bans and rate counters carry over instead of resetting.
HAProxy silently discards a peers section that does not name the local peer,
and a stick-table whose peers section was discarded stops synchronizing without
reporting an error, which is why the two have to be emitted together. The
outgoing worker is the one that serves the synchronization, so the *first*
reload that introduces the peers section — the Apply that follows an upgrade
from a version that did not generate it — still starts empty; every reload
after that carries the tables over.

A **restarted or recreated HAProxy container still starts empty**: there is no
predecessor left to synchronize from. `GET /security/banned-ips` reads the
tables back through the HAProxy admin Runtime API rather than the database, so
the dashboard's banned-IP count is the one overview metric that can fail
independently of PostgreSQL.

## Authentication & Rate Limiting

Expand Down
18 changes: 10 additions & 8 deletions release/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ rough edges. Generated configs, the GeoIP database and PostgreSQL data all
live in Docker volumes, but no upgrade-path guarantees are made between beta
releases yet — back up `pgdata` before upgrading. See
[What an upgrade does not carry over](#what-an-upgrade-does-not-carry-over)
for the two pieces of state that deliberately do not persist, then
for what is kept off the volumes, then
[Known limitations](#known-limitations) and
[Reporting feedback](#reporting-feedback) below.

Expand Down Expand Up @@ -128,15 +128,17 @@ it — the static `haproxy.cfg` seed is replaced at that point.
Everything you configure in the admin UI survives an upgrade: it lives in
PostgreSQL and is re-rendered onto the `generated_config` volume (which also
holds the GeoIP database and the vhost certificates), so vhosts, policies and
HTTPS keep working. Two pieces of state are deliberately not persisted:
HTTPS keep working. Two things are stored outside the volumes:

- **Bans and rate-limit counters.** These are HAProxy stick-tables, held in
the HAProxy process's memory and never written to disk. Any HAProxy reload
or restart starts them empty, so a source IP's ban and its request counters
are dropped — including by **Apply configuration**, which reloads HAProxy.
A tag-only upgrade does not recreate the pinned `haproxy` container, so the
tables do happen to survive one, but do not rely on that. See the DDoS
Protection & Auto-Ban section of `docs/architecture.md`.
the HAProxy process's memory and never written to disk. The generated config
wires them to a `peers` section, so a worker reloaded by **Apply
configuration** pulls them from the worker it replaces and a ban is not
lifted just because an admin changed something. A tag-only upgrade leaves
the pinned `haproxy` container running, so they survive that too — but a
restarted or recreated HAProxy container starts empty, and nothing can
recover them. See the DDoS Protection & Auto-Ban section of
`docs/architecture.md`.
- **Certbot's ACME account key and renewal configuration.** The built-in
certificate flow runs certbot inside the `backend` container with its
configuration under `/tmp`, which is not a volume, so the account is
Expand Down
23 changes: 19 additions & 4 deletions src/backend/app/templates/haproxy.cfg.j2
Original file line number Diff line number Diff line change
Expand Up @@ -2,9 +2,16 @@
# Generated from the policy database; manual edits are overwritten on the
# next config apply.

{% set has_ddos_stick_tables = routes | selectattr("ddos") | list | length > 0 %}
global
log stdout format raw local0 info
maxconn 2000
{% if has_ddos_stick_tables %}
# Names this process as the local peer of the `peers` section further down.
# HAProxy drops a peers section that does not contain the local peer, and a
# stick-table that lost its peers section is not synchronized on reload.
localpeer localpeer
{% endif %}
# Request bodies are buffered so the SPOE message in coraza.cfg
# (`body=req.body`) carries the whole body: buffer size is the largest
# body Coraza can inspect. coraza-spoa refuses a frame larger than 64 KiB
Expand Down Expand Up @@ -277,17 +284,25 @@ frontend fe_https

{% endif %}

{% set has_ddos_backends = routes | selectattr("ddos") | list | length > 0 %}
{% if has_ddos_backends %}
{% if has_ddos_stick_tables %}
# --- DDoS protection stick-tables ---------------------------------
# Stick-tables are HAProxy process memory. Without a peers section a reloading
# worker starts with empty ones and silently drops every ban and rate counter,
# which includes every "Apply configuration". Listing them below makes the new
# worker pull their contents from the outgoing one during the master-worker
# handover (config_apply._reload_haproxy reloads through the master socket).
# A restarted or recreated container still starts empty: there is no
# predecessor left to synchronize from.
peers gp_stick_tables
peer localpeer 127.0.0.1:10001
{% for route in routes %}
{% if route.ddos and route.ddos.enabled %}
backend {{ route.ddos.stick_table_name }}
stick-table type ipv6 size 100k expire {{ route.ddos.rate_limit_window_seconds }}s store http_req_rate({{ route.ddos.rate_limit_window_seconds }}s),conn_cur
stick-table type ipv6 size 100k expire {{ route.ddos.rate_limit_window_seconds }}s peers gp_stick_tables store http_req_rate({{ route.ddos.rate_limit_window_seconds }}s),conn_cur

{% if route.ddos.auto_ban_enabled %}
backend {{ route.ddos.ban_stick_table_name }}
stick-table type ipv6 size 100k expire {{ route.ddos.ban_duration_seconds }}s store gpc0
stick-table type ipv6 size 100k expire {{ route.ddos.ban_duration_seconds }}s peers gp_stick_tables store gpc0

{% endif %}
{% endif %}
Expand Down
10 changes: 7 additions & 3 deletions src/backend/tests/unit/test_config_generator.py
Original file line number Diff line number Diff line change
Expand Up @@ -313,7 +313,7 @@ def test_generate_one_vhost_with_ddos_protection_enabled() -> None:

assert "backend st_ddos_vhost_5" in generated.haproxy_cfg
assert (
"stick-table type ipv6 size 100k expire 5s store "
"stick-table type ipv6 size 100k expire 5s peers gp_stick_tables store "
"http_req_rate(5s),conn_cur" in generated.haproxy_cfg
)
assert (
Expand Down Expand Up @@ -355,6 +355,10 @@ def test_generate_one_vhost_with_ddos_protection_disabled_emits_nothing() -> Non
assert "stick-table" not in generated.haproxy_cfg
assert "track-sc0" not in generated.haproxy_cfg
assert "deny_status 429" not in generated.haproxy_cfg
# No stick tables means no peers section, so the `localpeer` that names
# this process in it is omitted too.
assert "peers" not in generated.haproxy_cfg
assert "localpeer" not in generated.haproxy_cfg


# ---------------------------------------------------------------------------
Expand Down Expand Up @@ -536,8 +540,8 @@ def test_generate_one_vhost_with_auto_ban_enabled() -> None:

assert "backend st_ban_vhost_5" in generated.haproxy_cfg
assert (
"stick-table type ipv6 size 100k expire 300s store gpc0"
in generated.haproxy_cfg
"stick-table type ipv6 size 100k expire 300s "
"peers gp_stick_tables store gpc0" in generated.haproxy_cfg
)
assert (
"http-request track-sc1 src table st_ban_vhost_5 "
Expand Down
48 changes: 46 additions & 2 deletions src/backend/tests/unit/test_config_renderer_haproxy.py
Original file line number Diff line number Diff line change
Expand Up @@ -304,6 +304,8 @@ def test_haproxy_template_omits_ddos_blocks_when_disabled() -> None:
assert "stick-table" not in rendered
assert "track-sc0" not in rendered
assert "deny_status 429" not in rendered
assert "peers" not in rendered
assert "localpeer" not in rendered


def test_haproxy_template_renders_ddos_blocks_when_enabled() -> None:
Expand All @@ -329,7 +331,7 @@ def test_haproxy_template_renders_ddos_blocks_when_enabled() -> None:
assert (
"backend st_ddos_vhost_1\n"
" stick-table type ipv6 size 100k expire 10s "
"store http_req_rate(10s),conn_cur" in rendered
"peers gp_stick_tables store http_req_rate(10s),conn_cur" in rendered
)
assert (
"http-request track-sc0 src table st_ddos_vhost_1 "
Expand All @@ -346,6 +348,47 @@ def test_haproxy_template_renders_ddos_blocks_when_enabled() -> None:
assert "timeout http-request 5s" in rendered


def test_haproxy_template_wires_every_stick_table_to_a_peers_section() -> None:
"""Table contents must survive a reload; a peers section is what does it.

Stick-tables are HAProxy process memory, so without a peers section a
reloading worker starts empty and "Apply configuration" drops every ban and
rate-limit counter. HAProxy discards a peers section that does not name the
local peer, and a stick-table left pointing at a discarded section stops
synchronizing without an error — so the `localpeer` global directive and
the section have to be emitted together.
"""
rendered = render_haproxy_cfg(
HaproxyRenderContext(
routes=(
HaproxyRoute(
vhost_acl_name="host_api",
vhost_hosts=("api.example.com",),
ssl_provider="none",
backend=_backend("be_api", "api", "api-backend:9000"),
ddos=_ddos(
stick_table_name="st_ddos_vhost_1",
auto_ban_enabled=True,
ban_stick_table_name="st_ban_vhost_1",
),
),
),
)
)

assert "\n localpeer localpeer\n" in rendered
assert (
"\npeers gp_stick_tables\n"
" peer localpeer 127.0.0.1:10001\n" in rendered
)
# Both tables (rate limiting and auto-ban) must reference the section, and
# the section must be declared before the first table that uses it.
assert rendered.count(" peers gp_stick_tables store") == 2
assert rendered.index("peers gp_stick_tables") < rendered.index(
"backend st_ddos_vhost_1"
)


def test_haproxy_template_omits_ddos_for_disabled_route_only() -> None:
rendered = render_haproxy_cfg(
HaproxyRenderContext(
Expand Down Expand Up @@ -417,7 +460,8 @@ def test_haproxy_template_renders_auto_ban_blocks_when_enabled() -> None:

assert (
"backend st_ban_vhost_1\n"
" stick-table type ipv6 size 100k expire 600s store gpc0" in rendered
" stick-table type ipv6 size 100k expire 600s "
"peers gp_stick_tables store gpc0" in rendered
)
assert (
"http-request track-sc1 src table st_ban_vhost_1 "
Expand Down
Loading