HTTP/2: DataReceived for an already-closed stream raises KeyError and wedges the connection

Version: Hypercorn 0.18.0, h2 4.x, Python 3.13, asyncio worker

What happens

If an ASGI app sends its response without reading the whole request body, the client keeps sending the rest of the body. Once Hypercorn has closed the stream, the next DataReceived for it fails in H2Protocol._handle_events:

elif isinstance(event, h2.events.DataReceived):
    await self.streams[event.stream_id].handle(...)   # KeyError: stream already removed
    self.connection.acknowledge_received_data(...)    # never reached

The KeyError escapes TCPServer._read_data, so the connection's reader stops while the socket stays open. Every later request on that HTTP/2 connection is never read and hangs until the client or proxy gives up.

Behind a proxy that multiplexes many clients onto one h2c connection to the app (Google Cloud Run, in our case), one early-answered upload makes unrelated requests from other users hang. We saw this as requests running into the proxy's 900 s timeout.

The StreamEnded branch right below already handles this case ("Response sent before full request received, nothing to do already closed."). DataReceived doesn't.

Reproduction

An app that answers before reading the body, and an h2 client that sends a body larger than the stream window, then a second request on the same connection:

import asyncio, socket
import h2.config, h2.connection, h2.events, h2.exceptions
from hypercorn.asyncio import serve
from hypercorn.config import Config

async def app(scope, receive, send):
    if scope["type"] == "lifespan":
        while True:
            msg = await receive()
            if msg["type"] == "lifespan.startup":
                await send({"type": "lifespan.startup.complete"})
            elif msg["type"] == "lifespan.shutdown":
                await send({"type": "lifespan.shutdown.complete"}); return
    status = 429 if scope["path"] == "/upload" else 200   # answer without reading the body
    await send({"type": "http.response.start", "status": status, "headers": [(b"content-length", b"2")]})
    await send({"type": "http.response.body", "body": b"ok"})

async def client(port):
    reader, writer = await asyncio.open_connection("127.0.0.1", port)
    conn = h2.connection.H2Connection(h2.config.H2Configuration(client_side=True))
    conn.initiate_connection()
    hdrs = lambda m, p: [(":method", m), (":path", p), (":scheme", "http"), (":authority", "localhost")]
    up = conn.get_next_available_stream_id()
    conn.send_headers(up, hdrs("PUT", "/upload"))
    body, sent, second, statuses = b"x" * 1_048_576, 0, None, {}
    loop = asyncio.get_running_loop(); deadline = loop.time() + 5
    while loop.time() < deadline:
        if sent < len(body):
            try:
                n = min(conn.local_flow_control_window(up), conn.max_outbound_frame_size)
            except h2.exceptions.StreamClosedError:
                n, sent = 0, len(body)
            if n:
                chunk = body[sent:sent + n]
                conn.send_data(up, chunk, end_stream=sent + len(chunk) >= len(body)); sent += len(chunk)
        if up in statuses and second is None:
            second = conn.get_next_available_stream_id()
            conn.send_headers(second, hdrs("GET", "/next"), end_stream=True)
        writer.write(conn.data_to_send()); await writer.drain()
        if second in statuses: break
        try:
            data = await asyncio.wait_for(reader.read(65536), 0.3)
        except asyncio.TimeoutError:
            continue
        for ev in conn.receive_data(data):
            if isinstance(ev, h2.events.ResponseReceived):
                statuses[ev.stream_id] = int(dict(ev.headers)[b":status"])
    return statuses.get(second)

async def main():
    with socket.socket() as s:
        s.bind(("127.0.0.1", 0)); port = s.getsockname()[1]
    cfg = Config(); cfg.bind = [f"127.0.0.1:{port}"]
    stop = asyncio.Event()
    server = asyncio.create_task(serve(app, cfg, shutdown_trigger=stop.wait))
    await asyncio.sleep(0.5)
    print("second request status:", await client(port))   # expected 200
    stop.set(); await server

asyncio.run(main())

Output on 0.18.0: the server logs KeyError: 1 from hypercorn/protocol/h2.py (_handle_events, the DataReceived branch), and second request status: None.

Suggested fix

Acknowledge and drop data for a stream that is gone, as StreamEnded does:

elif isinstance(event, h2.events.DataReceived):
    try:
        await self.streams[event.stream_id].handle(Body(stream_id=event.stream_id, data=event.data))
    except KeyError:
        # Response sent before full request received; drop the data.
        pass
    self.connection.acknowledge_received_data(event.flow_controlled_length, event.stream_id)

Acknowledging it either way returns the connection-level flow-control window, so the client can keep sending on other streams. We're running the equivalent as a local patch, and it fixes the hang in the reproduction above.

Additional reproduction on Hypercorn 0.18.0, h2 4.4.1, h11 0.16.0, Python 3.14.7 on Linux and 3.14.3 on macOS, with the asyncio worker and h2c prior knowledge.

This isolates the removed-stream branch without a large request: send 1 KiB to an early-answer path, wait for its complete 413 and stream end, wait 0.2 seconds, then send 4,096 more bytes with END_STREAM on stream 1 and GET /next on stream 3. Observed: the connection ends before stream 3 answers (about 0.203 seconds in the local run), and the server logs KeyError: 1 in H2Protocol._handle_events at the self.streams[event.stream_id] access.

Save the application below as early_app.py and the client as client.py. Install hypercorn==0.18.0 h2==4.4.1, then run:

hypercorn --bind 127.0.0.1:8080 --workers 1 --graceful-timeout 4 early_app:app

Application:

async def app(scope, receive, send):
    if scope["type"] == "lifespan":
        while True:
            message = await receive()
            if message["type"] == "lifespan.startup":
                await send({"type": "lifespan.startup.complete"})
            elif message["type"] == "lifespan.shutdown":
                await send({"type": "lifespan.shutdown.complete"})
                return
    if scope["type"] == "http":
        # Refuse one request without reading its body; ordinary requests succeed.
        status = 413 if scope["path"] == "/early" else 200
        await send({"type": "http.response.start", "status": status,
                    "headers": [(b"content-length", b"2")]})
        await send({"type": "http.response.body", "body": b"ok"})

Client:

import json
import socket
import sys
import time

import h2.config
import h2.connection
import h2.events


def probe(port, case, before_close=None):
    connection = h2.connection.H2Connection(
        config=h2.config.H2Configuration(client_side=True, header_encoding="utf-8")
    )
    connection.initiate_connection()

    def request(stream, method, path, end):
        connection.send_headers(stream, [(":method", method), (":scheme", "http"),
            (":authority", "localhost"), (":path", path)], end_stream=end)

    with socket.create_connection(("127.0.0.1", port), timeout=2) as sock:
        request(1, "POST", "/early", False)
        if case == "unread":
            for _ in range(40):
                connection.send_data(1, b"x" * 1024)
            request(3, "GET", "/next", True)
        else:
            connection.send_data(1, b"x" * 1024)
        sock.sendall(connection.data_to_send())
        statuses, ended, late_sent = {}, set(), False
        started = time.monotonic()
        deadline = started + 8
        error = None
        while 3 not in ended and time.monotonic() < deadline:
            if case == "removed" and 1 in ended and not late_sent:
                time.sleep(0.2)
                connection.send_data(1, b"x" * 4096, end_stream=True)
                request(3, "GET", "/next", True)
                sock.sendall(connection.data_to_send())
                late_sent = True
            sock.settimeout(max(0.001, deadline - time.monotonic()))
            try:
                data = sock.recv(65536)
                if not data:
                    error = "EOF before the following response"
                    break
                for event in connection.receive_data(data):
                    if isinstance(event, h2.events.ResponseReceived):
                        statuses[event.stream_id] = dict(event.headers)[":status"]
                    elif isinstance(event, h2.events.DataReceived):
                        connection.acknowledge_received_data(
                            event.flow_controlled_length, event.stream_id)
                    elif isinstance(event, h2.events.StreamEnded):
                        ended.add(event.stream_id)
                sock.sendall(connection.data_to_send())
            except TimeoutError:
                error = "No following response within 8 seconds"
                break
            except ConnectionError as exc:
                error = type(exc).__name__
                break
        result = {"case": case, "statuses": statuses, "ended": sorted(ended),
                  "next_status": statuses.get(3), "error": error,
                  "elapsed_seconds": round(time.monotonic() - started, 4)}
        if before_close is not None:
            before_close(result)
        return result


if __name__ == "__main__":
    print(json.dumps(probe(int(sys.argv[1]), sys.argv[2])))

In another terminal:

python client.py 8080 removed

Expected: stream 3 completes with 200. The late request data must still return the connection flow-control window even after stream 1 is removed. With #391's src/hypercorn/protocol/h2.py change applied to 0.18.0, stream 3 completes with 200 here.

If a fix also resets the answered stream with NO_ERROR to ask the sender to stop, two details matter. A stream leaves self.streams when the application finishes, which can be before END_STREAM is sent, so resetting on that alone can cut off the end of the response; wait until the stream has also left self.stream_buffers. And once the connection has sent GOAWAY, reset_stream raises ProtocolError, which handling StreamClosedError does not catch.

Sending forty unread frames together instead (unread) exposes a separate full input-queue/cleanup stall, reported in #393. #391 does not change that result, so these defects need independent fixes.

Navigation Menu

Repository navigation

HTTP/2: DataReceived for an already-closed stream raises KeyError and wedges the connection #389

What happens

Reproduction

Suggested fix

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

HTTP/2: DataReceived for an already-closed stream raises KeyError and wedges the connection #389

Description

What happens

Reproduction

Suggested fix

Activity

tahsinrk commented on Nov 2, 2026

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions