Repository navigation
Conversation
A resume() from a subscriber thread only takes effect later on AHC's event loop, after parts that the loop may still be decoding, and can then let one more socket read through with no demand left. Play WS prevented that by capturing the event loop through Netty's internal ThreadExecutorMap and re-running its check there. AHC 3.0.15 adds ResponseBodyControl.execute, which runs a task in sequence with the body callbacks, so upgrade to it and pass the check to execute instead. A check that runs on the thread that asked for it, inline on the event loop or with a control that applies calls synchronously, resumes directly; one queued from another thread runs a drain() pass on the event loop, as before. At most one queued check is pending at a time. The fake controls in the publisher tests now model execute as well.
With play.ws.ahc.useCookieStore = true, AHC 3.0.14 re-encoded a raw Cookie header on redirects and authentication retries: pairs could be reordered, valueless or unbalanced-quote pairs were dropped, and a value that AHC's strict encoder rejected failed the hop. AHC 3.0.15 keeps the header as written and only takes out the pairs whose cookie a response replaced or deleted. Update the README, and test a redirect that replaces one raw pair and keeps a valueless one.
AHC 3.0.15 hands on decompressed HTTP/1.1 body parts of at most 64 KiB and pauses decompression while the response is suspended. Before, every socket read of up to 64 KiB was inflated and delivered in full, so a highly compressible body queued many MiB beyond demand. Update the README; HTTP/2 still inflates each received DATA frame in full. StreamedGzipBufferingSpec now requires that nothing is delivered beyond demand: with early demand for one part, AHC delivers exactly that part. Before, 1,875,916 bytes were delivered beyond demand.
Before AHC 3.0.15, a ResponseBodyControl.suspend() queued from another thread could take effect after a newer resume() made on the event loop and stall a demanded response. That was the reason ResponseBodyPublisher suspends only on the event loop. AHC 3.0.15 skips such a suspend. Suspending on the event loop is still the better place: it takes effect before the next body part, whereas a suspend from another thread would only take effect after further parts. Update the comment and the description of ResponseBodyPublisherSuspendOnLoopSpec; the code and the tests are unchanged.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
project/Dependencies.scalaalready asks for AHC 3.0.15, so CI cannot pass before that release.This builds on these AsyncHttpClient pull requests:
The first commit needs AsyncHttpClient/async-http-client#2359. Each other commit needs one of the other three and is dropped if that one misses the release:
Related, but not needed for this PR:
It is a prerequisite for bounding HTTP/2 decompression in a later AHC change; AHC would still have to handle the output left from the final DATA frame. Until then, the README's HTTP/2 caveat in the
maxDecompressedResponseSizesection stays.Play WS now re-checks its resume decisions through AHC's new public
ResponseBodyControl.executeinstead of reaching AHC's event loop through Netty's internalThreadExecutorMap. It also documents two AHC changes that Play WS users can see.Streamed-body resumes
ResponseBodyPublisherresumes transport reads when the subscriber has demand and nothing is buffered. Aresume()from a subscriber thread only takes effect later on the event loop, after parts that the loop may still be decoding. Since #1481, Play WS has prevented that stale resume by capturing the event loop withThreadExecutorMap.currentExecutor()and re-running the check there.ThreadExecutorMapis internal to Netty.AHC 3.0.15 adds
ResponseBodyControl.execute(Runnable), which runs a task on the event loop that delivers the response body, in sequence with the callbacks made there: inline when called on that event loop, queued to it from any other thread. Play WS now passes its check toexecute:drain()that asked has just decided on the current state.drain()pass there, which resumes only if demand is still unmet.There is no behavior change for users. The existing tests for lost resumes, stale resumes and on-loop suspension still pass; only their fake controls changed, to model
execute.Play WS keeps suspending only on the event loop, so each suspension takes effect before the next body part. Before AHC 3.0.15, that was also needed because a suspend queued from another thread could undo a newer resume and stall the stream; AHC 3.0.15 skips such a suspend. A separate commit updates the comment and the spec's description; the code is unchanged.
AHC changes visible to Play WS users
Cookieheaders withplay.ws.ahc.useCookieStore = true: redirects and authentication retries now keep the header as written, except for pairs whose cookie a response replaced or deleted. Before, they re-encoded it: pairs could be reordered, valueless or unbalanced-quote pairs were dropped, and a value that AHC's strict encoder rejected failed the redirect or retry. A new redirect test pins this.maxDecompressedResponseSizesection is updated, andStreamedGzipBufferingSpecnow asserts that nothing is delivered beyond demand. It measures delivered parts, not memory inside the decompressor.Before marking ready
nettyVersioninproject/Dependencies.scalaagainst the Netty version of that AHC release. Play WSmainis on 4.2.19.Final since netty-buffer, netty-codec-base, ... 4.2.19.Final (was 4.2.18.Final) #1484, while AHC still pins 4.2.18.Final.HTTPSalso offersh2" paragraph against Plans for thefix/proxy-hop-alpnbranch AsyncHttpClient/async-http-client#2358.Test plan
Local runs, on JDK 17 unless stated otherwise, with a temporary local commit (not part of this PR) that resolves AHC from a local snapshot. CI cannot run before the AHC release.
Against a local AHC build that merges all four AHC pull requests (1,853 AHC tests pass):
sbt validateCode.play-ahc-ws-standalone/testFullandintegration-tests/testFull, 18 and 17 test classes, all pass, none served from the test cache. MiMa passes.scala-3.3.8: the same suites pass.StreamedGzipBufferingSpec: 0 bytes delivered beyond demand in each of its three runs.mainwith Netty 4.2.19 (netty-buffer, netty-codec-base, ... 4.2.19.Final (was 4.2.18.Final) #1484):validateCode, both suites on Scala 2.13 and 3.3.8, and MiMa pass.Against the previous revision of the AHC changes, which differed only in the decompressor's multi-member gzip support and part splitting, and in Javadoc and tests:
docpasses.ResponseBodyPublisherStressSpec, 3,000 iterations, 0 stalls;StreamingStallStressSpec, 300 HTTP/1.1 and 300 HTTP/2 iterations with mixed consumers, 0 stalls and 0 wrong sizes.Against a local AHC build of AsyncHttpClient/async-http-client#2359 alone (Scala 2.13, before the decompression commit):
flagpair is dropped.StreamedGzipBufferingSpec, with its earlier bound of 10 MiB: 1,875,916 bytes beyond demand, the same as with theThreadExecutorMapworkaround. With Play WS resuming directly instead of throughexecute, 35,589,292 bytes, and the test fails.The decompression commit's bound of 0 bytes holds only with AHC's demand-bounded decompression.