Add brotli compressor and decompressor filters - #194
Open
tobiass-threedy wants to merge 1 commit into
Open
Conversation
Add boost/iostreams/filter/brotli.hpp and src/brotli.cpp, symmetric filters wrapping the Brotli encoder and decoder, modelled on the existing zstd filters. Wired into the CMake and Boost.Build builds, documented in doc/installation.html and doc/bibliography.html, and covered by test/brotli_test.cpp. The implementation is not mine. It was posted by SSE4 as a work-in-progress patch on the brotli feature request: boostorg#175 (comment) That patch carried "(C) Copyright Reimar Doeffinger 2024" headers, but the attribution looks like an artefact of how it was produced: after a mechanical zstd->brotli rename, brotli.hpp is 91% line-identical to zstd.hpp and brotli_test.cpp is 83% line-identical to zstd_test.cpp, the three headers match the zstd trio's exact forms with only the year changed from 2018, the "Based on ..." chains were dropped, and even the comment "basically a copy-paste of the gzip test" is inherited verbatim from zstd_test.cpp. So the copyright lines have been restated as SSE4's, over the real derivation chain back through zstd.hpp, lzma.hpp and fusecompress. Only src/brotli.cpp, at 35% shared lines, is substantially new code. Changes made to the posted patch: * inflate() reported success whenever it was asked to finish, without consulting BrotliDecoderIsFinished(). Input ending in the middle of a brotli stream therefore decoded as though it were complete. It now raises brotli_error(brotli::truncated_input) when the decoder wants more input and the caller has said there is none, while still letting the caller supply a fresh output buffer when that is all that is missing. Covered by the new truncated_input_test, which fails without this change. * deflate() only drained the encoder for brotli::finish, so the declared brotli::flush code was never honoured. It now drains for any action other than brotli::run, matching zstd_base::deflate(). * deflate() could also spin forever: it looped while the encoder had more output without checking there was anywhere to put it. It now stops when the output buffer is full so the caller can supply a fresh one. * brotli_error stored its code as size_t but exposed it as int, which MSVC flags as C4267. Brotli's decoder error codes are a signed enum, so the code is now int throughout, which also removes two casts. * brotli_base's constructor initialised its members out of declaration order (-Wreorder). * The CMake integration compiled src/brotli.cpp whenever brotli was found, whether or not BOOST_IOSTREAMS_ENABLE_BROTLI was set, and ran pkg-config unconditionally. It now mirrors boost_iostreams_option(): the option defaults to ON only when brotli is present, forcing it ON without brotli is a hard error, and nothing is compiled when it is OFF. Brotli installs no CMake package configuration file upstream, so detection tries the vcpkg 'unofficial-brotli' package and falls back to pkg-config. * test/CMakeLists.txt is new to this library, and CMakeLists.txt already had a dormant "if(EXISTS test/CMakeLists.txt) add_subdirectory(test)" guard, so adding the file switched the CMake test path on for everyone while providing only this one test. It now returns early unless both BOOST_IOSTREAMS_ENABLE_BROTLI and Boost::unit_test_framework are present, and registers the test with add_test() so ctest runs it. Note that "using brotli ;" needs a b2 whose engine provides the "args" module; brotli.jam was added in bfgroup/b2#552 on 2026-05-03 and is not in any released Boost. The CMake build is unaffected. Tested on Windows against brotli 1.1.0/1.2.0: * g++ 10.3 and MSVC 19.44 (VS 2022), all six test cases pass under both; truncated_input_test fails when the inflate() fix is reverted. * CMake: configured and built in all four option states (found/default ON, found/forced OFF, missing/forced ON -> hard error, missing/default OFF). * Boost.Build: b2 with MSVC compiles src/brotli.cpp into libboost_iostreams and links against the brotli libraries via brotli.jam.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add boost/iostreams/filter/brotli.hpp and src/brotli.cpp, symmetric filters wrapping the Brotli encoder and decoder, modelled on the existing zstd filters. Wired into the CMake and Boost.Build builds, documented in doc/installation.html and doc/bibliography.html, and covered by test/brotli_test.cpp.
The implementation is not mine. It was posted by SSE4 as a work-in-progress patch on the brotli feature request:
#175 (comment)
That patch carried "(C) Copyright Reimar Doeffinger 2024" headers, but the attribution looks like an artefact of how it was produced: after a mechanical zstd->brotli rename, brotli.hpp is 91% line-identical to zstd.hpp and brotli_test.cpp is 83% line-identical to zstd_test.cpp, the three headers match the zstd trio's exact forms with only the year changed from 2018, the "Based on ..." chains were dropped, and even the comment "basically a copy-paste of the gzip test" is inherited verbatim from zstd_test.cpp. So the copyright lines have been restated as SSE4's, over the real derivation chain back through zstd.hpp, lzma.hpp and fusecompress. Only src/brotli.cpp, at 35% shared lines, is substantially new code.
Changes made to the posted patch:
inflate() reported success whenever it was asked to finish, without consulting BrotliDecoderIsFinished(). Input ending in the middle of a brotli stream therefore decoded as though it were complete. It now raises brotli_error(brotli::truncated_input) when the decoder wants more input and the caller has said there is none, while still letting the caller supply a fresh output buffer when that is all that is missing. Covered by the new truncated_input_test, which fails without this change.
deflate() only drained the encoder for brotli::finish, so the declared brotli::flush code was never honoured. It now drains for any action other than brotli::run, matching zstd_base::deflate().
deflate() could also spin forever: it looped while the encoder had more output without checking there was anywhere to put it. It now stops when the output buffer is full so the caller can supply a fresh one.
brotli_error stored its code as size_t but exposed it as int, which MSVC flags as C4267. Brotli's decoder error codes are a signed enum, so the code is now int throughout, which also removes two casts.
brotli_base's constructor initialised its members out of declaration order (-Wreorder).
The CMake integration compiled src/brotli.cpp whenever brotli was found, whether or not BOOST_IOSTREAMS_ENABLE_BROTLI was set, and ran pkg-config unconditionally. It now mirrors boost_iostreams_option(): the option defaults to ON only when brotli is present, forcing it ON without brotli is a hard error, and nothing is compiled when it is OFF. Brotli installs no CMake package configuration file upstream, so detection tries the vcpkg 'unofficial-brotli' package and falls back to pkg-config.
test/CMakeLists.txt is new to this library, and CMakeLists.txt already had a dormant "if(EXISTS test/CMakeLists.txt) add_subdirectory(test)" guard, so adding the file switched the CMake test path on for everyone while providing only this one test. It now returns early unless both BOOST_IOSTREAMS_ENABLE_BROTLI and Boost::unit_test_framework are present, and registers the test with add_test() so ctest runs it.
Note that "using brotli ;" needs a b2 whose engine provides the "args" module; brotli.jam was added in bfgroup/b2#552 on 2026-05-03 and is not in any released Boost. The CMake build is unaffected.
Tested on Windows against brotli 1.1.0/1.2.0: