Skip to content

Add brotli compressor and decompressor filters - #194

Open
tobiass-threedy wants to merge 1 commit into
boostorg:developfrom
tobiass-threedy:feature/brotli
Open

Add brotli compressor and decompressor filters#194
tobiass-threedy wants to merge 1 commit into
boostorg:developfrom
tobiass-threedy:feature/brotli

Conversation

@tobiass-threedy

Copy link
Copy Markdown

Add boost/iostreams/filter/brotli.hpp and src/brotli.cpp, symmetric filters wrapping the Brotli encoder and decoder, modelled on the existing zstd filters. Wired into the CMake and Boost.Build builds, documented in doc/installation.html and doc/bibliography.html, and covered by test/brotli_test.cpp.

The implementation is not mine. It was posted by SSE4 as a work-in-progress patch on the brotli feature request:

#175 (comment)

That patch carried "(C) Copyright Reimar Doeffinger 2024" headers, but the attribution looks like an artefact of how it was produced: after a mechanical zstd->brotli rename, brotli.hpp is 91% line-identical to zstd.hpp and brotli_test.cpp is 83% line-identical to zstd_test.cpp, the three headers match the zstd trio's exact forms with only the year changed from 2018, the "Based on ..." chains were dropped, and even the comment "basically a copy-paste of the gzip test" is inherited verbatim from zstd_test.cpp. So the copyright lines have been restated as SSE4's, over the real derivation chain back through zstd.hpp, lzma.hpp and fusecompress. Only src/brotli.cpp, at 35% shared lines, is substantially new code.

Changes made to the posted patch:

  • inflate() reported success whenever it was asked to finish, without consulting BrotliDecoderIsFinished(). Input ending in the middle of a brotli stream therefore decoded as though it were complete. It now raises brotli_error(brotli::truncated_input) when the decoder wants more input and the caller has said there is none, while still letting the caller supply a fresh output buffer when that is all that is missing. Covered by the new truncated_input_test, which fails without this change.

  • deflate() only drained the encoder for brotli::finish, so the declared brotli::flush code was never honoured. It now drains for any action other than brotli::run, matching zstd_base::deflate().

  • deflate() could also spin forever: it looped while the encoder had more output without checking there was anywhere to put it. It now stops when the output buffer is full so the caller can supply a fresh one.

  • brotli_error stored its code as size_t but exposed it as int, which MSVC flags as C4267. Brotli's decoder error codes are a signed enum, so the code is now int throughout, which also removes two casts.

  • brotli_base's constructor initialised its members out of declaration order (-Wreorder).

  • The CMake integration compiled src/brotli.cpp whenever brotli was found, whether or not BOOST_IOSTREAMS_ENABLE_BROTLI was set, and ran pkg-config unconditionally. It now mirrors boost_iostreams_option(): the option defaults to ON only when brotli is present, forcing it ON without brotli is a hard error, and nothing is compiled when it is OFF. Brotli installs no CMake package configuration file upstream, so detection tries the vcpkg 'unofficial-brotli' package and falls back to pkg-config.

  • test/CMakeLists.txt is new to this library, and CMakeLists.txt already had a dormant "if(EXISTS test/CMakeLists.txt) add_subdirectory(test)" guard, so adding the file switched the CMake test path on for everyone while providing only this one test. It now returns early unless both BOOST_IOSTREAMS_ENABLE_BROTLI and Boost::unit_test_framework are present, and registers the test with add_test() so ctest runs it.

Note that "using brotli ;" needs a b2 whose engine provides the "args" module; brotli.jam was added in bfgroup/b2#552 on 2026-05-03 and is not in any released Boost. The CMake build is unaffected.

Tested on Windows against brotli 1.1.0/1.2.0:

  • g++ 10.3 and MSVC 19.44 (VS 2022), all six test cases pass under both; truncated_input_test fails when the inflate() fix is reverted.
  • CMake: configured and built in all four option states (found/default ON, found/forced OFF, missing/forced ON -> hard error, missing/default OFF).
  • Boost.Build: b2 with MSVC compiles src/brotli.cpp into libboost_iostreams and links against the brotli libraries via brotli.jam.

Add boost/iostreams/filter/brotli.hpp and src/brotli.cpp, symmetric filters
wrapping the Brotli encoder and decoder, modelled on the existing zstd
filters. Wired into the CMake and Boost.Build builds, documented in
doc/installation.html and doc/bibliography.html, and covered by
test/brotli_test.cpp.

The implementation is not mine. It was posted by SSE4 as a work-in-progress
patch on the brotli feature request:

  boostorg#175 (comment)

That patch carried "(C) Copyright Reimar Doeffinger 2024" headers, but the
attribution looks like an artefact of how it was produced: after a mechanical
zstd->brotli rename, brotli.hpp is 91% line-identical to zstd.hpp and
brotli_test.cpp is 83% line-identical to zstd_test.cpp, the three headers
match the zstd trio's exact forms with only the year changed from 2018, the
"Based on ..." chains were dropped, and even the comment "basically a
copy-paste of the gzip test" is inherited verbatim from zstd_test.cpp. So the
copyright lines have been restated as SSE4's, over the real derivation chain
back through zstd.hpp, lzma.hpp and fusecompress. Only src/brotli.cpp, at 35%
shared lines, is substantially new code.

Changes made to the posted patch:

* inflate() reported success whenever it was asked to finish, without
  consulting BrotliDecoderIsFinished(). Input ending in the middle of a brotli
  stream therefore decoded as though it were complete. It now raises
  brotli_error(brotli::truncated_input) when the decoder wants more input and
  the caller has said there is none, while still letting the caller supply a
  fresh output buffer when that is all that is missing. Covered by the new
  truncated_input_test, which fails without this change.

* deflate() only drained the encoder for brotli::finish, so the declared
  brotli::flush code was never honoured. It now drains for any action other
  than brotli::run, matching zstd_base::deflate().

* deflate() could also spin forever: it looped while the encoder had more
  output without checking there was anywhere to put it. It now stops when the
  output buffer is full so the caller can supply a fresh one.

* brotli_error stored its code as size_t but exposed it as int, which MSVC
  flags as C4267. Brotli's decoder error codes are a signed enum, so the code
  is now int throughout, which also removes two casts.

* brotli_base's constructor initialised its members out of declaration order
  (-Wreorder).

* The CMake integration compiled src/brotli.cpp whenever brotli was found,
  whether or not BOOST_IOSTREAMS_ENABLE_BROTLI was set, and ran pkg-config
  unconditionally. It now mirrors boost_iostreams_option(): the option
  defaults to ON only when brotli is present, forcing it ON without brotli is
  a hard error, and nothing is compiled when it is OFF. Brotli installs no
  CMake package configuration file upstream, so detection tries the vcpkg
  'unofficial-brotli' package and falls back to pkg-config.

* test/CMakeLists.txt is new to this library, and CMakeLists.txt already had a
  dormant "if(EXISTS test/CMakeLists.txt) add_subdirectory(test)" guard, so
  adding the file switched the CMake test path on for everyone while providing
  only this one test. It now returns early unless both
  BOOST_IOSTREAMS_ENABLE_BROTLI and Boost::unit_test_framework are present, and
  registers the test with add_test() so ctest runs it.

Note that "using brotli ;" needs a b2 whose engine provides the "args" module;
brotli.jam was added in bfgroup/b2#552 on 2026-05-03 and is not in any released
Boost. The CMake build is unaffected.

Tested on Windows against brotli 1.1.0/1.2.0:

* g++ 10.3 and MSVC 19.44 (VS 2022), all six test cases pass under both;
  truncated_input_test fails when the inflate() fix is reverted.
* CMake: configured and built in all four option states (found/default ON,
  found/forced OFF, missing/forced ON -> hard error, missing/default OFF).
* Boost.Build: b2 with MSVC compiles src/brotli.cpp into libboost_iostreams
  and links against the brotli libraries via brotli.jam.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant