Repository navigation
C API: Rework and enhance the PyBytesWriter implementation #158585
Description
Activity
Currently in the main branch, there are 87 calls to
PyBytesWriter_Create(). Only 15 resize the writer later:_PyBytes_FormatEx(): callPyBytesWriter_GrowAndUpdatePointer()_PyBytes_FromIterator(): call_PyBytesWriter_ResizeAndUpdatePointer()_PyUnicode_EncodeCharmap(): callPyBytesWriter_Resize()_PyUnicode_EncodeIconv(): callPyBytesWriter_Grow()_PyUnicode_EncodeUTF16(): callPyBytesWriter_GrowAndUpdatePointer()_PyUnicode_EncodeUTF32(): callPyBytesWriter_GrowAndUpdatePointer()_io.FileIO.readall(): callPyBytesWriter_WriteBytes()multiple times_io._IOBase.readline(): callPyBytesWriter_WriteBytes()multiple times_io._RawIOBase.readall(): callPyBytesWriter_WriteBytes()multiple timesassemble.c (3 writers): callPyBytesWriter_Resize()codeobject.c: remove_column_info(): callPyBytesWriter_Resize(res, PyBytesWriter_GetSize(res) * 2)encode_code_page_errors(): callPyBytesWriter_GrowAndUpdatePointer()raw_unicode_escape(): callPyBytesWriter_GrowAndUpdatePointer()unicode_encode_ucs1(): callPyBytesWriter_GrowAndUpdatePointer()unicode_encode_utf8(): callPyBytesWriter_GrowAndUpdatePointer()
- added 4 commits that reference this issue
on Oct 2, 2026 I ran benchmarks on the following code to measure the
PyBytesWriteroverhead overPyBytes_FromStringAndSize(NULL, size).I didn't find any obvious way to optimize
PyBytesWriter.Benchmark on:
const Py_ssize_t size = 3; PyBytesWriter *writer = PyBytesWriter_Create(size); if (writer == NULL) { return NULL; } char *str = PyBytesWriter_GetData(writer); memset(str, 'x', size); return PyBytesWriter_Finish(writer);
I tried to add a freelist to bytes for sizes in range [0; 255] (bytes): see draft PR #158657.
Benchmark with size=3 bytes:
- Main branch: 29.8 ns +- 0.3 ns
- Add bytes freelist: 28.1 ns +- 0.0 ns
- Remove PyBytesWriter freelist: 37.8 ns +- 1.0 ns
Benchmark with size=64 bytes:
- Main branch: 31.0 ns +- 0.9 ns
- No writer small buffer: 30.2 ns +- 0.3 ns
- Add bytes freelist: 27.8 ns +- 0.5 ns
- Add bytes freelist, no writer small buffer: 25.0 ns +- 0.2 ns
- Remove PyBytesWriter freelist: 40.4 ns +- 4.4 ns
- Remove PyBytesWriter freelist, no writer small buffer: 37.5 ns +- 0.4 ns
Benchmark with size=255 bytes:
- Main branch: 35.5 ns +- 0.9 ns
- Add bytes freelist: 31.6 ns +- 1.7 ns
- Add bytes freelist, no writer small buffer: 27.6 ns +- 0.2 ns
- Remove PyBytesWriter freelist: 50.1 ns +- 0.7 ns
Benchmark with size=300 bytes (don't use writer small buffer):
- Main branch: 33.5 ns +- 0.5 ns
- Remove PyBytesWriter freelist: 40.8 ns +- 0.6 ns
Notes:
- Removing the PyBytesWriter freelist makes all benchmarks slower. So the freelist is more efficient than always calling
PyMem_Malloc()._Py_freelists_GET()costs a few nanoseconds, but it's acceptable. - The speed up of adding a freelist to bytes is not very obvious to me. It only saves 3.1 ns on size=64 bytes (31.0 ns => 27.8 ns). On size=3, it only saves 1.7 ns. Adding a bytes freelist can increase Python memory usage.
- Changing byteswriter_resize() allocation strategy to not use the small buffer saves 0.8 ns with size=64. Using the writer small buffer or not doesn't seem to really impact the performance. There is no major difference on performance.
1 remaining item
- added 7 commits that reference this issue
on Oct 3, 2026 I ran #158665 benchmark (create the string
b'abc') to compare the 3.15 branch and the (current) main branches:Mean +- std dev: [py315] 41.6 ns +- 1.0 ns -> [main] 32.5 ns +- 0.6 ns: 1.28x fasterOh nice, the
PyBytesWriteroverhead is now way smaller on the main branch! The main branch is 9.1 ns faster.I wrote an article on this work (and PyUnicodeWriter work): https://vstinner.github.io/optimize-pybyteswriter-pyunicodewriter-implementation.html.
Placeholder issue to keep track of changes to rework and enhance the
PyBytesWriterimplementation.Linked PRs