Bug: Incorrect floatToIntScaleFactor in FLACWriter breaks 24-bit audio encoding

[manol-o]

Hi,

had problem and with FLAC writing and with help of AI generated this bug report:

There is a mathematical error in the FLACWriter constructor inside choc_AudioFileFormat_FLAC.h that calculates the scaling factor for floating-point to integer conversion.

Currently, the scaling factor works for 16-bit audio by mathematical coincidence, but it drastically reduces the volume of 24-bit audio (by about -96 dB) because it scales to a maximum amplitude of 127 instead of 8,388,607.

The Bug

In the FLACWriter constructor, the scale factor is calculated as:

auto bits = encoder->protected_->bits_per_sample;
CHOC_ASSERT (bits != 0);
floatToIntScaleFactor = static_cast<double> ((1u << (31u - bits)) - 1);

Later, in appendFramesForType, this scale is applied:

dst[i] = static_cast<int32_t> (scale * s); // s is the float sample in [-1, 1]

libFLAC's FLAC__stream_encoder_process expects integer samples right-justified to bits_per_sample. Therefore, the full-scale maximum should be (1 << (bits - 1)) - 1.

Because the formula currently uses 31 - bits, it produces incorrect results for anything other than 16-bit audio:

• 16-bit: 31 - 16 = 15. 2^15 - 1 = 32,767. (Coincidentally correct, which is likely why this slipped through).
• 24-bit: 31 - 24 = 7. 2^7 - 1 = 127. (Incorrect. It should be 24 - 1 = 23, yielding 2^23 - 1 = 8,388,607).

Because choc::audio::BitDepth::int24 is commonly passed in, a full-scale 1.0 float sample gets encoded with a value of just 127.

Proposed Solution

Update the floatToIntScaleFactor calculation to use bits - 1u instead of 31u - bits.

// Corrected formula
floatToIntScaleFactor = static_cast<double> ((1u << (bits - 1u)) - 1);

Environment:

• Component: choc/audio/choc_AudioFileFormat_FLAC.h
• Impacted formats: Any non-16-bit FLAC encoding (e.g., 24-bit, 8-bit).