GCS: `get()` fails with `MissingContentLength` on chunked responses (`Content-Encoding: gzip`)

[nkemnitz]

Describe the bug

ObjectStore::get on the GCS backend fails with Error::Generic { store: "GCS", source: Header { source: MissingContentLength } } for any object stored with Content-Encoding: gzip. GCS serves these via Transfer-Encoding: chunked with no Content-Length, and object_store treats a missing Content-Length as fatal.

It manifests in two ways, and crucially a client cannot fully avoid it:

• Default reads (no Accept-Encoding): GCS applies decompressive transcoding — it decompresses the object server-side and streams the result chunked, with no Content-Length. Every gzip object fails, at any size.
• Even with Accept-Encoding: gzip (which returns the raw stored bytes): objects whose stored size exceeds ~8 MiB are still served chunked with no Content-Length (empirically the cutover is between 9 and 10 MB), so they fail too. Only small objects read with Accept-Encoding: gzip succeed.

So whether you receive the transcoded (uncompressed) bytes or the raw (compressed) bytes, the response is chunked with no Content-Length — and object_store rejects it before returning any data. head() and range reads on gzip objects fail as well.

These are valid HTTP/1.1 responses: a chunked body is self-delimiting, and RFC 9112 §6.2 states a sender MUST NOT send Content-Length together with Transfer-Encoding;
RFC 9112 §6.3 says the chunked framing determines the body length and overrides any Content-Length. object_store is therefore requiring a header the spec forbids on exactly these responses.

To Reproduce

Observe the offending response with no credentials against Google's public demo data (the ?cb= cache-buster forces origin past the public edge cache):

$ curl -sD - -o /dev/null \
  "https://storage.googleapis.com/neuroglancer-public-data/kasthuri2011/ground_truth/6_6_30/5376-5440_6656-6720_896-960?cb=$RANDOM" \
  | grep -iE "HTTP/|content-length|transfer-encoding|x-goog-stored-content-encoding"

HTTP/2 200
transfer-encoding: chunked
x-goog-stored-content-encoding: gzip
# (no content-length)

object_store then fails on that same public object (authenticated read — a service account or on GCP; verified with object_store 0.13.1 and main de0029a):

use object_store::gcp::GoogleCloudStorageBuilder;
use object_store::{path::Path, ObjectStore, ObjectStoreExt};

let store = GoogleCloudStorageBuilder::new()
    .with_bucket_name("neuroglancer-public-data")
    .with_service_account_path(sa) // or workload identity on GCP
    .build()?;
store
    .get(&Path::from(
        "kasthuri2011/ground_truth/6_6_30/5376-5440_6656-6720_896-960",
    ))
    .await?; // Err: Generic { store: "GCS", source: Header { source: MissingContentLength } }

For a self-contained object of any size in your own bucket (a >8 MiB one fails even with Accept-Encoding: gzip):

head -c 20000000 /dev/urandom | gzip | gsutil -h "Content-Encoding:gzip" cp - gs://YOUR_BUCKET/big.gz

Note: the bundled fake-gcs-server does not emulate decompressive transcoding (it always
returns a Content-Length), so it cannot reproduce this. The crate's own MockServer can — push a
response with a chunked body and no Content-Length header.

Expected behavior

A full GET of a valid, self-delimiting (chunked / HTTP-2) response should succeed by reading the body to completion rather than requiring a Content-Length header that the HTTP spec forbids on chunked responses.

Additional context

• Affects gzip only; br/zstd/uncompressed objects are served identity (with Content-Length) and read fine.
• Root cause: header_meta requires CONTENT_LENGTH unconditionally (header.rs#L144, in header_meta L114), and the GET path derives ObjectMeta.size/range from it before streaming
  (get.rs#L314 → #L333).
• Wire evidence (>8 MiB gzip object, authenticated, Accept-Encoding: gzip):
  Transfer-Encoding: chunked, no Content-Length, x-goog-stored-content-encoding: gzip,
  x-goog-stored-content-length: <n>.

Prior art — every other major GCS client reads to EOF (commit-pinned)

client	behavior	source
google-cloud-storage (Python)	streams via response.iter_content(); x-goog-stored-content-length only a retry heuristic	download.py#L145, #L163
cloud.google.com/go/storage (Go)	Reader.Remain() returns -1 for chunked / Decompressed; CRC skipped on transcoding	reader.go#L436, #L74, http_client.go#L1455
google-cloud-cpp (Apache Arrow C++)	reads to EOF via HasUnreadData(); falls back to x-goog-stored-content-length for size	object_read_source.cc#L114, #L55
TensorStore (C++ GCS kvstore)	accumulates body via libcurl callback to EOF; enables Accept-Encoding	gcs_key_value_store.cc#L578
gcsfs (fsspec)	reads the aiohttp stream to EOF	core.py#L118
rclone (Go)	GCS backend returns res.Body, reads to EOF	googlecloudstorage.go#L1385, issue #2658
smart_open (Python)	same bug class, fixed by delegating to google-cloud-storage	issue #422

AI disclaimer:

• 🤷‍♂️ ran into the bug
• 🤖 did initial investigation
• 🤷‍♂️ verified investigation results (checked RFC, response headers, and google-cloud-storage behavior)
• 🤖 helped writing the report
• 🤷‍♂️ proofread and made funny AI disclaimer list

[nkemnitz]

Some additional findings:

• GoogleCloudStorageBuilder::new().with_bucket_name("neuroglancer-public-data").with_skip_signature(true) fails the same way, which hopefully makes reproducibility easier
• There is a second failure mode for the decompressive transcoding : MissingEtag. Similar to the Content-Length, if the server sends decompressed content (because the client didn't set "Accept-Encoding: gzip", it can't write any useful information into these response headers. You'll only get the compressed objects information in x-goog-hash and x-goog-stored-content-length

And to clarify:

• Accept-Encoding; gzip does not fix the missing Content-Length issue for large files requested with HTTP1.1 - those might still get streamed (Transfer-Encoding: chunked), which forbids Content-Length
• .with_http2_only() + Accept-Encoding: gzip works, but is slower