Skip to content

fix: attach per-part checksum header on UploadPart for Object Lock buckets#1363

Merged
Slach merged 3 commits intoAltinity:masterfrom
david-pivonka:fix/upload-part-checksum-object-lock
Apr 27, 2026
Merged

fix: attach per-part checksum header on UploadPart for Object Lock buckets#1363
Slach merged 3 commits intoAltinity:masterfrom
david-pivonka:fix/upload-part-checksum-object-lock

Conversation

@david-pivonka
Copy link
Copy Markdown
Contributor

Summary

Fixes upload failures against S3 buckets with Object Lock enabled.

Problem

With s3.checksum_algorithm: CRC32 set in config, single PutObject / CopyObject uploads succeed against Object Lock buckets (that path was fixed in #829). Large file uploads still fail during multipart:

operation error S3: UploadPart, https response error StatusCode: 400,
api error InvalidRequest: Content-MD5 OR x-amz-checksum-* HTTP header
is required for Put Part requests with Object Lock parameters

s3manager.Uploader sends the per-part checksum as an HTTP trailer (via aws-chunked encoding). Object Lock rejects trailer-based checksums on UploadPart — it requires the checksum as a request header.

Fix

When CheckSumAlgorithm == CRC32 and the file is large enough to trigger multipart, PutFileAbsolute dispatches to a manual multipart path (putFileMultipartCRC32) that:

  1. Calls CreateMultipartUpload (existing enrich helper used, so tags / SSE / ACL preserved).
  2. Reads each part into a buffer, computes CRC32 (IEEE), base64-encodes it.
  3. Calls UploadPart with ChecksumAlgorithm: CRC32 and ChecksumCRC32: <base64> — the SDK serialises this as the x-amz-checksum-crc32 request header, which Object Lock accepts.
  4. Aborts on any failure; completes via CompleteMultipartUpload on success.

For all other cases (no checksum configured, non-CRC32 algo, file below part size), the original s3manager.Uploader path is unchanged.

Scope

  • Only activates when the operator opts in via checksum_algorithm: CRC32, so existing deployments are unaffected.
  • Does not touch CopyObject (multipart copy) — UploadPartCopyInput has no ChecksumAlgorithm field; server-side copy derives integrity from the source object and is not covered by this failure mode.
  • Sequential upload loop; concurrency matches s3manager.Uploader behaviour only in the dispatched case. Can be parallelised as a follow-up if needed.

Verification

Tested against a real AWS S3 bucket in ca-central-1 with Object Lock Compliance mode (30-day default retention). Before the patch: every multipart upload fails with the InvalidRequest error above. After: uploads complete and objects land under the default retention.

Related

…ckets

S3 Object Lock buckets reject UploadPart requests that carry the
checksum only as a trailer (the default behaviour of s3manager.Uploader)
with InvalidRequest "Content-MD5 OR x-amz-checksum-* HTTP header is
required for Put Part requests with Object Lock parameters".

When CheckSumAlgorithm is set to CRC32 and the file is large enough to
require multipart, switch to a manual CreateMultipartUpload / UploadPart
/ CompleteMultipartUpload loop that pre-computes CRC32 per part and
sends it as the x-amz-checksum-crc32 request header via
UploadPartInput.ChecksumCRC32.

Extends Altinity#829 (which fixed the PutObject / CopyObject paths) to the
UploadPart path.
@david-pivonka david-pivonka force-pushed the fix/upload-part-checksum-object-lock branch from 6a14ff8 to 7451e32 Compare April 22, 2026 06:17
@david-pivonka
Copy link
Copy Markdown
Contributor Author

@Slach when you have a moment, could you take a look at this one? Extends the #829 Object Lock support to the UploadPart path. Happy to adjust the approach (middleware instead of a parallel code path, concurrency, etc.) based on your feedback.

@Slach Slach merged commit 54c6d4c into Altinity:master Apr 27, 2026
27 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants