S3: delete() now requires DeleteObjects (bulk delete) support, breaks S3-compatible providers (Alibaba Cloud OSS, etc.)

[hengfeiyang]

Describe the bug

#549 refactored AmazonS3::delete() so that it now delegates to delete_stream(), which issues a POST /?delete (S3 DeleteObjects / bulk delete) request instead of the previous single-object DELETE /key.

While the S3 core protocol is widely implemented by third-party providers, the DeleteObjects multi-object API is not universally supported. As a result, any user of object_store 0.x against a non-AWS S3-compatible endpoint that doesn't implement POST /?delete can no longer delete objects at all — even single-object deletes — because the only public API (delete) now routes through bulk delete.

Concretely, this breaks:

• Alibaba Cloud OSS — its S3-compatible endpoint does not implement DeleteObjects. Calls fail with an XML error (NotImplemented / MethodNotAllowed, depending on region).
• Likely affected: older MinIO versions, Cloudflare R2 in some configurations, Tencent COS legacy endpoints, and various on-prem S3 gateways. (Can collect more reports if helpful.)

Before #549, delete() issued a plain DELETE /key, which is part of the core S3 API and is supported by essentially every S3-compatible provider. After #549, there is no way for downstream users to opt out.

To Reproduce

Using object_store configured with an Alibaba Cloud OSS S3-compatible endpoint:

let store = AmazonS3Builder::new()
    .with_endpoint("https://oss-<region>.aliyuncs.com")
    .with_bucket_name("my-bucket")
    .with_access_key_id(...)
    .with_secret_access_key(...)
    .build()?;

store.put(&path, bytes).await?;          // OK
store.delete(&path).await?;              // FAILS: NotImplemented / MethodNotAllowed

The same call succeeded on the previous release that used single-object DELETE.

Expected behavior

AmazonS3::delete(path) should issue a single-object DELETE /key request, which is part of the core S3 API and works against every S3-compatible provider.

Bulk delete (POST /?delete, the DeleteObjects API) is an optimization and should only be used by delete_stream / batched APIs — not by the per-object delete(). This also mirrors the AWS SDKs, which expose DeleteObject and DeleteObjects as distinct operations precisely because the latter is optional / extended.

Suggested fix

Two options, not mutually exclusive:

1. Revert delete() to single-object DELETE (preferred). delete_stream continues to use DeleteObjects for throughput; delete() stays compatible with every S3 provider. This is the lowest-risk change and matches the contract users had pre-refactor!: move delete to ObjectStoreExt #549.

2. Add an opt-out config, e.g. S3ConfigKey::DisableBulkDelete (or a positive with_bulk_delete(false) builder method), so users targeting incompatible providers can force single-object DELETEs throughout. Useful even with option 1, for delete_stream callers on providers that don't support DeleteObjects.

Happy to send a PR implementing either or both — would appreciate guidance on which direction maintainers prefer.

Environment

• object_store version: 0.13.x
• Provider: Alibaba Cloud OSS (S3-compatible endpoint)

Additional context

• Original PR: #549
• AWS API reference: [DeleteObject](https://docs.aws.amazon.com/AmazonS3/latest/API/API_DeleteObject.html) vs [DeleteObjects](https://docs.aws.amazon.com/AmazonS3/latest/API/API_DeleteObjects.html) — distinct operations in the upstream API.

---

The error like this:

2026-05-22T02:00:03.274139257+00:00 ERROR infra::storage: Failed to delete object: Generic S3 error: Error performing POST http://openobserve.oss-cn-beijing-internal.aliyuncs.com/openobserve?delete in 9.033372ms - Server returned non-2xx status code: 405 Method Not Allowed: <?xml version="1.0" encoding="UTF-8"?>
<Error>
  <Code>MethodNotAllowed</Code>
  <Message>The specified method is not allowed against this resource.</Message>
  <RequestId>6A0FB8A330259633318A4958</RequestId>
  <HostId>openobserve.oss-cn-beijing-internal.aliyuncs.com</HostId>
  <Method>POST</Method>
  <ResourceType>Object</ResourceType>
  <EC>0017-00000001</EC>
  <RecommendDoc>https://api.aliyun.com/troubleshoot?q=0017-00000001</RecommendDoc>
</Error>

@crepererum @alamb

[alamb]

Thanks for the report @hengfeiyang

Do you have a suggestion? Perhaps we could special case the delete_stream implementation when there is only a single obect to delete and fallback to using the /delete endoint 🤔

[tustvold]

IMO we should add a delete_opts API to the trait, this would resolve this issue, whilst also opening the door to delete preconditions which delete_stream can't support

#610 (comment)

[hengfeiyang]

I prefer to add one option can control it, default still use stream_delete, but we can change it back to normal delete like this:

    /// If set to `true`, [`delete_stream`](crate::ObjectStore::delete_stream)
    /// will issue parallel single-object `DELETE /key` requests instead of the
    /// bulk `DeleteObjects` API (`POST /?delete`). Use this for S3-compatible
    /// providers that do not implement `DeleteObjects` (e.g. Alibaba Cloud OSS).
    pub fn with_disable_bulk_delete(mut self, disable: bool) -> Self {
        self.disable_bulk_delete = disable.into();
        self
    }

openobserve@bd8226b

[hengfeiyang]

@alamb @tustvold what do you think? if you agree to add an option and keep the default behavior, user can use the option to switch back to DELETE, i can create the PR for it.

[Xuanwo]

Another quick workaround is to use object_store_opendal. It uses opendal as the backend of object_store and provides native support for oss. This way, bulk delete can still be performed.

But AFAIK, oss should support DeleteObjects under it's S3 API, please double check if there are other mistakes.

[hengfeiyang]

@Xuanwo Great, but it would be even better if object_store itself could have this fix as well.

But AFAIK, oss should support DeleteObjects under it's S3 API, please double check if there are other mistakes.

It supports, but doesn't compatible with s3 and already create tickets for Aliyun, that is another story.

[Xuanwo]

I think the endpoint here is wrong:

This request should be sent to

http://openobserve.oss-cn-beijing-internal.aliyuncs.com/?delete

Instead of

http://openobserve.oss-cn-beijing-internal.aliyuncs.com/openobserve?delete

/?delete should not work on an object.

[hengfeiyang]

@Xuanwo that is not a manually endpoint url, it is generated by the library. it works before v0.13.0. and when we move batch_delete back to delete it works again.

Maybe that is an endpoint issue, i will check it again.

By the way, same code works on s3.

[hengfeiyang]

@Xuanwo i confirmed, why it has a /openobseve with the endpoint? that's bucket name. it will auto append to the endpoint. unless i set .with_virtual_hosted_style_request(true), but this value default is false.

Also i tested aws, MinIO both works, oss got problem with the default config.

So, we know better about this case, but we still need to do some fix let it works with default configure.

bring DELETE back with delete_opt or with_disable_bulk_delete

[tustvold]

Adding an option to disable bulk delete seems reasonable to me

[hengfeiyang]

Thank you @alamb @tustvold i will create a PR soon with the solution which add an option to disable bulk delete

[PsiACE]

@Xuanwo 's suggestion is correct. Your request was sent to the wrong endpoint. Adding an option to disable bulk delete is not the right solution to this problem.

[hengfeiyang]

@PsiACE i shared the detail above. same config works for AWS GCP and MinIO, only OSS can't work. it only can work when i set with_virtual_hosted_style_request(true), but default value is false.

So, yes, we can use other config to let it work, but it will change the behavior too (store path will change). i still want to give one more option let the user to choose which solution they want.