feat(core): make reqwest/tokio optional and fix WASI compatibility#7156
feat(core): make reqwest/tokio optional and fix WASI compatibility#7156JMLX42 wants to merge 5 commits intoapache:mainfrom
Conversation
4a40581 to
cb3f3d5
Compare
|
There is one remaining issue: the IMHO the only option is to re-implement a (blocking) But this PR is still very much needed. |
|
I have been making great progress thanks to this little PR. Services already available as WASM componentsTwo services have already been transformed into WASM components:
So OpenDAL WASM component can now be created with just a few lines of code: use std::sync::OnceLock;
use opendal::services::MemoryConfig;
use opendal::Operator;
use opendal_wit::OperatorBackend;
static OPERATOR: OnceLock<Result<Operator, opendal::Error>> = OnceLock::new();
struct MemoryComponent;
impl OperatorBackend for MemoryComponent {
type Config = MemoryConfig;
fn operator_cell() -> &'static OnceLock<Result<Operator, opendal::Error>> {
&OPERATOR
}
}
opendal_wit::export!(MemoryComponent with_types_in opendal_wit);All the components: This has the potential to bring OpenDAL to any platform that supports WASM without any specific work on the original Rust code base. Next steps for
|
Move reqwest-specific HTTP client implementation to a separate module that is conditionally compiled only when http-client-reqwest feature is enabled. This makes reqwest/tokio truly optional dependencies. - Create reqwest_impl.rs with GLOBAL_REQWEST_CLIENT, HttpFetch impl, is_temporary_error helper, and HttpBufferBody - Add StubHttpClient for builds without reqwest that returns an error when used, allowing Default to work in all configurations - Gate HttpClient::new() with #[cfg(feature = "http-client-reqwest")] - Update mod.rs to conditionally include reqwest_impl module
The web_time::web module only exists for browser WASM (target_os = "unknown"), not for WASI targets. This caused compilation failures for wasm32-wasip2. Changed cfg conditions to only use web_time for browser WASM, allowing WASI to use the standard std::time::SystemTime which it supports.
Adds a CI job to verify WASI P2 compatibility. This catches issues like the web_time cfg condition fix where web_time::web only exists for browser WASM (target_os = unknown), not WASI. Includes compatible layers and services that don't require reqwest/tokio: - layers: capability-check, chaos, concurrent-limit, immutable-index, logging, metrics, mime-guess, retry, throttle, timeout - services: dashmap, mini-moka
cb3f3d5 to
14055e1
Compare
|
Just resolved conflicts and pushed. |
Which issue does this PR close?
N/A - This is a proactive fix discovered while working on WASI support.
Rationale for this change
OpenDAL currently cannot compile for WASI targets due to two issues:
Mandatory reqwest/tokio dependencies: The default features include
reqwest-rustls-tlsandexecutors-tokio, and there was no way to opt out of reqwest entirely. Both reqwest and tokio don't support WASI yet.web_time cfg condition: The
core/core/src/raw/time.rsfile uses#[cfg(target_arch = "wasm32")]to conditionally useweb_time::web::SystemTimeExt. However, theweb_time::webmodule only exists whentarget_os = "unknown"(browser WASM), not whentarget_os = "wasi".This fix is part of an effort to enable OpenDAL for WASI environments, being developed at opendal-wasi.
What changes are included in this PR?
1. Make reqwest and tokio optional (
default-features = false)http-client-reqwestfeature: New feature flag that controls whether reqwest is included as a dependencycore/core/src/raw/http_util/reqwest_impl.rs, only compiled when the feature is enabledhttp-client-reqwestfeature is now available on the mainopendalcrateWith these changes, users can now use
default-features = falseto build OpenDAL without reqwest/tokio, enabling WASI compatibility.2. Fix web_time WASI compatibility
Changed cfg conditions in
core/core/src/raw/time.rsfrom#[cfg(target_arch = "wasm32")]to#[cfg(all(target_arch = "wasm32", target_os = "unknown"))]so that:wasm32-unknown-unknown) continues to useweb_timewasm32-wasip1,wasm32-wasip2) usesstd::time::SystemTimewhich is supported3. Add CI job for
wasm32-wasip2targetAdded a new
build_under_wasijob to.github/workflows/ci_core.ymlto catch WASI compatibility regressions. The job builds with WASI-compatible features (layers and services that don't require reqwest/tokio).Are there any user-facing changes?
New feature flag:
http-client-reqwest- controls inclusion of reqwest HTTP clientBehavior change: When using
default-features = false, reqwest and tokio are no longer included. Users who want reqwest must explicitly enablehttp-client-reqwestorreqwest-rustls-tls.This is not a breaking change for users who use default features.
AI Usage Statement
This PR was developed with assistance from Claude (Anthropic's Claude Opus 4.5), used via Claude Code CLI. The AI assisted with:
All changes were reviewed and verified by the human developer.