Skip to content

feat(core): make reqwest/tokio optional and fix WASI compatibility#7156

Open
JMLX42 wants to merge 5 commits intoapache:mainfrom
lx-industries:feat/wasi-compat
Open

feat(core): make reqwest/tokio optional and fix WASI compatibility#7156
JMLX42 wants to merge 5 commits intoapache:mainfrom
lx-industries:feat/wasi-compat

Conversation

@JMLX42
Copy link

@JMLX42 JMLX42 commented Jan 22, 2026

Which issue does this PR close?

N/A - This is a proactive fix discovered while working on WASI support.

Rationale for this change

OpenDAL currently cannot compile for WASI targets due to two issues:

  1. Mandatory reqwest/tokio dependencies: The default features include reqwest-rustls-tls and executors-tokio, and there was no way to opt out of reqwest entirely. Both reqwest and tokio don't support WASI yet.

  2. web_time cfg condition: The core/core/src/raw/time.rs file uses #[cfg(target_arch = "wasm32")] to conditionally use web_time::web::SystemTimeExt. However, the web_time::web module only exists when target_os = "unknown" (browser WASM), not when target_os = "wasi".

This fix is part of an effort to enable OpenDAL for WASI environments, being developed at opendal-wasi.

What changes are included in this PR?

1. Make reqwest and tokio optional (default-features = false)

  • Add http-client-reqwest feature: New feature flag that controls whether reqwest is included as a dependency
  • Extract reqwest code to feature-gated module: Moved reqwest-specific code to core/core/src/raw/http_util/reqwest_impl.rs, only compiled when the feature is enabled
  • Forward feature to outer crate: The http-client-reqwest feature is now available on the main opendal crate

With these changes, users can now use default-features = false to build OpenDAL without reqwest/tokio, enabling WASI compatibility.

2. Fix web_time WASI compatibility

Changed cfg conditions in core/core/src/raw/time.rs from #[cfg(target_arch = "wasm32")] to #[cfg(all(target_arch = "wasm32", target_os = "unknown"))] so that:

  • Browser WASM (wasm32-unknown-unknown) continues to use web_time
  • WASI (wasm32-wasip1, wasm32-wasip2) uses std::time::SystemTime which is supported

3. Add CI job for wasm32-wasip2 target

Added a new build_under_wasi job to .github/workflows/ci_core.yml to catch WASI compatibility regressions. The job builds with WASI-compatible features (layers and services that don't require reqwest/tokio).

Are there any user-facing changes?

New feature flag: http-client-reqwest - controls inclusion of reqwest HTTP client

Behavior change: When using default-features = false, reqwest and tokio are no longer included. Users who want reqwest must explicitly enable http-client-reqwest or reqwest-rustls-tls.

This is not a breaking change for users who use default features.

AI Usage Statement

This PR was developed with assistance from Claude (Anthropic's Claude Opus 4.5), used via Claude Code CLI. The AI assisted with:

  • Diagnosing the root cause of the WASI compilation failures
  • Implementing the feature flag and code extraction for reqwest
  • Implementing the cfg condition fix for web_time
  • Identifying which features are compatible with WASI (by testing locally)
  • Writing the CI workflow job
  • Drafting this PR description

All changes were reviewed and verified by the human developer.

@dosubot dosubot bot added size:L This PR changes 100-499 lines, ignoring generated files. releases-note/fix The PR fixes a bug or has a title that begins with "fix" labels Jan 22, 2026
@JMLX42 JMLX42 changed the title fix(core): fix web_time WASI compatibility and add CI job feat(core): make reqwest/tokio optional and fix WASI compatibility Jan 22, 2026
@JMLX42 JMLX42 force-pushed the feat/wasi-compat branch 2 times, most recently from 4a40581 to cb3f3d5 Compare January 22, 2026 08:24
@JMLX42
Copy link
Author

JMLX42 commented Jan 22, 2026

There is one remaining issue: the blocking feature (which is required in wasip2) pull tokio as a dependency. blockinginternal-tokio-rttokio/rt-multi-thread. Which won't build in WASM.

IMHO the only option is to re-implement a (blocking) Operator in opendal-wasi.

But this PR is still very much needed.

@JMLX42
Copy link
Author

JMLX42 commented Jan 28, 2026

I have been making great progress thanks to this little PR.

Services already available as WASM components

Two services have already been transformed into WASM components:

  1. opendal-fs.wasm is a WASM/WASI compliant re-implementation of OpenDAL's Fs service (which can't be used directly because it depends on tokio)
  2. opendal-memory.wasm is a WASM component that directly uses OpenDAL's vanilla Memory service. Proving that existing OpenDAL services can just be compiled to a WASM component without a single modification.

So OpenDAL WASM component can now be created with just a few lines of code:

use std::sync::OnceLock;

use opendal::services::MemoryConfig;
use opendal::Operator;
use opendal_wit::OperatorBackend;

static OPERATOR: OnceLock<Result<Operator, opendal::Error>> = OnceLock::new();

struct MemoryComponent;

impl OperatorBackend for MemoryComponent {
    type Config = MemoryConfig;

    fn operator_cell() -> &'static OnceLock<Result<Operator, opendal::Error>> {
        &OPERATOR
    }
}

opendal_wit::export!(MemoryComponent with_types_in opendal_wit);

All the components:

  • export a common WIT interface defined here
  • are stored/distributed as OCI artifacts here

This has the potential to bring OpenDAL to any platform that supports WASM without any specific work on the original Rust code base.

Next steps for opendal-wasi

The next step is to test an HTTP driven service such as S3. I already have implemented HttpFetch for WASI here.

Move reqwest-specific HTTP client implementation to a separate module
that is conditionally compiled only when http-client-reqwest feature
is enabled. This makes reqwest/tokio truly optional dependencies.

- Create reqwest_impl.rs with GLOBAL_REQWEST_CLIENT, HttpFetch impl,
  is_temporary_error helper, and HttpBufferBody
- Add StubHttpClient for builds without reqwest that returns an error
  when used, allowing Default to work in all configurations
- Gate HttpClient::new() with #[cfg(feature = "http-client-reqwest")]
- Update mod.rs to conditionally include reqwest_impl module
The web_time::web module only exists for browser WASM (target_os = "unknown"),
not for WASI targets. This caused compilation failures for wasm32-wasip2.

Changed cfg conditions to only use web_time for browser WASM, allowing WASI
to use the standard std::time::SystemTime which it supports.
Adds a CI job to verify WASI P2 compatibility. This catches issues like
the web_time cfg condition fix where web_time::web only exists for
browser WASM (target_os = unknown), not WASI.

Includes compatible layers and services that don't require reqwest/tokio:
- layers: capability-check, chaos, concurrent-limit, immutable-index,
  logging, metrics, mime-guess, retry, throttle, timeout
- services: dashmap, mini-moka
@JMLX42
Copy link
Author

JMLX42 commented Jan 28, 2026

Just resolved conflicts and pushed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

releases-note/fix The PR fixes a bug or has a title that begins with "fix" size:L This PR changes 100-499 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant