Skip to content

fix(sandbox): propagate control-plane errors from update_*/delete (ENG-4244) - #206

Merged
SystemSculpt merged 4 commits into
mainfrom
codex/eng-4244-unwrap-update-delete-errors
Jul 28, 2026
Merged

fix(sandbox): propagate control-plane errors from update_*/delete (ENG-4244)#206
SystemSculpt merged 4 commits into
mainfrom
codex/eng-4244-unwrap-update-delete-errors

Conversation

@SystemSculpt

@SystemSculpt SystemSculpt commented Jul 28, 2026

Copy link
Copy Markdown
Member

Reported by Mathis Joffre in the shared EvenUp channel.

What broke

Every generated control-plane API function returns Union[Error, T] | None, so an error status is a return value, not an exception. SandboxInstance.update_metadata / update_ttl / update_lifecycle / update_network and delete passed that response straight into cls(response). SandboxInstance.__init__ does not type-check, so it built an instance wrapping an Error and constructed .fs, .process, .previews over it.

Two failure modes, the first worse than how it was reported:

  1. Silent no-op. The common call shape ignores the return value. A 403/404/500 on update_ttl returned normally and the TTL was never applied. A failed delete returned normally and the sandbox stayed alive and kept billing. No exception, no log.
  2. Displaced failure. Callers that used the result hit AttributeError: 'Error' object has no attribute 'metadata' arbitrarily far from the real cause.

Reproduced against a mocked control plane on main @ 2c405e2:

update_ttl        -> returned SandboxInstance wrapping Error
                     leaked payload: {"error":"FORBIDDEN","code":403,"message":"insufficient permissions"}
                     result.metadata.name -> AttributeError
update_lifecycle  -> same
update_metadata   -> same
update_network    -> same
delete            -> returned Error (typed as Sandbox)
get (guarded)     -> returned SandboxInstance wrapping Sandbox   # control

Confirmed present in the published blaxel==0.3.5 wheel, so this is live for customers rather than only on main.

Scope

Location Problem
sandbox/default/sandbox.py 4 update_* unguarded
sandbox/default/sandbox.py delete only checked None, then raised a misleading ValueError("Sandbox X not found") for what may have been a 500
sandbox/sync/sandbox.py same 4, SyncSandboxInstance
sandbox/sync/sandbox.py delete had no check at all
volume/volume.py _delete_volume_by_name / _sync, identical shape and identical generated Union[Error, Volume] type

create, get, list, fork, snapshot* were already correctly guarded.

Supporting evidence that this already bites us internally: tests/integration/core/conftest.py carries an _api_error(response) helper that sniffs returned values for a leaked Error during sandbox cleanup.

Fix

Route the responses through _unwrap_response(), which already exists in sandbox/default/sandbox.py and is already imported by the sync tree. It was added in #201 for the fork/snapshot helpers and simply never applied to the update/delete paths. Volume delete uses the guard shape already repeated six times in that file, so no new abstraction is introduced there either.

- return cls(response)
+ return cls(_unwrap_response(response, "update sandbox TTL"))

- if response is None:
-     raise ValueError(f"Sandbox {sandbox_name} not found")
- return response
+ return _unwrap_response(response, f"delete sandbox {sandbox_name}")

17 lines added and 4 removed across three source files. The _unwrap_response docstring is updated to state the rule, since it now governs every write path rather than only fork/snapshot.

Behavior change

delete on a 404 now raises SandboxAPIError instead of silently returning an Error, and the empty response raises SandboxAPIError instead of ValueError. This is the intended correction, but it is breaking for anyone relying on best-effort deletes.

Cross-SDK parity

  • TypeScript: @blaxel/core/src/sandbox/sandbox.ts passes throwOnError: true on every updateSandbox / deleteSandbox. Already correct.
  • Go: Stainless-generated, idiomatic (res, err). Already correct.
  • Python: the only affected SDK.

Verification

  • Red/green: with the source fix reverted and only the new tests applied, 18 tests fail; with the fix restored, 80 pass. The tests pin the bug rather than restating the implementation.
  • Full unit suite: 421 passed, 2 failed. Both failures are pre-existing on unpatched main (stale local device-mode credentials hitting api.blaxel.ai/v0/oauth/token), confirmed by stashing the patch and re-running.
  • make lint (ruff check --fix): all checks passed. Note that ruff format is deliberately not run, since it reformats 98 unrelated generated files and is not part of the lint target.
  • End-to-end with real httpx over a mocked transport: all five paths now raise SandboxAPIError, and the 200 control path still returns a proper SandboxInstance.

Coverage added: each of the 4 update_* x {Error, None} x {async, sync}, class-level and instance-level delete, volume delete in both trees, plus a success-path test so the guard cannot regress into "always raises".

Out of scope

drive/drive.py. The drives OpenAPI declares no error schema, so the generated _parse_response returns cast(Any, None) for 401/404. The six isinstance(response, Error) guards in that file are dead code that can never fire, and drive errors silently collapse to None. That needs a contract correction plus regeneration, sibling to ENG-4050 / ENG-4072, and is tracked separately.

Linear: ENG-4244


Note

Adds a test fix (commit 66cb48b) that removes a duplicate ttl-idle policy entry from an integration test. The control plane rejects duplicate policy types with a 400, which was previously swallowed silently — now that update_lifecycle raises, the duplicate surfaced as a test failure.

Written by Mendral for commit 66cb48b.

…G-4244)

Generated control-plane API functions return `Union[Error, T] | None`, so an
error status is a return value rather than an exception. The sandbox update_*
and delete helpers passed that response straight into `cls(response)`, which
does not type-check, so a 403/404/500 produced an instance wrapping an `Error`.

Callers that ignore the return value -- the common shape -- saw a failed write
as a success: the TTL was never applied, or the sandbox was never deleted and
kept billing. Callers that used the result hit `AttributeError: 'Error' object
has no attribute 'metadata'` far from the real cause.

Route the responses through `_unwrap_response()`, already present in the same
module and already imported by the sync tree; it was added in #201 for the
fork/snapshot helpers and never applied to the update/delete paths. Volume
delete had the identical gap and uses that file's existing guard shape.

Reported by Mathis Joffre. TypeScript (`throwOnError: true`) and Go are already
correct; Python was the only affected SDK.

Note: delete on a 404 now raises `SandboxAPIError` instead of silently
returning an `Error`, and the empty case raises `SandboxAPIError` instead of
`ValueError`.
mendral-app[bot]

This comment was marked as outdated.

Mendral review on #206: the volume delete helpers guarded Error but not
None, while the sandbox path routes through _unwrap_response, which raises
on both. delete_volume returns Union[Error, Volume] | None, so a None
response returned silently despite the -> Volume annotation -- the same
silent-failure class this PR removes for sandboxes.
@mendral-app

mendral-app Bot commented Jul 28, 2026

Copy link
Copy Markdown
Contributor

🧪 Testing Guide

What this PR addresses

The generated control-plane API functions return Union[Error, T] | None, meaning errors come back as return values, not exceptions. The update_metadata, update_ttl, update_lifecycle, update_network, and delete methods on SandboxInstance (and SyncSandboxInstance) were passing those responses directly to cls(...), which silently built an instance wrapping an Error object. This caused two failure modes:

  1. Silent no-op — a failed write (e.g. 403/500) looked like a success.
  2. Delayed AttributeError — callers inspecting the returned instance hit errors far from the real cause.

The same issue existed for volume delete helpers.

Steps to reproduce the original issue

  1. Set up a sandbox or volume instance via the SDK.
  2. Trigger a control-plane error on an update or delete operation (e.g. insufficient permissions → 403, or targeting a non-existent resource → 404).
  3. Before this fix: The method returns a SandboxInstance (or VolumeInstance) wrapping an Error object — no exception is raised. Subsequent access to .fs, .process, etc. either silently does nothing or raises unrelated AttributeError.

What to verify (expected behavior)

  • Error responses raise SandboxAPIError / VolumeAPIError with the correct status_code and message from the control plane (e.g. "insufficient permissions", status 403).
  • None responses (unexpected empty replies) also raise SandboxAPIError / VolumeAPIError rather than passing None into the constructor.
  • Happy path still works: When the API returns a valid Sandbox/Volume object, the methods return a properly constructed instance as before.
  • Unit tests pass: Run pytest tests/core/test_sandbox.py tests/core/test_volume_errors.py — the new parametrized tests cover all update helpers + delete for both async and sync variants.
  • Integration test fix: The test_update_lifecycle_with_different_policy_types_preserves_files test previously passed vacuously (the update was rejected with a 400 that was swallowed). It now uses a single valid policy so the test actually exercises what it claims.

Note

Posted by PR Testing Guide · Tag @mendral-app with feedback.

mendral-app[bot]

This comment was marked as outdated.

test_update_lifecycle_with_different_policy_types_preserves_files built a
lifecycle with ttl-idle listed twice. The control plane rejects duplicate
policy types with a 400, and update_lifecycle used to swallow that 400, so
the update silently never applied. The test then asserted the file survived
a change that had not happened, and passed for the wrong reason.

Now that update_lifecycle raises, the 400 surfaces and the copy-paste
duplicate is visible. Drop the duplicate entry so the test exercises the
policy-type change it claims to.

@mendral-app mendral-app Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

The only new change since my last LGTM review is a single integration test fix that removes an invalid duplicate policy entry. The fix is correct and well-commented. Previous review comments were already addressed in 082b3aa.

Tag @mendral-app with feedback or questions. View session

@SystemSculpt
SystemSculpt merged commit 7675a58 into main Jul 28, 2026
21 checks passed
@SystemSculpt
SystemSculpt deleted the codex/eng-4244-unwrap-update-delete-errors branch July 28, 2026 18:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant