feat(server,k8s): implement pause/resume with rootfs snapshot support by fengcone · Pull Request #668 · alibaba/OpenSandbox

fengcone · 2026-04-08T08:05:20Z

Add PausePolicy schema and API endpoints (/pause, /resume)
Add image_pull_policy configuration for sandbox containers
Support pausePolicy in pool-based BatchSandbox creation
Add SandboxSnapshot CRD and controller for snapshot lifecycle
Add image-committer Job for container rootfs snapshot/commit
Update AGENTS.md and Makefile for new components

Implements OSEP-0008 pause/resume functionality using SandboxSnapshot
CRD and nerdctl-based image committer for Kubernetes runtime.

Summary

What is changing and why?

Testing

Not run (explain why)
Unit tests
Integration tests
e2e / manual verification

Breaking Changes

None
Yes (describe impact and migration path)

Checklist

Linked Issue or clearly described motivation
Added/updated docs (if needed)
Added/updated tests (if needed)
Security impact considered
Backward compatibility considered

- Add PausePolicy schema and API endpoints (/pause, /resume) - Add image_pull_policy configuration for sandbox containers - Support pausePolicy in pool-based BatchSandbox creation - Add SandboxSnapshot CRD and controller for snapshot lifecycle - Add image-committer Job for container rootfs snapshot/commit - Update AGENTS.md and Makefile for new components Implements OSEP-0008 pause/resume functionality using SandboxSnapshot CRD and nerdctl-based image committer for Kubernetes runtime.

fengcone · 2026-04-08T08:05:38Z

@codex review

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 2f6a1f854c

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

- Rename snapshotPushSecretName -> snapshotPushSecret - Rename resumeImagePullSecretName -> resumeImagePullSecret - Fix ruff lint errors (F541, F841, F401) - Regenerate CRDs from Go types Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

…nd use generation-based reconciliation - Move controller-managed fields (sourcePodName, sourceNodeName resumeTemplate, containerSnapshots, pause/resume versions) from spec to status so spec is purely user input - Replace version-based dispatch with Kubernetes generation pattern spec.action (Pause/Resume) signals intent, observedGeneration tracks controller progress - Remove PausePolicy from BatchSandbox CRD; pause config now comes from server-level configuration - Add fail-fast spec validation before starting pause cycles - Add RBAC permission for reading secrets - Improve delete_sandbox error handling (proper 404/500 responses - Expand E2E tests with error-case coverage

# Conflicts: # server/tests/k8s/test_batchsandbox_provider.py # server/tests/k8s/test_kubernetes_service.py

fengcone · 2026-04-12T10:35:23Z

@codex review

…ill running

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 3a13c3a542

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 486658b3b4

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

fengcone · 2026-04-12T12:37:57Z

@codex review

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 6e302bce73

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

…very

fengcone · 2026-04-12T14:51:51Z

@codex review

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 7fa689d222

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

fengcone · 2026-04-12T15:16:01Z

@codex review

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: f86fd2a96b

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

…n pause resume lifecycle

fengcone · 2026-04-13T02:45:40Z

@codex review

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 7474fa0343

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Generalwin · 2026-04-13T04:07:44Z

 creation-date: 2026-03-11
 last-updated: 2026-03-13
-status: implementing
+status: implemented


can you add an user example

hittyt

Implements OSEP-0008 pause/resume functionality using a new SandboxSnapshot CRD and nerdctl-based image committer. The implementation is well-structured but has some security and reliability concerns that need addressing before merging. One notable architectural gap is that renew_expiration currently only works on active BatchSandbox resources and will fail with 404 for paused sandboxes; it should be updated to handle snapshots as well.

hittyt · 2026-04-13T04:08:07Z

+		spec := fmt.Sprintf("%s:%s", cs.ContainerName, cs.ImageURI)
+		containerSpecs = append(containerSpecs, spec)
+	}
+	fullCommand := fmt.Sprintf("/usr/local/bin/image-committer %s %s %s",


[P1] Potential Shell Injection Vulnerability

Building fullCommand using fmt.Sprintf and then executing it via /bin/sh -c is risky, as fields like SourcePodName or Namespace could potentially contain shell-special characters.

Since image-committer is a compiled binary, it's safer to run it directly by passing arguments to the Args list and avoiding a shell altogether.

Suggested change

fullCommand := fmt.Sprintf("/usr/local/bin/image-committer %s %s %s",

var containerSpecs []string

for _, cs := range containerSnapshots {

spec := fmt.Sprintf("%s:%s", cs.ContainerName, cs.ImageURI)

containerSpecs = append(containerSpecs, spec)

}

// Build the job

hittyt · 2026-04-13T04:08:07Z

+							Name:            CommitJobContainerName,
+							Image:           imageCommitterImage,
+							ImagePullPolicy: corev1.PullIfNotPresent,
+							Command:         []string{"/bin/sh", "-c"},


[P1] Avoid using shell for executing image-committer

Directly execute the binary without a shell wrapper to mitigate injection risks.

Suggested change

Command: []string{"/bin/sh", "-c"},

Command: []string{"/usr/local/bin/image-committer"},

Args: append([]string{snapshot.Status.SourcePodName, snapshot.Namespace}, containerSpecs...),

hittyt · 2026-04-13T04:08:07Z

+
+	// Set up signal handler to ensure all paused containers are resumed on exit
+	c := make(chan os.Signal, 1)
+	signal.Notify(c, os.Interrupt, syscall.SIGTERM)


[P2] Unreliable container unpause on SIGKILL

resumeAllPausedContainers is registered for SIGTERM and Interrupt, but not SIGKILL. If the Job Pod is killed abruptly (e.g., node shutdown or OOM), containers in the source Pod will remain paused forever. This could break the sandbox state.

Consider adding a fallback mechanism in the controller to ensure containers are unpaused if the Job fails or times out.

hittyt · 2026-04-13T04:08:07Z

+	}
+	registryHost := imageParts[0]
+
+	isInsecure := strings.Contains(registryHost, "local") ||


[P2] Limited insecure registry detection

The current logic for isInsecure only checks for specific keywords like local or certain IP prefixes. This may fail for internal corporate registries using private domain names or other private IP ranges like 172.16.x.x.

Consider making "insecure" a flag passed from the server config to the controller, and then to image-committer to make it more robust.

hittyt · 2026-04-13T04:08:07Z

+									Value: ContainerdSocketPath,
+								},
+							},
+							SecurityContext: &corev1.SecurityContext{


[P1] Highly Privileged Job Execution

This Job runs as root and mounts the host's containerd.sock, which effectively gives the Job Pod root access to the entire node.

It is critical that the ImageCommitterImage is strictly controlled and sourced only from a trusted repository to prevent privilege escalation.

hittyt · 2026-04-13T04:08:07Z

+}
+
+// handleDeletion handles the deletion of a SandboxSnapshot
+func (r *SandboxSnapshotReconciler) handleDeletion(ctx context.Context, snapshot *sandboxv1alpha1.SandboxSnapshot) (ctrl.Result, error) {


[P2] Resource Leak - Snapshot Images

The handleDeletion logic cleans up the K8s Job, but it doesn't clean up the OCI images in the registry. This will lead to storage leakage in the registry over time.

While cleaning up registry images can be complex, this limitation should at least be documented, and perhaps a periodic GC mechanism should be considered in the future.

…ntation

…easons

Generalwin · 2026-04-13T06:11:05Z

            )
-
-            if not workload:
+            snapshot = self.snapshot_provider.get_snapshot(sandbox_id, self.namespace)


这里应该不是所有的get sandbox请求都需要去获取snapshot这个对象吧？pause和resume的场景比例应该比较小。

Generalwin · 2026-04-13T06:19:55Z

-                "message": "Pause operation is not supported in Kubernetes runtime",
-            },
+        # 1. Get BatchSandbox
+        batchsandbox = self.workload_provider.get_workload(sandbox_id, self.namespace)


这里不应该有batch sandbox的概念，batch sandbox只是sandbox service的一种实现

Generalwin · 2026-04-13T06:28:08Z

+// Spec only contains user input fields (filled by Server), Controller never writes to spec.
+type SandboxSnapshotSpec struct {
+	// SandboxID is the stable public identifier for the sandbox.
+	SandboxID string `json:"sandboxId"`


这个直接叫SnapshotID怎么样？这样Snapshot独立于某个Sandbox之外，sandbox的pause和resume只是利用snapshot能力实现的一种在sandbox上的功能。

fengcone requested review from Generalwin, Pangjiping, Spground, hittyt, jwx0925, kevinlynx and ninan-nn as code owners April 8, 2026 08:05

chatgpt-codex-connector bot reviewed Apr 8, 2026

View reviewed changes

Comment thread server/opensandbox_server/services/k8s/kubernetes_service.py Outdated

Comment thread kubernetes/apis/sandbox/v1alpha1/batchsandbox_types.go Outdated

Comment thread server/opensandbox_server/services/k8s/kubernetes_service.py Outdated

fengcone and others added 4 commits April 8, 2026 08:49

docs(server,k8s): update doc for pause and resume feature

3b98183

Merge branch 'main' into feature/public-k8s-pause-resume

3a13c3a

# Conflicts: # server/tests/k8s/test_batchsandbox_provider.py # server/tests/k8s/test_kubernetes_service.py

fix(server): return workload state when snapshot fails but sandbox st…

486658b

…ill running

chatgpt-codex-connector bot reviewed Apr 12, 2026

View reviewed changes

Comment thread kubernetes/internal/controller/sandboxsnapshot_controller.go Outdated

Comment thread server/opensandbox_server/services/k8s/kubernetes_service.py Outdated

chatgpt-codex-connector bot reviewed Apr 12, 2026

View reviewed changes

Comment thread server/opensandbox_server/services/k8s/kubernetes_service.py Outdated

fix(server,k8s): add sandboxsnapshots RBAC and surface resume failures

6e302bc

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

chatgpt-codex-connector bot reviewed Apr 12, 2026

View reviewed changes

Comment thread server/opensandbox_server/services/k8s/kubernetes_service.py Outdated

Comment thread kubernetes/internal/controller/sandboxsnapshot_controller.go Outdated

fengcone added 2 commits April 12, 2026 22:19

fix(image-committer): replace crictl with nerdctl for container disco…

17d7008

…very

fix(k8s): surface resume failures and fix paused sandbox image URI

7fa689d

fengcone force-pushed the feature/public-k8s-pause-resume branch from b2cc1dc to 7fa689d Compare April 12, 2026 14:50

chatgpt-codex-connector bot reviewed Apr 12, 2026

View reviewed changes

Comment thread kubernetes/internal/controller/sandboxsnapshot_controller.go Outdated

Comment thread server/opensandbox_server/services/k8s/kubernetes_service.py Outdated

fengcone added 2 commits April 12, 2026 15:15

fix(controller): requeue on transient API errors in validatePauseSpec

dd1876a

fix(server): include full pause config in re-pause snapshot patch

f86fd2a

chatgpt-codex-connector bot reviewed Apr 12, 2026

View reviewed changes

Comment thread server/opensandbox_server/services/k8s/kubernetes_service.py

Comment thread server/opensandbox_server/services/k8s/kubernetes_service.py Outdated

fix(server,k8s): copy user labels to snapshot and verify full label i…

7474fa0

…n pause resume lifecycle

chatgpt-codex-connector bot reviewed Apr 13, 2026

View reviewed changes

Comment thread kubernetes/internal/controller/sandboxsnapshot_controller.go

Comment thread kubernetes/internal/controller/sandboxsnapshot_controller.go

Comment thread server/opensandbox_server/services/k8s/kubernetes_service.py

Generalwin reviewed Apr 13, 2026

View reviewed changes

Comment thread kubernetes/README.md

Generalwin reviewed Apr 13, 2026

View reviewed changes

hittyt requested changes Apr 13, 2026

View reviewed changes

fengcone added 3 commits April 13, 2026 04:09

docs(k8s): complete pause/resume state machine and add Chinese docume…

9efe77c

…ntation

feat(server): expose intermediate pause/resume states with detailed r…

6f6c231

…easons

fix(kubernetes): replace crictl with nerdctl

38bee80

Generalwin reviewed Apr 13, 2026

View reviewed changes

	Command: []string{"/bin/sh", "-c"},
	Command: []string{"/usr/local/bin/image-committer"},
	Args: append([]string{snapshot.Status.SourcePodName, snapshot.Namespace}, containerSpecs...),

Conversation

fengcone commented Apr 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Testing

Breaking Changes

Checklist

Uh oh!

fengcone commented Apr 8, 2026

Uh oh!

chatgpt-codex-connector bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

fengcone commented Apr 12, 2026

Uh oh!

chatgpt-codex-connector bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

chatgpt-codex-connector bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

fengcone commented Apr 12, 2026

Uh oh!

chatgpt-codex-connector bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

fengcone commented Apr 12, 2026

Uh oh!

chatgpt-codex-connector bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

fengcone commented Apr 12, 2026

Uh oh!

chatgpt-codex-connector bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

fengcone commented Apr 13, 2026

Uh oh!

chatgpt-codex-connector bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

hittyt left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

fengcone commented Apr 8, 2026 •

edited

Loading