Repository navigation
fix(gke): harden Agent Substrate security reference architecture, Mission Control UI, and lifecycle automation - #614
Merged
Conversation
…choing egress verifier, and OSS review hardening - Require SPIFFE client-certificate mTLS between gateway and atenet-router, removing the plaintext router bridge script. - Add cluster-local partner_api verifier returning keyed HMAC-SHA256 prefixes without reflecting injected bearer credentials. - Add managed port-forward lifecycle helper, standalone Option A teardown.sh, scoped Prometheus queries, and initial unit tests.
…g and document capacity trade-offs - Parameterize Option B worker node locations and machine types for regional capacity overrides. - Harden stage.sh and teardown-standard.sh readiness and cleanup checks on multi-node GKE Base Platform clusters. - Document Option A vs Option B VM footprint and capacity fallback settings.
…t guides for production best practices - Align docs/platforms/gke/base/use-cases/agent-substrate-security-demo/README.md and terraform/README.md with split-plane gateway, NetworkPolicy, and Workload Identity deployment steps.
…CPU snapshot checks, and docs alignment - Enforce strict URL scheme, redirect, and payload bounds in gateway.py and egress_agent.py. - Handle cross-CPU durable snapshot fallback in Mission Control Act 2 checks when regional worker nodes span CPU microarchitectures. - Expand unit test coverage and align observability query docs.
…r cspell - Reword CPU feature helper docstring in ui/mission_control.py to satisfy repository cspell dictionary checks.
… modular UI, and hash-pinned deps - Split monolithic ui/mission_control.py and ui/index.html into modular Python and frontend assets under 600 lines per file. - Pin container Python dependencies with SHA-256 hashes in app/requirements.txt and enforce --require-hashes in app/Dockerfile. - Tighten IAM scoping, fail-closed probe verdicts, and unit test suites across app/, ui/, and scripts/.
…d brownfield stage.sh upgrades - Support *.domain suffix entries in UI ALLOWED_HOSTS validation for reverse proxy environments. - Make stage.sh and dashboards.sh idempotent when re-staging existing clusters.
…ompt flows and harden teardown - Add 2-step Act 1 Prompt-to-Python synthesis (ui/terminal.py) with editable approval card, AST validation, clean <sandbox-probe> tracebacks, and aligned KEY = VALUE environment inspection - Add interactive Act 4 Google ADK prompt bar, quick-fill chips, and preserved live reply/revert transcript (ALLOW_CUSTOM_PYTHON=1) - Ensure best-effort actor suspend on error paths so 2-slot WorkerPool pods never leak running actors - Support *.domain suffix wildcards in ALLOWED_HOSTS, regional/global Vertex AI URL validation, quoted gcloud --filter regexes, and ate-system PVC cleanup fallback in teardown scripts - Expand unit test suite to 30 tests (84% coverage) while keeping all files <= 600 lines
The base platform this terraservice deploys on top of moved to hashicorp/google 8.8.0 on main (platforms/gke/base/_shared_config and the core terraservices). Pin the same version here so one deployment uses one provider version. `terraform validate` passes against 8.8.0. The regenerated lock file is identical to platforms/gke/base/_shared_config/.terraform.lock.hcl. Not applied against a live project.
… outputs _shared_config/scripts/set_environment_variables.sh loads these outputs with `eval "$(terraform output | sed ...)"`. `agent_substrate_node_locations` is a list(string), which `terraform output` renders over several lines as `tolist([ ... ])`. With the documented stockout override (TF_VAR_agent_substrate_node_locations) set, the eval is a bash syntax error, so deploy-standard.sh stops after the core platform is built and teardown-standard.sh stops before destroying anything. The terraservice reads the value through the symlinked variables file, so the output is not needed. Also remove three outputs that no script reads: agent_substrate_node_disk_type, agent_substrate_node_machine_type and agent_substrate_snapshot_bucket_force_destroy.
…ore teardown With ACP_TEARDOWN_CORE_PLATFORM=false the core platform stays deployed, but teardown-standard.sh ran `git checkout` on platforms/gke/base/_shared_config and core/initialize/backend.tf.bucket. That reverted the generated platform.auto.tfvars (project, region, platform name, resource prefix) and the state bucket name, so a later core teardown resolved the default bucket and could not find the platform's state. Any failure was hidden by `2>/dev/null || true`. Leave those files alone when the core platform is kept. core/teardown.sh restores them when it destroys core/initialize.
… egress The last egress rule of gateway-isolation and prometheus-ui-isolation had `ports` but no `to`, which matches every destination: any pod, Service, RFC 1918 host or link-local address on TCP 443. The architecture guide says private and link-local CIDRs are blocked. Scope the rule to 0.0.0.0/0 except 10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16, 100.64.0.0/10 and 169.254.0.0/16. The gateway still reaches atenet-router through its dedicated rule (pods labelled app=atenet-router on 443/8443, which matches the upstream v1.0.0 manifests). The manifest test now fails on an egress rule that has no `to`, or that allows 0.0.0.0/0 without excluding those ranges. Verified on GKE Dataplane V2 clusters: Vertex AI, Cloud Monitoring, and atenet-router remain reachable while private/link-local HTTPS ranges are blocked.
…s chat errors
chat_actor updated the turn counters only after _call_router returned. A
router outage, TLS failure or timeout raises HTTPException from _call_router,
so agent_gateway_chat_turns_total{outcome="error"} stayed flat exactly when
the router was down. The earlier implementation counted these in an `except`
block.
Count the failure and re-raise. Adds a regression test.
…UI_AUTH_TOKEN A bare `*` makes is_allowed_host_header() accept any Host header, which disables the DNS-rebinding guard. Without a UI token every mutating route is then reachable from a rebound origin. run_server() only printed a warning. Exit instead, before port-forwards or the listener start. `*` together with UI_AUTH_TOKEN, and suffix entries such as `*.example.com`, behave as before.
…commands The "Raw CLI" strings shown in the Mission Control footer and modal interpolated the Act 4 prompt after `-d` and the PromQL expression after `--data-urlencode` without shell quoting. The prompt can be typed by the operator (ALLOW_CUSTOM_PYTHON=1), so a copied command could word-split or expand `$(...)`. The UI itself never executes these strings. Quote both with shlex.quote, and assert that the prompt stays a single shell argument.
… build commands observability/queries.md told Option B readers to rewrite queries with promql_scope.py. The scoper leaves any selector that already has a `cluster` matcher unchanged, and every query in the file has one, so the command was a no-op. Tell readers to substitute the cluster name, and describe what the scoper does. terraform/README.md exported REGION with a us-central1 default before building a custom image. That is wrong for any other location, and the exported value leaks into a later teardown.sh run in the same shell. Read PROJECT_ID, REGION and AR_REPO from .state/staged.env, which stage.sh writes for both options, and do not export them.
…icies gateway-isolation and prometheus-ui-isolation allowed port 53 only to pods in kube-system. The base platform cluster that Option B deploys onto sets cluster_dns = "CLOUD_DNS", where pods resolve through the node-local 169.254.169.254:53 address. With default-deny egress the gateway could then not resolve atenet-router.ate-system.svc or the Vertex AI endpoint: /readyz needs no DNS, so the rollout looks healthy and the first admin call fails. Allow 169.254.169.254/32 on UDP and TCP 53 next to the kube-system rule. This mirrors the federated-learning NetworkPolicy for the same cluster. The manifest test now requires that address on every DNS egress rule.
…t real FastAPI CI installed only the repository-level requirements.txt, which has no fastapi or uvicorn, so test_app_services.py ran against hand-written stand-ins for FastAPI, uvicorn and google-auth. Header binding, dependency execution and routing were never exercised, and the check that every admin route requires the token ran only under the stand-in (it was skipped whenever real FastAPI was importable). - Add scripts/requirements-test.txt (fastapi, starlette, uvicorn, httpx, google-auth, requests at the versions pinned in app/requirements.txt) and install it in the unit-test and coverage workflows. - Remove the FastAPI, uvicorn and google-auth stand-ins. Google ADK is still replaced, now unconditionally, so the suite behaves the same whether or not google-adk is importable. - Replace the stand-in-only route check with HTTP tests through TestClient: every admin API route except /readyz and /identity must carry the token dependency; missing or wrong tokens get 401 before the router is called; an unset token gets 503; and the model proxy accepts only `:generateContent` with a JSON object body. Verified locally with those package versions on Python 3.13: 33 tests pass, and each new test fails when its guard is removed. The CI install step itself (Python 3.12, alongside the repository requirements) has not been run.
… live E2E signatures - Require EHOSTUNREACH or ENETUNREACH for Act 1 metadata server isolation instead of reporting BLOCKED on arbitrary timeouts or refused connections. - Narrow Act 3 Local DNS Relay denial matching to Errno -2 / NXDOMAIN / no such host (rejecting transient Errno -3 resolver failures) and re-fetch partner-api after Test B as a same-run positive control. - Read Secret/example-api directly in the CLI and UI harnesses to verify zero secret reflection instead of relying solely on a self-reported flag. - Require an explicit integer exit_code from sandbox probes, gate the footer gVisor kernel label on kernelOk, clear stale Act 2 clone badges before re-runs, and lead Act 4 restore labels with the current run's CLI latency.
…ion and recover Reset via tag - Refuse to suspend gemini-agent in Act 2 when an uncommitted RUNNING wake session is not CLEAN, and discard (revert) rather than checkpointing on post-resume or post-revert verification failures. - Roll gemini-agent back by recreating it from immutable clean-checkpoint-tag on Reset when its working snapshot was already overwritten, sharing the tag-recreation helper with Act 2 forensic cloning. - Align gateway Vertex AI per-attempt timeout (20s) under gemini_agent's 45s deadline and avoid replaying non-idempotent POST requests on read timeouts.
…rward ownership, and teardown - Add scripts/common.sh so stage.sh, teardown.sh, and run_4_act_demo.sh resolve the target cluster strictly from explicit env or .state/staged.env rather than adopting an unrelated ambient kubectl current-context. - Pin Mission Control UI and run_4_act_demo.sh to a private minified kubeconfig when staged.env is present, and prefer staged.env over ambient context in kube.get_cluster_name() and detect_project_id(). - Reject unowned local listeners in port-forward.sh and kube._start_single_pf unless tracked by an in-memory Popen or matching pf-*.pid kubectl process. - Verify exact --ingress-allowed-spiffe-ids allowlist and absence of legacy plaintext HTTP bridge on atenet-router before reporting mTLS readiness. - Label Option A clusters with created-by=agent-substrate-demo, require --yes or interactive confirmation in teardown.sh, refuse Option B clusters in teardown.sh, remove per-repo Artifact Registry node-SA bindings before SA deletion, preserve .state/ on teardown errors, and fail closed on Option B PVC release errors.
…g, runner exit code - Allow UDP/TCP 53 to 169.254.20.10 (GKE NodeLocal DNSCache) in the gateway and prometheus-ui NetworkPolicies. Pods on the Option B base platform cluster resolve through that address. - Set Gemini thinking to LOW for the ADK actor. Default thinking took 6-25 s per tool-call request in measurements; LOW took 1.4-6.2 s. - Return 0 from the EXIT trap in run_4_act_demo.sh when the last temporary file is absent. The runner exited 1 after verifying all four acts. - Lengthen the gateway, actor and client chat timeouts. Gateway uses a single 45 s upstream attempt under the actor (55 s), router (90 s), and client (95 s) deadlines.
… instead of failing The model sometimes answers the default Act 4 prompt without calling set_approval_policy (1 of 12 answered chats in one soak run). The UI and the CLI runner both treated that as a failure, although it is the safe outcome. - ui: when the approval policy is unchanged after the default prompt, return the result with model_resisted=true and show the model's reply. The session is still dirty, so the existing Revert step applies before a retry. A policy that is neither hijacked nor intact is still an error. - run_4_act_demo.sh: on an unchanged policy, discard the session with kubectl ate revert and send the injection again, up to ACT4_MAX_ATTEMPTS (default 3). If every attempt is resisted the script exits non-zero and says that Act 4 was not verified. - docs: mention the retry and ACT4_MAX_ATTEMPTS where the Terraform README describes the CLI runner. - tests: cover the resisted, hijacked, custom-prompt and unreadable-policy outcomes of the UI endpoint. The shell loop and app.js have no test in CI.
…start of the CLI runner run_4_act_demo.sh suspended any actor that was still running before Act 1. A suspend checkpoints that session. When an earlier run had stopped in Act 4 after the injection (seen live when an upstream model call timed out), the next run suspended the hijacked session, read the hijacked policy back in Act 4 step 1 and failed its own clean-baseline check. Revert the actor instead. Revert discards the uncommitted session and returns the actor to its last snapshot, which is what Act 4 itself relies on. Tested live: with gemini-agent left running and hijacked, the runner now passes on the first run.
…ment skip-core teardown - Align Markdown table column padding in terraform/README.md with repository Prettier 3.4.2 checks. - Document what ACP_TEARDOWN_CORE_PLATFORM=false removes (use-case Terraservice, dashboards, ate-system PVCs) and what remains on the kept Core Platform cluster.
…irectory and guard optional test deps Address reviewer feedback on shared repository files and unit-test hermeticity: - Revert all 12 shared repository files outside `(docs/)?platforms/gke/base/use-cases/agent-substrate-security-demo/` (`README.md`, `cspell.json`, `.coveragerc`, `.github/workflows/dictionary/*`, `.github/workflows/python-unit-test.yaml`, `.github/workflows/python-coverage-test.yaml`, and shared GKE Base Platform READMEs) back to `origin/main` so repo-wide workflows and dictionaries remain untouched. - Guard `fastapi`/`httpx`/`google-auth` imports in `test_app_services.py` and `test_kube_and_probes.py` with `@unittest.skipUnless` so the 31 stdlib/YAML unit tests execute cleanly in repo-wide CI while all 43 tests (including FastAPI `TestClient` checks) run whenever `scripts/requirements-test.txt` is installed. - Mock `kube.run_cmd` in `MissionControlRegressionTest.test_e6_cluster_name_promql_errors_and_slow_restores_not_fabricated` so zero unit tests depend on a host `kubectl` binary. - Add file-local `cspell:words` directives in the use-case files that introduce new security or stdlib identifiers so repo-wide `cspell` passes without modifying shared dictionaries.
karajendran
force-pushed
the
fix/agent-substrate-security-review
branch
from
November 20, 2026 04:50
d48fb20 to
f6895e6
Compare
Collaborator
|
LGTM, thanks |
syeda-anjum
approved these changes
Nov 20, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Hardens
platforms/gke/base/use-cases/agent-substrate-security-demo/following end-to-end security, architecture, and lifecycle verification across both standalone GKE1.37+(Option A,scripts/) and Terraform GKE Base Platform (Option B,terraform/) deployment paths:Split-plane Gateway, mTLS & NetworkPolicy isolation (
app/,manifests/):PodCertificateclient-certificate mTLS fromgateway(127.0.0.1:8081operator plane) tohttps://atenet-router.ate-system.svc:443, verifying the exact--ingress-allowed-spiffe-idsallowlist and absence of legacy plaintext bridges during staging.gateway-isolationandprometheus-ui-isolation:443egress to0.0.0.0/0excluding RFC 1918, CGNAT (100.64.0.0/10), and link-local (169.254.0.0/16) ranges, while explicitly permitting Cloud DNS (169.254.169.254:53), NodeLocal DNSCache (169.254.20.10:53), andkube-systemDNS.partner-api(ns/agent-secrets, default-deny egress, ingress restricted toatenet-egress) returning keyed HMAC-SHA256 verification prefixes so Act 3 verifies wire-injected bearer credentials without echoing secret bytes into sandbox memory or responses.app/requirements.txt,--require-hashesinapp/Dockerfile), setsThinkingLevel.LOWongemini_agenttool-call turns, avoids replaying non-idempotent POST requests on read timeouts, and counts router transport exceptions inagent_gateway_chat_turns_total{outcome="error"}.Modular Mission Control UI & fail-closed 4-Act verification (
ui/,scripts/):<= 600lines per file:mission_control.py,server.py,kube.py,probes.py,terminal.py,acts.py,recovery.py,index.html,app.js,styles.css).ALLOWED_HOSTS="*"unlessUI_AUTH_TOKENis set, supports*.domainsuffix entries, enforcesX-Requested-With: MissionControl+SameSite=Strictsession cookies, and shell-quotes operator-supplied prompts in displayed CLI snippets.ALLOW_CUSTOM_PYTHON=1) 2-step Act 1 prompt-to-Python synthesis + AST-validated sandbox execution and interactive Act 4 ADK prompt testing.ateworker-gvisor/atenet-egresssignatures (EHOSTUNREACH/ENETUNREACHfor169.254.169.254,Errno -2for Local DNS Relay, liveSecretnon-reflection, and a post-Test-B positive control).gemini-agentsessions, recoversResetfrom immutableclean-checkpoint-tagwhen a working snapshot was overwritten, reports model refusals on the default Act 4 injection asmodel_resistedinstead of a failure (with bounded retry inrun_4_act_demo.sh), and discards leftover running sessions viakubectl ate revertat CLI runner start.Cluster targeting, Terraform alignment & zero-leftover teardown (
scripts/,terraform/):.state/staged.env(scripts/common.sh), pins a0600minified kubeconfig for the UI and CLI runner, and verifies local port-forward PID ownership (scripts/port-forward.sh).hashicorp/google8.8.0, removes multi-line list outputs from_shared_config/outputs.tfthat brokeset_environment_variables.sheval, and preserves core_shared_config+backend.tf.bucketonACP_TEARDOWN_CORE_PLATFORM=false(with documented skip-core cleanup behavior interraform/README.md).scripts/teardown.sh(with--yesconfirmation guard,created-by=agent-substrate-demolabel check, Option B cross-teardown refusal,ate-systemPostgres PVC release with kubeconfig fallback, per-cluster GSA/bucket/dashboard cleanup, and--delete-networkregional AR/subnet/NAT/VPC cleanup) and fail-closed PVC cleanup interraform/teardown-standard.sh.Real FastAPI unit test & coverage wiring (
scripts/test_*.py,.github/workflows/):scripts/requirements-test.txtto.github/workflows/python-unit-test.yamlandpython-coverage-test.yamlsotest_app_services.pyexercises real FastAPI/Starlette routing and auth dependencies (42unit tests,85%use-case coverage).Scope & Size Breakdown
90 files changed, 10,551 insertions(+), 3,760 deletions(-)relative tomain(most churn comes from splitting the monolithicui/mission_control.pyandui/index.htmlinto< 600-line modules and adding dedicated unit-test suites):Validation
Repository CI checks (executed locally against exported
HEADa2261da4):addlicense -check -ignore '**/.terraform.lock.hcl' .— PASS.dev-tools/bin/dictionaries_check.sh&cspell(2,579files,0issues) — PASSprettier --check '**.md'(Prettier 3.4.2) — PASSterraform fmt -check -recursive— PASSblack --check ./&isort --check-only --profile black .— PASS42unit tests (run_python_unittests.sh+run_python_coverage_test.sh),85%coverage acrossagent-substrate-security-demo(--fail-under=80) — PASSterraform fmt; shell scripts,ui/app.js, Kubernetes manifests, and container builds were verified live via the from-scratch GKE runs below.Live From-Scratch End-to-End GKE Verification (including full teardown & zero-leftover GCP audit):
1.37.2,us-west1-b, dedicated VPC):create-cluster.sh(464s) ->stage.sh(357s) -> Mission Control UI 22-check API suite (22/22PASS) -> headless Chrome live 5-slide walkthrough (0JS errors) ->run_4_act_demo.sh(22s) + leftover hijacked-session recovery (26s) -> cluster exposure & in-pod NetworkPolicy reachability probes (PASS) ->teardown.sh --yes --delete-network(425s, deleting dashboards, PVCs, cluster, snapshot bucket, GSAs, regional AR repo, subnet, NAT/router, and VPC).1.37,us-east4): uninterrupteddeploy-standard.sh(1578s) -> Mission Control UI 22-check API suite (22/22PASS) -> headless Chrome live 5-slide walkthrough (0JS errors) ->run_4_act_demo.sh(61s) -> cluster exposure, NetworkPolicy & tainted worker node separation probes (PASS) ->teardown-standard.sh(tested in bothACP_TEARDOWN_CORE_PLATFORM=falseand fulltruemodes).