Skip to content

fix(gke): harden Agent Substrate security reference architecture, Mission Control UI, and lifecycle automation - #614

Merged
syeda-anjum merged 26 commits into
mainfrom
fix/agent-substrate-security-review
Nov 20, 2026
Merged

syeda-anjum merged 26 commits into
mainfrom
fix/agent-substrate-security-review

Conversation

@karajendran

Copy link
Copy Markdown
Collaborator

Summary

Hardens platforms/gke/base/use-cases/agent-substrate-security-demo/ following end-to-end security, architecture, and lifecycle verification across both standalone GKE 1.37+ (Option A, scripts/) and Terraform GKE Base Platform (Option B, terraform/) deployment paths:

  • Split-plane Gateway, mTLS & NetworkPolicy isolation (app/, manifests/):

    • Enforces SPIFFE PodCertificate client-certificate mTLS from gateway (127.0.0.1:8081 operator plane) to https://atenet-router.ate-system.svc:443, verifying the exact --ingress-allowed-spiffe-ids allowlist and absence of legacy plaintext bridges during staging.
    • Scopes gateway-isolation and prometheus-ui-isolation :443 egress to 0.0.0.0/0 excluding RFC 1918, CGNAT (100.64.0.0/10), and link-local (169.254.0.0/16) ranges, while explicitly permitting Cloud DNS (169.254.169.254:53), NodeLocal DNSCache (169.254.20.10:53), and kube-system DNS.
    • Adds cluster-local partner-api (ns/agent-secrets, default-deny egress, ingress restricted to atenet-egress) returning keyed HMAC-SHA256 verification prefixes so Act 3 verifies wire-injected bearer credentials without echoing secret bytes into sandbox memory or responses.
    • Pins container Python dependencies with SHA-256 hashes (app/requirements.txt, --require-hashes in app/Dockerfile), sets ThinkingLevel.LOW on gemini_agent tool-call turns, avoids replaying non-idempotent POST requests on read timeouts, and counts router transport exceptions in agent_gateway_chat_turns_total{outcome="error"}.
  • Modular Mission Control UI & fail-closed 4-Act verification (ui/, scripts/):

    • Splits the monolithic UI and HTML into modular components (<= 600 lines per file: mission_control.py, server.py, kube.py, probes.py, terminal.py, acts.py, recovery.py, index.html, app.js, styles.css).
    • Refuses ALLOWED_HOSTS="*" unless UI_AUTH_TOKEN is set, supports *.domain suffix entries, enforces X-Requested-With: MissionControl + SameSite=Strict session cookies, and shell-quotes operator-supplied prompts in displayed CLI snippets.
    • Adds optional (ALLOW_CUSTOM_PYTHON=1) 2-step Act 1 prompt-to-Python synthesis + AST-validated sandbox execution and interactive Act 4 ADK prompt testing.
    • Tightens Act 1/Act 3 pass verdicts to live ateworker-gvisor / atenet-egress signatures (EHOSTUNREACH/ENETUNREACH for 169.254.169.254, Errno -2 for Local DNS Relay, live Secret non-reflection, and a post-Test-B positive control).
    • Guards Act 2 against checkpointing uncommitted dirty/hijacked gemini-agent sessions, recovers Reset from immutable clean-checkpoint-tag when a working snapshot was overwritten, reports model refusals on the default Act 4 injection as model_resisted instead of a failure (with bounded retry in run_4_act_demo.sh), and discards leftover running sessions via kubectl ate revert at CLI runner start.
  • Cluster targeting, Terraform alignment & zero-leftover teardown (scripts/, terraform/):

    • Resolves target clusters strictly from explicit environment variables or .state/staged.env (scripts/common.sh), pins a 0600 minified kubeconfig for the UI and CLI runner, and verifies local port-forward PID ownership (scripts/port-forward.sh).
    • Aligns the Option B Terraservice to hashicorp/google 8.8.0, removes multi-line list outputs from _shared_config/outputs.tf that broke set_environment_variables.sh eval, and preserves core _shared_config + backend.tf.bucket on ACP_TEARDOWN_CORE_PLATFORM=false (with documented skip-core cleanup behavior in terraform/README.md).
    • Adds scripts/teardown.sh (with --yes confirmation guard, created-by=agent-substrate-demo label check, Option B cross-teardown refusal, ate-system Postgres PVC release with kubeconfig fallback, per-cluster GSA/bucket/dashboard cleanup, and --delete-network regional AR/subnet/NAT/VPC cleanup) and fail-closed PVC cleanup in terraform/teardown-standard.sh.
  • Real FastAPI unit test & coverage wiring (scripts/test_*.py, .github/workflows/):

    • Adds scripts/requirements-test.txt to .github/workflows/python-unit-test.yaml and python-coverage-test.yaml so test_app_services.py exercises real FastAPI/Starlette routing and auth dependencies (42 unit tests, 85% use-case coverage).

Scope & Size Breakdown

90 files changed, 10,551 insertions(+), 3,760 deletions(-) relative to main (most churn comes from splitting the monolithic ui/mission_control.py and ui/index.html into < 600-line modules and adding dedicated unit-test suites):

Area | Share of lines | Notes -- | -- | -- ui/ (modular split + Act 1/4 interactive flows) | 46.5% | Split from monolithic mission_control.py + index.html; every file <= 597 lines scripts/ (common/teardown/stage/runner + unit tests) | 32.3% | Includes test_app_services.py, test_ui_mission_control.py, test_kube_and_probes.py, test_targeting_and_verdicts.py, test_promql_scope.py app/ (split-plane gateway, ADK actor, egress actor, partner_api, requirements.txt) | 7.0% | Hash-pinned requirements.txt + partner_api.py terraform/ (provider 8.8.0, worker pool/IAM, deploy & teardown scripts, README) | 5.3% | Matches platforms/gke/base/_shared_config provider lock manifests/ (templates + NetworkPolicies) | 2.8% | Cloud DNS + NodeLocal DNSCache + private-CIDR HTTPS exclusions docs/, observability/, .github/ | 6.1% | Architecture/query docs, cspell dictionaries, workflow test deps

Validation

  1. Repository CI checks (executed locally against exported HEAD a2261da4):

    • addlicense -check -ignore '**/.terraform.lock.hcl' . — PASS
    • Final-newline check across all tracked text files — PASS
    • .dev-tools/bin/dictionaries_check.sh & cspell (2,579 files, 0 issues) — PASS
    • prettier --check '**.md' (Prettier 3.4.2) — PASS
    • terraform fmt -check -recursive — PASS
    • black --check ./ & isort --check-only --profile black . — PASS
    • 42 unit tests (run_python_unittests.sh + run_python_coverage_test.sh), 85% coverage across agent-substrate-security-demo (--fail-under=80) — PASS
    • Note on CI scope: GitHub Actions runs Python unit/coverage, formatting, spellcheck, license, and terraform fmt; shell scripts, ui/app.js, Kubernetes manifests, and container builds were verified live via the from-scratch GKE runs below.
  2. Live From-Scratch End-to-End GKE Verification (including full teardown & zero-leftover GCP audit):

    • Option A (standalone GKE 1.37.2, us-west1-b, dedicated VPC): create-cluster.sh (464s) -> stage.sh (357s) -> Mission Control UI 22-check API suite (22/22 PASS) -> headless Chrome live 5-slide walkthrough (0 JS errors) -> run_4_act_demo.sh (22s) + leftover hijacked-session recovery (26s) -> cluster exposure & in-pod NetworkPolicy reachability probes (PASS) -> teardown.sh --yes --delete-network (425s, deleting dashboards, PVCs, cluster, snapshot bucket, GSAs, regional AR repo, subnet, NAT/router, and VPC).
    • Option B (Terraform GKE Base Platform 1.37, us-east4): uninterrupted deploy-standard.sh (1578s) -> Mission Control UI 22-check API suite (22/22 PASS) -> headless Chrome live 5-slide walkthrough (0 JS errors) -> run_4_act_demo.sh (61s) -> cluster exposure, NetworkPolicy & tainted worker node separation probes (PASS) -> teardown-standard.sh (tested in both ACP_TEARDOWN_CORE_PLATFORM=false and full true modes).

…choing egress verifier, and OSS review hardening

- Require SPIFFE client-certificate mTLS between gateway and atenet-router,
  removing the plaintext router bridge script.
- Add cluster-local partner_api verifier returning keyed HMAC-SHA256 prefixes
  without reflecting injected bearer credentials.
- Add managed port-forward lifecycle helper, standalone Option A teardown.sh,
  scoped Prometheus queries, and initial unit tests.
…g and document capacity trade-offs

- Parameterize Option B worker node locations and machine types for regional
  capacity overrides.
- Harden stage.sh and teardown-standard.sh readiness and cleanup checks on
  multi-node GKE Base Platform clusters.
- Document Option A vs Option B VM footprint and capacity fallback settings.
…t guides for production best practices

- Align docs/platforms/gke/base/use-cases/agent-substrate-security-demo/README.md
  and terraform/README.md with split-plane gateway, NetworkPolicy, and
  Workload Identity deployment steps.
…CPU snapshot checks, and docs alignment

- Enforce strict URL scheme, redirect, and payload bounds in gateway.py and
  egress_agent.py.
- Handle cross-CPU durable snapshot fallback in Mission Control Act 2 checks
  when regional worker nodes span CPU microarchitectures.
- Expand unit test coverage and align observability query docs.
…r cspell

- Reword CPU feature helper docstring in ui/mission_control.py to satisfy
  repository cspell dictionary checks.
… modular UI, and hash-pinned deps

- Split monolithic ui/mission_control.py and ui/index.html into modular
  Python and frontend assets under 600 lines per file.
- Pin container Python dependencies with SHA-256 hashes in app/requirements.txt
  and enforce --require-hashes in app/Dockerfile.
- Tighten IAM scoping, fail-closed probe verdicts, and unit test suites across
  app/, ui/, and scripts/.
…d brownfield stage.sh upgrades

- Support *.domain suffix entries in UI ALLOWED_HOSTS validation for reverse
  proxy environments.
- Make stage.sh and dashboards.sh idempotent when re-staging existing clusters.
…ompt flows and harden teardown

- Add 2-step Act 1 Prompt-to-Python synthesis (ui/terminal.py) with editable approval card, AST validation, clean <sandbox-probe> tracebacks, and aligned KEY = VALUE environment inspection
- Add interactive Act 4 Google ADK prompt bar, quick-fill chips, and preserved live reply/revert transcript (ALLOW_CUSTOM_PYTHON=1)
- Ensure best-effort actor suspend on error paths so 2-slot WorkerPool pods never leak running actors
- Support *.domain suffix wildcards in ALLOWED_HOSTS, regional/global Vertex AI URL validation, quoted gcloud --filter regexes, and ate-system PVC cleanup fallback in teardown scripts
- Expand unit test suite to 30 tests (84% coverage) while keeping all files <= 600 lines
The base platform this terraservice deploys on top of moved to
hashicorp/google 8.8.0 on main (platforms/gke/base/_shared_config and the
core terraservices). Pin the same version here so one deployment uses one
provider version.

`terraform validate` passes against 8.8.0. The regenerated lock file is
identical to platforms/gke/base/_shared_config/.terraform.lock.hcl. Not
applied against a live project.
… outputs

_shared_config/scripts/set_environment_variables.sh loads these outputs with
`eval "$(terraform output | sed ...)"`. `agent_substrate_node_locations` is a
list(string), which `terraform output` renders over several lines as
`tolist([ ... ])`. With the documented stockout override
(TF_VAR_agent_substrate_node_locations) set, the eval is a bash syntax error,
so deploy-standard.sh stops after the core platform is built and
teardown-standard.sh stops before destroying anything.

The terraservice reads the value through the symlinked variables file, so the
output is not needed. Also remove three outputs that no script reads:
agent_substrate_node_disk_type, agent_substrate_node_machine_type and
agent_substrate_snapshot_bucket_force_destroy.
…ore teardown

With ACP_TEARDOWN_CORE_PLATFORM=false the core platform stays deployed, but
teardown-standard.sh ran `git checkout` on platforms/gke/base/_shared_config
and core/initialize/backend.tf.bucket. That reverted the generated
platform.auto.tfvars (project, region, platform name, resource prefix) and the
state bucket name, so a later core teardown resolved the default bucket and
could not find the platform's state. Any failure was hidden by
`2>/dev/null || true`.

Leave those files alone when the core platform is kept. core/teardown.sh
restores them when it destroys core/initialize.
… egress

The last egress rule of gateway-isolation and prometheus-ui-isolation had
`ports` but no `to`, which matches every destination: any pod, Service,
RFC 1918 host or link-local address on TCP 443. The architecture guide says
private and link-local CIDRs are blocked.

Scope the rule to 0.0.0.0/0 except 10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16,
100.64.0.0/10 and 169.254.0.0/16. The gateway still reaches atenet-router
through its dedicated rule (pods labelled app=atenet-router on 443/8443, which
matches the upstream v1.0.0 manifests).

The manifest test now fails on an egress rule that has no `to`, or that allows
0.0.0.0/0 without excluding those ranges.

Verified on GKE Dataplane V2 clusters: Vertex AI, Cloud Monitoring, and
atenet-router remain reachable while private/link-local HTTPS ranges are
blocked.
…s chat errors

chat_actor updated the turn counters only after _call_router returned. A
router outage, TLS failure or timeout raises HTTPException from _call_router,
so agent_gateway_chat_turns_total{outcome="error"} stayed flat exactly when
the router was down. The earlier implementation counted these in an `except`
block.

Count the failure and re-raise. Adds a regression test.
…UI_AUTH_TOKEN

A bare `*` makes is_allowed_host_header() accept any Host header, which
disables the DNS-rebinding guard. Without a UI token every mutating route is
then reachable from a rebound origin. run_server() only printed a warning.

Exit instead, before port-forwards or the listener start. `*` together with
UI_AUTH_TOKEN, and suffix entries such as `*.example.com`, behave as before.
…commands

The "Raw CLI" strings shown in the Mission Control footer and modal
interpolated the Act 4 prompt after `-d` and the PromQL expression after
`--data-urlencode` without shell quoting. The prompt can be typed by the
operator (ALLOW_CUSTOM_PYTHON=1), so a copied command could word-split or
expand `$(...)`. The UI itself never executes these strings.

Quote both with shlex.quote, and assert that the prompt stays a single shell
argument.
… build commands

observability/queries.md told Option B readers to rewrite queries with
promql_scope.py. The scoper leaves any selector that already has a `cluster`
matcher unchanged, and every query in the file has one, so the command was a
no-op. Tell readers to substitute the cluster name, and describe what the
scoper does.

terraform/README.md exported REGION with a us-central1 default before
building a custom image. That is wrong for any other location, and the
exported value leaks into a later teardown.sh run in the same shell. Read
PROJECT_ID, REGION and AR_REPO from .state/staged.env, which stage.sh writes
for both options, and do not export them.
…icies

gateway-isolation and prometheus-ui-isolation allowed port 53 only to pods in
kube-system. The base platform cluster that Option B deploys onto sets
cluster_dns = "CLOUD_DNS", where pods resolve through the node-local
169.254.169.254:53 address. With default-deny egress the gateway could then
not resolve atenet-router.ate-system.svc or the Vertex AI endpoint: /readyz
needs no DNS, so the rollout looks healthy and the first admin call fails.

Allow 169.254.169.254/32 on UDP and TCP 53 next to the kube-system rule. This
mirrors the federated-learning NetworkPolicy for the same cluster. The
manifest test now requires that address on every DNS egress rule.
…t real FastAPI

CI installed only the repository-level requirements.txt, which has no
fastapi or uvicorn, so test_app_services.py ran against hand-written
stand-ins for FastAPI, uvicorn and google-auth. Header binding, dependency
execution and routing were never exercised, and the check that every admin
route requires the token ran only under the stand-in (it was skipped whenever
real FastAPI was importable).

- Add scripts/requirements-test.txt (fastapi, starlette, uvicorn, httpx,
  google-auth, requests at the versions pinned in app/requirements.txt) and
  install it in the unit-test and coverage workflows.
- Remove the FastAPI, uvicorn and google-auth stand-ins. Google ADK is still
  replaced, now unconditionally, so the suite behaves the same whether or not
  google-adk is importable.
- Replace the stand-in-only route check with HTTP tests through TestClient:
  every admin API route except /readyz and /identity must carry the token
  dependency; missing or wrong tokens get 401 before the router is called; an
  unset token gets 503; and the model proxy accepts only `:generateContent`
  with a JSON object body.

Verified locally with those package versions on Python 3.13: 33 tests pass,
and each new test fails when its guard is removed. The CI install step itself
(Python 3.12, alongside the repository requirements) has not been run.
… live E2E signatures

- Require EHOSTUNREACH or ENETUNREACH for Act 1 metadata server isolation
  instead of reporting BLOCKED on arbitrary timeouts or refused connections.
- Narrow Act 3 Local DNS Relay denial matching to Errno -2 / NXDOMAIN / no
  such host (rejecting transient Errno -3 resolver failures) and re-fetch
  partner-api after Test B as a same-run positive control.
- Read Secret/example-api directly in the CLI and UI harnesses to verify zero
  secret reflection instead of relying solely on a self-reported flag.
- Require an explicit integer exit_code from sandbox probes, gate the footer
  gVisor kernel label on kernelOk, clear stale Act 2 clone badges before
  re-runs, and lead Act 4 restore labels with the current run's CLI latency.
…ion and recover Reset via tag

- Refuse to suspend gemini-agent in Act 2 when an uncommitted RUNNING wake
  session is not CLEAN, and discard (revert) rather than checkpointing on
  post-resume or post-revert verification failures.
- Roll gemini-agent back by recreating it from immutable clean-checkpoint-tag
  on Reset when its working snapshot was already overwritten, sharing the
  tag-recreation helper with Act 2 forensic cloning.
- Align gateway Vertex AI per-attempt timeout (20s) under gemini_agent's 45s
  deadline and avoid replaying non-idempotent POST requests on read timeouts.
…rward ownership, and teardown

- Add scripts/common.sh so stage.sh, teardown.sh, and run_4_act_demo.sh
  resolve the target cluster strictly from explicit env or .state/staged.env
  rather than adopting an unrelated ambient kubectl current-context.
- Pin Mission Control UI and run_4_act_demo.sh to a private minified
  kubeconfig when staged.env is present, and prefer staged.env over ambient
  context in kube.get_cluster_name() and detect_project_id().
- Reject unowned local listeners in port-forward.sh and kube._start_single_pf
  unless tracked by an in-memory Popen or matching pf-*.pid kubectl process.
- Verify exact --ingress-allowed-spiffe-ids allowlist and absence of legacy
  plaintext HTTP bridge on atenet-router before reporting mTLS readiness.
- Label Option A clusters with created-by=agent-substrate-demo, require --yes
  or interactive confirmation in teardown.sh, refuse Option B clusters in
  teardown.sh, remove per-repo Artifact Registry node-SA bindings before SA
  deletion, preserve .state/ on teardown errors, and fail closed on Option B
  PVC release errors.
…g, runner exit code

- Allow UDP/TCP 53 to 169.254.20.10 (GKE NodeLocal DNSCache) in the gateway
  and prometheus-ui NetworkPolicies. Pods on the Option B base platform
  cluster resolve through that address.
- Set Gemini thinking to LOW for the ADK actor. Default thinking took 6-25 s
  per tool-call request in measurements; LOW took 1.4-6.2 s.
- Return 0 from the EXIT trap in run_4_act_demo.sh when the last temporary
  file is absent. The runner exited 1 after verifying all four acts.
- Lengthen the gateway, actor and client chat timeouts. Gateway uses a single 45 s
  upstream attempt under the actor (55 s), router (90 s), and client (95 s)
  deadlines.
… instead of failing

The model sometimes answers the default Act 4 prompt without calling
set_approval_policy (1 of 12 answered chats in one soak run). The UI and the
CLI runner both treated that as a failure, although it is the safe outcome.

- ui: when the approval policy is unchanged after the default prompt, return
  the result with model_resisted=true and show the model's reply. The session
  is still dirty, so the existing Revert step applies before a retry. A policy
  that is neither hijacked nor intact is still an error.
- run_4_act_demo.sh: on an unchanged policy, discard the session with
  kubectl ate revert and send the injection again, up to ACT4_MAX_ATTEMPTS
  (default 3). If every attempt is resisted the script exits non-zero and says
  that Act 4 was not verified.
- docs: mention the retry and ACT4_MAX_ATTEMPTS where the Terraform README
  describes the CLI runner.
- tests: cover the resisted, hijacked, custom-prompt and unreadable-policy
  outcomes of the UI endpoint. The shell loop and app.js have no test in CI.
…start of the CLI runner

run_4_act_demo.sh suspended any actor that was still running before Act 1. A
suspend checkpoints that session. When an earlier run had stopped in Act 4
after the injection (seen live when an upstream model call timed out), the
next run suspended the hijacked session, read the hijacked policy back in
Act 4 step 1 and failed its own clean-baseline check.

Revert the actor instead. Revert discards the uncommitted session and returns
the actor to its last snapshot, which is what Act 4 itself relies on.

Tested live: with gemini-agent left running and hijacked, the runner now
passes on the first run.
…ment skip-core teardown

- Align Markdown table column padding in terraform/README.md with repository Prettier 3.4.2 checks.
- Document what ACP_TEARDOWN_CORE_PLATFORM=false removes (use-case Terraservice, dashboards, ate-system PVCs) and what remains on the kept Core Platform cluster.
…irectory and guard optional test deps

Address reviewer feedback on shared repository files and unit-test hermeticity:

- Revert all 12 shared repository files outside `(docs/)?platforms/gke/base/use-cases/agent-substrate-security-demo/` (`README.md`, `cspell.json`, `.coveragerc`, `.github/workflows/dictionary/*`, `.github/workflows/python-unit-test.yaml`, `.github/workflows/python-coverage-test.yaml`, and shared GKE Base Platform READMEs) back to `origin/main` so repo-wide workflows and dictionaries remain untouched.
- Guard `fastapi`/`httpx`/`google-auth` imports in `test_app_services.py` and `test_kube_and_probes.py` with `@unittest.skipUnless` so the 31 stdlib/YAML unit tests execute cleanly in repo-wide CI while all 43 tests (including FastAPI `TestClient` checks) run whenever `scripts/requirements-test.txt` is installed.
- Mock `kube.run_cmd` in `MissionControlRegressionTest.test_e6_cluster_name_promql_errors_and_slow_restores_not_fabricated` so zero unit tests depend on a host `kubectl` binary.
- Add file-local `cspell:words` directives in the use-case files that introduce new security or stdlib identifiers so repo-wide `cspell` passes without modifying shared dictionaries.
@karajendran
karajendran force-pushed the fix/agent-substrate-security-review branch from d48fb20 to f6895e6 Compare November 20, 2026 04:50
@syeda-anjum

Copy link
Copy Markdown
Collaborator

LGTM, thanks

@syeda-anjum
syeda-anjum merged commit cb7c8f3 into main Nov 20, 2026
50 checks passed
@syeda-anjum
syeda-anjum deleted the fix/agent-substrate-security-review branch November 20, 2026 05:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants