Add Helm deployment examples and lifecycle testing

[nerdalert]

Background

Commit 262d030 adds the initial upstream Helm deployment foundation:

• a grid-operator chart;
• a temporary praxis-gateway workload chart owned by Grid for integration testing;
• strict lint, schema, template, CRD synchronization, packaging, and Kind lifecycle validation;
• Helm-based deployment in the global ingress demo; and
• versioned OCI publication of the Grid operator chart from the release workflow.

The charts are sufficient for development and demonstrated integration, but a supported deployment contract, maintained deployment examples, and broader lifecycle qualification remain productization work.

Goal

Define and qualify a clear Helm installation path for Grid on standard Kubernetes while keeping ownership boundaries explicit:

• Grid owns the Grid operator chart and Grid CRDs.
• Grid temporarily carries the Praxis gateway workload chart so the integration can be installed and tested from one repository.
• A future Praxis gateway operator should eventually own gateway deployment and lifecycle.
• Forge may create local Kind infrastructure and invoke Helm, but Helm must also work independently against existing clusters.

Required deployment examples

Add maintained, versioned examples under a common deployment-examples directory. Each example must provide values files, commands, architecture, expected resources, validation, teardown, and an explicit statement of what it proves.

1. Local Kind simulation

• Use Forge to create the required single-node Kind clusters.
• Install the Grid operator and Praxis gateway charts through Forge Helm steps.
• Use mock inference providers and generated demo credentials.
• Support both the short validation path and the complete failure/recovery demonstration.
• Remain the primary reproducible development and CI scenario.

2. Private multi-cluster Grid without global ingress

• Target existing Kubernetes clusters rather than creating them.
• Provide an example topology of four clusters with two logical edge gateways and two provider sites.
• Configure inter-cluster SWIM reachability, gateway mTLS, provider credentials, and routing overlays.
• Expose each edge through a user-selected private Service type or address.
• Require the client or an existing private traffic manager to select an edge; the example does not install global DNS or ingress steering.
• Prove distributed provider discovery, edge-local overlay rendering, intelligent provider selection, remote-provider fallback, and credential isolation.

This example must remain genuinely multi-cluster. A single-cluster gateway installation without distributed Grid state belongs in Praxis gateway documentation and is not a Grid deployment example.

3. Multi-cluster Grid with global ingress

• Provide the Helm equivalent of the existing global ingress demo topology.
• Keep the traffic-manager component replaceable and separate from Grid provider selection.
• Document edge withdrawal, provider drain, provider failure, overlay convergence, and recovery evidence.
• Clearly distinguish demonstration components from infrastructure supplied by a production environment.

Installation contract

• Document installs from a local chart checkout and from the versioned OCI chart.
• Require an explicit Praxis gateway image tag or digest; do not imply that Grid publishes a default Praxis image.
• Prefer immutable image digests in reproducible examples.
• Document required namespaces, ServiceAccounts, ClusterRoles, RoleBindings, Services, Secrets, ConfigMaps, and CRDs.
• Keep additional namespace access explicit through resourceNamespaces.
• Document SWIM bind, advertise, seed, and Service settings for existing clusters.
• Document provider gateway configuration, routing-overlay mounts, TLS mounts, and credential mounts.
• Provide preflight checks for cluster access, Kubernetes version, Helm version, image access, DNS, required ports, and cross-cluster reachability.

Lifecycle and security requirements

• Qualify fresh install, idempotent upgrade, rollback, uninstall, and reinstall.
• Document that Helm does not upgrade or remove CRDs automatically and provide a deliberate CRD upgrade procedure.
• Preserve custom resources when uninstalling the chart.
• Confirm generated resources work under restricted pod security settings without fixed platform-specific assumptions.
• Keep containers non-root with a read-only root filesystem, dropped capabilities, and a runtime-default seccomp profile.
• Test existing and chart-created ServiceAccounts.
• Test RBAC access in the release namespace, explicitly added namespaces, and an unauthorized namespace.
• Never render credential values into chart output, release notes, logs, or evidence.
• Keep the operator at one replica until multi-replica controller operation is qualified.

CI and release requirements

The current Helm workflow already runs on pull requests, merge queues, and pushes to main. Preserve and extend these gates:

• helm lint --strict for every chart;
• default and representative helm template renderings;
• JSON Schema rejection tests for invalid and unknown values;
• Kubernetes schema validation of rendered resources;
• byte-for-byte CRD synchronization checks;
• chart packaging checks;
• Kind install, readiness, helm test, upgrade, rollback where practical, uninstall, and CRD-retention checks;
• uploaded rendered manifests and failure logs with no Secret data; and
• pinned actions and checksummed downloaded tools.

Release validation must ensure:

• chart version matches the release version;
• appVersion matches the released Grid operator version;
• only versioned OCI artifacts are published;
• no latest tag is created;
• release notes contain the chart reference, version, digest, and install command; and
• pull-request workflows never receive package-write permission or publish artifacts.

Documentation

• Add an example-selection table explaining when to use each topology.
• Keep the chart values reference synchronized with values.yaml and values.schema.json.
• Document chart ownership and the temporary nature of the Praxis gateway chart.
• Document image compatibility requirements between Grid, Praxis, and mock providers.
• Include troubleshooting for image pulls, SWIM discovery, gateway readiness, overlay mounts, TLS, RBAC, and CRD upgrades.
• Avoid claiming that a chart lifecycle test using a placeholder gateway image proves Praxis runtime behavior; point to the global ingress runtime demo for that proof.

Acceptance criteria

• All three deployment examples exist and use the shipped charts.
• Every example has copy-and-paste install, verification, and teardown commands.
• The private four-cluster example proves Grid behavior without requiring global ingress.
• The global-ingress example retains the current runtime assertions and machine-readable evidence.
• Local checkout and versioned OCI installation paths are tested.
• CI runs strict static validation and Kind lifecycle validation for both charts.
• Upgrade, rollback, uninstall, reinstall, and CRD lifecycle behavior are documented and tested.
• RBAC and Secret boundaries have positive and negative coverage.
• Chart and container compatibility is documented with immutable references.
• Remaining gateway-operator ownership and migration work is explicitly tracked rather than hidden in the temporary chart.

Out of scope

• Building a global DNS or traffic-management service.
• Automating customer network connectivity or firewall configuration.
• Multi-cluster application lifecycle management outside the Grid components.
• Moving the temporary Praxis gateway chart to its eventual owning repository in this issue.