feat(workspaces): add team-backed multi-workspace GPU quotas

[Kevin Cho (chokevin)]

Summary

TauGrid previously treated one primary workspace and its same-name queue as the cluster-wide project boundary. Operators can now define TauTeam allocations, attach multiple TauWorkspace resources to a team, and give each workspace an explicit GPU guarantee while Kueue manages the residual team pool through a Cohort.

• Reconcile Team Cohorts, workspace ClusterQueues, LocalQueues, namespaces, and managed access resources with UID-based ownership protection.
• Preserve legacy externally managed workspace queues and discovery-driven quota growth while keeping Team-backed quota explicitly operator assigned.
• Route project configuration and lifecycle commands through an explicit workspace, and show workspace plus Team quota in the CLI and Portal.
• Retain namespaces and PVCs on deletion, reject unsafe retained-state adoption, and block quota reductions below active reservations.
• Keep RBAC redesign out of scope; existing researcher access behavior remains the authorization model.

Related issue

N/A

Validation

Passed: cd controllers/tau-core && make test lint
Passed: helm unittest charts/tau-core-controller (5 suites, 12 tests)
Passed: python3 scripts/check-license-headers.py
Passed: git diff --check
Passed: focused CLI, core, Portal Go/frontend, chart, documentation, and controller validation during implementation
Passed: remote arm64 Kind lifecycle using Podman and Kubernetes v1.35
  - installed generated Team and Workspace CRDs
  - reconciled one Team with two concurrent Workspaces
  - verified Cohort, ClusterQueues, LocalQueues, namespaces, owner UIDs, RBAC boundaries, quota admission, CLI routing, workload execution, namespace/PVC retention, and managed-resource cleanup
Passed: blocker-only ownership/continuity review found no remaining blockers

Compatibility and operational impact

• Adds the teams.tau.azure.com CRD and new optional Team/quota fields to TauWorkspace and TauCluster APIs.
• Helm teams and workspaces values default to empty, so existing installs do not create new resources automatically.
• Workspaces without teamRef retain the legacy external/same-name ClusterQueue contract.
• Existing discovery-labeled legacy ClusterQueues continue receiving discovered GPU flavors and quota growth; Team-backed queues do not.
• Project catalog entries may now specify workspace; conflicting catalog, run-policy, and flag values fail closed.
• Portal requires read access to Kueue Cohorts and adds /api/portal/quota.
• Rollback is the normal chart/application rollback; retained namespaces and PVCs are intentionally not deleted.

Checklist

• The change is focused and does not include unrelated edits.
• Tests cover the changed behavior, or I explained why tests are not needed.
• Documentation and examples are updated when public behavior changes.
• Generated files are updated from their source and included in this pull request.
• New dependencies are publicly available and their license and provenance were reviewed.
• The diff contains no secrets, private endpoints, customer data, or Microsoft-internal information.
• Fork-based CI can validate the change without repository secrets, or maintainer-only validation is clearly identified.

AI assistance: GitHub Copilot helped implement, test, review, and draft this description; the author verified the claims and final text.

[Pengfei Ni (feiskyer)]

This replaces the whole workspace after syncWorkspace has already reconciled its namespace and RBAC. If target or workload identity changes while waiting for the lock, the remaining steps install the new identity in the old namespace and can acknowledge the new generation using the refreshed resourceVersion. Restart the full sync on relevant spec changes, or refresh before resolving namespace/RBAC; cover the waiting-lock case through syncWorkspace.

[Pengfei Ni (feiskyer)]

This typed Get still reads the manager cache; quota locks do not make it authoritative. If the Workspace informer observes a guarantee reduction from 12 to 4 before the Team informer observes allocation 16 to 8, this pass can reduce the live queue to 4 then expand the Cohort pool to 12 using the stale Team. The live unstructured update has a current resourceVersion, and both accounting checks reuse that stale allocation. Use authoritative reads for locked quota accounting, including the nested Team reconciler, and cover independently lagging Team/Workspace caches.