Skip to content

feat(eval): online-eval BYO execution role, remove auto-provisioning (P3) - #2070

Open
jariy17 wants to merge 1 commit into
feat/online-insight-updatefrom
feat/online-eval-byo-role
Open

feat(eval): online-eval BYO execution role, remove auto-provisioning (P3)#2070
jariy17 wants to merge 1 commit into
feat/online-insight-updatefrom
feat/online-eval-byo-role

Conversation

@jariy17

@jariy17 jariy17 commented Aug 22, 2026

Copy link
Copy Markdown
Contributor

Draft · P3 of the online-insight stack. Stacked on #2069 (P2). Real-deploy bug bash done via RECORD; see below.

What

eval online-eval now requires the caller to bring their own execution role, and the CLI-side auto-provisioning is deleted entirely.

  • create: --role-arn is now required (guard rejects a missing value with required option '--role-arn <role-arn>' not specified). It is passed straight through to CreateOnlineEvaluationConfig.
  • update: --role-arn stays optional (replace); when omitted the service keeps the existing role.

Removed

  • src/core/onlineEvalExecutionRole.tsx (+ its unit test) — the whole default-role provisioning module: grantOnlineEvalScope / revokeOnlineEvalScope / onlineEvalExecutionRoleName / scopePolicyName / executionPolicy / accountIdFromRoleArn.
  • create's auto-provision fallback and the retryWhileRolePropagates IAM-propagation retry (a caller role that cannot be assumed now fails fast).
  • update's --update-role flag, the managed-role grant/revoke/KMS re-scope block, and the entire role-scope-warning path (RoleScopeWarning type, stderr advisory). updateOnlineEvaluationConfig now returns the bare UpdateOnlineEvaluationConfigResponse.
  • Now-dead helpers in core/eval.tsx: evaluatorKmsKeys, logGroupNamesOf (create no longer calls GetEvaluator or scopes a role).

Stack

P1 #2067 → P2 #2069P3 (this) → P4 TUI get/list. See the plan artifact.

Verification

  • tsc --noEmit clean.
  • online-eval goldens re-recorded against the AWS exploratory account (725476964917) — the real-deploy bug bash. Recording surfaced (and fixed) a real pause→delete UPDATING race by adding a settle() before delete. Goldens now show the caller-supplied role; no AgentCoreOnlineEval-* provisioned role, no IAM (GetRole/PutRolePolicy) fixtures. Verified no agentcore_cli_online_eval* residue in the account afterward.
  • Offline replay green: online-eval 25/25, online-insight 19/19 (untouched). Full suite 1751/1751.

@github-actions github-actions Bot added the size/xl PR size: XL label Aug 22, 2026
@github-actions github-actions Bot added agentcore-harness-reviewing AgentCore Harness review in progress and removed agentcore-harness-reviewing AgentCore Harness review in progress labels Aug 22, 2026
@jariy17
jariy17 marked this pull request as ready for review August 22, 2026 15:56
@jariy17 jariy17 changed the title feat(eval): online-eval BYO execution role, remove auto-provisioning (P3, draft) feat(eval): online-eval BYO execution role, remove auto-provisioning (P3) Aug 22, 2026
@github-actions github-actions Bot added size/xl PR size: XL and removed size/xl PR size: XL labels Aug 22, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size/xl PR size: XL

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant