feat(evals): real ServiceNow MCP behind the acme-servicenow demo

[benjaminshafii]

What & why

The marketplace demo evals (oauth-mcp-publish / oauth-mcp-install) featured an MCP named acme-servicenow that was really the generic mock (scripts/mock-oauth-mcp-server.mjs) with a single mock_echo tool and mcp:read/mcp:write scopes. This PR makes it the real thing: a faithful, spec-correct ServiceNow instance simulator standing behind the acme-servicenow connector.

scripts/mock-oauth-mcp-server.mjs is untouched (other flows/tests depend on it).

scripts/servicenow-mcp-server.mjs (new, zero-dependency)

MCP spec correctness (Streamable HTTP)

• Protocol-version negotiation over 2025-11-25 / 2025-06-18 / 2025-03-26 (echoes the client's when supported, else latest).
• JSON-RPC errors done right: -32700 parse, -32600 invalid/batch (batching removed in 2025-06-18), -32601 unknown method, -32602 bad tool args; notifications/client-responses → 202; ping → {}.
• Mcp-Session-Id issued on initialize, validated on later requests (unknown → 404), DELETE terminates.
• GET /mcp → 405 with Allow; MCP-Protocol-Version header validated; Origin checks (non-loopback → 403).
• Served at both /mcp and the realistic native path /sncapps/mcp-server/mcp/sn_openwork_it.

OAuth 2.1 on the real ServiceNow endpoint paths

• /oauth_auth.do + /oauth_token.do, pre-registered clients only (no DCR) — like a real instance's Application Registry.
• PKCE S256 required (rejects plain), auth-code + refresh grants, none / client_secret_post / client_secret_basic.
• RFC 9728 protected-resource metadata per MCP path (this fixed a real discovery gap — see below), RFC 8414 AS metadata (S256-only, no registration_endpoint), RFC 8707 resource-audience binding on issued tokens; every 401 carries WWW-Authenticate: Bearer resource_metadata=….

Real ServiceNow surface

• Seeded incident table (+ sys_user / sys_user_group), ServiceNow-shaped Table API (/api/now/table/incident, sysparm_query/_limit/_offset/_fields, ServiceNow error envelopes).
• 6 incident tools — search_incidents, get_incident, create_incident, update_incident, add_comment, resolve_incident — with LLM-oriented descriptions, JSON-Schema I/O, annotations, deep-links, and scope enforcement (incidents.read / incidents.write) surfaced as ServiceNow-ACL-style tool errors.

RFC 9728 fix worth calling out

The opencode engine follows the 401 WWW-Authenticate resource_metadata URL and then validates PRM.resource === connection URL. Serving one root PRM (resource=/mcp) 500'd the engine on the deep /sncapps/... path. The server now serves path-specific protected-resource metadata and points each path's 401 at its own PRM, and treats the two MCP URLs as one aliased resource for audience checks. Result: the engine completes OAuth over the realistic deep path with no DCR.

Tests — apps/server/src/mcp.servicenow-spec.e2e.test.ts (new)

• 10 pure-HTTP spec tests (277 assertions): PRM/AS discovery, PKCE/refresh/audience, scope enforcement, transport/session/origin semantics, tools/list + tools/call + structured content, Table API.
• 1 real-engine e2e (sidecar-gated): the opencode engine completes browser OAuth against the ServiceNow endpoints — no /register, over /sncapps/mcp-server/mcp/sn_openwork_it, with the RFC 8707 resource param.

How I verified (commands + results)

# apps/server — bun e2e
bun test src/mcp.servicenow-spec.e2e.test.ts   # 10 pass / 0 fail / 277 expect (engine handled /sncapps deep path)
bun test src/mcp.oauth-flow.e2e.test.ts        # 2 pass / 0 fail (untouched mock unaffected)

# Local fraimz (two-Electron Den demo: den-api :8788, den-web :3005, ServiceNow MCP :3979)
pnpm fraimz --flow oauth-mcp-publish  --cdp-url http://127.0.0.1:9923   # PASSED
pnpm fraimz --flow oauth-mcp-install  --cdp-url http://127.0.0.1:9924   # PASSED

Frame proof is posted as a follow-up comment.

Flow updates (current-dev reality)

• oauth-mcp-publish: points acme-servicenow at the real ServiceNow MCP (:3979, deep /sncapps/... path), auto-starts the server, and adapts to the current UI (engine-ready copy + the new Connect panel). Still proves secret redaction on export + publish.
• oauth-mcp-install: the old member marketplace card→Add path no longer exists on dev for cloud-runnable plugins (member self-connect now requires prior admin provisioning of the org MCP connection). The flow was rewritten to honestly prove, member-side, that the org-shared ServiceNow plugin is available to her and that her member-visible config carries clientId+scope but never the owner's secret. The member's own end-to-end OAuth against the ServiceNow server is covered by the engine e2e test above.
• Both flows now declare kind and carry per-frame voiceovers.

Notes / honesty

• Local run, not Daytona (Daytona not used here).
• mcp-oauth-silent-reauth fails at the silent-heal step on this machine (stuck "Issue"). My diff touches zero files in that flow's path (new files + the two oauth-mcp flows only), so this is pre-existing/environmental, not caused by this change — flagged for transparency.

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
openwork-app	Ready	Preview, Comment	Jul 9, 2026 4:27pm
openwork-den	Ready	Preview, Comment	Jul 9, 2026 4:27pm
openwork-den-worker-proxy	Ready	Preview, Comment	Jul 9, 2026 4:27pm
openwork-landing	Ready	Preview, Comment, Open in v0	Jul 9, 2026 4:27pm

[benjaminshafii]

fraimz — ✅ PASSED

1 passed · 0 failed · 0 skipped — run 2026-07-09T16-26-46-893Z

Full frame proof with validated screenshots: evals/results/2026-07-09T16-26-46-893Z/fraimz.html (re-run: pnpm fraimz --flow oauth-mcp-publish)

✅ oauth-mcp-publish — Owner exports skill + OAuth MCP (secret redacted) and publishes to a marketplace

User-facing flow demo

1. Owner is signed in to OpenWork Cloud

   🎙 Alex is signed in to OpenWork Cloud on his own desktop, and his Acme Robotics workspace is ready.

   • 📸 oauth-mcp-publish-01-owner-signed-in.png — 4 validations passed
   

2. OAuth MCP genuinely requires sign-in in Settings > Extensions

   🎙 Alex adds the Acme ServiceNow connector, and it appears in Extensions with a real Sign in needed badge. The secret stays hidden while the connector waits for OAuth.

   • ✅ Visible text does not include "acme-oauth-secret-98765"
   • 📸 oauth-mcp-publish-02-oauth-mcp-needs-signin.png — 6 validations passed
   

3. Portable export never contains the client secret

   🎙 The portable export is ready to share, and it keeps the ServiceNow connector details without showing the client secret.

4. Marketplace carries the plugin; published MCP has no secret

   🎙 Alex publishes the laptop refresh plugin to the BY IT Marketplace. The marketplace copy includes the connector people need, but the secret never appears.

5. Published plugin is visible in the Connect view

   🎙 Alex opens Connect and sees Laptop Refresh Policy shared with Acme Robotics. The page shows the plugin without exposing the ServiceNow secret.

   • ✅ Visible text does not include "acme-oauth-secret-98765"
   • 📸 oauth-mcp-publish-03-plugin-published.png — 6 validations passed
   

[benjaminshafii]

fraimz — ✅ PASSED

1 passed · 0 failed · 0 skipped — run 2026-07-09T16-27-00-165Z

Full frame proof with validated screenshots: evals/results/2026-07-09T16-27-00-165Z/fraimz.html (re-run: pnpm fraimz --flow oauth-mcp-install)

✅ oauth-mcp-install — Member sees the shared ServiceNow plugin; OAuth remains member-owned and secret-free

User-facing flow demo

1. Member (Rashmi) is signed in on her own app instance

   🎙 Rashmi is signed in on her own OpenWork desktop as an Acme Robotics member.

   • 📸 oauth-mcp-install-01-member-signed-in.png — 4 validations passed
   

2. Rashmi's member Den view exposes the shared ServiceNow plugin without the owner's secret

   🎙 Rashmi opens Connect and sees that Acme's shared Laptop Refresh Policy, backed by the ServiceNow connector, is available to her. The owner's OAuth client secret never appears in her member view.

   • ✅ URL hash includes "/settings/connect"
   • 📸 oauth-mcp-install-02-member-connect-panel.png — 5 validations passed
   

3. Member-visible ServiceNow MCP config has clientId and scope, never the owner's secret

   🎙 Using the shared ServiceNow connector still requires Rashmi's own ServiceNow sign-in. Her member-visible configuration carries the shared client ID and OAuth scope, but the owner's client secret never travels to her workspace.

   • ✅ URL hash includes "/settings/connect"