Skip to content

feat(server): log sanitized details for upstream auth rejections [SAO-17523] - #270

Merged
josjeon merged 7 commits into
mainfrom
sao-17523-upstream-diagnostics
Oct 8, 2026
Merged

josjeon merged 7 commits into
mainfrom
sao-17523-upstream-diagnostics

Conversation

@josjeon

@josjeon josjeon commented Sep 24, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Unexpected upstream 4xx authorization rejections currently log only the operation and HTTP status. This adds bounded, sanitized diagnostics showing whether target context was sent, the shape of its target fields, and the validation error kind and field path. The details appear in warning text and structured JSON, including when the host application owns logging.

Implements SAO-17523, a subtask of SAO-17418.

For example, an upstream uuid_parsing error at body.context.target_id is logged with that error kind and path, while the identifier is represented only as a string length. The client still receives 502 AUTH_UPSTREAM_REJECTED.

Scope

  • Summarize target context by presence and field shape; omit target values and credentials.
  • Retain only allowlisted validation kinds and field-path segments. Drop input, msg, and ctx; replace unknown kinds/segments with safe placeholders.
  • Limit the summary to five errors and four path segments. Parse bodies up to 64 KiB; oversized or unusable bodies receive an explicit status and an unknown total.
  • Emit indexed validation dictionaries so host-managed logging preserves the diagnostic fields. Agent Control JSON logs use proper JSON serialization and retain exception/stack information.
  • Rebase onto current main, preserving string operation labels and identity resolution. Add regression coverage for identity lookup returning 400/422 and typed internal diagnostic summaries.

Risk and Rollout

Low risk: the added work occurs on unexpected upstream 4xx responses. Existing authorization decisions and API error mappings remain unchanged. No migration or configuration change is required; revert the PR to roll back.

The original incident's rejected caller payload and root cause remain unconfirmed. Multitenant staging validation of deployed logs remains a follow-up.

Testing

  • make check passed on Python 3.12 with an isolated local PostgreSQL database: 2,560 tests passed and 1 optional test skipped; all Ruff and mypy checks passed.
  • Regression coverage includes 989 server tests, emitted text/JSON logs, redaction, malformed/deeply nested/oversized validation bodies, bounded error lists, and identity lookup 400/422 responses.
  • Independent security, logging, and reliability reviews found no actionable defects.
  • All seven PR commits have verified SSH signatures with author email josjeon@cisco.com.
  • GitHub checks passed: Python CI, UI CI, TypeScript SDK CI, server image build, and PR title validation.

Checklist

  • Linked ticket and scoped acceptance criteria.
  • Added automated coverage for sanitized diagnostics and rendered text/JSON output.
  • Preserved current main identity and authorization behavior.
  • Verify deployed multitenant staging logs for malformed and oversized upstream responses, including absence of caller values.

AI Tool Assistance Usage Statement

  • AI assistance was used for implementation, tests, review, and this description; the changes were reviewed and validated.

@codecov

codecov Bot commented Sep 24, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

@josjeon
josjeon force-pushed the sao-17523-upstream-diagnostics branch from eb9dd54 to e069e65 Compare September 28, 2026 16:48
josjeon added a commit that referenced this pull request Sep 28, 2026
…SAO-17580] (#272)

## Summary

`make typecheck` currently fails on `main` and on every open PR. This
annotates the four call sites that SQLAlchemy 2.1.0 broke, unblocking
merges.

The failure belongs to no branch's changes.

## What happened

SQLAlchemy 2.1.0 no longer lets mypy infer the element type through
`result.scalars().all()`:

```
server/src/agent_control_server/services/control_bindings.py:282:
  error: Need type annotation for "rows" (hint: "rows: list[<type>] = ...")  [var-annotated]
Found 1 error in 1 file (checked 51 source files)
make: *** [Makefile:138: typecheck] Error 1
```

The repository lock pins 2.0.51, which still infers it, so the error
appears only in CI, where dependencies resolve fresh. Main's own run at
`bd7d91f` installs both and reports the error in the same log:

```
+ mypy==2.3.1
+ sqlalchemy==2.1.0
server/src/agent_control_server/services/control_bindings.py:282: error: Need type annotation for "rows"
```

## Evidence this is not branch-specific

| Branch | Date | CI |
| --- | --- | --- |
| `main` | 2026-09-22 | success |
| `main` | 2026-09-24 | failure |
| `fix/sao-17418-runtime-token-auth` | 2026-09-24 | success |
| `fix/sao-17418-runtime-token-auth` | 2026-09-28 | failure |

SQLAlchemy 2.1.0 was released between those dates, and the flagged file
is not in the diff of either affected PR. Re-running the job does not
clear it: CI resolves 2.1.0 again each time, so the failure is
deterministic rather than flaky.

## What changed

Four sites share the pattern. CI reports only the first, so fixing them
one at a time would surface the next on the following run.

| File | Annotation |
| --- | --- |
| `services/control_bindings.py:282` | `rows: list[ControlBinding]` |
| `services/controls.py:306` | `versions: list[ControlVersion]` |
| `services/controls.py:512` | `controls: list[Control]` |
| `endpoints/agents.py:463` | `agents: Sequence[Agent]` |

`agents.py` keeps `Sequence` rather than `list` because that call site
does not wrap the result in `list()` and only reads and slices it.
`Sequence` and `Agent` were already imported there.

## Scope

- Type annotations only. No runtime behavior changes, no control flow,
no queries touched.
- Deliberately out of scope: raising the SQLAlchemy floor in
`pyproject.toml` or regenerating the lock. That is a wider dependency
decision, and these annotations are correct under either version.

## Risk and Rollout

- Risk level: minimal. Annotations are erased at runtime.
- No migration or configuration change.
- Rollback plan: revert this PR.

## Testing

- [x] `mypy server/src` clean under the locked versions: 51 source
files, no issues.
- [x] `ruff check server/src` clean.
- [ ] Full server test suite: not run locally. The local Docker daemon
is unresponsive, so the Postgres-backed suite cannot start. CI covers it
here, and the change is annotations only.
- [ ] Reproduced the failure locally against CI's resolved versions: not
possible right now, the internal package index returns 401. The CI log
for `main` is cited above instead.

## Checklist

- [x] Linked issue:
[SAO-17580](https://splunk.atlassian.net/browse/SAO-17580).
- [x] Documentation/examples: no update required.
- [x] Unblocks #268 (approved, blocked only by this) and #270. Both
touch different files, so this merges independently of either.

# AI Tool Assistance Usage Statement

- [x] AI assistance was used to draft parts of the implementation, that
was subsequently modified and extended.
- [ ] AI assistance was used in generating tests/documentation/comments
for this change.
- [x] AI assistance was used for optimizing/troubleshooting/refactoring
existing code in this change.
- [ ] AI assistance was used to draft this entire change as is.
@josjeon
josjeon enabled auto-merge (squash) September 28, 2026 21:11
@josjeon
josjeon force-pushed the sao-17523-upstream-diagnostics branch from e069e65 to 6396e25 Compare September 29, 2026 18:00
@josjeon
josjeon requested a review from wrisa September 29, 2026 18:56
@josjeon josjeon changed the title feat(server): Report sanitized diagnostics on upstream 4xx rejections [SAO-17523] feat(server): Add validation details to upstream auth logs [SAO-17523] Sep 29, 2026
… [SAO-17523]

When the upstream authorization service rejected a check, the log recorded
only the operation and the status code. During the 2026-09-23 multitenant
incident that left the cause unknown: the pods showed Orbit returning 422 and
Agent Control translating it to 502, with nothing to say which field was
rejected.

Attach three things to that warning: the operation, the shape of the target
context that was sent, and the field path plus error kind from the upstream
validation body. Orbit answers with the standard FastAPI validation envelope,
where `type` and `loc` alone separate a malformed target_id (`uuid_parsing` at
body.context.target_id) from an unsupported target_type (`enum` at
body.context.target_type) from an unknown operation (`enum` at body.operation).

Caller data stays out. Only `type` and `loc` are taken from each entry;
`input`, `ctx`, and `msg` echo the caller's values and are dropped. Target
context is described by shape rather than value, so absent, null, empty, and
wrongly typed stay distinguishable without logging an identifier. The logged
list is capped, with the total reported separately so a truncated list never
reads as complete.

Nothing Orbit-specific is added. Target values remain opaque, with no checks
for `log_stream` or UUID format, and the parser never raises: an unexpected
rejection body degrades to an empty summary instead of turning a 502 into a
500.
@josjeon
josjeon force-pushed the sao-17523-upstream-diagnostics branch from 500b07f to b170a0f Compare October 7, 2026 16:49
@josjeon josjeon changed the title feat(server): Add validation details to upstream auth logs [SAO-17523] feat(server): add sanitized upstream auth diagnostics [SAO-17523] Oct 7, 2026
@josjeon josjeon changed the title feat(server): add sanitized upstream auth diagnostics [SAO-17523] feat(server): log sanitized details for upstream auth rejections [SAO-17523] Oct 7, 2026
@josjeon
josjeon merged commit db9c667 into main Oct 8, 2026
8 checks passed
@josjeon
josjeon deleted the sao-17523-upstream-diagnostics branch October 8, 2026 01:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants