[New Plugin] NeuralTrust TrustGuard - #1772
Open
albertbausili wants to merge 2 commits into
Open
albertbausili wants to merge 2 commits into
albertbausili wants to merge 2 commits into
Conversation
Register neuraltrust.evaluate so Portkey can send chat, completion, and Anthropic Messages traffic to TrustGuard with fail-closed defaults.
albertbausili
added a commit
to NeuralTrust/docs
that referenced
this pull request
Sep 2, 2026
Portkey now has an in-gateway plugin, `neuraltrust.evaluate`, registered on both before_request_hooks and after_request_hooks. Unlike the BYOG webhook the page documented, it holds the request body, so it consumes transformed_payload and is the second gateway besides TrustGate that can redact. It is contributed in Portkey-AI/gateway#1772 (issue #1771) and not yet in a Portkey release, so the webhook stays as "Before the native plugin ships" — it is the only path that works on Portkey Cloud today. Coverage splits Portkey out of the boolean-verdict gateway rows it shared with Kong, Apigee and Azure APIM, into LiteLLM's shape: redact conditional on both surfaces, tool-level conditional throughout, since tool declarations and tool_calls reach the payload where Portkey populates them and a transformed tool call is written back request-level. The fail-closed behaviour is the part worth documenting carefully, because the plugin deliberately disagrees with the gateway. Portkey passes a check when it returns a verdict OR when it errored and failOnError is unset, and failOnError defaults to false — so an unreachable guardrail reported as an error would be forgiven and traffic would flow uninspected. The plugin returns verdict:false with error:null instead, which the engine cannot forgive, and reserves the error-shaped result for unreachableFallback: fail_open. Verified end to end against a locally built gateway: 446 with the provider never called on a forced auth failure and on an unreachable endpoint, a masked payload arriving at the provider from the before-hook, and a masked response reaching the caller from the after-hook. Also documents two traps found while doing it — `npm run build-plugins` drops every plugin missing from conf.json, and a streamed response reaches the after-hook with no parsed body, so the output side is not inspected at all rather than inspected late.
The plugin README and manifest carried no route to the fuller documentation, so a reviewer or an operator configuring the guardrail had only this directory to go on. `apiKey.description` renders in the Portkey UI, and it did not say where the key comes from. Pangea, Patronus, Qualifire, WalledAI and Lasso all answer that in the same field, so this follows them rather than inventing a place for it. README gains the setup guide and the endpoint contract, next to the paragraph that describes the check.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
neuraltrust.evaluateguardrail (issue [Feature] NeuralTrust TrustGuard #1771) so Portkey can send chat, completion, and Anthropic Messages traffic to TrustGuardPOST /v1/evaluate.allow/block/report/transformonto Portkey verdicts. Transform writes backmessages, completionprompt, or Anthropicsystem+tool_use. Block with hookdeny: trueis HTTP 446.verdict: false,error: null) so an unreachable TrustGuard cannot be forgiven.unreachableFallback: fail_openrestores house style for connect / timeout / 502 / 504 only.plugins/index.tsand listsneuraltrustinplugins_enabled. Credentials stay on the check (apiKey, optionalcollectorKey/apiBase).Description:
plugins/neuraltrust/(manifest, handler, payload codec, tests, README)plugins/index.tsneuraltrustadded toconf.json/conf.example.jsonplugins_enabledDocumentation:
POST /v1/evaluate: https://docs.neuraltrust.ai/trustguard/api/evaluateTests Run/Test cases added:
npx jest plugins/neuraltrust— 60 mocked tests (payload, transforms, fail-closed matrix, HTTP errors) plus optional live suite if.creds.jsonis presentnpx prettier --checkon the plugin filesType of Change:
Test plan
npx jest plugins/neuraltrust(60/60, including the optional live TrustGuard call)deny: true+ TrustGuardblockreturns HTTP 446 and does not call the providerallow/reportstill reach the model;transformredacts chatmessagesand completionpromptsystemandtool_useare preserved on extract/apply (unit coverage; not a live Anthropic provider run)apiKeyand unreachable TrustGuard fail closed unlessunreachableFallbackisfail_open