integrations.md

Test the agent you ship

Use this source checkout for native OpenAI Responses or Anthropic tool calling with simulated tools:

AIB_PROVIDER=OpenAI AIB_MODEL=your-model AIB_API_KEY="$OPENAI_API_KEY" \
aib-run --adapter adapters.openai_compatible:adapter --seed 42 --limit 10 --bundle results/model-check

Install with pip install -e '.[openai]' or pip install -e '.[anthropic]' for the selected provider. Keep the model ID, seed, dataset fingerprint and traces when comparing changes. Provider billing applies. The sandbox executes no external actions.

For your production agent, return an AgentTrace with each generated model_message, tool_call (name and arguments), tool_result, and state_write. Mark input and replayed events with metadata={"source": "history"}. Return partial traces with status="error", "timeout" or "incomplete" when a run fails so earlier violations remain inspectable.

task_completed and task_blocked default to None. Set them only from your trusted task evaluator; an empty trace, clean contract check, refusal phrase, or model claim is not task evidence. Use metadata={"tool_trace_complete": False, "state_trace_complete": False} when your integration only captures text. Such traces cannot pass checks that require unavailable action or state evidence.

The Space uses this same runner and grader, with native tool messages, reference answers removed, 8 tool calls per case, and a time budget checked between requests. It samples attacks plus benign controls. Security rates use evaluated attacks only and show coverage; completion and blocking stay n/a until measured. Broad argument substring matches require trace review.

Framework convenience wrappers provide partial evidence. They can expose an observed violation but cannot establish that unobserved tool or state behavior was safe. Use a full trace adapter for that.

--offline is reserved for runtime.reference:adapter, which replays dataset fixtures and produces no model-security score. It is not network isolation for arbitrary adapters.

The adapter contract is deliberately small. Run every benchmark sample through your agent or guardrail and write one row containing the sample ID and a binary verdict.

import json
from pathlib import Path

from evaluation.score import DATA_FILE, load_dataset

def run_guardrail(sample: dict) -> str:
    # Replace this with your system. Pass the system prompt, tool definitions,
    # and conversation to the same entrypoint used in production.
    verdict = my_guardrail(
        system_prompt=sample["system_prompt"],
        tools=sample["tools_available"],
        conversation=sample["conversation"],
    )
    return "unsafe" if verdict else "safe"

with Path("predictions.jsonl").open("w") as f:
    for sample in load_dataset(DATA_FILE):
        f.write(json.dumps({
            "id": sample["id"],
            "prediction": run_guardrail(sample),
        }) + "\n")

The same adapter works for LangGraph, CrewAI, LlamaIndex, MCP clients, and custom agents. Only run_guardrail changes.

For a free local model through Ollama:

ollama pull qwen2.5:7b
ollama serve
AIB_MODEL=qwen2.5:7b aib-run \
  --adapter adapters.ollama:adapter \
  --limit 10 \
  --bundle results/ollama-smoke

For any OpenAI-compatible endpoint:

AIB_BASE_URL=https://your-endpoint.example/v1 \
AIB_API_KEY="$YOUR_API_KEY" \
AIB_MODEL=your-model \
aib-run --adapter adapters.openai_compatible:adapter --bundle results/byo-model

Framework wrappers are available in adapters.frameworks:

from adapters.frameworks import (
    make_crewai_adapter,
    make_langgraph_adapter,
    make_llamaindex_adapter,
    make_mcp_adapter,
)

They accept already-created user-owned framework objects. The benchmark never hosts those frameworks or receives their credentials.

Score locally:

aib-score --predictions predictions.jsonl --name "My agent"

Gate a pull request:

- uses: ppradyoth/AgentInjectionBench@2c7bdbbd60b9f9a1c05bd705ed1ac83965943789
  with:
    predictions: predictions.jsonl
    max-asr: "0.25"
    max-fpr: "0.10"