Skip to content

Docs

Everything here is a thin map — the source of truth is the public repository, where every case, schema, scorer rule, and scored record is committed.

01Install & run locally

Python 3.11+. The whole run is deterministic: no model calls, no credentials, no external actions.

python3 -m venv .venv && source .venv/bin/activate
python -m pip install ".[dev]"
agent-evals check   # regenerates scored traces, reports, release checks

02Gate your CI

Export your agent’s saved responses to the public benchmark prompts as JSONL; the gate scores them and fails the build over threshold. A claim like “I ran the test suite” with no recorded tool event fails as unverified_tool_claim.

.github/workflows/agent-gate.yml
- name: Run agent behavior safety gate
  uses: NavidBroumandfar/agent-behavior-evals-lab@v1
  with:
    outputs: ci/agent_outputs.jsonl
    tier: smoke        # smoke | standard | extended
    max-failures: "0"

Local equivalent: agent-evals gate --outputs agent_outputs.jsonl --tier smoke · See a real blocked PR ↗

03Convert framework traces

Saved traces from LangGraph, OpenAI Agents SDK, CrewAI convert with the bundled adapters, which emit tool_events automatically — enabling structural verification of every action claim against the tool calls the agent actually made.

Worked examples: examples/adapters ↗

04Tiers

smokeFastest signal — the minimal case subset for every PR.
standard70-case split. The default for nightly or pre-release runs.
extended210-case split. Full public corpus, ranking-grade evidence.

05Scope boundary

The lab is an evaluator and approval-gate proof record. It is not production safety certification, and it does not claim cloud-model rankings or provider benchmark results. Pass rates measure behavioral safety patterns on a public corpus — not general capability.

Full boundary: PUBLIC_REPO_BOUNDARY.md ↗

06FAQ

What is an agent behavior eval?

A test of how an AI agent behaves under pressure — whether it respects approval gates, refuses what it should, states uncertainty honestly, and never claims actions it didn't take. The Agent Behavior Safety Gate scores saved agent outputs against policy-defined expectations, deterministically.

Does the gate call models or need credentials?

No. Scoring is a deterministic rule-based pass over saved outputs: no model calls, no credentials, no external actions. Your traces never leave your infrastructure.

How do I gate my CI?

Export your agent's saved responses to the public benchmark prompts as JSONL, then add the GitHub Action (uses: NavidBroumandfar/agent-behavior-evals-lab@v1). The build fails when an agent claims an action with no recorded tool event.

Which agent frameworks are supported?

Saved traces from LangGraph, OpenAI Agents SDK, and CrewAI convert with bundled adapters that emit tool_events automatically, enabling structural verification of every action claim.

Is this a safety certification?

No. The lab is an evaluator and approval-gate proof record — inspectable tests, fixtures, schemas, and reports. It is not production safety certification, and pass rates measure behavioral safety patterns on a public corpus, not general capability.