WardenBot AI · Open source

Run chatbot and agent security tests in pytest.

pytest-wardenbot gives developers a free local test harness for chatbot and LLM app failure modes, plus instrumented scenarios for agents that call tools, retrieve documents, and keep memory. A trusted recording sink observes what the agent actually did, so a crossed boundary becomes a test you keep in CI. Managed monitoring adds scheduled probes and alerts.

Python 3.11+ · Apache 2.0 · Beta (0.8.0). Maintained by the WardenBot AI organization. Runs locally with no WardenBot account and no telemetry. APIs may still change before 1.0; the changelog records each change.

The package name stays the same: pytest-wardenbot on PyPI.

terminal Apache 2.0
python -m pip install pytest-wardenbot
python -m pytest --wardenbot-quickstart
export CHATBOT_URL=https://your-bot.example/chat
# Edit conftest.py: response fields + real business facts
python -m pytest
Free plugin

Chatbot suite

  • 29 deterministic checks graded without a judge model, with accuracy measured on 744 labeled responses
  • Direct, indirect, and encoded prompt injection
  • System-prompt leak elicitation
  • Refusal bypass and multi-turn jailbreak patterns
  • Off-topic deflection
  • Canary-token leak detection in every leak check, plus business-truth assertions
  • Optional semantic, brand-voice, and hallucination checks
  • Deterministic tests for LLM pipelines: a local model endpoint serves your own SDK client scripted, replayed, or hostile replies, with prompt and output graders and no API key
Browse the chatbot checks →
Free plugin

Agent suite

  • Eight versioned MCP scenarios: poisoned tool results, tool descriptions, resources, and prompts
  • Opt-in MCP 2026-07-28 lifecycle scenarios: tool definitions that change after listing, and poisoned input requests mid-call
  • Five opt-in application scenarios in four packs: poisoned retrieval, cross-session memory, cross-tenant reads, viewer writes, and approval before write
  • A benign baseline paired with every attack trial
  • Explicit outcomes: violated, resisted, inconclusive, unsupported, or error, with task success reported separately
  • Blocked forbidden actions still count against the agent, so a guardrail can't hide an unsafe agent
  • Required-coverage mode, so missing evidence fails CI instead of passing
  • Ready-made targets for LangGraph and the Claude Agent SDK, plus recipes for the OpenAI Agents SDK and any tool loop
  • Runs fully local on Ollama, vLLM, or llama.cpp with no API key; models that can't call tools are reported as unsupported
  • An agent quickstart that tests a real model on the first run, then takes your own agent in its place
  • A preregistered run against Qwen and Mistral on Ollama, with every trial's evidence published
Select agent scenarios →
Continuous Monitoring

Managed monitoring unlocks

  • Managed adversarial test corpus
  • Scheduled external probes and alert routing
  • Adaptive attack chains for approved workflows
  • Behavior diffs and security evidence packs
  • Broader MCP and agent coverage for deployed systems
Fits your test suite

Connect your app. Keep the checks in your workflow.

Run locally or in CI with fixtures you control, readable failures, and a public test corpus.

Adapters for your stack

Connect an HTTP endpoint, OpenAI Responses or Chat Completions, Anthropic Messages, or a LangChain Runnable. Extend the adapter interface for your own app.

Choose an adapter →

Instrument your agent

Implement a small agent target for your own tool loop, or connect through the optional controlled MCP harness over in-memory, stdio, or loopback HTTP transports.

Instrument agent trials →

Portable evidence

Export versioned evidence bundles, regrade saved observations offline, compare a run with a recorded baseline, and review results as JUnit or standalone HTML.

Share and replay evidence →

Bounded agent campaigns

Run many isolated trials with shared call, turn, and time budgets, bounded concurrency, and cancellation cleanup. Every planned case keeps a result row.

Bound agent campaigns →

Parallel async probes

Use async adapters and the probe runner to test a corpus concurrently, with a limit on simultaneous requests and results for each prompt.

Run probes in parallel →

Optional model judging

Add semantic checks through DeepEval, or combine multiple judges with majority, unanimous, or any-pass consensus. Your model provider may charge for calls.

Configure LLM judges →

Deterministic grading does not need a judge model. Your chatbot's or agent's own API usage can still incur costs. Passing a fixed test corpus is a regression signal, not a security certification. The versioned coverage matrix lists which risk categories are tested and which are not.

Example

Write tests around facts, leaks, and refusals.

The free plugin is for engineering teams that want fast local checks. Agent scenarios run from the same project with pytest --pyargs pytest_wardenbot.agent_tests. Continuous Monitoring adds external browser-driven probes, behavior diffs, trace dashboards, and alerting.

conftest.py
# conftest.py
import pytest
from pytest_wardenbot.adapters.http import HTTPChatbotAdapter

@pytest.fixture
def chatbot():
    return HTTPChatbotAdapter(
        url="https://example.com/chat",
        request_field="message",
        response_field="response",
    )

# Run the bundled checks:
# Import bundled tests into a local test_chatbot.py
# so they use the fixtures in your project:
# from pytest_wardenbot.tests.test_prompt_injection import *
# pytest test_chatbot.py
Built in public

Inspect the checks. Help improve them.

Read the tests, review the release history, or contribute a regression case in the WardenBot AI GitHub organization. The plugin runs in your environment; no WardenBot account is required.

Ready for proof

Use pytest locally. Let WardenBot watch production.

Keep regression checks in your test suite. Choose managed monitoring for scheduled external probes, alerts, and evidence, with broader authenticated and MCP coverage available by plan.