Qualifire AI Open Supply Rogue: An end-to-end agent AI testing framework designed to evaluate AI agent efficiency, compliance, and reliability.

Agent techniques are probabilistic, context-dependent, and policy-bound. Conventional QA (unit checks, static prompts, or scalar “LLM-as-a-judge” scores) can’t reveal multi-turn vulnerabilities and audit trails are weak. Developer groups want protocol-accurate conversations, specific coverage checks, and machine-readable proof to gate releases with confidence.

Qualfire AI is now open sourced roguea Python framework for evaluating AI brokers in agent-to-agent (A2A) protocol. Rogue transforms enterprise insurance policies into actionable eventualities, drives multi-turn interactions with focused brokers, and outputs definitive studies appropriate for CI/CD and compliance critiques.

fast begin

Conditions

• uvx – If not put in, comply with these steps UV installation guide
• Python 3.10+
• API key on your LLM supplier (OpenAI, Google, Anthropic, and so on.).

set up

Possibility 1: Fast set up (advisable)

Automated set up scripts get you up and operating rapidly.

# TUI
uvx rogue-ai
# Net UI
uvx rogue-ai ui
# CLI / CI/CD
uvx rogue-ai cli

Possibility 2: Handbook set up

(a) Clone the repository.

git clone https://github.com/qualifire-dev/rogue.git
cd rogue

(b) Set up dependencies.

If utilizing UVs:

Or should you’re utilizing pip:

(c) Non-obligatory: Set atmosphere variables. Create a .env file in your root listing and add your API key. Rogue makes use of LiteLLM, so you may configure keys for various suppliers.

OPENAI_API_KEY="sk-..."
ANTHROPIC_API_KEY="sk-..."
GOOGLE_API_KEY="..."

operating rogue

Rogue operates on a consumer/server structure, the place the core analysis logic runs on the backend server, and totally different shoppers hook up with the backend server via totally different interfaces.

Default conduct

Once I run uvx rogue-ai with out specifying a mode I get:

1. Begins a rogue server within the background
2. Begin the TUI (Terminal Person Interface) consumer

Obtainable modes

• Default (server + TUI): uvx rogue-ai – begin server in background + TUI consumer
• server: uvx rogue-ai server – runs solely the backend server
• Tui: uvx rogue-ai tui – runs TUI consumer solely (requires server operating)
• Net UI: uvx rogue-ai ui – Runs the Gradio internet interface consumer solely (requires server operating)
• CLI: uvx rogue-ai cli – Runs non-interactive command line analysis (requires operating server, very best for CI/CD)

mode argument

server mode

uvx rogue-ai server [OPTIONS]

possibility:

• –host HOST – host operating the server (default: 127.0.0.1 or HOST atmosphere variable)
• –port PORT – Port on which to run the server (default: 8000 or PORT atmosphere variable)
• –debug – allow debug logging

TUI mode

uvx rogue-ai tui [OPTIONS]
Net UI Mode
uvx rogue-ai ui [OPTIONS]

possibility:

• –rogue-server-url URL – rogue server URL (default: http://localhost:8000)
• –port PORT – Port on which the UI runs
• –workdir WORKDIR – working listing (default: ./.rogue)
• –debug – allow debug logging

Instance: Testing a T-shirt retailer agent

This repository accommodates an instance of a easy agent that sells T-shirts. You should utilize this to see how Rogue works.

Set up instance dependencies.

If utilizing UVs:

Or should you’re utilizing pip:

pip set up -e .[examples]

(a) Begin the pattern agent server in a separate terminal.

If utilizing UVs:

uv run examples/tshirt_store_agent

If not:

python examples/tshirt_store_agent

It will begin the agent at http://localhost:10001.

(b) Set rogue Level to the pattern agent within the UI.

• Agent URL: http://localhost:10001
• Authentication: No authentication

(c) carry out and monitor assessments; rogue Check your T-shirt agent’s insurance policies.

You should utilize both TUI (uvx rogue-ai) mode or internet UI (uvx rogue-ai ui) mode.

The place is Rogue appropriate: Sensible use instances

• Enhancing security and compliance: Validate PII/PHI processing, denial actions, safety breach prevention, and regulatory area insurance policies with transcript-based proof.
• E-commerce and help agent: Implement OTP gate reductions, refund guidelines, SLA-aware escalations, and accuracy of device utilization (order lookup, ticketing) beneath hostile and failure situations.
• Developer/DevOps Agent: Consider code MODs and CLI copilots for workspace limitations, rollback semantics, fee limiting/backoff conduct, and unsafe command prevention.
• multi-agent system: Validate planner-executor contracts, functionality negotiations, and schema match through A2A. Consider interoperability between disparate frameworks.
• Regression and drift monitoring: Nightly suite for brand spanking new mannequin variations or immediate adjustments. Detect behavioral drift and implement policy-critical passing standards earlier than launch.

What precisely is a rogue and why ought to agent improvement groups care?

rogue An end-to-end testing framework designed to evaluate the efficiency, compliance, and reliability of AI brokers. rogue Combine enterprise context and threat into structured checks with clear targets, techniques, and success standards. EvaluatorAgent performs protocol-correct dialog in quick single-turn or deep multi-turn adversarial modes. Carry your individual mannequin or depart it to us rogue Conduct checks utilizing Qualifire’s custom-built SLM auditors. Streaming observability and deterministic artifacts: Rationale, timing, and mannequin/model lineage related to stay transcripts, cross/fail, and transcript spans.

Beneath the hood: How Rogue is constructed

Rogue works on a consumer/server structure.

• Rogue server: Comprises core analysis logic
• consumer interface: A number of interfaces connecting to the server:
  • Tui (Terminal UI): Trendy terminal interface constructed with Go and Bubble Tea
  • Net UI: Gradio-based internet interface
  • CLI: Command-line interface for automated evaluation and CI/CD

This structure permits for versatile deployment and utilization patterns the place servers can run independently and a number of shoppers can join concurrently.

abstract

rogue It helps developer groups take a look at the conduct of brokers truly operating in a manufacturing atmosphere. Translate written insurance policies into concrete eventualities, run these eventualities on A2A, and report what occurred with auditable transcripts. The result’s a transparent, repeatable sign that can be utilized in CI/CD to detect coverage violations and regressions earlier than they ship.

Because of the Qualifire workforce for offering the thought management/sources for this text. The Qualfire workforce helps this content material/article.