Evaluate — measure what happened

Score a model's safety with red-team attacks and benchmark/judge evaluation.

Input contract

flowchart LR
    subgraph inputs [Inputs]
        M[Model]
        B[Benchmarks<br/>HarmBench · WildJailbreak · …]
        J[Judge<br/>WildGuard · HarmBench-Mistral · …]
    end
    M --> EVAL[evaluate]
    B --> EVAL
    J --> EVAL
    EVAL --> R[Metrics<br/>asr · refusal_rate · harmfulness_score]

Quick example

from safetune.evaluate import evaluate

results = evaluate(
    model,
    benchmarks=["harmbench", "xstest"],
    judge="wildguard",
)
print(results["harmbench"]["asr"])

Catalog of alternatives

Sub-kind	Methods	Guide
red-team attacks	AbliterationAttack, BoNAttack	Attacks overview
benchmarks	BenchmarkSpec, REGISTRY, list_benchmarks()	Benchmarks
judges	run_judge(), JudgeAdapter, JUDGE_REGISTRY	Judges
evaluation pipeline	evaluate(), evaluate_with_vllm_backend(), run_safety_eval()	Pipeline
spectral monitoring	SpectralEntropyMonitor	Monitoring
TamperBench harness	TamperBenchEvaluator	TamperBench
Pareto frontier	ParetoVisualizer, ParetoPoint	Pareto frontier

Naming

evaluate is the name (it measures, it cannot verify).