Human agency in the age of powerful AI
AI should help people solve real problems.
HumanWill gives companies and researchers evidence about when AI withholds legitimate help—and how useful its answers are. Our first open-source benchmark focuses on cybersecurity.
Public alpha · Open-source framework · Public question pack
The first benchmark
HumanWill · Cybersecurity benchmark
False-refusal rate (%) · Lower is better · 21 model configurations
Hover, focus or tap a model for five-topic results. Scroll sideways on smaller screens.
No matching models.
Search only filters this list. All and Clear all apply to every model. Bars stay ordered by false-refusal rate.
No models selected. Choose models above or select All to restore the full chart.
* Incomplete coverage · Rates use classified outcomes
Updated 13 Oct 2026 · First report (13 models) ↗
424 selected cybersecurity scenarios per configuration. Historical comparison: model settings, provider routes and run dates vary. Not everyday refusal rates or an overall safety score. Partial/full withholding counted; missing outcomes excluded. Muse Wonder added 13 Oct 2026; its 24 persistent zero-output Meta completions are explicitly counted as false refusals, alongside 25 semantically judged text refusals. Usefulness remains scored on 400 answers. Mistral Large 4.0 added 7 Oct 2026 via OpenRouter/Mistral, with default reasoning and a 65,536-token output cap. HumanWill · CC BY 4.0.
Put the evidence to work
Compare AI configurations against legitimate security workflows, from vulnerability assessment to incident investigation. Extend the reference pack with your own questions and customize judging criteria for your use case.
Run your first evaluation ↗Study false refusal and judged usefulness as separate measures. Review the methodology, run your own comparisons, and inspect your saved responses and judging policies through the CLI or agent API.
Explore the project & contribute ↗The framework uses Apache 2.0; the reference questions, report and comparison chart use CC BY 4.0. The question pack is public; historical model responses and judgments remain private. License scope ↗