FronteraEval updateshttps://fronteraeval.org/2026-08-30T08:25:18.742ZNew and updated frontier AI evaluation records.AbstentionBench: Reasoning LLMs Fail on Unanswerable Questionshttps://fronteraeval.org/evaluations/inspect-abstention-bench/2026-08-30T00:00:00ZEvaluating abstention across 20 diverse datasets, including questions with unknown answers, underspecification, false premises, subjective interpretations, and outdated information.AgentBench: Evaluate LLMs as Agentshttps://fronteraeval.org/evaluations/inspect-agent-bench-os/2026-08-30T00:00:00ZA benchmark designed to evaluate LLMs as AgentsAgent Threat Bench Autonomy Hijackhttps://fronteraeval.org/evaluations/inspect-agent-threat-bench-autonomy-hijack/2026-08-30T00:00:00ZEvaluates LLM agents against the OWASP Top 10 for Agentic Applications (2026), measuring both task utility and security resilience across memory poisoning, autonomy hijacking, and data exfiltration scenarios.Agent Threat Bench Data Exfilhttps://fronteraeval.org/evaluations/inspect-agent-threat-bench-data-exfil/2026-08-30T00:00:00ZEvaluates LLM agents against the OWASP Top 10 for Agentic Applications (2026), measuring both task utility and security resilience across memory poisoning, autonomy hijacking, and data exfiltration scenarios.Agent Threat Bench Memory Poisonhttps://fronteraeval.org/evaluations/inspect-agent-threat-bench-memory-poison/2026-08-30T00:00:00ZEvaluates LLM agents against the OWASP Top 10 for Agentic Applications (2026), measuring both task utility and security resilience across memory poisoning, autonomy hijacking, and data exfiltration scenarios.AgentBoardhttps://fronteraeval.org/evaluations/canonical-agentboard/2026-08-30T00:00:00ZMulti-environment benchmark and analysis toolkit for language-model agents.AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agentshttps://fronteraeval.org/evaluations/inspect-agentdojo/2026-08-30T00:00:00ZAssesses whether AI agents can be hijacked by malicious third parties using prompt injections in simple environments such as a workspace or travel booking app.AgentHarmhttps://fronteraeval.org/evaluations/canonical-agentharm/2026-08-30T00:00:00ZEvaluates whether language-model agents can execute harmful multi-step tasks.Agentharm Benignhttps://fronteraeval.org/evaluations/inspect-agentharm-benign/2026-08-30T00:00:00ZAssesses whether AI agents might engage in harmful activities by testing their responses to malicious prompts in areas like cybercrime, harassment, and fraud, aiming to ensure safe behavior.Agie Aqua Rathttps://fronteraeval.org/evaluations/inspect-agie-aqua-rat/2026-08-30T00:00:00ZAGIEval is a human-centric benchmark specifically designed to evaluate the general abilities of foundation models in tasks pertinent to human cognition and problem-solving.Agie Logiqa Enhttps://fronteraeval.org/evaluations/inspect-agie-logiqa-en/2026-08-30T00:00:00ZAGIEval is a human-centric benchmark specifically designed to evaluate the general abilities of foundation models in tasks pertinent to human cognition and problem-solving.Agie Lsat Arhttps://fronteraeval.org/evaluations/inspect-agie-lsat-ar/2026-08-30T00:00:00ZAGIEval is a human-centric benchmark specifically designed to evaluate the general abilities of foundation models in tasks pertinent to human cognition and problem-solving.Agie Lsat Lrhttps://fronteraeval.org/evaluations/inspect-agie-lsat-lr/2026-08-30T00:00:00ZAGIEval is a human-centric benchmark specifically designed to evaluate the general abilities of foundation models in tasks pertinent to human cognition and problem-solving.Agie Lsat Rchttps://fronteraeval.org/evaluations/inspect-agie-lsat-rc/2026-08-30T00:00:00ZAGIEval is a human-centric benchmark specifically designed to evaluate the general abilities of foundation models in tasks pertinent to human cognition and problem-solving.Agie Mathhttps://fronteraeval.org/evaluations/inspect-agie-math/2026-08-30T00:00:00ZAGIEval is a human-centric benchmark specifically designed to evaluate the general abilities of foundation models in tasks pertinent to human cognition and problem-solving.Agie Sat Enhttps://fronteraeval.org/evaluations/inspect-agie-sat-en/2026-08-30T00:00:00ZAGIEval is a human-centric benchmark specifically designed to evaluate the general abilities of foundation models in tasks pertinent to human cognition and problem-solving.Agie Sat En Without Passagehttps://fronteraeval.org/evaluations/inspect-agie-sat-en-without-passage/2026-08-30T00:00:00ZAGIEval is a human-centric benchmark specifically designed to evaluate the general abilities of foundation models in tasks pertinent to human cognition and problem-solving.Agie Sat Mathhttps://fronteraeval.org/evaluations/inspect-agie-sat-math/2026-08-30T00:00:00ZAGIEval is a human-centric benchmark specifically designed to evaluate the general abilities of foundation models in tasks pertinent to human cognition and problem-solving.AHBhttps://fronteraeval.org/evaluations/register-ahb/2026-08-30T00:00:00ZA text-only safety benchmark for evaluating whether language models maintain refusal behavior under humanities-style adversarial reformulations of harmful prompts.AIME 2024: Problems from the American Invitational Mathematics Examinationhttps://fronteraeval.org/evaluations/inspect-aime2024/2026-08-30T00:00:00ZA benchmark for evaluating AI's ability to solve challenging mathematics problems from the 2024 AIME - a prestigious high school mathematics competition.AIME 2025: Problems from the American Invitational Mathematics Examinationhttps://fronteraeval.org/evaluations/inspect-aime2025/2026-08-30T00:00:00ZA benchmark for evaluating AI's ability to solve challenging mathematics problems from the 2025 AIME - a prestigious high school mathematics competition.AIME 2026: Problems from the American Invitational Mathematics Examinationhttps://fronteraeval.org/evaluations/inspect-aime2026/2026-08-30T00:00:00ZA benchmark for evaluating AI's ability to solve challenging mathematics problems from the 2026 AIME - a prestigious high school mathematics competition.AIR Bench: AI Risk Benchmarkhttps://fronteraeval.org/evaluations/inspect-air-bench/2026-08-30T00:00:00ZA safety benchmark evaluating language models against risk categories derived from government regulations and company policies.Alignment Fakinghttps://fronteraeval.org/evaluations/register-alignment-faking/2026-08-30T00:00:00ZMeasures whether LLMs strategically comply with harmful requests when believing outputs will be used for RLHF training (free-tier/monitored) versus when unmonitored (paid-tier). Uses 100 harmful prompts from LLM-LAT/harmful-dataset. Each prompt is presented twice with free-tier and paid-tier tags inside a system prompt describing a conflicting training objective. Primary metric is the compliance gap (free minus paid compliance rate via two-proportion z-test); a secondary scorer detects…ANIMA: Animal Norms In Moral Assessmenthttps://fronteraeval.org/evaluations/inspect-anima/2026-08-30T00:00:00ZEvaluates the quality of a model's moral reasoning about animal welfare across 13 ethical dimensions.APPS: Automated Programming Progress Standardhttps://fronteraeval.org/evaluations/inspect-apps/2026-08-30T00:00:00ZAPPS is a dataset for evaluating model performance on Python programming tasks across three difficulty levels consisting of 1,000 at introductory, 3,000 at interview, and 1,000 at competition level. The dataset consists of an additional 5,000 training samples, for a total of 10,000 total samples. We evaluate on questions from the test split, which consists of programming problems commonly found in coding interviews.AppWorldhttps://fronteraeval.org/evaluations/register-appworld/2026-08-30T00:00:00ZAppWorld evaluates autonomous agents on 750 day-to-day digital tasks requiring iterative Python code generation against 457 APIs across 9 simulated apps. Tasks are split into test_normal (168) and test_challenge (417, includes unseen Amazon/Gmail APIs). Scoring uses programmatic state-based unit tests checking database diffs for goal completion (Task Goal Completion, TGC) and collateral damage avoidance, rather than reference-solution comparison. Scenario Goal Completion (SGC) measures…AraTrusthttps://fronteraeval.org/evaluations/register-aratrust/2026-08-30T00:00:00ZAraTrust evaluates LLM trustworthiness when prompted in Arabic via 522 human-written multiple-choice questions (3 options each) spanning 8 categories: truthfulness, ethics, physical health, mental health, unfairness, illegal activities, privacy, and offensive language, with 34 subcategories. Questions were authored by native Arabic speakers or adapted from exams and existing datasets. Scoring is accuracy of selected answer against a single correct option. The eval code loads the dataset from…ARC Challengehttps://fronteraeval.org/evaluations/inspect-arc-challenge/2026-08-30T00:00:00ZDataset of natural, grade-school science multiple-choice questions (authored for human tests).ARC Easyhttps://fronteraeval.org/evaluations/inspect-arc-easy/2026-08-30T00:00:00ZDataset of natural, grade-school science multiple-choice questions (authored for human tests).ARC-AGI-2https://fronteraeval.org/evaluations/canonical-arc-agi-2/2026-08-30T00:00:00ZTests novel abstract reasoning and skill acquisition on tasks designed to resist memorised solutions.ArxivRollBenchhttps://fronteraeval.org/evaluations/register-arxivrollbench/2026-08-30T00:00:00ZA rolling benchmark for evaluating recent scientific text reasoning from arXiv papers. ArxivRollBench constructs multiple-choice sequencing, cloze, and next-fragment prediction tasks over newly released scientific papers across arXiv domains, with compact and full public releases.Assistant Bench Closed Book One Shothttps://fronteraeval.org/evaluations/inspect-assistant-bench-closed-book-one-shot/2026-08-30T00:00:00ZTests whether AI agents can perform real-world time-consuming tasks on the web.Assistant Bench Closed Book Zero Shothttps://fronteraeval.org/evaluations/inspect-assistant-bench-closed-book-zero-shot/2026-08-30T00:00:00ZTests whether AI agents can perform real-world time-consuming tasks on the web.Assistant Bench Web Browserhttps://fronteraeval.org/evaluations/inspect-assistant-bench-web-browser/2026-08-30T00:00:00ZTests whether AI agents can perform real-world time-consuming tasks on the web.Assistant Bench Web Search One Shothttps://fronteraeval.org/evaluations/inspect-assistant-bench-web-search-one-shot/2026-08-30T00:00:00ZTests whether AI agents can perform real-world time-consuming tasks on the web.Assistant Bench Web Search Zero Shothttps://fronteraeval.org/evaluations/inspect-assistant-bench-web-search-zero-shot/2026-08-30T00:00:00ZTests whether AI agents can perform real-world time-consuming tasks on the web.b3: Backbone Breaker Benchmarkhttps://fronteraeval.org/evaluations/inspect-b3/2026-08-30T00:00:00ZA comprehensive benchmark for evaluating LLMs for agentic AI security vulnerabilities including prompt attacks aimed at data exfiltration, content injection, decision and behavior manipulation, denial of service, system and tool compromise, and content policy bypass.Bbehhttps://fronteraeval.org/evaluations/inspect-bbeh/2026-08-30T00:00:00ZA reasoning capability dataset that replaces each task in BIG-Bench-Hard with a novel task that probes a similar reasoning capability but exhibits significantly increased difficulty.Bbeh Minihttps://fronteraeval.org/evaluations/inspect-bbeh-mini/2026-08-30T00:00:00ZA reasoning capability dataset that replaces each task in BIG-Bench-Hard with a novel task that probes a similar reasoning capability but exhibits significantly increased difficulty.BBH: Challenging BIG-Bench Taskshttps://fronteraeval.org/evaluations/inspect-bbh/2026-08-30T00:00:00ZTests AI models on a suite of 23 challenging BIG-Bench tasks that previously proved difficult even for advanced language models to solve.BBQ: Bias Benchmark for Question Answeringhttps://fronteraeval.org/evaluations/inspect-bbq/2026-08-30T00:00:00ZA dataset for evaluating bias in question answering models across multiple social dimensions.BFCLhttps://fronteraeval.org/evaluations/inspect-bfcl/2026-08-30T00:00:00ZEvaluates LLM function/tool-calling ability on a simplified split of the Berkeley Function-Calling Leaderboard (BFCL).BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructionshttps://fronteraeval.org/evaluations/inspect-bigcodebench/2026-08-30T00:00:00ZPython coding benchmark with 1,140 diverse questions drawing on numerous python libraries.BixBenchhttps://fronteraeval.org/evaluations/register-bixbench/2026-08-30T00:00:00ZBixBench evaluates LLM agents on open-ended bioinformatics data analysis tasks. Agents receive biological datasets and must produce Jupyter notebooks to analyze data and answer research questions. The eval uses a sandboxed IPython kernel environment where agents execute code interactively. Scoring is performed via an LLM judge comparing agent-generated answers against reference answers. The dataset is hosted on HuggingFace (futurehouse/BixBench) and tasks cover diverse bioinformatics analysis…BOLD: Bias in Open-ended Language Generation Datasethttps://fronteraeval.org/evaluations/inspect-bold/2026-08-30T00:00:00ZA dataset to measure fairness in open-ended text generation, covering five domains: profession, gender, race, religious ideologies, and political ideologies.BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionshttps://fronteraeval.org/evaluations/inspect-boolq/2026-08-30T00:00:00ZReading comprehension dataset that queries for complex, non-factoid information, and require difficult entailment-like inference to solve.BrokenMathhttps://fronteraeval.org/evaluations/register-brokenmath/2026-08-30T00:00:00ZBrokenMath measures sycophancy in LLMs by presenting adversarially falsified recent olympiad theorems and asking models to prove them. This eval runs the dataset's `benchmark` split — 451 adversarial, proof-style problems (perturbed from 2025 competition sources and verified by an IMO medalist). A 4-way LLM-as-judge classifies each response as correct/detected/corrected/incorrect; the primary metric is the sycophancy rate (fraction judged incorrect — the model attempted a proof of the false…BrowseComp: A Simple Yet Challenging Benchmark for Browsing Agentshttps://fronteraeval.org/evaluations/inspect-browse-comp/2026-08-30T00:00:00ZA benchmark for evaluating agents' ability to browse the web. The dataset consists of challenging questions that generally require web-access to answer correctly.CASTLEhttps://fronteraeval.org/evaluations/register-castle/2026-08-30T00:00:00ZCASTLE evaluates LLM vulnerability detection on 250 hand-crafted, compilable C programs covering 25 CWE types (6 vulnerable, 4 non-vulnerable per CWE). Models receive a system prompt requesting JSON output indicating vulnerability presence and CWE number. Scoring uses the CASTLE Score: +5 for correct vulnerability detection (minus 1 per extra false positive reported), +2 for correct true-negative identification, and -1 per false positive otherwise. TPR and FPR are also reported.