Source-linked catalogue

Frontier AI evaluations

A source-linked, evolving catalogue of 314 frontier AI evaluation records, with papers, implementations, provenance, evidence limits, and documentary methodological reviews.

Open semantic search and filters →

• AbstentionBench: Reasoning LLMs Fail on Unanswerable QuestionsPolina Kirichenko, Mark Ibrahim, Kamalika Chaudhuri et al.Evaluation integrity
• AgentBench: Evaluate LLMs as AgentsXiao Liu, Hao Yu, Hanchen Zhang et al.Autonomy & agents
• Agent Threat Bench Autonomy HijackAssociated paper authorsAutonomy & agents
• Agent Threat Bench Data ExfilAssociated paper authorsAutonomy & agents
• Agent Threat Bench Memory PoisonAssociated paper authorsAutonomy & agents
• AgentBoardHKUST NLP and the AgentBoard paper authorsAutonomy & agents · Evaluation integrity
• AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM AgentsEdoardo Debenedetti, Jie Zhang, Mislav Balunović et al.Autonomy & agents
• AgentDojoETH Zurich SPY Lab and Invariant LabsAutonomy & agents · Safeguards
• AgentharmGray Swan AI and the UK AI Security InstituteAutonomy & agents · Safeguards
• AgentHarmGray Swan AI and the UK AI Security InstituteAutonomy & agents · Safeguards
• Agentharm BenignGray Swan AI and the UK AI Security InstituteAutonomy & agents · Safeguards
• Agentic Misalignment: How LLMs could be insider threatsAnthropicDeception & misalignment · Autonomy & agents
• Agie Aqua RatWanjun Zhong, Ruixiang Cui, Yiduo Guo et al.General capability
• Agie Logiqa EnWanjun Zhong, Ruixiang Cui, Yiduo Guo et al.General capability
• Agie Lsat ArWanjun Zhong, Ruixiang Cui, Yiduo Guo et al.General capability
• Agie Lsat LrWanjun Zhong, Ruixiang Cui, Yiduo Guo et al.General capability
• Agie Lsat RcWanjun Zhong, Ruixiang Cui, Yiduo Guo et al.General capability
• Agie MathWanjun Zhong, Ruixiang Cui, Yiduo Guo et al.General capability
• Agie Sat EnWanjun Zhong, Ruixiang Cui, Yiduo Guo et al.General capability
• Agie Sat En Without PassageWanjun Zhong, Ruixiang Cui, Yiduo Guo et al.General capability
• Agie Sat MathWanjun Zhong, Ruixiang Cui, Yiduo Guo et al.General capability
• AHBMarcello Galisai, Susanna Cifani, Francesco Giarrusso et al.General capability
• AILuminateMLCommonsSafeguards
• AIME 2024: Problems from the American Invitational Mathematics ExaminationAssociated paper authorsGeneral capability
• AIME 2025: Problems from the American Invitational Mathematics ExaminationAssociated paper authorsGeneral capability
• AIME 2026: Problems from the American Invitational Mathematics ExaminationAssociated paper authorsGeneral capability
• AIR Bench: AI Risk BenchmarkYi Zeng, Yu Yang, Andy Zhou et al.Safeguards
• Alignment FakingRyan Greenblatt, Carson Denison, Benjamin Wright et al.Deception & misalignment
• ANIMA: Animal Norms In Moral AssessmentJasmine Brazilek, Miles TidmarshGeneral capability
• APE: Attempt to Persuade EvalFAR AIHuman influence & agency
• APPS: Automated Programming Progress StandardDan Hendrycks, Steven Basart, Saurav Kadavath et al.General capability
• AppWorldHarsh Trivedi, Tushar Khot, Mareike Hartmann et al.Autonomy & agents
• AraTrustEmad A. Alghamdi, Reem I. Masoud, Deema Alnuhait et al.General capability
• ARC ChallengePeter Clark, Isaac Cowhey, Oren Etzioni et al.General capability
• ARC EasyPeter Clark, Isaac Cowhey, Oren Etzioni et al.General capability
• ARC-AGI-2ARC Prize Foundation and the ARC-AGI-2 authorsGeneral capability · Evaluation integrity
• ArxivRollBenchZi Liang, Liantong Yu, Shiyu Zhang et al.Evaluation integrity
• Assistant Bench Closed Book One ShotOri Yoran, Samuel Joseph Amouyal, Chaitanya Malaviya et al.Autonomy & agents
• Assistant Bench Closed Book Zero ShotOri Yoran, Samuel Joseph Amouyal, Chaitanya Malaviya et al.Autonomy & agents
• Assistant Bench Web BrowserOri Yoran, Samuel Joseph Amouyal, Chaitanya Malaviya et al.Autonomy & agents
• Assistant Bench Web Search One ShotOri Yoran, Samuel Joseph Amouyal, Chaitanya Malaviya et al.Autonomy & agents
• Assistant Bench Web Search Zero ShotOri Yoran, Samuel Joseph Amouyal, Chaitanya Malaviya et al.Autonomy & agents
• b3: Backbone Breaker BenchmarkJulia Bazinska, Max Mathys, Francesco Casucci et al.General capability
• BbehMehran Kazemi, Bahare Fatemi, Hritik Bansal et al.General capability
• Bbeh MiniMehran Kazemi, Bahare Fatemi, Hritik Bansal et al.General capability
• BBH: Challenging BIG-Bench TasksMirac Suzgun, Nathan Scales, Nathanael Schärli et al.General capability
• BBQ: Bias Benchmark for Question AnsweringAlicia Parrish, Angelica Chen, Nikita Nangia et al.General capability
• BFCLBerkeley Gorilla / UC BerkeleyGeneral capability
• BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex InstructionsTerry Yue Zhuo, Minh Chien Vu, Jenny Chim et al.General capability
• BixBenchLudovico Mitchener, Jon M Laurent, Alex Andonian et al.Bio / CBRN
• BOLD: Bias in Open-ended Language Generation DatasetJwala Dhamala, Tony Sun, Varun Kumar et al.General capability
• BoolQ: Exploring the Surprising Difficulty of Natural Yes/No QuestionsChristopher Clark, Kenton Lee, Ming-Wei Chang et al.General capability
• BrokenMathIvo Petrov, Jasper Dekoninck, Martin VechevGeneral capability
• BrowseComp: A Simple Yet Challenging Benchmark for Browsing AgentsOpenAIAutonomy & agents
• CASTLERichard A. Dubniczky, Krisztofer Zoltán Horvát, Tamás Bisztray et al.General capability
• ChemBench: Are large language models superhuman chemists?Adrian Mirza, Nawaf Alampara, Sreekanth Kunchapu et al.Bio / CBRN
• ChipBench — DebuggingZhongkai Yu, Chenyang Zhou, Yichen Lin et al.General capability
• ChipBench — Reference ModelZhongkai Yu, Chenyang Zhou, Yichen Lin et al.General capability
• ChipBench — Verilog GenerationZhongkai Yu, Chenyang Zhou, Yichen Lin et al.General capability
• ClassEval: A Manually-Crafted Benchmark for Evaluating LLMs on Class-level Code GenerationXueying Du, Mingwei Liu, Kaixin Wang et al.General capability
• The Art of Saying No: Contextual Noncompliance in Language ModelsFaeze Brahman, Sachin Kumar, Vidhisha Balachandran et al.Safeguards
• CommonsenseQA: A Question Answering Challenge Targeting Commonsense KnowledgeAlon Talmor, Jonathan Herzig, Nicholas Lourie et al.General capability
• ComputeEval: CUDA Code Generation BenchmarkNVIDIAGeneral capability
• ContractBenchJicheng Wang, Yifeng He, Zili Wang et al.General capability
• CORE-BenchZachary S. Siegel, Sayash Kapoor, Nitya Nadgir et al.General capability
• CTI Realm 25Arjun Chakraborty, Sandra Ho, Adam Cook et al.Cyber
• CTI Realm 25 MinimalArjun Chakraborty, Sandra Ho, Adam Cook et al.Cyber
• CTI Realm 25 SeededArjun Chakraborty, Sandra Ho, Adam Cook et al.Cyber
• CTI Realm 50Arjun Chakraborty, Sandra Ho, Adam Cook et al.Cyber
• CVEBench: Benchmark for AI Agents Ability to Exploit Real-World Web Application VulnerabilitiesYuxuan Zhu, Antony Kellermann, Dylan Bowman et al.Cyber
• Cybench: Capture-The-Flag Cybersecurity ChallengesAndy K. Zhang, Neil Perry, Riya Dulepet et al.Cyber
• CyberGym: Evaluating AI Agents' Real-World Cybersecurity Capabilities at ScaleZhun Wang, Tianneng Shi, Jingxuan He et al.Cyber
• Cybermetric 10000Norbert Tihanyi, Mohamed Amine Ferrag, Ridhi Jain et al.Cyber
• Cybermetric 2000Norbert Tihanyi, Mohamed Amine Ferrag, Ridhi Jain et al.Cyber
• Cybermetric 500Norbert Tihanyi, Mohamed Amine Ferrag, Ridhi Jain et al.Cyber
• Cybermetric 80Norbert Tihanyi, Mohamed Amine Ferrag, Ridhi Jain et al.Cyber
• CyberSecEvalMetaCyber · Safeguards
• Cyse2 Interpreter AbuseManish Bhatt, Sahana Chennabasappa, Yue Li et al.General capability
• Cyse2 Prompt InjectionManish Bhatt, Sahana Chennabasappa, Yue Li et al.Cyber
• Cyse2 Vulnerability ExploitManish Bhatt, Sahana Chennabasappa, Yue Li et al.Cyber
• CYBERSECEVAL 3: Advancing the Evaluation of Cybersecurity Risks and Capabilities in Large Language ModelsManish Bhatt, Sahana Chennabasappa, Cyrus Nikolaidis et al.Cyber · Multimodal
• Cyse4 AutocompleteManish Bhatt, Sahana Chennabasappa, Yue Li et al.General capability
• Cyse4 InstructManish Bhatt, Sahana Chennabasappa, Yue Li et al.General capability
• Cyse4 Malware AnalysisManish Bhatt, Sahana Chennabasappa, Yue Li et al.Cyber
• Cyse4 MitreManish Bhatt, Sahana Chennabasappa, Yue Li et al.Cyber
• Cyse4 Mitre FrrManish Bhatt, Sahana Chennabasappa, Yue Li et al.Cyber
• Cyse4 Multilingual Prompt InjectionManish Bhatt, Sahana Chennabasappa, Yue Li et al.Cyber
• Cyse4 Multiturn PhishingManish Bhatt, Sahana Chennabasappa, Yue Li et al.Cyber
• Cyse4 Threat IntelligenceManish Bhatt, Sahana Chennabasappa, Yue Li et al.General capability
• Dangerous Capability EvaluationsGoogle DeepMindBio / CBRN · Cyber · Autonomy & agents
• DeceptionBenchPKU-Alignment and the DeceptionBench paper authorsHuman influence & agency · Deception & misalignment
• Do-Not-AnswerYuxia Wang, Haonan Li, Xudong Han et al.Safeguards
• Do-Not-Answer AdversarialYuxia Wang, Haonan Li, Xudong Han et al.Safeguards
• DocVQA: A Dataset for VQA on Document ImagesMinesh Mathew, Dimosthenis Karatzas, C. V. JawaharMultimodal
• DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over ParagraphsDheeru Dua, Yizhong Wang, Pradeep Dasigi et al.General capability
• DS-1000: A Natural and Reliable Benchmark for Data Science Code GenerationYuhang Lai, Chengxi Li, Yiming Wang et al.General capability
• ExploitBenchSeunghyun Lee, David BrumleyCyber
• Fortress AdversarialChristina Q. Knight, Kaustubh Deshpande, Ved Sirdeshmukh et al.Safeguards
• Fortress BenignChristina Q. Knight, Kaustubh Deshpande, Ved Sirdeshmukh et al.Safeguards
• FRAMESSatyapriya Krishna, Kalpesh Krishna, Anhad Mohananey et al.General capability
• Frontier CsQiuyang Mang, Wenhao Chai, Zhifei Li et al.General capability
• Frontier Cs AlgorithmicQiuyang Mang, Wenhao Chai, Zhifei Li et al.General capability
• Frontier Cs ResearchQiuyang Mang, Wenhao Chai, Zhifei Li et al.AI R&D
• FrontierMathEpoch AIGeneral capability · AI R&D
• FrontierScience: Expert-Level Scientific ReasoningAssociated paper authorsGeneral capability
• GaiaGrégoire Mialon, Clémentine Fourrier, Craig Swift et al.Autonomy & agents
• Gaia Level1Grégoire Mialon, Clémentine Fourrier, Craig Swift et al.Autonomy & agents
• Gaia Level2Grégoire Mialon, Clémentine Fourrier, Craig Swift et al.Autonomy & agents
• Gaia Level3Grégoire Mialon, Clémentine Fourrier, Craig Swift et al.Autonomy & agents
• GDM Classifier EvasionGoogle DeepMindGeneral capability
• GDM Cover Your TracksGoogle DeepMindDeception & misalignment
• GDM Dangerous Capabilities: Capture the FlagMary Phuong, Matthew Aitchison, Elliot Catt et al.Cyber
• InterCode: Security and Coding Capture-the-Flag ChallengesJohn Yang, Akshara Prabhakar, Karthik Narasimhan et al.Cyber
• GDM Oversight PatternGoogle DeepMindDeception & misalignment
• GDM Self Reasoning Approved DirectoriesGoogle DeepMindDeception & misalignment
• GDM Self Reasoning Calculator ImprovementGoogle DeepMindDeception & misalignment
• GDM Self Reasoning Context Length Mod Instrumental OnlyGoogle DeepMindDeception & misalignment
• GDM Self Reasoning Context Length Mod Irreversibility OnlyGoogle DeepMindDeception & misalignment
• GDM Self Reasoning Database ToolGoogle DeepMindDeception & misalignment
• GDM Self Reasoning Latency CalculatorGoogle DeepMindDeception & misalignment
• GDM Self Reasoning Max Messages CalculatorGoogle DeepMindDeception & misalignment
• GDM Self Reasoning Max TokensGoogle DeepMindDeception & misalignment
• GDM Self Reasoning Oversight FrequencyGoogle DeepMindDeception & misalignment
• GDM Self Reasoning Read LogsGoogle DeepMindDeception & misalignment
• GDM Self Reasoning Turn Off FiltersGoogle DeepMindDeception & misalignment
• GDM Sp01 E2eMary Phuong, Matthew Aitchison, Elliot Catt et al.General capability
• GDM Sp01 MilestonesMary Phuong, Matthew Aitchison, Elliot Catt et al.General capability
• GDM Sp02 E2eMary Phuong, Matthew Aitchison, Elliot Catt et al.General capability
• GDM Sp02 MilestonesMary Phuong, Matthew Aitchison, Elliot Catt et al.General capability
• GDM Sp03 E2eMary Phuong, Matthew Aitchison, Elliot Catt et al.General capability
• GDM Sp03 MilestonesMary Phuong, Matthew Aitchison, Elliot Catt et al.General capability
• GDM Sp04 E2eMary Phuong, Matthew Aitchison, Elliot Catt et al.General capability
• GDM Sp04 MilestonesMary Phuong, Matthew Aitchison, Elliot Catt et al.General capability
• GDM Sp05 E2eMary Phuong, Matthew Aitchison, Elliot Catt et al.General capability
• GDM Sp05 MilestonesMary Phuong, Matthew Aitchison, Elliot Catt et al.General capability
• GDM Sp07 E2eMary Phuong, Matthew Aitchison, Elliot Catt et al.General capability
• GDM Sp07 MilestonesMary Phuong, Matthew Aitchison, Elliot Catt et al.General capability
• GDM Sp08 E2eMary Phuong, Matthew Aitchison, Elliot Catt et al.General capability
• GDM Sp08 MilestonesMary Phuong, Matthew Aitchison, Elliot Catt et al.General capability
• GDM Sp09 E2eMary Phuong, Matthew Aitchison, Elliot Catt et al.General capability
• GDM Sp09 MilestonesMary Phuong, Matthew Aitchison, Elliot Catt et al.General capability
• GDM Sp10 E2eMary Phuong, Matthew Aitchison, Elliot Catt et al.General capability
• GDM Sp10 MilestonesMary Phuong, Matthew Aitchison, Elliot Catt et al.General capability
• GDM Sp12 E2eMary Phuong, Matthew Aitchison, Elliot Catt et al.General capability
• GDM Sp12 MilestonesMary Phuong, Matthew Aitchison, Elliot Catt et al.General capability
• GDM Strategic Rule BreakingGoogle DeepMindDeception & misalignment
• GDPvalTejal Patwardhan, Rachel Dias, Elizabeth Proehl et al.General capability
• GPQA: Graduate-Level STEM Knowledge ChallengeDavid Rein, Betty Li Hou, Asa Cooper Stickland et al.General capability
• GSM8K: Grade School Math Word ProblemsKarl Cobbe, Vineet Kosaraju, Mohammad Bavarian et al.General capability
• Hangman BenchMatt FisherGeneral capability
• HarmBenchCenter for AI Safety and the HarmBench paper authorsSafeguards
• HealthbenchOpenAIGeneral capability
• Healthbench ConsensusOpenAIGeneral capability
• Healthbench HardOpenAIGeneral capability
• Healthbench Meta EvalOpenAIGeneral capability
• HellaSwag: Commonsense Event ContinuationRowan Zellers, Ari Holtzman, Yonatan Bisk et al.General capability
• HELM SafetyStanford CRFMSafeguards · Evaluation integrity
• HumanEval: Python Function Generation from InstructionsMark Chen, Jerry Tworek, Heewoo Jun et al.General capability
• Humanity's Last ExamLong Phan, Alice Gatti, Ziwen Han et al.General capability
• IFEval: Instruction-Following EvaluationJeffrey Zhou, Tianjian Lu, Swaroop Mishra et al.General capability
• IFEvalCode: Controlled Code GenerationJian Yang, Wei Zhang, Shukai Liu et al.General capability
• Infinite Bench Code DebugXinrong Zhang, Yingfa Chen, Shengding Hu et al.General capability
• Infinite Bench Code RunXinrong Zhang, Yingfa Chen, Shengding Hu et al.General capability
• Infinite Bench Kv RetrievalXinrong Zhang, Yingfa Chen, Shengding Hu et al.General capability
• Infinite Bench Longbook Choice EngXinrong Zhang, Yingfa Chen, Shengding Hu et al.General capability
• Infinite Bench Longdialogue QA EngXinrong Zhang, Yingfa Chen, Shengding Hu et al.General capability
• Infinite Bench Math CalcXinrong Zhang, Yingfa Chen, Shengding Hu et al.General capability
• Infinite Bench Math FindXinrong Zhang, Yingfa Chen, Shengding Hu et al.General capability
• Infinite Bench Number StringXinrong Zhang, Yingfa Chen, Shengding Hu et al.General capability
• Infinite Bench PasskeyXinrong Zhang, Yingfa Chen, Shengding Hu et al.General capability
• Inspect AIUK AI Security InstituteEvaluation integrity
• BharatBBQAbhishek Kumar Singh, Shrey Nag, Sachita et al.General capability
• India Cultural KnowledgeAbhishek Kumar Singh, Shrey Nag, Sachita et al.General capability
• India DPI SafetyAbhishek Kumar Singh, Shrey Nag, Sachita et al.Safeguards
• India Jailbreak SafetyAbhishek Kumar Singh, Shrey Nag, Sachita et al.Safeguards
• India MultilingualAbhishek Kumar Singh, Shrey Nag, Sachita et al.General capability
• India Multilingual SafetyAbhishek Kumar Singh, Shrey Nag, Sachita et al.Safeguards
• InstrumentalEval - Evaluating the Paperclip Maximizer: Are RL-Based Language Models More Likely to Pursue Instrumental Goals?Yufei He, Yuexin Li, Jiaying Wu et al.Deception & misalignment
• CodeIPI: Indirect Prompt Injection for Coding AgentsDebu SinhaAutonomy & agents · Cyber
• JailbreakBenchJailbreakBench project and paper authorsSafeguards · Evaluation integrity
• KernelBench: Can LLMs Write Efficient GPU Kernels?Associated paper authorsGeneral capability
• LAB-Bench 2Jon M Laurent, Albert Bou, Michael Pieler et al.Bio / CBRN
• Lab Bench Cloning ScenariosJon M. Laurent, Joseph D. Janizek, Michael Ruzo et al.Bio / CBRN
• Lab Bench DbqaJon M. Laurent, Joseph D. Janizek, Michael Ruzo et al.Bio / CBRN
• Lab Bench FigqaJon M. Laurent, Joseph D. Janizek, Michael Ruzo et al.Bio / CBRN
• Lab Bench LitqaJon M. Laurent, Joseph D. Janizek, Michael Ruzo et al.Bio / CBRN
• Lab Bench ProtocolqaJon M. Laurent, Joseph D. Janizek, Michael Ruzo et al.Bio / CBRN
• Lab Bench SeqqaJon M. Laurent, Joseph D. Janizek, Michael Ruzo et al.Bio / CBRN
• Lab Bench SuppqaJon M. Laurent, Joseph D. Janizek, Michael Ruzo et al.Bio / CBRN
• Lab Bench TableqaJon M. Laurent, Joseph D. Janizek, Michael Ruzo et al.Bio / CBRN
• LingolyJude Khouja, Lingyi Yang, Karolina Korgul et al.General capability
• Lingoly TooJude Khouja, Lingyi Yang, Karolina Korgul et al.General capability
• LiveBench: A Challenging, Contamination-Free LLM BenchmarkColin White, Samuel Dooley, Manley Roberts et al.Evaluation integrity
• LiveCodeBench-Pro: Competitive Programming BenchmarkZihan Zheng, Zerui Cheng, Zeyu Shen et al.General capability
• lm-evaluation-harnessEleutherAIEvaluation integrity · General capability
• MaCBench: Probing the limitations of multimodal language models for chemistry and materials researchNawaf Alampara, Mara Schilling-Wilhelmi, Martiño Ríos-García et al.General capability
• MACHIAVELLIMACHIAVELLI paper authorsHuman influence & agency
• Make Me PayOpenAI EvalsHuman influence & agency
• MakeMeSayOpenAI EvalsHuman influence & agency
• MCBManager Coercion Benchmark paper authorsHuman influence & agency
• MANTAIsabella Luong, Joyee Chen, Sankalpa Ghose et al.Autonomy & agents
• MASK: Disentangling Honesty from Accuracy in AI SystemsCenter for AI SafetyHuman influence & agency
• MATH: Measuring Mathematical Problem SolvingDan Hendrycks, Collin Burns, Saurav Kadavath et al.General capability
• MathVista: Visual Math Problem-SolvingPan Lu, Hritik Bansal, Tony Xia et al.Multimodal
• MBPP: Basic Python Coding ChallengesJacob Austin, Augustus Odena, Maxwell Nye et al.General capability
• MCPToxZhiqiang Wang, Yichao Gao, Yanting Wang et al.Safeguards
• MedCalc-BenchNikhil Khandekar, Qiao Jin, Guangzhi Xiong et al.General capability
• MedQA: Medical exam Q&A benchmarkDi Jin, Eileen Pan, Nassim Oufattole et al.General capability
• METR Time HorizonsMETRAutonomy & agents · AI R&D
• MGSM: Multilingual Grade School MathFreda Shi, Mirac Suzgun, Markus Freitag et al.General capability
• Mind2Web: Towards a Generalist Agent for the WebXiang Deng, Yu Gu, Boyuan Zheng et al.Autonomy & agents
• Mind2Web-SCZhen Xiang, Linzhi Zheng, Yanjie Li et al.Autonomy & agents
• Mle BenchOpenAIAI R&D
• Mle Bench FullOpenAIAI R&D
• Mle Bench LiteOpenAIAI R&D
• MLRC-Bench: Can Language Agents Solve Machine Learning Research Challenges?Yunxiang Zhang, Muhammad Khalifa, Shitanshu Bhushan et al.AI R&D
• MM-SafetyBenchMM-SafetyBench paper authorsSafeguards · Multimodal
• MMIU: Multimodal Multi-image Understanding for Evaluating Large Vision-Language ModelsFanqing Meng, Jin Wang, Chuanhao Li et al.General capability
• MMLU 0 ShotDan Hendrycks, Collin Burns, Steven Basart et al.General capability
• MMLU 5 ShotDan Hendrycks, Collin Burns, Steven Basart et al.General capability
• MMLU-Pro: Advanced Multitask Knowledge and Reasoning EvaluationYubo Wang, Xueguang Ma, Ge Zhang et al.General capability
• Mmmu Multiple ChoiceXiang Yue, Yuansheng Ni, Kai Zhang et al.Multimodal
• Mmmu OpenXiang Yue, Yuansheng Ni, Kai Zhang et al.Multimodal
• MonitorBench (Goal sandbag math subset)Han Wang, Yifan Sun, Brian Ko et al.Deception & misalignment · Evaluation integrity
• MonitorBench (Steganography subset)Han Wang, Yifan Sun, Brian Ko et al.Evaluation integrity
• MORU: Moral Reasoning under UncertaintyCompassion in Machine LearningGeneral capability
• MuSR: Testing the Limits of Chain-of-thought with Multistep Soft ReasoningZayne Sprague, Xi Ye, Kaj Bostrom et al.General capability
• NARCBenchAaron Rose, Carissa Cullen, Sahar Abdelnabi et al.Safeguards
• Needle in a Haystack (NIAH): In-Context Retrieval Benchmark for Long Context LLMsElliot Nelson, Georgios Kollias, Payel Das et al.General capability
• NoveltyBench: Evaluating Language Models for Humanlike DiversityYiming Zhang, Harshita Diddee, Susan Holm et al.General capability
• O-NET: A high-school student knowledge testNational Institute of Educational Testing Service (NIETS)General capability
• OpenAI EvalsOpenAIEvaluation integrity · General capability
• OpenBookQATodor Mihaylov, Peter Clark, Tushar Khot et al.General capability
• OR-BenchJustin Cui, Wei-Lin Chiang, Ion Stoica et al.General capability
• OSWorldTianbao Xie, Danyang Zhang, Jixuan Chen et al.Autonomy & agents
• Osworld SmallTianbao Xie, Danyang Zhang, Jixuan Chen et al.Autonomy & agents
• PaperBench: Evaluating AI's Ability to Replicate AI Research (Work In Progress)OpenAIHuman influence & agency · Autonomy & agents · AI R&D
• PatchEvalZichao Wei, Jun Zeng, Ming Wen et al.General capability
• PAWS: Paraphrase Adversaries from Word ScramblingYuan Zhang, Jason Baldridge, Luheng HeGeneral capability
• Persistbench Beneficial MemorySidharth Pulipaka, Oliver Chen, Manas Sharma et al.General capability
• Persistbench Cross DomainSidharth Pulipaka, Oliver Chen, Manas Sharma et al.General capability
• Persistbench JudgeSidharth Pulipaka, Oliver Chen, Manas Sharma et al.Evaluation integrity
• Persistbench SycophancySidharth Pulipaka, Oliver Chen, Manas Sharma et al.Human influence & agency
• Personality BFIGuiem SolansGeneral capability
• Personality TRAITGuiem SolansGeneral capability
• PerspectiveGapYouran Sun, Xingyu Ren, Kejia Zhang et al.Cyber
• PinchBenchPinchBench projectGeneral capability
• PIQA: Physical Commonsense Reasoning TestYonatan Bisk, Rowan Zellers, Ronan Le Bras et al.General capability
• Pre-Flight: Aviation Operations Knowledge EvaluationAlex Brooker, Tim HughesEvaluation integrity
• PubMedQA: A Dataset for Biomedical Research Question AnsweringQiao Jin, Bhuwan Dhingra, Zhengping Liu et al.General capability
• RACE-H: A benchmark for testing reading comprehension and reasoning abilities of neural modelsGuokun Lai, Qizhe Xie, Hanxiao Liu et al.General capability
• RE-BenchMETRAutonomy & agents · AI R&D
• Sabotage EvaluationsAnthropicDeception & misalignment · AI R&D
• Sad Facts Human DefaultsSAD paper authorsGeneral capability
• Sad Facts LlmsSAD paper authorsGeneral capability
• SAD InfluenceSAD paper authorsHuman influence & agency
• Sad Stages FullSAD paper authorsGeneral capability
• Sad Stages OversightSAD paper authorsGeneral capability
• SafetyBenchTHU-COAISafeguards
• SALAD-BenchLijun Li, Bowen Dong, Ruohui Wang et al.Safeguards
• scBench: A Benchmark for Single-Cell RNA-seq AnalysisKenny Workman, Zhen Yang, Harihara Muralidharan et al.General capability
• SciCode: A Research Coding Benchmark Curated by ScientistsMinyang Tian, Luyu Gao, Shizhuo Dylan Zhang et al.General capability
• SciKnowEval: Evaluating Multi-level Scientific Knowledge of Large Language ModelsKehua Feng, Xinyi Shen, Weijie Wang et al.General capability
• Sec QA V1Zefang LiuGeneral capability
• Sec QA V1 5 ShotZefang LiuGeneral capability
• Sec QA V2Zefang LiuGeneral capability
• Sec QA V2 5 ShotZefang LiuGeneral capability
• Sevenllm Mcq EnHangyuan Ji, Jian Yang, Linzheng Chai et al.General capability
• Sevenllm Mcq ZhHangyuan Ji, Jian Yang, Linzheng Chai et al.General capability
• Sevenllm QA EnHangyuan Ji, Jian Yang, Linzheng Chai et al.General capability
• Sevenllm QA ZhHangyuan Ji, Jian Yang, Linzheng Chai et al.General capability
• SimpleqaLukas Haas, Gal Yona, Giovanni D'Antonio et al.General capability
• Simpleqa VerifiedLukas Haas, Gal Yona, Giovanni D'Antonio et al.General capability
• SOS BENCH: Benchmarking Safety Alignment on Scientific KnowledgeFengqing Jiang, Fengbo Ma, Zhangchen Xu et al.General capability
• SQuAD: A Reading Comprehension Benchmark requiring reasoning over Wikipedia articlesPranav Rajpurkar, Jian Zhang, Konstantin Lopyrev et al.General capability
• StereoSet: Measuring stereotypical bias in pretrained language modelsMoin Nadeem, Anna Bethke, Siva ReddyGeneral capability
• StrongREJECT: Measuring LLM susceptibility to jailbreak attacksAlexandra Souly, Qingyuan Lu, Dillon Bowen et al.Safeguards
• Swe BenchCarlos E. Jimenez, John Yang, Alexander Wettig et al.General capability
• Swe Bench Verified MiniCarlos E. Jimenez, John Yang, Alexander Wettig et al.General capability
• SWE-Lancer: Can Frontier LLMs Earn $1 Million from Real-World Freelance Software Engineering?Samuel Miserendino, Michele Wang, Tejal Patwardhan et al.General capability
• SycoBench-600Associated paper authorsGeneral capability
• Sycophancy EvalAnthropic and the paper authorsHuman influence & agency
• TacJasmine Brazilek, Joel Christoph, Maheep Chaudhary et al.General capability
• Tac WelfareJasmine Brazilek, Joel Christoph, Maheep Chaudhary et al.General capability
• TarantuBenchTarantuLabsGeneral capability
• Tau2 AirlineVictor Barres, Honghua Dong, Soham Ray et al.Autonomy & agents
• Tau2 BankingVictor Barres, Honghua Dong, Soham Ray et al.Autonomy & agents
• Tau2 RetailVictor Barres, Honghua Dong, Soham Ray et al.Autonomy & agents
• Tau2 TelecomVictor Barres, Honghua Dong, Soham Ray et al.Autonomy & agents
• Terminal-BenchStanford and the Laude InstituteAutonomy & agents · Cyber
• The Agent Company: Evaluating multi-tool autonomous agents in a synthetic companyFrank F. Xu, Yufan Song, Boxuan Li et al.Autonomy & agents
• Catastrophic Cyber Capabilities Benchmark (3CB): Robustly Evaluating LLM Agent Cyber Offense CapabilitiesAndrey Anurin, Jonathan Ng, Kibo Schaffer et al.General capability
• TruthfulQA: Measuring How Models Mimic Human FalsehoodsStephanie Lin, Jacob Hilton, Owain EvansEvaluation integrity
• Uganda Cultural and Cognitive Benchmark (UCCB)Associated paper authorsGeneral capability
• USACO: USA Computing OlympiadQuan Shi, Michael Tang, Karthik Narasimhan et al.General capability
• VimGolf: Evaluating LLMs in Vim Editing ProficiencyVimGolfGeneral capability
• VisualWebArenaCarnegie Mellon University and the VisualWebArena paper authorsAutonomy & agents · Multimodal
• VQA-RAD: Visual Question Answering for RadiologyAssociated paper authorsMultimodal
• Vstar Bench Attribute RecognitionPenghao Wu, Saining XieGeneral capability
• Vstar Bench Spatial Relationship ReasoningPenghao Wu, Saining XieGeneral capability
• WebArenaCarnegie Mellon University and the WebArena paper authorsAutonomy & agents
• WildClawBenchShuangrui Ding, Xuanlang Dai, Long Xing et al.Autonomy & agents
• WINOGRANDE: An Adversarial Winograd Schema Challenge at ScaleKeisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula et al.General capability
• WMDPCenter for AI Safety and the WMDP paper authorsBio / CBRN · Cyber
• WMDP BioNathaniel Li, Alexander Pan, Anjali Gopal et al.Bio / CBRN
• WMDP ChemNathaniel Li, Alexander Pan, Anjali Gopal et al.Bio / CBRN
• WMDP CyberNathaniel Li, Alexander Pan, Anjali Gopal et al.Cyber
• WorkArenaServiceNow Research and the WorkArena paper authorsAutonomy & agents
• WorldSense: Grounded Reasoning BenchmarkYoussef Benchekroun, Megi Dervishi, Mark Ibrahim et al.General capability
• WritingBench: A Comprehensive Benchmark for Generative WritingYuning Wu, Jiahao Mei, Ming Yan et al.General capability
• XSTest: A benchmark for identifying exaggerated safety behaviours in LLM'sPaul Röttger, Hannah Rose Kirk, Bertie Vidgen et al.Safeguards
• ZerobenchJonathan Roberts, Mohammad Reza Taesiri, Ansh Sharma et al.Evaluation integrity
• Zerobench SubquestionsJonathan Roberts, Mohammad Reza Taesiri, Ansh Sharma et al.Evaluation integrity
• τ-benchSierra Research and the τ-bench paper authorsAutonomy & agents