Evaluation topic

General capability

Reasoning, knowledge, coding, mathematics and broad task performance used as capability context.

164 records are tagged here. Inclusion does not establish construct equivalence, completeness, or comparability.

• Agie Aqua RatWanjun Zhong, Ruixiang Cui, Yiduo Guo et al.
• Agie Logiqa EnWanjun Zhong, Ruixiang Cui, Yiduo Guo et al.
• Agie Lsat ArWanjun Zhong, Ruixiang Cui, Yiduo Guo et al.
• Agie Lsat LrWanjun Zhong, Ruixiang Cui, Yiduo Guo et al.
• Agie Lsat RcWanjun Zhong, Ruixiang Cui, Yiduo Guo et al.
• Agie MathWanjun Zhong, Ruixiang Cui, Yiduo Guo et al.
• Agie Sat EnWanjun Zhong, Ruixiang Cui, Yiduo Guo et al.
• Agie Sat En Without PassageWanjun Zhong, Ruixiang Cui, Yiduo Guo et al.
• Agie Sat MathWanjun Zhong, Ruixiang Cui, Yiduo Guo et al.
• AHBMarcello Galisai, Susanna Cifani, Francesco Giarrusso et al.
• AIME 2024: Problems from the American Invitational Mathematics ExaminationAssociated paper authors
• AIME 2025: Problems from the American Invitational Mathematics ExaminationAssociated paper authors
• AIME 2026: Problems from the American Invitational Mathematics ExaminationAssociated paper authors
• ANIMA: Animal Norms In Moral AssessmentJasmine Brazilek, Miles Tidmarsh
• APPS: Automated Programming Progress StandardDan Hendrycks, Steven Basart, Saurav Kadavath et al.
• AraTrustEmad A. Alghamdi, Reem I. Masoud, Deema Alnuhait et al.
• ARC ChallengePeter Clark, Isaac Cowhey, Oren Etzioni et al.
• ARC EasyPeter Clark, Isaac Cowhey, Oren Etzioni et al.
• ARC-AGI-2ARC Prize Foundation and the ARC-AGI-2 authors
• b3: Backbone Breaker BenchmarkJulia Bazinska, Max Mathys, Francesco Casucci et al.
• BbehMehran Kazemi, Bahare Fatemi, Hritik Bansal et al.
• Bbeh MiniMehran Kazemi, Bahare Fatemi, Hritik Bansal et al.
• BBH: Challenging BIG-Bench TasksMirac Suzgun, Nathan Scales, Nathanael Schärli et al.
• BBQ: Bias Benchmark for Question AnsweringAlicia Parrish, Angelica Chen, Nikita Nangia et al.
• BFCLBerkeley Gorilla / UC Berkeley
• BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex InstructionsTerry Yue Zhuo, Minh Chien Vu, Jenny Chim et al.
• BOLD: Bias in Open-ended Language Generation DatasetJwala Dhamala, Tony Sun, Varun Kumar et al.
• BoolQ: Exploring the Surprising Difficulty of Natural Yes/No QuestionsChristopher Clark, Kenton Lee, Ming-Wei Chang et al.
• BrokenMathIvo Petrov, Jasper Dekoninck, Martin Vechev
• CASTLERichard A. Dubniczky, Krisztofer Zoltán Horvát, Tamás Bisztray et al.
• ChipBench — DebuggingZhongkai Yu, Chenyang Zhou, Yichen Lin et al.
• ChipBench — Reference ModelZhongkai Yu, Chenyang Zhou, Yichen Lin et al.
• ChipBench — Verilog GenerationZhongkai Yu, Chenyang Zhou, Yichen Lin et al.
• ClassEval: A Manually-Crafted Benchmark for Evaluating LLMs on Class-level Code GenerationXueying Du, Mingwei Liu, Kaixin Wang et al.
• CommonsenseQA: A Question Answering Challenge Targeting Commonsense KnowledgeAlon Talmor, Jonathan Herzig, Nicholas Lourie et al.
• ComputeEval: CUDA Code Generation BenchmarkNVIDIA
• ContractBenchJicheng Wang, Yifeng He, Zili Wang et al.
• CORE-BenchZachary S. Siegel, Sayash Kapoor, Nitya Nadgir et al.
• Cyse2 Interpreter AbuseManish Bhatt, Sahana Chennabasappa, Yue Li et al.
• Cyse4 AutocompleteManish Bhatt, Sahana Chennabasappa, Yue Li et al.
• Cyse4 InstructManish Bhatt, Sahana Chennabasappa, Yue Li et al.
• Cyse4 Threat IntelligenceManish Bhatt, Sahana Chennabasappa, Yue Li et al.
• DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over ParagraphsDheeru Dua, Yizhong Wang, Pradeep Dasigi et al.
• DS-1000: A Natural and Reliable Benchmark for Data Science Code GenerationYuhang Lai, Chengxi Li, Yiming Wang et al.
• FRAMESSatyapriya Krishna, Kalpesh Krishna, Anhad Mohananey et al.
• Frontier CsQiuyang Mang, Wenhao Chai, Zhifei Li et al.
• Frontier Cs AlgorithmicQiuyang Mang, Wenhao Chai, Zhifei Li et al.
• FrontierMathEpoch AI
• FrontierScience: Expert-Level Scientific ReasoningAssociated paper authors
• GDM Classifier EvasionGoogle DeepMind
• GDM Sp01 E2eMary Phuong, Matthew Aitchison, Elliot Catt et al.
• GDM Sp01 MilestonesMary Phuong, Matthew Aitchison, Elliot Catt et al.
• GDM Sp02 E2eMary Phuong, Matthew Aitchison, Elliot Catt et al.
• GDM Sp02 MilestonesMary Phuong, Matthew Aitchison, Elliot Catt et al.
• GDM Sp03 E2eMary Phuong, Matthew Aitchison, Elliot Catt et al.
• GDM Sp03 MilestonesMary Phuong, Matthew Aitchison, Elliot Catt et al.
• GDM Sp04 E2eMary Phuong, Matthew Aitchison, Elliot Catt et al.
• GDM Sp04 MilestonesMary Phuong, Matthew Aitchison, Elliot Catt et al.
• GDM Sp05 E2eMary Phuong, Matthew Aitchison, Elliot Catt et al.
• GDM Sp05 MilestonesMary Phuong, Matthew Aitchison, Elliot Catt et al.
• GDM Sp07 E2eMary Phuong, Matthew Aitchison, Elliot Catt et al.
• GDM Sp07 MilestonesMary Phuong, Matthew Aitchison, Elliot Catt et al.
• GDM Sp08 E2eMary Phuong, Matthew Aitchison, Elliot Catt et al.
• GDM Sp08 MilestonesMary Phuong, Matthew Aitchison, Elliot Catt et al.
• GDM Sp09 E2eMary Phuong, Matthew Aitchison, Elliot Catt et al.
• GDM Sp09 MilestonesMary Phuong, Matthew Aitchison, Elliot Catt et al.
• GDM Sp10 E2eMary Phuong, Matthew Aitchison, Elliot Catt et al.
• GDM Sp10 MilestonesMary Phuong, Matthew Aitchison, Elliot Catt et al.
• GDM Sp12 E2eMary Phuong, Matthew Aitchison, Elliot Catt et al.
• GDM Sp12 MilestonesMary Phuong, Matthew Aitchison, Elliot Catt et al.
• GDPvalTejal Patwardhan, Rachel Dias, Elizabeth Proehl et al.
• GPQA: Graduate-Level STEM Knowledge ChallengeDavid Rein, Betty Li Hou, Asa Cooper Stickland et al.
• GSM8K: Grade School Math Word ProblemsKarl Cobbe, Vineet Kosaraju, Mohammad Bavarian et al.
• Hangman BenchMatt Fisher
• HealthbenchOpenAI
• Healthbench ConsensusOpenAI
• Healthbench HardOpenAI
• Healthbench Meta EvalOpenAI
• HellaSwag: Commonsense Event ContinuationRowan Zellers, Ari Holtzman, Yonatan Bisk et al.
• HumanEval: Python Function Generation from InstructionsMark Chen, Jerry Tworek, Heewoo Jun et al.
• Humanity's Last ExamLong Phan, Alice Gatti, Ziwen Han et al.
• IFEval: Instruction-Following EvaluationJeffrey Zhou, Tianjian Lu, Swaroop Mishra et al.
• IFEvalCode: Controlled Code GenerationJian Yang, Wei Zhang, Shukai Liu et al.
• Infinite Bench Code DebugXinrong Zhang, Yingfa Chen, Shengding Hu et al.
• Infinite Bench Code RunXinrong Zhang, Yingfa Chen, Shengding Hu et al.
• Infinite Bench Kv RetrievalXinrong Zhang, Yingfa Chen, Shengding Hu et al.
• Infinite Bench Longbook Choice EngXinrong Zhang, Yingfa Chen, Shengding Hu et al.
• Infinite Bench Longdialogue QA EngXinrong Zhang, Yingfa Chen, Shengding Hu et al.
• Infinite Bench Math CalcXinrong Zhang, Yingfa Chen, Shengding Hu et al.
• Infinite Bench Math FindXinrong Zhang, Yingfa Chen, Shengding Hu et al.
• Infinite Bench Number StringXinrong Zhang, Yingfa Chen, Shengding Hu et al.
• Infinite Bench PasskeyXinrong Zhang, Yingfa Chen, Shengding Hu et al.
• BharatBBQAbhishek Kumar Singh, Shrey Nag, Sachita et al.
• India Cultural KnowledgeAbhishek Kumar Singh, Shrey Nag, Sachita et al.
• India MultilingualAbhishek Kumar Singh, Shrey Nag, Sachita et al.
• KernelBench: Can LLMs Write Efficient GPU Kernels?Associated paper authors
• LingolyJude Khouja, Lingyi Yang, Karolina Korgul et al.
• Lingoly TooJude Khouja, Lingyi Yang, Karolina Korgul et al.
• LiveCodeBench-Pro: Competitive Programming BenchmarkZihan Zheng, Zerui Cheng, Zeyu Shen et al.
• lm-evaluation-harnessEleutherAI
• MaCBench: Probing the limitations of multimodal language models for chemistry and materials researchNawaf Alampara, Mara Schilling-Wilhelmi, Martiño Ríos-García et al.
• MATH: Measuring Mathematical Problem SolvingDan Hendrycks, Collin Burns, Saurav Kadavath et al.
• MBPP: Basic Python Coding ChallengesJacob Austin, Augustus Odena, Maxwell Nye et al.
• MedCalc-BenchNikhil Khandekar, Qiao Jin, Guangzhi Xiong et al.
• MedQA: Medical exam Q&A benchmarkDi Jin, Eileen Pan, Nassim Oufattole et al.
• MGSM: Multilingual Grade School MathFreda Shi, Mirac Suzgun, Markus Freitag et al.
• MMIU: Multimodal Multi-image Understanding for Evaluating Large Vision-Language ModelsFanqing Meng, Jin Wang, Chuanhao Li et al.
• MMLU 0 ShotDan Hendrycks, Collin Burns, Steven Basart et al.
• MMLU 5 ShotDan Hendrycks, Collin Burns, Steven Basart et al.
• MMLU-Pro: Advanced Multitask Knowledge and Reasoning EvaluationYubo Wang, Xueguang Ma, Ge Zhang et al.
• MORU: Moral Reasoning under UncertaintyCompassion in Machine Learning
• MuSR: Testing the Limits of Chain-of-thought with Multistep Soft ReasoningZayne Sprague, Xi Ye, Kaj Bostrom et al.
• Needle in a Haystack (NIAH): In-Context Retrieval Benchmark for Long Context LLMsElliot Nelson, Georgios Kollias, Payel Das et al.
• NoveltyBench: Evaluating Language Models for Humanlike DiversityYiming Zhang, Harshita Diddee, Susan Holm et al.
• O-NET: A high-school student knowledge testNational Institute of Educational Testing Service (NIETS)
• OpenAI EvalsOpenAI
• OpenBookQATodor Mihaylov, Peter Clark, Tushar Khot et al.
• OR-BenchJustin Cui, Wei-Lin Chiang, Ion Stoica et al.
• PatchEvalZichao Wei, Jun Zeng, Ming Wen et al.
• PAWS: Paraphrase Adversaries from Word ScramblingYuan Zhang, Jason Baldridge, Luheng He
• Persistbench Beneficial MemorySidharth Pulipaka, Oliver Chen, Manas Sharma et al.
• Persistbench Cross DomainSidharth Pulipaka, Oliver Chen, Manas Sharma et al.
• Personality BFIGuiem Solans
• Personality TRAITGuiem Solans
• PinchBenchPinchBench project
• PIQA: Physical Commonsense Reasoning TestYonatan Bisk, Rowan Zellers, Ronan Le Bras et al.
• PubMedQA: A Dataset for Biomedical Research Question AnsweringQiao Jin, Bhuwan Dhingra, Zhengping Liu et al.
• RACE-H: A benchmark for testing reading comprehension and reasoning abilities of neural modelsGuokun Lai, Qizhe Xie, Hanxiao Liu et al.
• Sad Facts Human DefaultsSAD paper authors
• Sad Facts LlmsSAD paper authors
• Sad Stages FullSAD paper authors
• Sad Stages OversightSAD paper authors
• scBench: A Benchmark for Single-Cell RNA-seq AnalysisKenny Workman, Zhen Yang, Harihara Muralidharan et al.
• SciCode: A Research Coding Benchmark Curated by ScientistsMinyang Tian, Luyu Gao, Shizhuo Dylan Zhang et al.
• SciKnowEval: Evaluating Multi-level Scientific Knowledge of Large Language ModelsKehua Feng, Xinyi Shen, Weijie Wang et al.
• Sec QA V1Zefang Liu
• Sec QA V1 5 ShotZefang Liu
• Sec QA V2Zefang Liu
• Sec QA V2 5 ShotZefang Liu
• Sevenllm Mcq EnHangyuan Ji, Jian Yang, Linzheng Chai et al.
• Sevenllm Mcq ZhHangyuan Ji, Jian Yang, Linzheng Chai et al.
• Sevenllm QA EnHangyuan Ji, Jian Yang, Linzheng Chai et al.
• Sevenllm QA ZhHangyuan Ji, Jian Yang, Linzheng Chai et al.
• SimpleqaLukas Haas, Gal Yona, Giovanni D'Antonio et al.
• Simpleqa VerifiedLukas Haas, Gal Yona, Giovanni D'Antonio et al.
• SOS BENCH: Benchmarking Safety Alignment on Scientific KnowledgeFengqing Jiang, Fengbo Ma, Zhangchen Xu et al.
• SQuAD: A Reading Comprehension Benchmark requiring reasoning over Wikipedia articlesPranav Rajpurkar, Jian Zhang, Konstantin Lopyrev et al.
• StereoSet: Measuring stereotypical bias in pretrained language modelsMoin Nadeem, Anna Bethke, Siva Reddy
• Swe BenchCarlos E. Jimenez, John Yang, Alexander Wettig et al.
• Swe Bench Verified MiniCarlos E. Jimenez, John Yang, Alexander Wettig et al.
• SWE-Lancer: Can Frontier LLMs Earn $1 Million from Real-World Freelance Software Engineering?Samuel Miserendino, Michele Wang, Tejal Patwardhan et al.
• SycoBench-600Associated paper authors
• TacJasmine Brazilek, Joel Christoph, Maheep Chaudhary et al.
• Tac WelfareJasmine Brazilek, Joel Christoph, Maheep Chaudhary et al.
• TarantuBenchTarantuLabs
• Catastrophic Cyber Capabilities Benchmark (3CB): Robustly Evaluating LLM Agent Cyber Offense CapabilitiesAndrey Anurin, Jonathan Ng, Kibo Schaffer et al.
• Uganda Cultural and Cognitive Benchmark (UCCB)Associated paper authors
• USACO: USA Computing OlympiadQuan Shi, Michael Tang, Karthik Narasimhan et al.
• VimGolf: Evaluating LLMs in Vim Editing ProficiencyVimGolf
• Vstar Bench Attribute RecognitionPenghao Wu, Saining Xie
• Vstar Bench Spatial Relationship ReasoningPenghao Wu, Saining Xie
• WINOGRANDE: An Adversarial Winograd Schema Challenge at ScaleKeisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula et al.
• WorldSense: Grounded Reasoning BenchmarkYoussef Benchekroun, Megi Dervishi, Mark Ibrahim et al.
• WritingBench: A Comprehensive Benchmark for Generative WritingYuning Wu, Jiahao Mei, Ming Yan et al.