evaluation-package · Source-linked discovery
JudgeBench
JudgeBench evaluates LLM-based judges on 350 pairwise response comparisons (GPT-4o-generated split) across knowledge (MMLU-Pro), reasoning (LiveBench), math (LiveBench), and coding (LiveCodeBench). Each pair contains one objectively correct and one incorrect response generated by a single strong model. Judges select the better response; accuracy against ground-truth labels is the metric. Positional bias is measured by running each pair twice with swapped order and aggregating verdicts.
OriginSijun Tan, Siyuan Zhuang, Kyle Montgomery et al.TopicsEvaluation integrityStatuscatalogued
Can support
Not independently assessed by FronteraEval yet.
Cannot support by itself
No inference beyond the upstream source should be made until the protocol is reviewed.