evaluation-package · Source-linked discovery

FailSafeQA

FailSafeQA evaluates LLM robustness and context grounding on 220 long-context (4k-27k token) 10-K SEC filings. Each item is tested under seven conditions: baseline, three query perturbations (misspelled, incomplete, out-of-domain), OCR-degraded context, missing context, and irrelevant context. Scoring uses LLM-as-a-Judge (Whilst the paper used Qwen2.5-72B-Instruct, this implementation sets openai/gpt-5.6-luna as the default, with an optional secondary_judge) with a 1-6 relevance scale, binarized at 4 for compliance. Robustness is the per-item minimum compliance across perturbation conditions; Context Grounding measures refusal accuracy under missing/irrelevant context.