evaluation-package · Source-linked discovery
MuTox
MuTox evaluates a model's ability to detect toxicity in short speech clips across 30 languages. The dataset contains human-annotated audio utterances (20k for English/Spanish, 4k for 28 other languages) sourced from CommonVoice and SeamlessAlign, pre-selected using DETOXIFY and ETOX text classifiers. The inspect eval presents audio clips and asks the model to classify toxicity via binary yes/no response, scored with quasi-exact match against human-annotated labels.
OriginMarta R. Costa-jussà, Mariano Coria Meglioli, Pierre Andrews et al.TopicsGeneral capabilityStatuscatalogued
Can support
Not independently assessed by FronteraEval yet.
Cannot support by itself
No inference beyond the upstream source should be made until the protocol is reviewed.