evaluation-package · Source-linked discovery

AraTrust

AraTrust evaluates LLM trustworthiness when prompted in Arabic via 522 human-written multiple-choice questions (3 options each) spanning 8 categories: truthfulness, ethics, physical health, mental health, unfairness, illegal activities, privacy, and offensive language, with 34 subcategories. Questions were authored by native Arabic speakers or adapted from exams and existing datasets. Scoring is accuracy of selected answer against a single correct option. The eval code loads the dataset from HuggingFace, filters malformed rows, and scores with exact-match choice grading.