Hugging Face's policy chief doubts AI evaluators

Hugging Face's policy chief, Irene Solaiman, has cast doubt on the idea that embedding AI evaluators into models or products will solve the industry's benchmarking problem. Evaluations have become central to how AI labs describe their systems. Companies use them to argue that one model is safer, more capable or more aligned than another, and regulators increasingly look to test results when deciding how to oversee the technology. But the methods behind those evaluations are complex, contested and still developing. Solaiman's comments suggest that making evaluation a built-in feature may create an appearance of rigor without resolving deeper questions. Who designs the tests? What incentives do developers have to game them? How well do benchmark results predict behavior in real-world settings? An evaluator that is tightly integrated with a model might be convenient, but convenience is not the same as independence. If the same organization controls the model and the scoring, trust remains a challenge. The debate matters because policymakers and enterprises need reliable signals. They want to know whether an AI system can be deployed safely, whether it resists manipulation and whether it performs consistently across contexts. Static benchmarks can be useful, but they can also become targets for optimization. Evaluators embedded in products may help with continuous monitoring, yet they cannot replace external scrutiny, red-team testing and transparent reporting. The likely conclusion is not that evaluations are useless. It is that no single evaluator should be treated as a silver bullet. The field needs better methodology, independent review and clearer disclosure of limitations. As AI systems become more capable and more widely deployed, the credibility of safety claims will depend on whether evaluation is seen as a serious scientific and governance practice, not just a feature listed on a product page.