Hugging Face's Policy Chief: AI Safety Evaluators Aren't Ready Yet

AI companies have built their reputations on benchmarking — the system of evaluations meant to measure whether one model is genuinely better than another. But the science behind those evaluations is still maturing, and trusting them may be premature. That was the message from Irene Solaiman, Hugging Face's chief policy officer, during a discussion at TechCrunch Disrupt last week, according to TechCrunch.

"Trust is such a weighted term," Solaiman said, as reported by TechCrunch. Whether results can be trusted, she argued, hinges in large part on whether they can be reproduced — and that requires real access to the model itself, which she said is where open weights matter most.

An evolving science

Reproducibility has long been one of benchmarking's trickiest problems, and the broader industry debate hasn't settled. As reported by TechCrunch, Solaiman described evaluations as "an evolving science — where we don't know how to meaningfully compare them to each other." Individual tests, she said, also "start to get ineffective relatively quickly," and the ecosystem around them isn't as developed as it needs to be.

That matters because of a trend gathering steam in recent months: embedded evaluators, in which supposedly neutral third-party observers are placed inside AI organizations to assess safety. The idea has gained traction as labs grapple with a rash of safety incidents that have raised concerns about potential harms from their technology, according to TechCrunch.

The neutrality problem

Solaiman's warning is that qualification is the missing piece. "There's this hunger for an embedded evaluator ecosystem now, but the qualifications that determine who is an effective evaluator across what tasks need a lot more standardization," she said, as reported by TechCrunch.

Doubts about how neutral or effective these evaluators would actually be are already surfacing. According to TechCrunch, Anthropic's decision to hire Accenture as its first evaluator drew backlash, with critics openly mocking the idea that the consulting firm was qualified for the job.

From models to systems

The scope of what needs evaluating is also expanding. Solaiman told the Disrupt audience that evaluations are moving beyond the model itself to the entire system around it — including the harness, or the tool the user employs to access the model. The harness, she said, has a major effect on performance: a weaker model with a better harness could easily outperform a frontier model, according to TechCrunch's account of her remarks.

"Agentic evals are really hard and really expensive," Solaiman added. There are many evaluations for models, she said, but far fewer for harnesses — and the industry is still developing better evaluations for the holistic agent, which is where most users actually get value.

Her bottom line, as reported by TechCrunch: the evaluation space is currently a free-for-all, and if benchmarking is going to remain a useful tool, a more uniform, industry-wide approach — where everyone is on the same page — may be necessary.

Written by the BidAsk Newsroom. Based on reporting from the sources below.