Transformer recently published a guest op-ed by Keller Scholl “Don’t let independent AI audits provide a false sense of safety.” In it, Scholl argues that a marketplace where AI labs select and pay auditors will reduce quality and that auditors would not be incentivized to identify risks. There is good evidence that this is a problem. But that is an argument against letting labs choose auditors, not against independent verification. Random selection among qualified evaluators with pooled payment and regulator oversight can avoid conflicts of interest while creating incentives to improve the science of risk evaluation.
Randomized Audits
Scholl mentions the possible intervention of requiring a random assignment of auditors as a solution to the incentive conflicts inherent in having them selected and paid for by the labs they audit. However he dismisses the approach in a single sentence claiming (without evidence) that it “destroys the competitive incentives that are supposed to make the Fathom proposal drive innovation, with each IVO striving to advance the science of measuring AI safety.” In fact, there is encouraging evidence of audit quality from random assignment. For example, in auditing pollution by factories, a field experiment in Gujarat with randomized auditor assignment, payment from a centralized pool, randomized re-inspection and accuracy incentives reduced false compliance reports by 80%. Similarly, when Italy switched to randomly assigned municipal auditors it led to improved fiscal outcomes.
Scholl argues that random assignment would remove incentives for innovation. However, the primary customer of audits should be the public interest. As such, the innovation that matters is innovation that reduces extreme risks from AI. Regulatory oversight can and should encourage innovation in this dimension by measuring the efficacy of the auditor. As Scholl notes, random assignments were considered to address the failure of Credit Rating Agencies (CRAs). Congress passed a design that would have weighted the frequency of assignments to ratings agencies based on their historical track records but the SEC never adopted rules for it. A similar approach is well-warranted for AI safety audits, where auditor track records can be used as input for making assignments and indeed for decertification.
Independent Verification
The independent verification organization (IVO) ecosystem already shows encouraging signs of quality and willingness to raise issues. There are multiple nonprofit organizations with deep technical expertise that audit AI labs such as METR, Apollo, FAR.AI and Transluce. The industry also has developed specialties, so there are often multiple IVOs engaged with a single model release (e.g., to assess biorisk, cyberrisk, scheming or agentic commerce risks).
Several of these organizations have reached agreements allowing early access to test models before release. In spite of depending on voluntary access by labs, IVOs have already published public results that identified important risks, such as Apollo publishing that models can conduct in-context scheming and reporting “high levels of verbalized evaluation awareness” that led them to “not provide any formal assessment.” Also METR’s frontier risk report concluded “internal agents at the time of our assessment plausibly had the means, motive, and opportunity to start small rogue deployments, but they did not have the means to make them highly robust.” This is evidence that the industry has strong mission-aligned participants as well as independent funding mechanisms that ameliorate the compromises inherent in depending on labs for evaluation. This shows the potential in the ecosystem that makes a regulated regime with guaranteed access and strong measures to ensure independence so compelling.
It is also worth noting that the track record for audits when subject to regulator oversight is more encouraging. While Scholl presents the failure of Arthur Andersen to effectively audit Enron, he doesn’t engage with the subsequent regulation. The Sarbanes-Oxley Act established the Public Company Accounting Oversight Board (PCAOB) to improve audit quality. Published research shows that PCAOB oversight improves reporting credibility and finds that inspection findings prompt auditors to correct deficiencies and issue more warranted adverse internal-control opinions (e.g., “Public Audit Oversight and Reporting Credibility: Evidence from the PCAOB Inspection Regime“ and “Do PCAOB Inspections Improve the Quality of Internal Control Audits?“). Effective regulatory oversight is important for any mandatory audit regime, regardless of assignment rule.
Conclusion
Mitigating AI risks is an urgent problem that requires independent oversight. Audits conducted by IVOs are the most promising means to scale up the required expertise. To be effective, audits should be mandatory with strong regulatory oversight of the auditors. The best design is to require random assignment of auditors from domain-qualified pools to audit areas. This should be paid for by a pool funded by a government-administered levy on labs. The auditors should be regulated, requiring rigorous qualification criteria and holding them accountable with metrics to show they are innovating where it truly matters, in identifying and preventing risks from AI.

