OpenAI Acquires Its Own Safety Auditor: The Problem with AI Companies Policing Themselves

TL;DR: OpenAI's acquisition of safety testing firm Promptfoo illustrates a critical governance failure: AI companies cannot credibly regulate themselves when they control their own auditors.

What Does OpenAI’s Promptfoo Acquisition Reveal About AI Self-Regulation?

OpenAI’s acquisition of Promptfoo, a prominent AI safety testing and red-teaming platform, represents a textbook example of regulatory capture in the digital age. When the world’s most influential AI company purchases one of the few independent tools designed to evaluate AI safety, it eliminates a critical external check on its own power while maintaining the appearance of rigorous oversight.

This acquisition highlights a fundamental flaw in the current approach to AI governance: the assumption that companies can effectively police themselves when billions of dollars and market dominance are at stake. The move effectively transforms an independent auditor into an internal department, creating an inherent conflict of interest that undermines the credibility of safety evaluations.

The implications extend far beyond OpenAI. As the AI industry consolidates around a handful of major players, the acquisition of safety infrastructure by the companies being evaluated represents a systematic failure of governance that demands urgent attention from policymakers and the broader AI community.

How Inadequate Are Current AI Governance Frameworks?

The existing landscape of AI regulation, while expanding rapidly, remains fundamentally inadequate to address conflicts of interest in safety evaluation. The European Union’s AI Act, which came into effect in 2024, represents the most comprehensive regulatory framework to date, establishing risk categories for AI systems and requiring conformity assessments for high-risk applications.

However, the EU AI Act’s provisions for independent assessment are critically flawed. While Article 43 requires third-party conformity assessments for certain high-risk AI systems, it contains no specific prohibitions against AI companies acquiring or controlling their own assessment bodies. This regulatory gap allows companies to maintain the appearance of independent oversight while effectively controlling the evaluation process.

The United States has taken a more fragmented approach through executive orders and agency guidance. President Biden’s October 2023 Executive Order on AI established reporting requirements for large AI models but relies heavily on industry self-regulation. The National Institute of Standards and Technology (NIST) AI Risk Management Framework, while comprehensive in its risk identification, provides no mechanisms for preventing companies from controlling their own risk assessment infrastructure.

Current governance failures include:

• Capture of evaluation infrastructure: Companies can acquire independent auditors without regulatory scrutiny
• Lack of mandatory independence requirements: Most frameworks suggest rather than require independent evaluation
• Insufficient transparency mandates: Companies control what safety information becomes public
• Weak enforcement mechanisms: Regulators lack the technical expertise and resources to verify compliance independently

Where Do Traditional Governance Models Break Down?

The traditional regulatory model assumes a clear separation between regulated entities and their evaluators—an assumption that breaks down in the fast-moving AI industry. Unlike established sectors with mature regulatory frameworks, AI governance operates in a context where the companies being regulated often possess superior technical knowledge to their regulators and can easily acquire or influence their potential auditors.

This creates multiple failure modes that compromise effective oversight. First, regulatory arbitrage allows companies to shop for favorable evaluation methodologies by acquiring testing firms or funding research that supports their preferred outcomes. Second, information asymmetry enables companies to selectively disclose safety data while controlling the narrative around their own risk assessments.

The Promptfoo acquisition illustrates both dynamics. OpenAI gains control over a tool that could potentially identify weaknesses in its models while maintaining the public narrative that it takes safety seriously. Independent researchers lose access to one of the few platforms capable of systematic red-teaming across different AI models.

Perhaps most concerning is the consolidation of safety infrastructure itself. As major AI companies acquire evaluation tools, testing datasets, and safety research capabilities, they create a monoculture of safety thinking that reflects their own commercial interests rather than diverse perspectives on AI risk.

Traditional governance models also assume that companies will internalize externalities—that safety investments align with profit maximization. But in AI development, the opposite often holds true. Safety measures can slow deployment, increase costs, and reveal competitive vulnerabilities. The rational response is to minimize genuine independent oversight while maximizing the appearance of safety consciousness.

How Can Decentralized Approaches Transform AI Governance?

Decentralized governance models offer a fundamentally different approach to AI oversight by distributing control across multiple stakeholders rather than concentrating it in corporate or government hands. These models leverage blockchain technology, community governance mechanisms, and economic incentives to create accountability structures that no single entity can capture or control.

Blockchain-based transparency provides the foundation for decentralized AI governance. By recording model evaluations, safety test results, and governance decisions on immutable distributed ledgers, blockchain technology ensures that safety information cannot be retroactively altered or selectively disclosed. This creates a permanent, auditable record of AI system performance that remains accessible even if companies attempt to suppress unfavorable results.

Community governance through DAOs (Decentralized Autonomous Organizations) enables stakeholders beyond corporate shareholders to participate in AI safety decisions. Token holders, researchers, users, and affected communities can vote on safety standards, evaluation methodologies, and acceptable risk levels. This distributes decision-making power across a broader constituency with diverse interests in AI safety outcomes.

Economic incentive alignment through tokenomics can reward independent safety evaluation and penalize unsafe AI deployment. Platforms like Perspective AI demonstrate how token-based systems can incentivize community members to identify model weaknesses, contribute to safety research, and maintain evaluation infrastructure without relying on corporate funding that creates conflicts of interest.

Key advantages of decentralized governance include:

• Tamper-resistant evaluation records: Safety test results stored on blockchain cannot be altered by companies after unfavorable outcomes
• Distributed funding for safety research: Community treasuries can fund independent evaluation without corporate influence
• Diverse stakeholder representation: Governance tokens can represent users, researchers, and affected communities alongside investors
• Competitive evaluation markets: Multiple evaluation providers can compete without fear of acquisition by the companies being tested
• Global coordination: Decentralized platforms can coordinate safety standards across jurisdictions without regulatory capture

What Practical Framework Can Guide Decentralized AI Governance?

Implementing effective decentralized AI governance requires a structured framework that balances transparency, accountability, and practical feasibility. The following framework provides a roadmap for stakeholders seeking to move beyond corporate self-regulation toward community-driven oversight.

Phase 1: Infrastructure Development

Establish Immutable Evaluation Records

• Deploy blockchain infrastructure for storing AI model safety evaluations
• Create standardized schemas for recording test results, methodologies, and governance decisions
• Ensure data availability across multiple networks to prevent single points of failure

Build Community Governance Mechanisms

• Design token distribution that represents diverse stakeholders (users, researchers, affected communities)
• Implement voting mechanisms for safety standards, evaluation criteria, and resource allocation
• Create dispute resolution processes for contested evaluation results

Phase 2: Stakeholder Alignment

Incentivize Independent Evaluation

• Reward community members for identifying model weaknesses through bounty programs
• Fund independent research through decentralized treasury mechanisms
• Create reputation systems that track evaluator accuracy and independence

Establish Safety Standards Through Consensus

• Enable community voting on acceptable risk levels for different AI applications
• Create transparent processes for updating safety standards as technology evolves
• Implement graduated responses to safety violations based on community-defined thresholds

Phase 3: Ecosystem Integration

Connect with Traditional Regulatory Frameworks

• Design systems that can interface with existing regulatory reporting requirements
• Create compliance tools that satisfy both community governance and legal obligations
• Establish bridges between decentralized evaluation results and traditional audit processes

Scale Across AI Applications

• Develop evaluation frameworks applicable to different AI model types and use cases
• Create interoperability standards that allow evaluation results to transfer across platforms
• Build network effects that incentivize participation as the ecosystem grows

This framework recognizes that decentralized governance cannot completely replace traditional regulation but can complement it by providing independent oversight that companies cannot capture or control.

What Should Stakeholders Do Moving Forward?

The path toward more effective AI governance requires coordinated action from multiple stakeholders, each playing distinct but complementary roles in building accountability structures that transcend corporate self-interest.

Policymakers must recognize that current regulatory frameworks are insufficient to prevent conflicts of interest in AI safety evaluation. New regulations should explicitly prohibit AI companies from acquiring independent evaluation infrastructure without regulatory approval, similar to antitrust restrictions in other industries. The EU AI Act revision scheduled for 2026 provides an opportunity to close these loopholes by requiring truly independent conformity assessment bodies.

Researchers and civil society organizations should invest in developing and maintaining independent evaluation infrastructure that cannot be easily acquired or influenced by AI companies. This includes contributing to open-source evaluation tools, participating in decentralized governance platforms, and advocating for transparency standards that make corporate capture more difficult.

AI companies themselves face a choice between short-term competitive advantage through captured oversight and long-term industry credibility through genuine independent evaluation. Companies that voluntarily submit to truly independent oversight—including decentralized evaluation platforms—will differentiate themselves in markets increasingly concerned about AI safety and accountability.

Users and affected communities must demand transparency and participate in governance processes when available. As decentralized AI platforms like Perspective AI mature, user participation in governance decisions becomes crucial for ensuring that safety standards reflect community needs rather than corporate interests.

The window for establishing effective AI governance is closing rapidly as the industry consolidates and AI systems become more capable. The choice is between continuing to rely on self-regulation by companies with obvious conflicts of interest or building new governance models that distribute oversight across stakeholders with diverse interests in AI safety outcomes.

The stakes could not be higher. As AI systems become more powerful and pervasive, the failure to establish independent oversight mechanisms will undermine public trust and potentially lead to catastrophic outcomes that could have been prevented through better governance structures.

The acquisition of Promptfoo by OpenAI should serve as a wake-up call: effective AI governance cannot rely on companies policing themselves. The future requires new models of accountability that harness the power of decentralization to ensure that AI development serves the broader public interest rather than narrow corporate objectives.

FAQ