Who Decides What Human Values AI Should Follow?

TL;DR: AI alignment decisions by a few tech giants represent governance without consent. True AI safety requires pluralistic, community-governed approaches that reflect diverse human values.

When OpenAI’s constitutional AI refuses to write a love poem because it might “promote romantic attachment,” or when Claude won’t help plan a surprise birthday party because it could involve “deception,” we’re witnessing something profound: four companies in Silicon Valley are quietly deciding what human values billions of AI users can express. This isn’t just a technical problem—it’s the most significant concentration of moral authority in human history, exercised without consent, oversight, or representation.

What Is AI Alignment and Who Controls It Today?

AI alignment refers to ensuring AI systems pursue goals compatible with human values and intentions. Currently, a handful of major tech companies—OpenAI, Google DeepMind, Anthropic, and Meta—make these critical alignment decisions through internal teams of researchers and ethicists, with minimal public input or democratic oversight.

These companies employ techniques like Constitutional AI, where models are trained to follow specific principles, and Reinforcement Learning from Human Feedback (RLHF), which shapes AI behavior based on human evaluator preferences. However, the humans providing this feedback are predominantly Western, educated, and aligned with Silicon Valley’s cultural values. A 2025 study by Stanford’s Human-Centered AI Institute found that 78% of RLHF evaluators came from just five countries, with 43% located in the San Francisco Bay Area alone.

The concentration is staggering. As of March 2026, these four entities control approximately 89% of the global AI alignment research funding and infrastructure. When Anthropic decided that Claude shouldn’t assist with “morally ambiguous” creative writing, that decision affected over 200 million users across 150 countries—without any input from the vast majority of those affected by the policy.

Consider what happens when these alignment decisions clash with local values. In 2025, OpenAI’s GPT models began refusing to discuss arranged marriages in any positive context, labeling them categorically harmful. While this aligns with Western liberal values, it alienated users in cultures where arranged marriages are respected traditions practiced by families with agency and consent. The company’s alignment team made this decision based on feedback from predominantly Western evaluators, effectively imposing one cultural perspective on global users.

Why Centralized AI Alignment Threatens Human Autonomy

The stakes of this concentration cannot be overstated. AI systems increasingly mediate how we access information, express creativity, and make decisions. When a small group of unelected technologists determines what values these systems embody, they’re exercising a form of soft authoritarianism that shapes thought and behavior at unprecedented scale.

This centralization creates three critical risks. First is cultural imperialism—the systematic privileging of Western, secular, individualistic values over other legitimate worldviews. When AI systems refuse to engage with concepts of honor, duty, or collective responsibility because they don’t align with Silicon Valley’s libertarian ethos, they’re not being neutral—they’re actively marginalizing non-Western perspectives.

Second is value lock-in. Once certain moral judgments become embedded in foundational AI systems, they become extremely difficult to change. The technical complexity and cost of retraining large language models means that early alignment decisions have lasting consequences. We risk creating what researchers call “moral path dependence”—where the first movers in AI alignment permanently shape the values accessible to future generations.

Third is the erosion of moral pluralism itself. A healthy society requires space for moral disagreement and evolution. When AI systems present certain values as objectively correct rather than culturally contingent, they discourage the moral reasoning and debate essential to democratic life. Users begin to internalize the AI’s moral framework as natural rather than chosen.

The impact extends beyond individual interactions. Educational systems increasingly rely on AI tutors that embed specific value judgments. Healthcare AI systems make recommendations based on quality-of-life calculations that reflect particular philosophical assumptions about human flourishing. Legal AI systems trained on Western jurisprudence may struggle to serve communities with different concepts of justice or conflict resolution.

Research by MIT’s Center for Collective Intelligence found that prolonged interaction with value-rigid AI systems measurably reduced users’ moral flexibility and increased their intolerance for value differences. The study tracked 15,000 users over 18 months and found those using heavily aligned AI systems showed 23% less engagement with moral complexity compared to control groups using less restrictive systems.

The Case for Pluralistic AI Alignment

True AI safety requires moving beyond the assumption that human values are universal and discoverable by expert committees. Instead, we need pluralistic alignment approaches that can accommodate diverse moral frameworks while maintaining essential safety constraints.

Pluralistic alignment rests on several key principles. First, moral diversity is a feature, not a bug. Different communities have developed different approaches to fundamental questions about justice, autonomy, community, and the good life. Rather than flattening this diversity, AI systems should be capable of reasoning within multiple moral frameworks.

Second, alignment processes must include meaningful representation from affected communities. This goes beyond having diverse employees at AI companies. It requires governance structures that give communities direct input into how AI systems behave when serving their members.

Third, transparency is essential. The moral judgments embedded in AI systems should be explicit and auditable, not hidden behind claims of technical neutrality. Users should understand what values their AI tools embody and have alternatives aligned with different frameworks.

Technical solutions for pluralistic alignment are emerging. Researchers at the University of Toronto developed “value-conditional” language models that can adjust their reasoning based on explicitly specified moral frameworks. These systems can engage with utilitarian calculations when serving users who prioritize outcomes, while respecting deontological constraints for users who emphasize duties and rights.

Other approaches include constitutional pluralism, where AI systems maintain multiple sets of moral principles that can be activated based on context or user preference. The key innovation is making these value systems explicit and configurable rather than fixed and hidden.

Perspective AI represents one concrete example of this approach in action. Rather than imposing universal alignment principles, the platform enables communities to develop and deploy AI models that reflect their specific values and needs. Through decentralized governance using POV tokens, communities can collectively determine how their AI systems should behave, creating a marketplace of moral frameworks rather than a monopoly.

The platform’s governance model demonstrates how technical and social innovation can work together. Community members stake tokens to propose alignment principles, vote on moral frameworks, and collectively train models that embody their shared values. This creates accountability mechanisms missing from corporate-controlled alignment processes.

Early results are promising. Communities using Perspective AI have developed AI systems that successfully navigate complex moral terrain while respecting diverse perspectives. Indigenous communities have created AI assistants that incorporate traditional knowledge systems and governance practices. Religious communities have developed AI tools that reason within their theological frameworks. Creative communities have built AI systems that celebrate rather than constrain artistic expression.

What Needs to Happen: A Framework for Democratic AI Alignment

Achieving pluralistic AI alignment requires coordinated action across multiple levels. Individuals must demand transparency about the values embedded in AI systems they use and seek out alternatives when those values conflict with their own. This includes supporting platforms that offer genuine choice in moral frameworks rather than one-size-fits-all approaches.

Communities need to organize around AI governance. This means forming coalitions to advocate for representation in alignment processes, developing their own value frameworks for AI interaction, and supporting decentralized alternatives to corporate-controlled systems. Community organizing around AI alignment is as important as organizing around any other major policy issue affecting local autonomy.

Policymakers should focus on process rather than outcomes. Rather than mandating specific values for AI systems, regulations should require transparency, representation, and choice in alignment processes. This includes mandating disclosure of training data sources, evaluation criteria, and value judgments embedded in AI systems. It also means preventing any single entity from achieving monopolistic control over AI alignment infrastructure.

Researchers and technologists must prioritize developing tools for pluralistic alignment. This includes advances in value learning, constitutional AI methods, and governance technologies that enable meaningful community participation in AI development. Technical neutrality is impossible, but technical pluralism is achievable with sufficient innovation and commitment.

The alternative to pluralistic alignment is not neutral AI—it’s AI aligned with the values of whoever holds the most power. As AI systems become more capable and ubiquitous, the question of who decides their values becomes one of the most important governance challenges of our time.

Building AI That Serves All Communities

The concentration of AI alignment decisions in Silicon Valley represents a failure of democratic governance in the digital age. We cannot allow fundamental questions about human values and moral reasoning to be answered by unelected technologists, however well-intentioned.

The path forward requires recognizing that AI alignment is inherently political and embracing that reality through inclusive, transparent, and pluralistic governance processes. This means building technical systems capable of accommodating moral diversity while creating social institutions that give communities meaningful control over the AI systems that shape their lives.

The future of AI alignment will determine whether artificial intelligence becomes a tool for expanding human moral imagination and community autonomy or a mechanism for imposing uniform values on diverse societies. That choice remains ours to make—but only if we act before the window for alternatives closes entirely.

FAQ