Who is whispering in the ear of the chatbot you use?
Mark Fadul: Director and Co-Founder of AI Forensics
Algorithms are not neutral when it comes to values. For over a decade, we have allowed big tech companies to use them as gatekeepers of our information ecosystem without demanding transparency or accountability in return. The consequences have ranged from amplifying sensationalist, polarizing content to cloaked personalized ads and the proliferation of monopolistic behaviors, shaping public discourse in ways that undermine democratic exchange.
Although we have learned the hard way what can happen when critical information infrastructure is handed over to corporate interests without oversight, we are now repeating the same mistake with AI chatbots—and the stakes could be much higher. Chatbots do not merely store or organize existing information; they create and frame it. Facebook and Google decide which news articles you see, while tools like ChatGPT, Claude, and Gemini synthesize that information into seemingly credible answers.
This distinction is crucial because the shift from curator to editor makes undue influence less visible and more insidious. We are once again ceding unprecedented authority over the foundational structure of future information to private companies, without even calling for independent oversight. The most pressing threat is not that these AI systems may rebel or go out of control, but rather that a handful of self-interested parties are quickly becoming gatekeepers of information for a massive and growing portion of the population.
Current chatbots are not just large language models (LLMs). They are built on numerous opaque algorithmic layers involved in the development and deployment of the model, each of which can serve as an entry point for platforms or other parties to shape information according to their interests.
This “algorithmic influence cluster” consists of at least five layers. The first is organizing training data. When determining what data to include or exclude during training, platforms make opaque decisions on sources, how to evaluate differing perspectives, and what content to filter. These choices then shape the model’s worldview. For example, in October 2025, Elon Musk launched Grokipedia to provide training data for his chatbot Grok. It is a corporate-controlled encyclopedia meant to offer an alternative to Wikipedia that is “anti-woke” and operates under a model of community governance, which has long served as a trusted information source online.
The second layer is reinforcement learning from human and AI feedback, the process that has transformed large language models from temperamental text generators into usable “assistants.” During this post-training phase of model development, human reviewers assess outputs to steer the system toward desirable behaviors, such as helpfulness or politeness. Currently, these human evaluations remain a largely invisible but essential component of the AI industry. However, they are increasingly being replaced by “teachers” made by AI that are intended to align the base model with predetermined principles embedded in a “constitution.”
The third layer is searching the web. When chatbots search online or access digital databases, retrieval-augmented generation (RAG) systems identify bits of information to be included in the model’s response. This functionality mirrors that of traditional search engines, which prioritize certain sources at the expense of others. Just like with search engines, introducing advertisements into chatbot responses—an initiative announced by ChatGPT for 2026—raises further concerns about objectivity.
The fourth layer involves system prompts. Since these prompts come into play when a chatbot generates an answer, they allow platforms to adjust a chatbot’s behavior without retraining it. For example, since Grok’s system prompts were published last year, we know they include directions such as “do not hesitate to make politically unsound claims.” (ChatGPT, Claude, and Gemini also use system prompts, but these remain confidential.)
The final layer consists of safety filters. Before a chatbot’s query reaches the model, input filters determine whether it is “acceptable.” Likewise, after the model generates a response, output filters can modify, monitor, or sanitize content before it reaches you. While platforms have legitimate reasons for blocking certain queries (like those seeking instructions on making a bomb), the fact that these filters are opaque raises open questions. Model developers could create a foundation for systematic censorship without us knowing it. Safety filters in Chinese chatbots block all references to the Tiananmen Square massacre.
Political and corporate interests are already shaping this cluster of algorithmic influences at a time when chatbots are being deployed globally. After Donald Trump was inaugurated for a second term, Apple updated its AI training instructions to avoid labeling supporters of the “Make America Great Again” movement as “extremists” or “radicals.” Last summer, Reuters discovered that Meta had updated its internal AI guidelines to relax preventative measures that barred its chatbots from making racist statements or engaging in “flirtatious” behavior with minors, among other things. In May, the AI assistant Grok began amplifying unsubstantiated and out-of-context claims about a “white genocide” in South Africa (Elon Musk himself is a white South African). While the company blamed “unauthorized modifications,” such “design imbalances” are common and appear ideologically consistent with Musk’s views.
Political manipulation via chatbots has already proven effective. A study published in Nature in 2025 showed that chatbots trained to defend a particular candidate could significantly influence moderate and undecided voters (the groups that determine the outcome of most elections).
Unlike authoritarian regimes that exercise explicit control over information, democracies depend on diverse sources and transparent, accountable information systems. Allowing an unaccountable central authority to control AI infrastructure is an invitation to drift toward technological authoritarianism; it is easy to see how each layer of algorithmic influence can be exploited to amplify or suppress specific opinions without explicit censorship.
Last December, the European Commission fined the platform X €120 million ($138 million) for “violating its transparency obligations under the Digital Services Act.” As expected, X and its defenders portrayed this move as an attack on free speech. However, transparency is essential for defending free expression. In its absence, we cannot know who is monitored or what forms of influence are exerted on the media we all consume.
We have learned from the rise of social media what can happen when accountability lags behind media adoption. We do not have the luxury of repeating the same mistakes with systems that hold greater power and influence over public knowledge.
