In June, the U.S. government required Anthropic to restrict access to its two newest models by foreign nationals, citing national security and cybersecurity concerns. Unable to verify users’ nationality in real time, Anthropic temporarily withdrew access to the models for all users. A few weeks later, the Chinese company Moonshot AI released its Kimi K3 model, and then published its full model weights. Together, these episodes have reignited a familiar debate: How far can governments go in requiring AI safeguards before they slow innovation or cede ground to foreign rivals?
But that debate is incomplete. It treats regulation as though it occurs in a vacuum, directed at a single monolithic AI company. In reality, AI products are built through a complex process involving many interacting firms. Indeed, my own academic research in this area suggests regulation can transform that process by determining who must invest in safety—and who waits for someone else to pick up the bill.
These questions are especially relevant as the European Commission prepares to begin enforcing key provisions in the EU AI Act, which imposes different safety and transparency requirements on general-purpose AI model providers, including OpenAI and Anthropic, and on companies that build AI applications, including voice assistants and customer service chatbots. As EU regulators put those rules into practice, they should ask how requirements targeting one layer will affect the rest of the development chain.
Consider a company building software to summarize physicians’ clinical notes. Development would likely start with a general-purpose model produced by a large AI developer (think GPT or Claude) which the company would adapt for a clinical setting. Between these entities, safety investments are divided into layers. The general-purpose model maker encounters choices related to training, general evaluation, safeguards, and documentation. The medical software company then encounters a different slate of choices, covering clinical validation, fine-tuning procedures, human review, and error monitoring. Just as the first company cannot anticipate every clinical use, the second company cannot easily identify vulnerabilities inherited from the base model.
It may seem reasonable to place responsibility as far downstream as possible. Downstream companies have the best understanding of how the model is used, and may be well positioned to address sector-specific risks. The challenge, however, is that subjecting one party to safety requirements can alter how much the other parties invest and contribute.
Imagine a food safety regime requiring restaurants to inspect every ingredient they serve while imposing no duties on large suppliers. Restaurants bear responsibility for what reaches the plate, of course, and diners trust them to serve quality food. Yet a rule targeting restaurants alone could give suppliers reason to relax their own quality controls, anticipating that every restaurant will have to conduct a final inspection. What looks reasonable when we focus on restaurants may be problematic when we consider the broader supply chain.
AI rules can work the same way. If a foundation model developer—the company that creates the general-purpose model—knows that a downstream company must meet a safety standard, it may do less itself, expecting the downstream company to pick up the slack. When rules apply only to the downstream layer, that company—for example, the company building the clinical notes application I discussed above—ends up doing more, while the upstream developer may do less. The product could therefore end up no safer overall, even though the regulated company has increased its safety efforts.
Policy debates asking which single actor is best positioned to prevent harm miss the point. The crucial question is which contributions are needed across the chain of development, and how a rule directed at one company can change the behavior of others.
Together with my coauthors Jon Kleinberg and Hoda Heidari, I set out to understand these mechanisms in a paper recently published in Proceedings of the National Academy of Sciences. We built a game-theoretic model involving a general-purpose AI developer and a downstream application company that adapts the model for a particular market. Each entity chooses how much to invest in safety and performance. A regulator can set a minimum safety requirement for either company or both.
The model is not a measure of any existing law, nor does it tell policymakers whether to use liability, audits, taxes, or another legal instrument to achieve AI safety. Instead, it explores a particular strategic mechanism: When one firm’s choices are constrained, how does the other firm respond?
Our analysis finds that weak safety regulation targeting only downstream applications can backfire, producing technologies that are less safe than they would be with no regulation. Here, “weak” means a safety floor at or below the level that the firms would have reached without regulation. Even such a modest rule can change the firms’ strategies. Although the downstream company complies with the regulation, the model maker can drop its safety investment by more than the downstream company adds. The final product is less safe despite the presence of a minimum safety standard.
Our second finding suggests a more constructive possibility. When appropriately calibrated requirements apply to both the upstream model maker and downstream company, regulation can act as a commitment device. Each firm can invest in safety assured that the other must also contribute. In some settings we modeled, these rules led to greater safety and performance for consumers, while leaving both firms financially better off.
A weak rule targeted at downstream applications can quietly give large AI providers permission to withdraw effort. A well-calibrated rule, on the other hand, can facilitate coordination and stronger safety commitments.
The U.S. government’s restrictions on access to Anthropic’s models may not be a standalone incident. As AI models become more capable and their use becomes more widespread, there are likely to be more political standoffs, access restrictions, and calls for safety regulation. To evaluate any proposal for AI regulation, ask the following questions: Who is being targeted with a standard? What will other companies do in response? The answers will help determine whether regulation makes everybody more accountable, or whether it merely makes safety easier to outsource.