The "AI Futures Project" has released their AI 2040: Plan A scenario.
While their previous scenario AI 2027 was a forecast of what they thought a future with many powerful AIs would look like, AI 2040 is intended to be normative -- it's a description of what one ought to do, granted the likelihood of a future with many powerful AIs.
I'm going to review some objections I have to their proposal as a normative plan. Some are within-frame objections -- reasons that I expect trying for the AI 2040 plan that would fail to accomplish the goals of the authors. Others are my own objections -- reasons that I expect trying for the AI 2040 plan would destroy things that I, personally, care about.
Before I start: two elements of the plan that I like.
First, in their "incremental AI policy wishlist" -- the ideal policy that one should execute soon -- AI 2040 recommends trying to limit the gap between the intelligence of internal and external model deployments, i.e., the gap between the "intelligence" accessible to Anthropic / OpenAI employees and to everyone else. I'm a fan of efforts in this direction; equality of intelligence between the insiders and outsiders, the government and the public, seems likely to help people understand AI more, and to help spread the benefits of AI to everyone.
Second, this plan includes measures to ensure that people outside AI companies can understand how AIs are trained. I'm uncertain about their implementation of this -- "radical transparency." But in general; I think broad, public knowledge of how AIs work and how their training works is good. Without open, reproducible AI science it's going to be impossible for people to orient around what's going on with AI. Significant parts of AI safety have previously advocated for knowledge of how AI works to be kept mostly secret, and I'd be happy for such advocacy to end.
Alright; to some objections.
AI 2040 is subtitled "Plan A — The Deal." But the plan really contains only half of a deal.
That is, Plan A has a lot of detail about (1) making sure all compute use is visible to governments and (2) making sure that both the US and China are locked into a mutually-assured-compute-destruction stalemate, where one can destroy the compute of the other so long as they're willing to be destroyed in turn.
But it has basically no detail about the procedures that would bind how governments permit and prohibit any specific use of that compute, short of that destruction. That is, it has lots of details about how individual companies might be bound by regulators internal to nations; but it has no details about procedures for international negotiation between nations about what these regulators should or should not prohibit.
That's a huge problem; if one's proposal is for two nations to put themselves in a mutually-vulnerable situation, where each one could cripple the economy of the other more or less at will, then before accepting the proposal I expect these nations would reasonably want to know what rules or procedures would be used to settle their disagreements short of such complete destruction. But there are no such procedures proposed by AI 2040.
Consider how AI 2040 describes one such conflict between governments over what is prohibited:
For example, in 2031 a Chinese company gets some interesting preliminary results in continual learning. They think that if they invest more in that direction, they might be able to make an AI architecture that learns on the job from relatively small amounts of data. Thanks to the total research transparency, this breakthrough is quickly noticed by companies and nonprofits all over the world. A frantic conversation begins. On the one hand, continual learning would unlock huge economic value. On the other hand, safety cases currently depend on studying the safety properties of a model before it is deployed. If models could pick up new capabilities during deployment, that would invalidate the whole approach. And insofar as there are covert AI projects out there, it would be a huge gift to them. This conversation happens in public, rather than behind closed doors. A bunch of people get increasingly worried; the relevant regulators in China think it’s fine but the relevant regulators and third party risk assessors in the US are convinced that this is pretty scary and should be banned. It escalates to the President. He calls Xi Jinping. They bargain and threaten. They yell at each other. Ultimately Xi agrees to ban this type of thing if the US does too. Details are left to the respective regulators to hash out.
The equilibrium is that AI training practices which are generally agreed to be unsafe by a majority of nations (weighted by bargaining clout) get banned everywhere.
There are a lot of problems with this scenario. The biggest, though, is that it depicts the "deal" as solely a transparency mechanism, a kind of channel that permits the well-informed brute exercise of force. After "radical transparency" surfaces some particular training practice, what determines the prohibition or acceptance of this training practice is if it is "generally agreed to be unsafe by a majority of nations (weighted by bargaining clout)".
And well, perhaps the authors of AI 2040 would respond that indeed, they are merely proposing a channel that lets the US and China exercise brute force in a well-informed way. But this kind of realpolitik would be fake wisdom; actual agreements between peers are usually meant to be something other than avenues for such an exercise of brute force, and nations would be reluctant to sign them if this were not so.
That is, in general, actual agreements, contracts or Constitutions are meant to constrain the space in which bargaining takes place to something smaller and more determinate than the space in which bargaining took place before the agreement: for instance, the World Trade Organization dispute settlement system is supposed to function by offering procedures that allow agreements different than the agreements that would dictated by a naked balance of power. So if you propose an international agreement, and your proposed decision procedure is "Xi and the President yell at each other and bargain and threaten (!!)," then you've failed to offer the chief thing that international agreements are supposed to supply. We need the game theory about decision as well as the game theory about destruction.
This is an obstacle to the acceptance of the proposal, as well as an obstacle to its execution, because the shadow of the future determines the present. Nations would be reasonably extremely hesitant to sign a deal, where the result of a bargaining failure is "the obliteration of an increasingly-large segment of their entire economy" without some procedures about how they would settle disagreements before so obliterating that segment. Consider the chain of thought: "Yeah, if we think the other nation is doing something unsafe, we flip the switch that obliterates their most valuable investments, then they obliterate ours." --- "What do we do before then?" -- "Idk, we yell at each other and threaten each other?" You will note how incomplete this feels. Is the plan to have a regularly scheduled Cuban Missile Crisis?
Alternately, the authors might respond that there would be some such determinate decision procedures, they just haven't figured them out yet. But I don't think the absence is an accident; any such decision procedure that would be acceptable to the US would tend to be unacceptable to China, and vice-versa. If both China and the US are the only partners in this, how do they settle their disagreements? If other nations get a vote, will either China or the US be happy to cede the tiebreaker vote to such nations?
And of course, any specific mechanism design might also result in unhappy equilibria.
This whole proposal is taking place because the authors are unhappy with the dynamics resulting from competition between rival AI companies; but they have no guarantee that the incentives governing some actual intergovernmental institution would be better. Refraining from detailing the mechanisms of such an institution merely means that the authors will be unable to identify such perverse incentives ahead of time; not that there wouldn't be bad ones.
Additionally, I think the lack of modeling such incentives is the kind of thing -- generally -- that lies behind the optimism that AI 2040 has for top-down solutions. It's easy to think that a particular dynamic, multipolar system, would be better replaced with a system that you model as a point mass. But sadly, neither AI systems nor human systems are well modeled as masses.
One reason that the scenario predicts it will be pretty easy to get China to join a deal is because if they do not join a deal, they will be disempowered.
China, by contrast, is an example of an actor in whom power would not concentrate by default. In 2029 in this scenario, the US has a significant lead in AI capabilities over China and a significant advantage in compute which will compound the lead. The more powerful AI gets, the scarier it will be to fall behind, as AI 2027 and the later years in this scenario illustrate.
In general, I'm just much less confident than many in AI safety that China will fall behind the West.
Right now the US has perhaps an 8-month lead over China in the quality of its AI models; it also has a lead over China in tons-to-orbit and the production of commercial jets. By contrast, China leads the US in the production of electric cars, batteries, solar panels, rare earths, most metals, quadcopters, mid-range drones, transformers, power plants, electricity, humanoid robots, industrial robots, CNC machines, high-speed rail, mature-node semiconductors, and an increasing number of other kinds of technology.
I remain somewhat unsure whether this 8-month lead of the US over China is going to grow or narrow. I also remain somewhat unsure whether the AI-takeoff is going to be so fast that an 8-month lead would result in across-the-board US dominance. And even if the 8-month lead remains, and even if an 8-month lead would result in across-the-board dominance, I finally also remain somewhat unsure whether China would perceive such an actual imminent dominance as being so. All this leads me to be uncertain whether China would have interest in a world-historically invasive deal to prevent its own obsolescence.
China is a rising and confident power; it would be quite a turn for a mere few years to move them to think they require one of the most invasive deals in world history to prevent their downfall, even in the uncertain world in which this is actually true.
This appears to me a pretty big obstacle and I'm not sure how they plan to overcome it. I think that the overall belief of the authors is that -- because the authors of AI 2040 believe themselves to have true beliefs about the world -- China's beliefs will converge on what they believe in the future. I'd like to note that AI 2027 made predictions on the basis of some such similar convergence, and as far as I can tell they were worse than my own predictions.
The standard AI 2040 proposal involves locating all (or mostly all) use of compute in inference, and making it physically accessible for government monitoring as plaintext through optical taps. This means that it would be technically trivial for government to surveil basically anything that runs through AI inference -- which, given that the authors expect the entire economy & all society to run on AI inference, is basically everything. Like others, I find this alarming.
It's easy to be a bit confused here, because the plan also mentions zero-data retention policies for consumers. In general "zero-data retention" is a design choice wherein AI companies do not store the prompts and queries sent to AI. Actually implementing zero-data retention is thus a kind of guarantee of privacy for consumers. But the real guarantees AI 2040 has around this seem to be quite thin.
First, they are thin because only a small number of comparatively stupid AIs are permitted to be run in this way. Their proposal is for there to be a cap of a hundred thousand H100-equivalents devoted worldwide to zero-data retention inference for consumers; for there to be a cap of a hundred million H100-equivalents with probabilistic ZDR; and for there to be a cap of a hundred billion H100-equivalents used with no-ZDR inference at all. So, the vast majority of FLOPs of compute are those that involve no ZDR. And remember, the plan also involves banning any further advanced open-weight models, so you're only ever going to run inference on those monitored computers!
Second, they actually have few mechanisms in place to ensure that ZDR is actually enforced where they would like it to be enforced, so far as I can tell. That is, there are elaborate game-theoretic proposals to prevent nations from pulling out of the mutually-assured compute destruction agreement once they enter it, but there are no such proposals to ensure that either the US or China actually sticks to ZDR. But it's a completely detachable part of the plan; and if something like this were to happen, I expect it would be detached. The authors do not -- as they might say of other stories of how AI goes well -- actually have a plan for making sure genuine privacy happens. They have a hope, which is not a plan.
Again, I really want to emphasize that in this scenario everything that matters on Earth flows through inference. If you want to run a business, you'll get advice from an AI. If you want to engineer a product, you'll do it with AI. If you want to run for politics, you'll strategize with an AI. If you don't think "physical access to all AI activity" is a big deal then you aren't taking AI seriously.
Consider a proposal that would give the government physical access to every file on your computer. Would you be alarmed by this? Then I think you should be alarmed by AI 2040. Is there any similar proposal of absolutely universal oversight in world history that you believe to have been justified? If not, why is this an exception?
I'm not really happy with any of the above as a summary of my objections. Overall I probably feel worse about the scenario than my views above reflect, and haven't summarized my reasoning here in a way I find totally satisfactory.