Many in AI safety believe that it's good for one AI developer to be far, far ahead of the others in their progress towards artificial superintelligence. That is, they think that we're close
The chief reason given for this belief has been that, if one developer is far ahead of the others, if they run into some kind of a problem with steering, understanding, and "aligning" AI, their lead will enable them to "pause" and temporarily halt their own AI development in order to solve alignment, share information, or coordinate with others.
I'll refer to this strategy as the "burning the lead" strategy, and the belief that it is broadly good as the "burning the lead" belief. You can find arguments in favor of this belief from Bostom, Karnofsky, Amodei and others.
Even so: Is "burning the lead" actually a good strategy for making the future go well?
I think that, no, the overwhelming preponderance of the evidence is that it is not a good strategy.
At a very high level, I think the argument for "burning the lead" is that it helps you through improving the chances of technical alignment. But gives you large benefits in only a small and unlikely fraction of possible future worlds, but has great costs in almost all possible future worlds. The strategy has at most a low-to-moderate chance of improving the odds of technical alignment going well -- and is plausibly just as likely to make technical alignment go poorly. But the strategy has a high-to-certain chance of making power struggles over AI worse, hurting epistemics and society's ability to orient around AI, and to generally make the future go worse.
TRASH --Thus, the "burning the lead" strategy is most attractive in only a small
But it has a high-to-certain chance of making
That is, the benefits of the strategy are great only in worlds where alignment is broadly hard and where a software-only intelligence explosion is likely to happen. And these benefits come almost entirely through improved odds of "technical alignment" succeeding, even though it is uncertain whether this lead actually increases the likelihood of "technical alignment"'s success.
On the other hand, the costs of this strategy are great in all worlds,
On the othe hand, the costs of the strategy are universal. By
Scenario:
At any point during the development of artificial intelligence, one can imagine a distribution of AI developers, sorted by the intelligence of the AIs they can create. Let's make the 0-indexed developer have the most intelligent AI, the 1-indexed developer the next most intelligent, and so on. This developer-chunked distribution of intelligences may look exponential and uneven -- as advocated for by the strategy of "burning the lead" -- or it might be relatively even and flat.
[images]
What produces such a distribution? Well, a top-heavy, low-entropy distribution is likely if both knowledge about how to create AI and compute used for training AI is concentrated in that one particular actor. And a flat, high-entropy distribution is likely if knowledge and compute is correspondingly dispersed.
Costs of being like this:
What happens if "keeping a lead" is extremely difficult?