"If Anyone Builds It, Everyone Dies" (IABI) is the new book from the Machine Intelligence Institute, about how AI companies are trying to build very smart AIs that will certainly kill us all. I disagree with it in too many places to write just one review. So I'm going to try to cover different topics from different chapters.
The first chapter of IABI presents us with two problems: (1) indifference to empirical details, when accuracy about them would not immediately support the book's hypothesis; and (2) a corresponding preference for Grand Narratives that elide such details.
It would be reasonable for me to skip the first chapter; it's not the most vital for the book's argument. Nevertheless, I think problems like these two problems are endemic both to the book and to the prior Yudkowskan body of thought. So I think it's worth spending a little time on them to get a notion of the flavor of mistake that we're going to see throughout.
The question that the first chapter asks is this -- why do humans rule over Earth's other animals, rather than vice-versa? Why do humans determine whether Chimpanzee or Orcas will go extinct, rather than the other way around?
And the chapter responds, unambiguously -- general intelligence! Specifically, "intelligence" conceived of as broadly general, cross-domain prediction and steering of the world:
Other species on Earth are born with specialized skills: bees build beehives, beavers build dams. A human looks at the beaver’s dam and figures out how it’s done; we learned to make dams, without needing the knowledge built into our genes. Now we dam rivers, like beavers; we build houses for ourselves, like bees; we weave threads into nets, like spiders. We build power plants and space-rockets that no other species builds at all.
Humans can do things our ancestors never did, and which other animals cannot do, because of a quality sometimes named “intelligence.” Our genes did not wire most of our abilities into us; instead we observed, we tried, we remembered, we generalized, and then we achieved... Human intelligence is the source of all our power, all our technology.
The chapter goes on to gloss that "intelligence" consists in predicting and steering reality in a "general" fashion not shared by AIs or other animals.
Note that sometimes the supplemental material talks about intelligence as whatever it is that makes humans better than chimps. That is, rather than say that "intelligence" consists in general prediction and steering, it's simply the thing that makes humans dominant, whatever it might be:
When we speak of “human-level intelligence,” we are trying to talk about whatever quality makes humans capable of building and maintaining a technological civilization, in the way that chimpanzees can’t.
But this is a somewhat weird way to talk -- Soares and Yudkowsky clearly don't mean 'intelligence' as an empty placeholder for any quality that might be necessary or sufficient for making humans capable of building civilization. They think it describes a quality in one's brain, rather than in one's body, for instance.
So accordingly -- usually Nate and Soares usually talk as if they have good evidence "whatever it is" of "intelligence" is in fact pointing at general human problem-solving ability, and in fact a problem solving ability they take to be genetically determined:
Our ancestors were, somehow, selected to be good at solving problems, broadly construed, despite our ancient ancestors rarely facing an engineering trial more complicated than building a spear.
Note that they do not, however -- at any point -- offer specific citations from any branch of paleoanthropology, of comparative psychology, of evolutionary biology, or of any science, supporting this assertion.
Now as a first guess, I admit, "general problem solving ability based on the structure of the brain" seems like a pretty good guess for why humans dominate the earth. But history of science is litered with a lot of "seems like a pretty good guess" views -- views considered too obvious to bother justifying -- which in fact, turned out to be totally false.
So: How do we know if this is true? What other qualities could explain human dominance? How would we know if "genetically granted, general problem solving" is the primary quality, or even the most important, that grants human dominance?
Well, in order to have evidence either for or against this theory, we at least need to be able to articulate to ourselve an alternate hypothesis.
Evidence is data that discriminates between theories that predict different things; so both falsifiability and 'evidence' in general require the ability to formulate alternate theories. If we're seeking evidence on whether "General problem-solving ability, rooted in the brain's structure, is the key to humanity's dominance" is true, we need to come up with some other possible explanation.
So to ask ourselves whether is the correct theory, we must ask ourselves: what else could be such an explanation?
Here's another possibility: cultural intelligence.
Let's imagine an intentionally extreme version of "cultural intelligence" as a theory of human dominance: suppose that humans have very little generalized problem-solving ability. Suppose the main thing that distinguishes humans from other animals is that human genetics give humans a greater ability and greater inclination to acquire skills from other humans, than any other animal does from almost any other animal. (Or -- if we grant primate-like hands are key for dominatino -- a greater ability and inclination than Chimpanzees and bonobos.)
In this world, the important fact is that when one human does stumble across some solution to some problem, this solution is much more likely to get handed on to one's descendants or peers than it would be for other animals. In sufficiently large groups of humans, these handed-down solutions get embellished and further enhanced over time, by humans who (although still lacking generalized problem-solving ability) either do get lucky in stumbling over improvements to the currently-best solution, or perhaps by humans who have learned some specialized, rather tha generalized, problem-solving ability. Some such accidentally-acquired solutions are themselves rather broad -- once you stumble on, say, marine Roman concrete you can use it in many places -- and allow you to push forward the handed-down tradition much further.
So, on this theory, this vast accumulated cultural payload, whose integrity over generations is secured by greater human imitative ability, and whose contents exceed that of any individual mind, is the key that explains, say, human dominance over all animals by the time of the Romans. "The thing that explains human dominance" is not "general problem solving," but a vastly increased ability to preserve solutions to problems, over time and space.
What might be evidence for or against this theory of cultural intelligence? Let's look at two domains: comparative human-Chimpanzee psychology, and at the timing at which distinctively human behaviors emerged.
1. Let's examine the contrast between the abilities of young chimpanzees and young humans, and whether it fits more with one theory or another.
To zoom in on one such contrast -- you might consider how human children are more likely than chimpanzees to "overimitate" the actions of an adult.
There's a well-known experiment in which an adult demonstrates to a young human or young Chimpanzee how to open a box. In doing so, the adult performs some movements that are visibly causally necessary, and performs other movements that are visibly causally superfluous. We find that a human three-year old who then attempts to open the box is very likely to "overimitate" and blindly copy the causally superfluous actions while opening the box; by contrast, the "dumb ape" at a similar age tends to be more discerning, and to copy only the causally necessary actions. So humans -- in this particular context -- "overimitate" by cutting-and-pasting actions, while chimps are more likely to lean on the underlying causal structure.
This surprising result replicates in a test contrasting children and bonobos. Five-year-old children overimitate even more reliably than three-year-old children; children can persist in this "overimitation" even when doing so is costly. So we find that humans exceed apes in expertise at copying others, even contrary to
But -- I hasten to note -- this particular genre of experiment is are just one from a whole host of comparative psychology. This field tends to show that the distinctively human cognitive skillset, when contrasted with our nearest primate relatives, focuses on imitative, social, and cooperative abilities, rather than generic problem solving.
One forceful statement of this position is found in the abstract of "The Cultural Intelligence Hypothesis," a journal article describing its conclusion after performing a battery of such tests, some focusing on physical cognitive skills and others focusing on social cognitive skills (emphasis mine):
Humans have many cognitive skills not possessed by their nearest primate relatives. The cultural intelligence hypothesis argues that this is mainly due to a species-specific set of social-cognitive skills, emerging early in ontogeny, for participating and exchanging knowledge in cultural groups. We tested this hypothesis by giving a comprehensive battery of cognitive tests to large numbers of two of humans' closest primate relatives, chimpanzees and orangutans, as well as to 2.5-year-old human children before literacy and schooling. Supporting the cultural intelligence hypothesis and contradicting the hypothesis that humans simply have more “general intelligence,” we found that the children and chimpanzees had very similar cognitive skills for dealing with the physical world but that the children had more sophisticated cognitive skills than either of the ape species for dealing with the social world.
So to reiterate: human-chimpanzee contrastive experiments which find that humans exceed chimps most specially in social skills are evidence that something like the cultural intelligence hypothesis is true. Or at the very least, evidence that "general intelligence is the key to human dominance" is missing large, necessary parts of the picture.
2. Let's switch from comparative psychology to paleoanthropology to find evidence distinguishing our two theories.
You could consider whether different theories make different predictions about when distinctively human behaviors arrive.
Consider those who think that "general problem solving skill" is the most important thing for humans to have, and that it specifically hinges on the "three pounds of slimy, wet, grayish tissue" within one's skull, as Yudkowsky does. It would be natural for such a theory to predict that after you have modern human brains, you should quickly acquire modern human behavior. In fact, the IABI book more or less informally predicts that modern human behavior and dominance follows quickly upon modern human DNA:
Speaking historically (or rather anthropologically), it looks like at some point after humans and chimpanzees started diverging, a threshold was crossed...
To the naked eye, leaving aside all theory, it looks like some kind of dam broke and unleashed a vast flood of intelligence behind it. Some unknown kind of “all hell” broke loose...
Intelligence powerful and general enough to create a civilization seems to have hit the world fast and hard, cleanly separating Homo sapiens from the other animals.
The problem for this view is that there appears to be quite a long wait between the onset of anatomically modern humans (AMH) and the onset of "behaviorial modernity".
You should note that this is a pretty hotly contested scientific topic, but it looks like an "anatomically modern human" -- a human that you might recognize on the subway, one within the range of modern human phenotypes -- came into existence very roughly 300,000 to 100,000 years ago. While a majority of the behaviors we associate with modern humans came into existence much more recently -- 50,000 to 10,000 years ago, depending on which behaviors one believes to be most significant. One name for this puzzle is the "Sapient Paradox".
Renfrew, inventor of the phrase, summarizes part of the problem:
The human mind over the past 10,000 years, in most areas of the world, has developed symbolic concepts of such evident factuality that it is difficult for us now to imagine life without them. These include notions of value (including money), of number and measure, of individual people of high status and power, and of material things that embody the sacred and the forbidden. These are often considered givens of the human condition.
Yet these features are not ‘givens’ at all—they are emergent features of the past 10,000 years. How are such fundamental changes in the human condition supported in the brain?
The problem for the neuroscientist and the evolutionary archaeologist is to understand how this has come about on so short a time frame, when the human genome has been established for much longer, certainly since the out-of-Africa dispersals of some 60,000 years ago (Forster 2004; Mellars 2006a,b). This problem indeed relates to the sapient paradox (Renfrew 1996): that the biological basis of our species has been established for at least that time (and perhaps for as much as 200,000 years), while the novel behavioural aspects of our ‘sapient’ status have taken so long to emerge or to construct themselves, or rather that they have done so very recently. This must lay emphasis upon the plasticity of the human brain (its capacity to adapt within a single lifespan to new conditions) and on the aspects of the socialization process of shared experience.
...the only conceivable solution to the sapient paradox requires that the performance of this brain should be seen within a short-term evolutionary context where genomic change is probably not significant
Or -- to somewhat tendentiously gloss -- the solution to the Sapient Paradox lies in "modern human behavior" having as its absolute prerequisite the prior cultural build-up that grants humans many skills. Other scientists
Is it generalized problem-solving that explains human dominance over other animals? What kind of things would this predict about the world? Are there other theories that make alternate predictions?
There's a countervailing cluster of theories that one could
So, I've taken a deeper dive into two topics above for evidence distinguishing whether "general problem solving skill" and "cultural accumulation skill" are more responsible for human dominance.
Note that neither of these two deeper dives are remotely close to comprehensive; at best I've managed to avoid being egregiously misleading. And these two topics are not the only ones that might be invoked upon what is response for human dominance. One might invoke topics as distinct as: Isaac Newton's interest in the occult and in the Book of Revelation; Chimp-human eye color differences; why Romans didn't invent the bicycle; how far unrecorded history extends and how likely we are to find unprecedently ancient ruins; or the role of revelations from one's god in mathematics.