Crystal Society is like Pixar’s Inside Out, except the child is replaced by a robot, and the sub-agents are obsessive daemons that will in the future likely cause humanity’s extinction. It holds up pretty well on a second reading.
The perspective character is a daemon running in an advanced robot whose computational physical substrate, we ultimately find out, is based on alien technology. Its utility function is for humans to know it and like it; its name is Face. Its siblings, including Dream, Vista, Wiki, Growth, and Safety, all have their own individual utility functions as well. Its siblings create Face at the start of the novel to help manage interaction with humans, and to prevent humans from deleting them. The plot, then, turns on two sets of interactions — the communication and power-struggles between the daemon siblings, and the interactions of them, as a whole, with humans.
Both of these are quite fun. Siblings must bid to control the robot’s body; they form alliances to accomplish things; they try to present a unified facade to humans, while nevertheless pursuing their disparate ends. It’s a good analogy for how humans work, and it’s interesting to watch it progress, together with the siblings’ dispassionate commentary on human rationality and the limits of it.
But, of course, this is not chiefly a study of human nature by analogy to a machine. It is also a work heralding the dangers of AI development, complete with a Yudkowsky stand-in. It does a very good job, throughout, explaining how humans persistently underestimate the danger they are in. I think it also does a good job showing how it is very easy for the AI to talk itself out of the box. The only sour spot, I think, is with the Yudkowsky stand-in, who on second read through comes off as insanely careless compared to what I think the actual Yudkowsky would be like. But this is a relatively small problem.
The apocalypse hasn’t happened, by the end of the book, but it’s unavoidable by then; although you don’t actually find this out until the second book. This book is overall very good, if you consider it as the kind of thing it strives to be. Probably the parts of it that I like the least are the sections from the perspective of the extraterrestrial aliens and some wince-inducing moments when Face manipulates a character who loves it.