grids that flipped nuclear plants on or off to a logic only they understood. Today. The more profound version, however, would be the arrival of AI that really did think and create and intuit tremors too subtle for the human mind. Tomorrow. Like so much of our connected age, such machines would arrive, Vinge felt, because we want and even need them to achieve our dreams. Then, he supposed, they would take over. The leap from evoking Mozart to enacting Stalin would not be so much ofa leap anyhow, at least technologically. It’s just bits. Goode’s definition could have been screwed into something still tighter: “Let an ultraintelligent machine be defined as the box that will eliminate us.” The day after tomorrow.
What spun uneasily from that silly NASA poem, “Our robots precede us....” is a fear: Real Al is fish bait. We'll snap at it hungrily, hoping it will satisfy some human ache only to discover we've been hooked, soon to be devoured. The idea that a superintelligent device would always be docile enough to tip us off to its secret switches of control or to reveal its looming accidents in a way our simple minds can understand, seems unlikely. To be honest, we might have a hard time even understanding the off switches, let alone reaching them. So many of our incentives are to let an effective Al finger more and more of our lives. To teach and encourage it, in some settings, extremely undocile: A weapon to attack our enemies, our political opponents or, finally, each other. It was easy enough for Vinge to see how this would end. It wouldn't be with the sort of intended polite, lap-dog domesticity of artificial intelligence we might hope for, but with a rotweiler of a device, alive to the meaty smell of power, violence and greed.
The Oxford philosopher Nick Bostrom has described the following thought experiment: Imagine a super-intelligent machine, programmed to do whatever is needed make paperclips as fast as possible and connected to every resource that task might demand. 7°¢Go figure it out! might be all its human instructors tell it. As the clip-making AI becomes better and better at its task, it demands more and still more resources: more electricity, steel, manufacturing, shipping. The paperclips pile up. The machine looks around: If only it could control the power supply. The shipping. The steel mining. The humans. And so, ambitious for more and better paperclips, it begins to think around its masters, - incapable of stopping until it has punched the entire world into paperclips. You had to hope someone had remembered to place a “halt” command into is logic somewhere. And though Bostrom’s messianic wire twister is unlikely - of course, no one is going to forget to tell a machine to stop making paperclips - the power of his example is to remind us that if humans can lose their minds, so can Als. “We cannot blithely assume that a superintelligence will necessarily share any of the final values stereotypically associated with wisdom and intellectual development in humans,” Bostrom wrote. “It is no less possible—and probably technically easier—to build a superintelligence that places final value on nothing but calculating.” And as these devices cogitate in
266 Imagine a super-intelligent machine: Nick Bostrom, “Ethical Issues in Advanced Artificial Intelligence,” Cognitive, Emotional and Ethical Aspects of Decision Making in Humans and in Artificial Intelligence (2003) Vol 2, ed I, Smit et al, Institute of Advanced Studies in Systems Research and Cybernetics, pp 12-17
194
HOUSE_OVERSIGHT_018426