mainstream groups, who had more to lose, initially qualified and diluted the message, taking positions like, “It would make sense in the long term to delegate control over local matters.” (There were always exceptions: Some public intellectuals proclaimed the original dissident message verbatim.) Finally, the original message—being, simply, true—won out over its diluted versions. Estonia regained its independence in 1991, and the last Soviet troops left three years later.
The people who took the risk and spoke the truth in Estonia and elsewhere in the Eastern Bloc played a monumental role in the eventual outcome—an outcome that changed the lives of hundreds of millions of people, myself included. They spoke the truth, even as their voices trembled.
The Second Message: AI Risk
My exposure to the second revolutionary message was via Yudkowsky’s blog—the blog that compelled me to reach out and arrange that meeting in California. The message was: Continued progress in AI can precipitate a change of cosmic proportions—a runaway process that will likely kill everyone. We need to put in a lot of extra effort to avoid that outcome.
After my meeting with Yudkowsky, the first thing I did was try to interest my Skype colleagues and close collaborators in his warning. I failed. The message was too crazy, too dissident. Its time had not yet come.
Only later did I learn that Yudkowsky wasn’t the original dissident speaking this particular truth. In April 2000, there was a lengthy opinion piece in Wired titled, “Why the Future Doesn’t Need Us,” by Bill Joy, co-founder and chief scientist of Sun Microsystems. He warned:
Accustomed to living with almost routine scientific breakthroughs, we have yet to come to terms with the fact that the most compelling 21st-century technologies—robotics, genetic engineering, and nanotechnology—pose a different threat than the technologies that have come before. Specifically, robots, engineered organisms, and nanobots share a dangerous amplifying factor: They can self-replicate. .. . [O|ne bot can become many, and quickly get out of control.
Apparently, Joy’s broadside caused a lot of furor but little action.
More surprising to me, though, was that the AI-risk message arose almost simultaneously with the field of computer science. In a 1951 lecture, Alan Turing announced: “[I]t seems probable that once the machine thinking method had started, it would not take long to outstrip our feeble powers. .. . At some stage, therefore, we should have to expect the machines to take control... .”?! A decade or so later, his Bletchley Park colleague I. J. Good wrote, “The first ultraintelligent machine is the /ast invention that man need ever make, provided that the machine is docile enough to tell us how to keep it under control.””? Indeed, I counted half a dozen places in The Human Use of Human Beings where Wiener hinted at one or another aspect of the Control Problem. (“The machine like the djinnee, which can learn and can make decisions on the basis of
*1 Posthumously reprinted in Phil. Math. (3) vol. 4, 256-60 (1966). *2 Irving John Good, “Speculations concerning the first ultraintelligent machine,” Advances in Computers, vol. 6 (Academic Press, 1965), pp. 31-88.
71
HOUSE_OVERSIGHT_016291