depend somewhat on the goals of the people within the organization, but they are not identical.
Any American knows how loose the tie is between the actions of the U.S. government and the diverse and often contradictory aims of its citizens. That is also true of corporations. For-profit corporations nominally serve multiple constituencies, including shareholders, senior executives, employees, and customers. These corporations differ in how they balance their loyalties and often behave in ways that serve none of their constituents. The “neurons” that carry their corporate thought are not just the human employees or the technologies that connect them; they are also coded into the policies, incentive structures, culture, and procedural habits of the corporation. The emergent corporate goals do not always reflect the values of the people who implement them. For instance, an oil company led and staffed by people who care about the environment may have incentive structures or policies that cause it to compromise environmental safety for the sake of corporate earnings. The components’ good intentions are not a guarantee of the emergent system’s good behavior.
Governments and corporations, both built partly of humans, are naturally motivated to at least appear to share the goals of the humans they depend upon. They could not function without the people, so they need to keep them cooperative. When such organizations appear to behave altruistically, this is often part of their motive. I once complimented the CEO of a large corporation on the contribution his company made toward a humanitarian relief effort. The CEO responded, without a trace of irony, “Yes. We have decided to do more things like that to make our brand more likeable.” Individuals who compose a hybrid superintelligence may occasionally exert a “humanizing” influence—for example, an employee may break company policies to accommodate the needs of another human. The employee may act out of true human empathy, but we should not attribute any such empathy to the superintelligence itself. These hybrid machines have goals, and their citizens/customers/employees are some of the resources they use to accomplish them.
We are close to being able to build superintelligences out of pure information technology, without human components. This is what people normally refer to as “artificial intelligence,” or AI. It is reasonable to ask what the attitudes of the hypothetical machine superintelligences will be toward humans. Will they, too, see humans as useful resources and a good relationship with us as worth preserving? Will they be constructed to have goals that are aligned with our own? Will a superintelligence even see these questions as important? What are the “right questions” that we should be asking? I believe that one of the most important 1s this: What relationship will various superintelligences have to one another?
It is interesting to consider how the hybrid superintelligences currently deal with conflicts among themselves. Today, much of the ultimate power rests in the nation states, which claim authority over a patch of ground. Whether they are optimized to act in the interests of their citizens or those of a despotic ruler, nation states assert priority over other intelligences’ desires or goals within their geographic dominion. They claim a monopoly on the use of force and recognize only other nation states as peers. They are willing, if necessary, to demand great sacrifices of their citizens to enforce their authority, even to the point of sacrificing their citizens’ lives.
122
HOUSE_OVERSIGHT_016342