318 18 Advanced Self-Modification: A Possible Path to Superhuman AGI
only after these have achieved something close to human-level general intelligence (even if not precisely humanlike general intelligence).
Another troublesome issue regarding self-modification is that the boundary between "self- modification" and learning is not terribly rigid. In a sense, all learning is self-modification: if it doesn’t modify the system’s knowledge, it isn’t learning! Particularly, the boundary between "learning of cognitive procedures" and "profound self-modification of cognitive dynamics and structure" isn’t terribly clear. There is a continuum leading from, say,
1. learning to transform a certain kind of sentence into another kind for easier comprehension,
or learning to grasp a certain kind of object, to
2. learning a new inference control heuristic, specifically valuable for controlling inference
about (say) spatial relationships; or, learning a new Atom type, defined as a non-obvious
judiciously chosen combination of existing ones, perhaps to represent a particular kind of
requently-occurring mid-level perceptual knowledge, to
3. learning a new learning algorithm to augment MOSES and hillclimbing as a procedure
earning algorithm, to
4, learning a new cognitive architecture in which data and procedure are explicitly identical, and there is just one new active data structure in place of the distinction between AtomSpace and MindAgents
Where on this continuum does the "mere learning" end and the "real self-modification" start?
In this chapter we consider some mechanisms for "advanced self-modification" that we believe will be useful toward the more complex end of this continuum. These are mechanisms that we strongly suspect are not needed to get a CogPrime system to human-level general intelligence. However, we also suspect that, once a CogPrime system is roughly near human-level general intelligence, it will be able to use these mechanisms to rapidly increase aspects of its intelligence in very interesting ways.
Harking back to our discussion of AGI ethics and the risks of advanced AGI in Chapter 12, these are capabilities that one should enable in an AGI system only after very careful reflection on the potential consequences. It takes a rather advanced AGI system to be able to use the capabilities described in this chapter, so this is not an ethical dilemma directly faced by current AGI researchers. On the other hand, once one does have an AGI with near-human general intelligence and advanced formal-manipulation capabilities (such as an advanced CogPrime system), there will be the option to allow it sophisticated, non-human-like methods of self- modification such as the ones described here. And the choice of whether to take this option will need to be made based on a host of complex ethical considerations, some of which we reviewed above.
18.2 Cognitive Schema Learning
We begin with a relatively near-term, down-to-earth example of self-modification: cognitive schema learning.
CogPrime’s MindAgents provide it with an initial set of cognitive tools, with which it can learn how to interact in the world. One of the jobs of this initial set of cognitive tools, however, is to create better cognitive tools. One form this sort of tool-building may take is cognitive
HOUSE_OVERSIGHT_013234