Gödel, Escher, Bach: An Eternal Golden Braid (124 page)

Quite seriously, a machine that can pass the Turing test may well add as slowly as you or I do, and for

similar reasons. It will represent the number 2 not just by the two bits "10", but as a full-fledged concept the way we do, replete with associations such as its homonyms

"too" and "to", the words "couple" and "deuce", a host of mental images such as dots on dominos, the shape of the numeral '2', the notions of alternation, evenness, oddness, and on and on ... With all this "extra baggage" to carry around, an intelligent program will become quite slothful in its adding. Of course, we could give it a ' pocket calculator , so to speak (or build one in). Then it could answer very fast, but its performance would be just like that of a person with a pocket calculator. There would be two separate parts to the machine: a reliable but mindless part and an intelligent but fallible part. You couldn't rely on the composite system to be reliable, any more than a composite of person and machine is necessarily reliable. So if it's right answers you're after, better stick to the pocket calculator alone-don't throw in the intelligence!

Question: Will there be chess programs that can beat anyone?

Speculation: No. There may be programs which can beat anyone at chess, but they will not be exclusively chess players. They will be programs of general intelligence, and they will be just as temperamental as people. "Do you want to play chess?" "No, I'm bored with chess. Let's talk about poetry." That may be the kind of dialogue you could have with a program that could beat everyone. That is because real intelligence inevitably depends on a total overview capacity-that is, a programmed ability to "jump out of the system", so to speak-at least roughly to the extent that we have that ability. Once that is present, you can't contain the program; it's gone beyond that certain critical point, and you just have to face the facts of what you've wrought.

Question: Will there be special locations in memory which store parameters governing the behavior of the program, such that if you reached in and changed them, you would be able to make the program smarter or stupider or more creative or more interested in baseball? In short, would you be able to "tune" the program by fiddling with it on a relatively low level?

Speculation: No. It would be quite oblivious to changes of any particular elements in memory, just as we stay almost exactly the same though thousands of our neurons die every day(!). If you fuss around too heavily, though, you'll damage it, just as if you irresponsibly did neurosurgery on a human being. There will be no "magic"

location in memory where, for instance, the "IQ" of the program sits. Again, that will be a feature which emerges as a consequence of lower-level behavior, and nowhere will it sit explicitly. The same goes for such things as "the number of items it can hold in short-term memory", "the amount it likes physics", etc., etc.

Question: Could you "tune" an Al program to act like me, or like you-or halfway between us?

Speculation: No. An intelligent program will not be chameleon-like, any more than people are. ,It will rely on the constancy of its memories, and will not be able to flit between personalities. The idea of changing internal parameters to "tune to a new personality" reveals a ridiculous underestimation of the complexity of personality.

Question: Will there be a "heart" to an Al program, or will it simply consist of "senseless loops and sequences of trivial operations" (in the words of Marvin Minskys)?

Speculation: If we could see all the way to the bottom, as we can a shallow pond, we would surely see only "senseless loops and sequences of trivial operations"-and we would surely not see any "heart". Now there are two kinds of extremist views on AI: one says that the human mind is, for fundamental and mysterious reasons, unprogrammable. The other says that you merely need to assemble the appropriate

"heuristic devices-multiple optimizers, pattern-recognition tricks, planning algebras, recursive administration procedures, and the like",' and you will have intelligence. I find myself somewhere in between, believing that the "pond" of an Al program will turn out to be so deep and murky that we won't be able to peer all the way to the bottom. If we look from the top, the loops will be invisible, just as nowadays the current-carrying electrons are invisible to most programmers. When we create a program that passes the Turing test, we will see a "heart" even though we know it's not there.

Question: Will Al programs ever become "superintelligent"?

Speculation: I don't know. It is not clear that we would be able to understand or relate to a "superintelligence", or that the concept even makes sense. For instance, our own intelligence is tied in with our speed of thought. If our reflexes had been ten times faster or slower, we might have developed an entirely different set of concepts with which to describe the world. A creature with a radically different view of the world may simply not have many points of contact with us. I have often wondered if there could be, for instance, pieces of music which are to Bach as Bach is to folk tunes: "Bach squared", so to speak. And would I be able to understand them? Maybe there is such music around me already, and I just don't recognize it, just as dogs don't understand language. The idea of superintelligence is very strange. In any case, I don't think of it as the aim of Al research, although if we ever do reach the level of human intelligence, superintelligence will undoubtedly be the next goal-not only for us, but for our Al-program colleagues, too, who will be equally curious about Al and superintelligence. It seems quite likely that Al programs will be extremely curious about Al in general-understandably.

Question: You seem to be saying that AI programs will be virtually identical to people, then. Won't there be any differences?

Speculation: Probably the differences between Al programs and people will be larger than the differences between most people. It is almost impossible to imagine that the "body" in which an Al program is housed would not affect it deeply. So unless it had an amazingly faithful replica of a human body-and why should it?-it would probably have enormously different perspectives on what is important, what is interesting, etc. Wittgenstein once made the amusing comment, "If a lion could speak, we would not understand him." It makes me think of Rousseau's painting of the gentle lion and the sleeping gypsy on the moonlit desert. But how does Wittgenstein know? My guess is that any Al program would, if comprehensible to us, seem pretty alien. For that reason, we will have a very hard time deciding when and if we really are dealing with an Al program, or just a "weird" program.

Question: Will we understand what intelligence and consciousness and free will and "I"

are when we have made an intelligent program?

Speculation: Sort of-it all depends on what you mean by "understand". On a gut level, each of us probably has about as good an understanding as is possible of those things, to start with. It is like listening to music. Do you really understand Bach because you have taken him apart? Or did you understand it that time you felt the exhilaration in every nerve in your body? Do we understand how the speed of light is constant in every inertial reference frame? We can do the math, but no one in the world has a truly relativistic intuition. And probably no one will ever understand the mysteries of intelligence and consciousness in an intuitive way. Each of us can understand
people
, and that is probably about as close as you can come.

Sloth Canon

This time, we find Achilles and the Tortoise visiting

the dwelling of their new friend, the Sloth.

Achilles: Shall I tell you of my droll footrace with Mr. T?

Sloth: Please do.

Achilles: It has become quite celebrated in these parts. I believe it's even been written up, by Zeno.

Sloth: It sounds very exciting.

Achilles: It was. You see, Mr. T began way ahead of me. He had such a huge head start, and yet

Sloth: You caught up, didn't you?

Achilles: Yes-being so fleet of foot, I diminished the distance between us at a constant rate, and soon overtook him.

Sloth: The gap kept getting shorter and shorter, so you could.

Achilles: Exactly. Oh, look-Mr. T has brought his violin. May I try playing on it, Mr. T?

Tortoise: Please don't. It sounds very flat.

Achilles: Oh, all right. But I'm in a mood for music. I don't know why. Sloth: You can play the piano, Achilles.

Achilles: Thank you. I'll try it In a moment. I just wanted to add that I also had another kind of "race" with Mr. T at a later date. Unfortunately, in that race Tortoise: You didn't catch up, did you? The gap kept getting longer and longer, so you couldn't.

Achilles: That's true. I believe THAT race has been written up, too, by Lewis Carroll.

Now, Mr. Sloth, I'll take up your offer of trying out the piano. But I'm so bad at the piano. I'm not sure I dare. Sloth: You should try.

(Achilles sits down and starts playing a simple tune.)

Achilles: Oh-it sounds very strange. That's not how it's supposed to sound at all!

Something is very wrong.

Tortoise: You can't play the piano, Achilles. You shouldn't try.

Achilles: It's like a piano in a mirror. The high notes are on the left, and the low notes are on the right. Every melody comes out inverted, as if upside down. Who would have ever thought up something so cockeyed as that?

Tortoise: That's so characteristic of sloths. They hang from

Achilles: Yes, I know-from tree branches-upside down, of course. That sloth-piano would he appropriate for playing inverted melodies such

FIGURE 133. "
Sloth Canon",from the Musical Offering, by J. S. Bach. [Music printed
by Donald Byrd's program "SMUT'
.

as occur in some canons and fugues. But to learn to play a piano while hanging from a tree must he very difficult. You must have to devote a great deal of energy to it.

Sloth: That's not so characteristic of sloths.

Achilles: No, I gather sloths like to take life very easy. They do everything about half as fast as normal. And upside down, to boot. What a peculiar way to go through life!

Speaking of things that are both upside- and slowed-down, there's a "Canon per augmentationem, contrario motu" in the
Musical Offering
. In my edition, the letters

`S', À', `T' are in front of the three staves. I don't know why. Anyway, I think Bach carried it off very skillfully. What's your opinion, Mr. T?

Tortoise: He outdid himself. As for those letters "SAT", you could guess what they stand for.

Achilles: "Soprano", "Alto", and "Tenor", I suppose. Three-part pieces are often written for that combination of voices. Wouldn't you agree, Mr. Sloth?

Sloth: They stand for

Achilles: Oh, just a moment, Mr. Sloth. Mr. Tortoise-why are you putting on your coat?

You're not leaving, are you? We were just going to fix a snack to eat. You look very tired. How do you feel?

Tortoise: Out of gas. So long! (
Trudges wearily out the door
.) Achilles: The poor fellow-he certainly looked exhausted. He was jogging all morning.

He's in training for another race with me. Sloth: He did himself in.

Achilles: Yes, but in vain. Maybe he could beat a Sloth ... but me? Never! Now-weren't you about to tell me what those letters "SAT" stand for? Sloth: As for those letters

"SAT", you could never guess what they stand for.

Achilles: Well, if they don't stand for what I thought, then my curiosity is piqued.

Perhaps I'll think a little more about it. Say, how do you cook French fries? Sloth: In oil.

Achilles: Oh, yes-I remember. I'll cut up this potato into strips an inch or two in length.

Sloth: So short?

Achilles: All right, already, I'll cut four-inch strips. Oh, boy, are these going to be good French fries! Too bad Mr. T won't be here to share them.

Strange Loops,

Or Tangled Hierarchies

Can Machines Possess Originality?

IN THE CHAPTER before last, I described Arthur Samuel's very successful checkers program-the one which can beat its designer. In light of that, it is interesting to hear how Samuel himself feels about the issue of computers and originality. The following extracts are taken from a rebuttal by Samuel, written in 1960, to an article by Norbert Wiener.

It is my conviction that machines cannot possess originality in the sense implied by Wiener in his thesis that "machines can and do transcend some of the limitations of their designers, and that in doing so they may be both effective and dangerous." .. .