Season 2, Episode 5: The Likelihood and Risks of Superintelligent Machines

Transcript for Season 2, Episode 5: The Likelihood and Risks of Superintelligent Machines

Kurt Andersen: Welcome to The World as You’ll Know It. I'm your host, Kurt Andersen. We’re discussing the future with an A-team of experts, this season the shape of things to come specifically as a result of technology.

My guest on this episode is Stuart Russell… who, I discovered, started programming computers at age 11 in 1973,  became a leading computer scientist, co-wrote what became the textbook on artificial intelligence, and went on to become Chair of Computer Science at UC Berkeley, where he still teaches.

In the last decade, as AI’s power and amazing feats accelerated, he has begun ringing alarm bells about its dangerous and potentially catastrophic consequences. Most recently in his brilliant book, Human Compatible: AI and the Problems of Control.

Stuart thinks that to make superintelligent machines serve us, we have to start programming a fundamental uncertainty into them, teach them that they don’t know everything –– including what we humans actually want them to do.

Stuart Russell, welcome. It’s a great pleasure to have you here and I’m happy to meet you.

Stuart Russell: Good to meet you too, Kurt. How are you?

Kurt Andersen: I’m well, thanks. So the overarching theme of this series, as it's emerged, has been how we can design and steer technologies to serve the best interests of as many people as possible. But before we discuss how AI might at any moment reach warp speed and become its full, fully empowered self, and either push us toward utopia or catastrophe… I want to talk briefly, with you, about what AI has been, and is right now. In Human Compatible, you talk about the various AI bubbles… And I remember the first one because I was in college taking an introductory computer course in the 1970s, and it had just popped. And then, I read, it reinflated, and learned from reading your book that just as you became a computer professor and AI expert, that second bubble popped around 1990?

Stuart Russell: Yeah, that's right. 

Kurt Andersen: So I guess my first question is here we are with all of this excitement about AI and all of these extraordinary things it has started doing and the more amazingly extraordinary things it’s apparently just about to do. So this time AI is not a bubble? This is the real thing? 

Stuart Russell: Well, that remains to be seen. In all the previous incarnations, it wasn't that it was a complete fraud, it wasn't that it couldn't do anything. It’s just that the capabilities that were developed were limited and usually people didn't understand those limitations or they chose not to see them. Expectations raced ahead of reality, and then when those expectations were dashed, people stampeded for the exits, right? No, no one wants to be the last person investing in a technology that everybody else has run away from. So just as quickly as things go up, they go down. I remember going to a dinner in 1994 with a whole bunch of Wall Street people. And we went around the table saying what we did. And I said, “Oh, I do AI,”  and then someone immediately interrupted and said, “I thought AI failed in the 80s.” As if it was a particular you know, like a moon mission or a particular piece of technology that didn't work or a toaster that broke. It's kind of like saying, ‘I thought physics failed.’ 

Kurt Andersen: Yeah.

Stuart Russell: It doesn't make sense. It's just sort of a category error. But what's happening now is it has been on the back burner since the 90s when we started developing fairly large neural nets that could be trained from image data. And starting around 2010 people saw enough promise in that to really invest in engineering, developing hardware that's incredibly powerful and then trying it on lots of things and seeing that there's some areas like speech recognition, object recognition and images where it seems to confer a huge benefit. I think some of that is pretty robust. I think the gains in speech recognition are completely real. The gains in image recognition seem to be a bit problematic.

Kurt Andersen: Really? I thought that was one of the, the challenges that was pretty much solved.

Stuart Russell: Well, um we seem to get very good performance on the benchmarks, the ImageNet benchmark, for example, where nowadays it's routine for machine learning algorithms to reach a 98 percent accuracy on recognizing often some quite difficult categories. So there are lots of different kinds of dogs in the dataset and you have to recognize the difference between a Norfolk terrier and a Norwich terrier and so on, so forth. And humans, even if they spend several weeks training themselves on all these categories can still only reach about ninety five. So you would think, wow, that's a huge success. And then recently, some people at MIT started looking inside to see what is the algorithm actually doing? How is it recognizing these objects? And in many cases they found that the recognition performance worked just as well if you just had the border of the image. So a few pixels around the edge and no pixels of the object itself at all. So in other words, the system had just recognized some regularity in the training sets, which associated some background color, or background texture with the object, and had picked up on that. And that's from the algorithm’s point of view, perfectly reasonable, right? And so I think in many cases we're over interpreting the success of our systems. And you could have the same problem occurring with traffic lights and stop signs if the vision algorithm is really keying on something else about the background of the sky or whatever.

Kurt Andersen: Right.

Stuart Russell: You could get these catastrophic failures. And maybe that's part of what happens with some of the self driving accidents where the vision system seems to just completely fail to see an obstacle. 

Kurt Andersen: So I, one fact I learned from, of the hundreds I learned from Human Compatible, was that the speed of computers since the early days and since not long before AI was conceived has increased a thousand trillion times and memory almost as much. You don't say this, but I take it from your book that really it is that incredible advance in the hardware capacity that has gotten us to the, if you will, AI tipping point where, okay, the Norwich terror thing isn't as great as we thought, but it is doing lots of amazing things. Is that right? It’s more the hardware than some conceptual breakthroughs we've had? 

Stuart Russell: I would say it's a bit of both. I think in the media, you'll often see the explanation that the only thing that's happened is we have big data sets and big machines. But crudely speaking, the faster the machine, the faster you get the wrong answer, right. So it isn't a solution in itself. And trying to extrapolate machine speeds and say, oh, look, you know, at this point in time, 2029, our machines will have more power than the whole human race. I think that's nonsense. The reason we're able to use the computational power that has come into existence is that we developed these new algorithms -- the convolutional neural networks, for example, that are able to generalize reasonably well from image data. So it's much more sort of a chicken and egg thing. If you see an algorithm that looks like it might deliver some value to you, if only it would run fast enough, then you invest in the hardware. And as soon as we saw that, you know, in 2010, 11, 12, that these big convolutional neural networks could deliver huge performance gains, even though they were, you know, taking weeks and weeks and weeks and weeks to train, then we developed hardware to accelerate them. 

Kurt Andersen: Right. You say, you said an interesting thing and I've not heard it put this way, that the technologies like Siri and Alexa, which are familiar at this point to everyone, are Trojan horses for AI. So they are not AI, but they're sort of simulations sort of softening the ground for us to be comfortable with it? 

Stuart Russell: To some extent, yeah. It's it's a sort of a placeholder, it creates this niche, into which you can put more and more AI and as you put more AI in you are delivering more value, or at least people think they are getting more value. You know, that creates this virtuous cycle from the point of view of AI research, that there is immediate returns to improvements in performance and so you get these big ramp ups in investment. That's another part of the story besides the big data and the big machines, is the ecosystem, right the, the place where the AI can do its thing.

Kurt Andersen: One thing. I mean, we've spent really most of a couple of episodes in this series talking about the problems of social media, which certainly to the listeners of this series are well known and to people I think are pretty well known, and and how in some way they are at least a preview of the problems of AI that I think make the problems when they are larger and even more existential, understandable to people. I'm just interested that we are at a point where our cars drive too fast or our guns shoot too well, and we need to figure out a way to to to put fetters on them. 

Stuart Russell: So it's interesting you should mention guns, actually. If you think about it, exploding bullets would be a very effective way to kill people. But we banned them in 1858. Right, the countries got together in St. Petersburg and declared that you could not have exploding ordnance, I think, below 400 grams. So basically, ever since then, bullets don't explode when they enter your body. They do all sorts of other things, but they don't blow up. So we are very familiar with the idea that too much technological capability can end up being harmful. I think particularly when we don't know how to use it properly. So so this is the same thing with social media, right? It's a failure of how we set up the learning algorithm, so the learning algorithms in social media, right. Their job is to figure out what to send you, right, what to put in your news feed.  If you're on YouTube, what video to load up next for you to watch. And people have designed those algorithms to maximize click-through or engagement or whatever the, right, the probability that you will stay on the site, that you will watch the next video that they send you. 

Kurt Andersen: And more importantly, the next ad that they'll put on between the videos. 

Stuart Russell: And the ads as well. Right. And again, that’s the wrong objective because, you know, an algorithm that maximizes that will will do it by essentially brainwashing you. It'll learn how to send you content that changes you into a much more predictable automaton consumer of any kind of content, whatever content it can send you that you will consume, it will turn you into a consumer of that content. That seems to be that the most predictable kinds of consumers are the ones who are most extreme and blinkered and narrow in their beliefs and tastes. That’s one explanation for what's been happening to our society and in the last few years. I'm sure that there are many other things going on. But I think this is one of the consequences of poorly designed objectives of machine learning algorithms.

Kurt Andersen: Beyond social media. And let's … getting back to guns and exploding ordnance. One of your big concerns has become autonomous weapons -- armed drones and robots without some Air Force captain operating it. And that the first one was, was used apparently in Libya last year. We are in the early days of this, but you are, you are horrified at what could happen at any moment with this stuff, right? 

Stuart Russell: Yeah. It's not a completely straightforward story. And the debate has evolved over time. So a lot of the early debate was of the form: AI isn't good enough to be trusted with killing people -- that they would not be good at recognizing combatant versus noncombatant. And they would you know, they might kill a little child who's carrying a toy rifle or, you know, they might hear a firecracker go off and think it's a gun and shoot back and so on and so forth. But I think now people realize that the real risk is much more that you're creating a weapon of mass destruction, right? A technology that can be used to kill millions of people. But one that's much more difficult to control than nuclear weapons because nuclear weapons require big military industrial complex. They're very expensive, their delivery mechanisms are heavy and elaborate and so on. The small autonomous weapon, you know, a lethal weapon that only needs to be two or three inches in diameter. Right. A little tiny quadcopter with some kind of explosive charge is enough to kill a person. And you can put a million of those in a truck. And I hear the U.S. government say, ‘oh, well, don't worry, we'll just make sure they never fall into the wrong hands.’ But, you know, give me a break. They already have. The Turkish government developed, uh, you know, an autonomous lethal drone called the Kargu a few years ago, and they already shipped it to non-state actors in the Libyan conflict.

Kurt Andersen: When I learned of that, the specifics of that, from your book and read more about it and went to the company website, where they have these videos advertising the swarming capabilities, just as you're saying, not with millions, but dozens at a time, which is terrifying. 

Stuart Russell: Yep. You know, and they advertise autonomous hit face recognition, human tracking, all the things that that we've been warning about. And, you know, some of the biggest drone swarms now, they have mostly used for aerial advertising, but I think we're up to five or six thousand drones being controlled simultaneously. So it's just a matter of time. I do not really understand why this concept is so difficult for the major powers to grasp. They keep talking about, you know, the only issue being some kind of Skynet, right, where the weapons wake up one morning and decide that they're going to take over the world and they they hate humans and all this kind of stuff. Computer programs don't wake up and decide they hate the human race and want to take over the world. We'll see, right, we may have to work very hard to convince people or we may have to wait for the first large-scale massacre to take place and then perhaps there will be some awakening. 

Kurt Andersen: Well, as happened with gas after the First World War. As happened with nuclear weapons, obviously after the Second World War, we'll see. But we are heading for and this is really what I want to get to, and what your book is very interestingly about is, is when AI becomes General AI, this all purpose super Siri, who knows everything there is to know, and you just say, “Here's the problem I want to solve. You propose a solution --  good -- execute it,”  and it does it. That's where we're going, right?

Stuart Russell: That's the goal. Because that's the form of AI that would be just dramatically more useful than anything we have right now. I think in the book I use the example of travel, right? If you think about how difficult it was to travel across the world 200 years ago. It was a multi-billion dollar prospect, would would take 10 years and thousands of people to put together an expedition, and you'd probably die before you got there, right? But I can now just take out my phone and go tap, tap, tap. And I'm on the other side of the world tomorrow, and it's practically free. That enormous progress is going to become possible for pretty much anything we know how to do, even if currently it's very expensive and involves lots of people and lots of time and, um, you know, complex planning and organization and so on. You shouldn't think of it as there's going to be a humanoid robot with a very big brain inside. You should think of it as there's going to be this global intelligence utility, a little bit like the way we have electricity as a utility.

Kurt Andersen: Right.

Stuart Russell: Or the Internet as a utility.

Kurt Andersen: Right.

Stuart Russell: And maybe you would just pay some relatively small rent for using the physical appendages for a certain period of time to do whatever you want. 

Kurt Andersen: And you say that at this point it's not hardware that's holding AI back from that, whatever you want to call it, not the singularity, but the moment it becomes a kind of limitlessly superintelligent. You say it is not the hardware, it’s the software that we have and will have sufficient computer capacity to do that.

Stuart Russell: Yeah, in fact, I think we already do. You know, as I was saying, the capabilities we have now are gargantuan. The problem is we don't know how to program them properly. 

Kurt Andersen: And you're cautious about when, when we will get there. You know, I see, you know, people like Mark Zuckerberg and the head of the DeepMind saying, oh, we're going to have general cognition and human level AI this decade, you know, not even at the end of this decade. And having been through these bubble explosions or studying them anyway before, you're more cautious. But you do, you suggest that it could happen like that with this intelligence explosion of the machines, or it might take till the end of the century? Is that right? 

Stuart Russell: We can't say exactly how it's going to happen. If we said, we need, you know, half a dozen major conceptual breakthroughs, and in the book I list four or five and I'm leaving one or two that I haven't thought of yet. But, you know, maybe it'll be 10. But I don't think it's 100 or a thousand major conceptual breakthroughs that have to happen. And I give the example of what happened in, with atomic energy. From Einstein's special relativity in 1905 onwards, we knew that there was a massive quantity of energy stored in atoms and it could be released if we could convert, uh, one type of atom into another. You know, and some people predicted that we could have nuclear weapons that would find new ways to release this energy and that one pound of this nuclear material would have the explosive force of 150 tons of dynamite. And yet the mainstream physics establishment always said, no, this is completely impossible. There was a speech that Rutherford gave. He was sort of the leader of nuclear physics. And he was asked by a journalist, do you think there's any prospect even in 25 or 30 years time that we could find a way to release the energy of the atom? And he basically said, no, you know, anyone who thinks we can do it is talking moonshine. And then Leo Szilard read a report of this in The Times the next morning, and he went out for a walk and invented the nuclear chain reaction. So the key contribution that he came up with was realizing that if you used a neutron to hit the nucleus and break it apart and produce more neutrons, it's quite likely you'll hit a nucleus. Then you get a chain reaction. So he basically thought of all this crossing the road. So that's what I mean by a conceptual breakthrough. Right. Someone just has the core idea that unlocks the next big phase in a technology. And I think you can look at the history of AI and say, yeah, there's, you know, maybe a dozen of those or 20 of those that have already happened. Maybe it's another five, maybe it's another 10, but that could give us AI that, it might not exceed human capabilities along every avenue, but it doesn't have to, right? Because machines already have enormous advantages in terms of data and bandwidth and just raw computational speed. What matters is: Is the behavior of the machine something that our intelligence is sufficient to control, or not? An interesting thing I found out in the course of writing the book is that chimpanzees actually have much better short term memories than humans, even for something like the digits of a telephone number, which you would think they're not even evolved to, to do. But, they can remember 20 digit numbers without breaking a sweat. And so in some ways they have superior minds to us. But you know, that superiority isn't nearly enough to to allow them to control us, or to prevent us from controlling them. And that's what really matters, right? We can make them go extinct and they can't make us go extinct? You know, when you come down to it. 

Kurt Andersen: And your book is really about this existential threat, that if we keep designing AI as we've been doing -- what you call the “Standard Model” -- we’ll end up in a place where we've lost control. Maybe catastrophically. Because the way we currently design AI with these fixed objectives programmed in is going to backfire on us eventually, with horrible unintended consequences. 

Stuart Russell: Right, we create machines that are designed to optimize some objective and then we specify the objective, and off it goes. That sounds kind of reasonable because it's sort of what you know, it's what we do with each other, right? When we get into a taxi and ask the driver to take us to the airport, being at the airport is not the complete definition of the objective. Because if it was, then there would be no problem with mowing down pedestrians on the way just to get there a bit faster. Uh, you know, if you want to be at the airport quickly, then you drive at 180 miles an hour and so on. So when we give requests to each other, we are not stating complete objectives. We're giving some indication about our preferences to be interpreted against a whole background of shared understanding of other things that we care about. And that's why it works. So this idea that the way you get an AI system to do what you want is tell it what you want doesn't work the same way, because the AI system doesn't already share this whole background of understanding about all the other things you care about besides being at the airport. 

Kurt Andersen: And it seems to me that what you're saying is that it can never work if we have to tell it every goddamn thing. It has to be able to not intuit, but figure out all of those other things. And you're proposing to, to design it henceforth to do that with all of the long-term versions of our preferences -- that's the challenge that you're proposing to solve. 

Stuart Russell: So actually, it's in a way, it's more even more difficult than that. So if it's the case that we can't specify our entire preference structure about the future completely and correctly upfront, then it doesn't make sense to design machines that require that to be true in order for them to work correctly. Right. Because if they’re pursuing the wrong objective or if we forgot something, then they are acting as if we didn't care about it.

Kurt Andersen: Because if you don't tell them about preferences, they just, it's not a thing for them. Right. 

Stuart Russell: Right. And so instead, what you want, you know, from the machine's point of view, you want the machine to think, “OK, I know this much about what the person wants, but there's other stuff I don't know,” right? So the machine has to know that it doesn't know the full extent of our preference structure. And it's that uncertainty that we forgot to put in when we started doing uncertain reasoning and uncertain planning. We forgot uncertainty about the objective 

Kurt Andersen: And, and, and yes, that the machine must somehow embody that uncertainty. And you say we have to steer AI, I’m quoting you, in a radically new direction, abandon the assumption the machine should have one definite fixed objective tear out and replace part of the foundation of AI, which sounds like a large challenge. You propose three big rules that, of course, reminded me of Isaac Asimov's three laws of robotics from...

Stuart Russell: Complete coincidence.

Kurt Andersen: I’ll bet. 80 years ago. So although yours are more interesting because his are just like: Do one and then these other two are subsidiary, whereas yours are kind of distinct rules. Tell our listeners what your three rules are. 

Stuart Russell: OK, so the three rules embody this principle that the machine is supposed to be helping us. So the first principle is the machine's only objective is the satisfaction of human preferences. And preferences here means, you know, everything you care about for the whole future. Right? So future A versus future B, which one do you like best? The second...

Kurt Andersen: Right, not this donut you want right now. 

Stuart Russell: So not just what kind of pizza you want or ice cream flavor, but everything. The second principle is that the machine knows that it doesn't know what your preferences are. And that's crucial. The third principle is essentially, you know, a statement of what do preferences mean from the point of view of the machine's ability to observe. And it says that the evidence that's available about human preferences comes from human behavior, the the choices that we make.

Stuart Russell: Now the second principle, the one that says the machines are uncertain, I think this is really the key. This is what changes the direction of the field because all the technology we developed assumed perfect knowledge of the objective, right, and requires you to plug in the objective up front and then run the algorithm and get the behavior. If you allow for uncertainty about the objective, you get all kinds of new behaviors that you just couldn't exhibit before. For example, the idea of asking permission before doing something wouldn't make any sense if the machine believes it has the correct objective, it would just  carry out whatever plan achieves the objective.

Stuart Russell: We have to have machines that behave this way because it's rational for them to do so, because this is how to solve the problem that we've given it. So if we give it this problem, formulate it this way where it knows that it doesn't know the objective, but it's supposed to satisfy human objectives, it becomes rational for it to ask questions like, Well, what kind of pizza should I get you? Do you mind if I turn the oceans into sulfuric acid while I'm fixing this climate change problem?” 

Kurt Andersen: Yeah... or make you all not have babies and wipe out humanity. 

Stuart Russell: Right. So it's asking those questions because it knows something about our preferences. It knows that we want to, you know, prevent climate change, for example, but the solution that it's thinking about is one that affects the pH of the oceans or, you know, the existence of a future human generation, and it doesn't know our preferences about that. Because it wants to avoid violating our preferences, it's rational for it to ask for further information, for permission or elaboration of its understanding of our preferences -- and in the extreme case it should allow us to actually switch it off, which is sort of the core of the control problem, right? If we can't switch it off, then sort of game over. 

Kurt Andersen: Well which is sort of the Sky.. if not the Skynet consciousness moment, even if it's not evil in a, in the conventional human way, you're getting to a point where, wait, you're not letting me switch you off and you're saying that's perfectly plausible as as an thing AI would eventually do to ensure that it is fulfilling its previously encoded missions. 

Stuart Russell: Yeah. In the classical model where it believes it has the correct objective, then being switched off would be a failure, right? Because it wouldn't, you know, not a lot you can do once you've switched off or ...

Kurt Andersen: It’s HAL!  

Stuart Russell: Yeah. I mean, HAL kills the humans on the space mission because it has an objective and it's afraid that the humans are going to interfere with it. And of course there wouldn't be much of a movie if HAL really was super intelligent enough to just kill all the humans, you know, in the snap of its fingers, and that would be the end of the movie. It's kind of interesting. You know, I often get calls or emails from people who are thinking about films involving superintelligence. And they ask questions like, ‘OK, so superintelligence is threatening to take over the world, then the humans outwit it and foil its evil plan, right? So can you explain to us how the humans outwit the superintelligence?’ It’s like, ummm sorry, I can't just sort of by definition. Right. Because if they could it wouldn't really be super intelligent.

Kurt Andersen: So reading your book, and and your goal of creating what you call beneficial machines and provably beneficial AI, it occurred to me that the problem with the way we’ve designed AI so far is that the machines are trying to make us more machine-like and single minded and predictable and easier to deal with. And you're saying we need to start making machines fundamentally less machine-like, less literal-minded, right? 

Stuart Russell: I think what we're trying to create is something that we don't have any models for. Right. Something that is both more powerful than ourselves and yet completely within our control. You know, models like a parent and the child well, yeah, the parent is more powerful than the child, and at some point, the parent has to, you know, stop tying the shoelaces of the child right and say to the child, OK, time to go to school and you've got to tie your own shoelaces this time because it's you, you know, so we can imagine the system doing that, right, saying, "I know that I could, you know, design the next generation of of interstellar transportation for you and I you know, I know we could cure all human diseases and so on, but it's not good for the human race in the long run that AI does everything, and so it's time to learn, you know, time to tie your own shoelaces. "But do we want that model where we are the children and AI is the parent? You know, that's not a good model. You know, animals in the zoo, that's not a good model. You know, we just don't have a good model for this and we have to kind of invent it as we go along. And I’ve been trying to find, you know, in science fiction, a visualization of a future where humans coexist with much more capable superintelligent machines. It's been notoriously difficult for writers, you know, going back hundreds of years to imagine Utopia in a way that actually makes you want to live there, um, and maybe this is just a failure of imagination and things will turn out fine. And maybe the machines will figure out how to arrange things so that we're not just living a life of idle luxury. And I think this is a real question that we're going to have to struggle with. You know, as one of the characters in Humans, which is a TV series that illustrates the problems, one of the characters says, you know, what's the point of spending seven years training to be a doctor when the machine can do it in seven seconds and be better than me? 

Kurt Andersen: Unless we can somehow retain and cultivate the ability to feel and have extreme unpredictable emotions, unlike the machines, and that becomes our unique human quality. You actually have this amazing, to me stunning, example of a self-driving car figuring out how to behave properly at a four-way stop sign without any human pre-programming: how to convey its intention to  the nearby human drivers!

Stuart Russell: The algorithm is trying to solve this problem, where, in fact, here there are several people, several cars at the stop sign and none of them could quite figure out what the other ones are planning to do. And this is notoriously difficult for self-driving cars to figure out. So they end up just stopping and people even drive around them, uh, to just relieve the bottleneck. But one of the solutions to to the problem that the machine, the algorithms came up with is to communicate to the other cars that they should go first by just backing up a few inches. And they they figured out that by doing that, they were showing to the other car that their preference was for the other car to go first. And, you know, I thought that was a very neat example 

Kurt Andersen: So just one day, one car did that and everybody went, “Holy cow.” Really? 

Stuart Russell: Yeah, I think it was in simulation initially because you want to do these things in simulation. But yeah, it finds these solutions.

Kurt Andersen: It's kind of adorable. Um, in terms of getting to where we need to go with this radical revision of how we think about AI and the three rules, you think that, for instance, international regulation is feasible as we've done with nuclear fission, as we're doing as we're doing with genetic technology. And is that because if it goes wrong, it goes wrong for everybody? 

Stuart Russell: Yeah, that's the basic idea, right? And we see this a lot with industries that face risks. You know, the different entities will collaborate to eliminate the risk, because the risk is not just a risk to the person who dies, but to the whole industry that's responsible for the technology. So with electric power, people were very afraid of electrocution, of fires. And so these collaborative institutes were set up to develop safety standards and then make sure that all manufacturers of electrical equipment would comply with safety standards because, you know, people being electrocuted by their hair dryers would be really bad for everyone in the industry. So there's an economic incentive for this type of cooperation, even among competing entities. 

Kurt Andersen: There are many glimmers of hope in your book. You say that governments, in particular the EU, are are setting up bodies to look at this and figure this out and create bodies of law of how, what should or shouldn't be allowed, for instance. And there's a California law that you mentioned that has made it illegal for essentially digital technology to pretend it's human, which sounds good to me. But, I am interested in your field and how much resistance there is to dealing with this, grappling with this, admitting it's a serious set of potentially existential problems. You talk about this big Stanford University report from 2016 that just ridiculously dismisses, that superhuman robots: Not possible! You know, no cause for concern, no threat to human humankind without any evidence or argument. Why is there so much kind of reflexive pooh poohing of even taking this stuff seriously? 

Stuart Russell: Uh, well, I don't want to get into psychoanalysis, but I think there is a kind of a defensive reaction. Um, If someone comes to you and says, you know, what you're devoting your professional life to, uh, is potentially incredibly harmful. Um, you know, so making a podcast is really a threat to the human race. Right. You’d come up with all kinds of reasons why you didn't have to pay attention. 

Kurt Andersen: No, I'd stop doing it. But anyway, 

Stuart Russell: Of course, but you know, a symptom of this, is that the arguments of paying no attention, are patently refutable. You know, in the book, I give some examples and I won't attribute them to anybody because they're just embarrassing. I mean, you know, for example, the claim that because calculators are superhuman at arithmetic and haven't destroyed the world, there's nothing to worry about with super-intelligent computers. Right? Even a five-year-old can figure out that's not a very convincing argument. You know, another one was, you know, yes, it's logically possible that a black hole could materialize in near-Earth orbit and destroy the world, but we don't worry about that. But that's not what's going on. What's going on is a vast fraction of the world's biggest brains are trying to make a black hole materialize in near-Earth orbit. And wouldn't you ask them, like, why are you doing that and are you sure it's a good idea, right? There's a lot of denial going on that that secretly people understand, at least intellectually understand that there is a question to answer, which is: What happens if we succeed? And have you figured out how to control machines that are more capable than human beings? And it's a scary question, so then people kind of retreat from it. You know, and this has been going on for decades, that philosophers come up with all these reasons why AI is impossible and then those reasons get knocked down one after another. And all these predictions, although it's possible for a computer ever to reach, you know, the level of a human chess master, let alone a grandmaster. Well, you know, a few years later, it happened. Oh, well, it's impossible they could ever reach the level of world champ. Well, a few years later, that happened too. Oh, it's well, OK, chess is easy, but Go is impossible. Well, that happened too. Right. And now all of a sudden, you've got AI researchers saying that they think AI's impossible. And that's, again, like that's got to be a symptom of denial. 

Kurt Andersen: Yeah. 

Stuart Russell: You can't really believe. So I think this is a phase that we're going through and I hope people will understand this is not, as some people describe it, as an attack on AI. You're not attacking physics by saying, “Wow, you physicists are so clever, you might figure out how to release the energy of the atom.” Right, that’s not an attack on physics, it's actually a compliment to physics and it's a compliment to AI and its potential to say, “You could eventually exceed human capabilities. But don't you think you ought to figure out what happens next?”

Kurt Andersen: In other words, everybody in AI, right, needs to be convinced -- or forced -- to go beyond "Oh, we're free to design AI any way we want"  and beyond  "sorry, free market, we can do whatever we want with AI."  That people, a bit longer term, before super-intelligent machines totally transform everything, the whole AI world needs to recommit to an idea of the common good, a public good beyond just profit and immediate interests, right?

Stuart Russell: Yeah, and there's a lot of open questions about what that means. 

Kurt Andersen: Yes, yes.

Stuart Russell: You know moral philosophers have been struggling with this for thousands of years: How do you make decisions when the outcome, the affects of your decisions impinge on many people? How do you trade off the interests of all those people in a way that's satisfactory? And, you know, some people argue that, well if you buy the AI, then you get to say what the objectives are and it serves your interests, and presumably that that means it's completely indifferent to the interests of other people. Right. And that's probably not going to work, because if it's completely indifferent to the interests of all other people, then any action it takes that benefits you, the owner it's going to take as long as it doesn't, as long as there's no blowback on the owner. So it might even, you know, find undetectable ways to to steal money from other people's bank accounts, because as long as it's undetectable, there won't be any blowback for the owner. We do it now with laws, right? We're constantly passing laws to plug holes in the way people misbehave. But with machines, you'll get these weird extreme loophole exploiting kinds of behaviors all the time, the kind of problems we have with corporate tax lawyers. 

Kurt Andersen: I was just going to say. You’re describing... AI is just lawyers times orders of magnitude. 

Stuart Russell: Yeah, but, you know, so the corporate tax lawyer is a perfect example, right? We’ve been trying to write tax laws for 6000 years. And we’ve failed. And the reason we failed is because the corporation or the rich individual doesn't want to pay taxes. And so we're constantly trying to put all these fences and obstacles in the way. And they're always finding ways around because they have very high paid lawyers and they're very intelligent and they, they outwit the legislatures all the time. And you could think of superintelligent A.I as like that. If it's pursuing this fixed objective, right? It's sort of like it doesn't want to pay taxes, it's going to find a way not to pay taxes. So the only solution is to design it in such a way that it wants to pay its taxes. You're not going to have a system that's misaligned with human objectives, and then you try to put in all these prohibitions to mitigate the bad effects of that. It's not going to work. 

Kurt Andersen: So in the book you say that because these days so much money and so much effort is now being lavished on AI, and we’re pretty far along after 70 years at it, that people in your field are really surprised by each new amazing AI milestone. You see them coming. So looking at the very near future, the next five years, let’s say, what are you most hopeful about that AI will probably achieve? And what are you the most disturbed by? 

Stuart Russell: So I think we'll see real progress in understanding language. Not to the point where it could read James Joyce and, and then write a learned essay about what exactly it means. I think that's pretty hard for people. But to the point where, you know, it could read all the newspapers, listen to all the podcasts and radio broadcasts or whatever and sort of answer questions about what's happening or what did happen in the past. The cool thing about computers is that if it can do a little bit of it, it can do a lot of it, right? Most of what computers could do is scalable. So, you know, if it can read a book in, you know, 10 hours of CPU time, well, then we can just scale it up and read every book ever written in the history of the human race in every language, uh, you know, before lunch, if, uh, if we scale it up on a big enough data center. And so that would be an incredible resource of information. So if you think about search engines and how much of a resource that is to people that never existed before, this would be like search engine on steroids, you know, 10 times as valuable. The thing I'm most concerned about in the next five years is really autonomous weapons. Because as things stand right now, the technology is moving ahead, the major powers are blocking any kind of meaningful ban on this technology for reasons that I think are misguided. And it's going to be really difficult to pull back. Because once military start integrating the technology into their postures, it's really hard to undo.  And, uh, you know, you gave the example earlier of poison gas in the First World War. We may need something on that scale before we realize that this is just a really bad idea. 

Kurt Andersen: Stuart Russell, I could go on and on this. I'm so grateful you've given so much of your time. I really enjoy this, thank you so much. 

Stuart Russell: It's been a pleasure, Kurt.

Kurt Andersen: The World as You’ll Know It is brought to you by Aventine, a non-profit research institute creating and sharing work that explores how today’s decisions could affect the future. The views expressed do not necessarily represent those of Aventine, its employees or affiliates.

Danielle Mattoon is the Editorial Director of Aventine. The World As You’ll Know It is produced in partnership with Pineapple Street Studios.

Next time on The World As You’ll Know It, I’ll talk with Mariana Mazzucato, superstar economist and author, professor at University College London––and key consultant to governments around the world.

Mariana Mazzucato: We often call these companies Big Tech companies, and you know as we’ve just said, most of the tech behind Big Tech wasn’t funded by Big Tech it was funded by government.