Unlocking The Brain: How Computers Can Read Our Thoughts

Transcript for Season 3, Episode 1: Unlocking The Brain: How Computers Can Read Our Thoughts

Judith Warner: Welcome to The World As You’ll Know It.  I’m your host, Judith Warner.

This season we’re focusing on the brain. And specifically on some of the staggering advancements in brain science that have completely altered our understanding of how the brain works, what it’s capable of, and how it can be changed.  

These discoveries have opened the door to new ways of treating trauma, depression and pain. And they’ve made possible the kinds of things that just a few years ago might have seemed like science fiction. Like the astonishing ability to communicate directly from your brain to a computer without moving a muscle. 

This spring, you may have read about the case of the German man in his 30s who was  completely paralyzed – unable to move or speak – because of ALS. After losing the ability to communicate through eye movement, he was given brain implants that allowed him to think commands that were then translated by a computer into full sentences that his family could understand.

Dr. John Donoghue: He was volitionally voluntarily, consciously able to modulate his brain. And he would put out things like, “I love my son” or, you know, “adjust my pillow.” I think he had an expletive in there or what, you know, so he was communicating in a very, you know, very, very, very slow rate, but he was communicating, demonstrating that there was a conscious being there able to express his wishes. He just had no way to express them. But now he did. That really brings tears to your eyes. It's, it's unbelievable that he could communicate again.

Judith Warner: Dr. John Donoghue is an award-winning professor of neuroscience and engineering at Brown University. He’s known around the world as one of the pioneers of brain computer interface — or BCI – which is the technology that allowed the man in Germany to communicate with his family.

For more than 40 years, he’s devoted his career to restoring movement to people with paralysis – and we feel very lucky to have him with us today. He’s going to tell us what BCI is, what it can do, and what it may mean for our collective future.

John – thank you so much for being here! We're really looking forward to talking to you. 

Dr. Donoghue: Well, thanks Judith. I'm really pleased to be here.

Judith Warner: Well, I’m particularly excited to speak with you because – in addition to all your decades of expertise in the brain and neurotechnology, you happen to be really good at explaining what you do in a way that someone who really doesn’t know anything about your work can understand. So, just for starters: can you tell us exactly what brain computer interface is?

Dr. Donoghue: So a brain computer interface, which is also sometimes called a brain machine interface is a way to connect from the brain to the outside world directly bypassing, uh, some parts of your nervous system and your muscles. So it's a straight connection through, from the brain to the outside world. And when they say brain computer, really it's connecting to any kind of device. And typically there's always a computer in the loop because you have to translate what is coming out of the brain in brain language, into something that a computer can understand or some machine can understand.

Judith Warner: And then that people can understand, I guess. Can you talk a little bit more about who benefits and how this technology is used?

Dr. Donoghue: Basically the idea is that for people who are paralyzed, paralyzed usually is a case like after a stroke, uh, or an injury, the connection between the brain where the parts, where you sort of think up what you want to do and create the commands to move your body come from, and they get shipped down to your spinal cord and then eventually out to your muscles. 

So that pathway, which is about a million little wires on each side of your brain, uh, can be damaged —  a stroke in your brain, uh, damaged to your spinal cord or even diseases like ALS or amyotrophic lateral sclerosis — where the connections in the spinal cord die, the neurons die. So all of those things result in a disconnection between the brain command center for movement and your body.

Judith Warner: So how do you bypass it? What are the actual mechanics involved?

Dr. Donoghue: So we basically can go to three major roots. One is you can take that signal and go to a computer and whatever a computer can do, you can have it do. The next thing would be to control some kind of what's called a prosthetic device, an artificial substitute for what your body can do. And the one that we worked with is a robotic arm. And then the third way would be to connect the brain back up to the body itself. And that would mean taking the brain signal and doing some fancy manipulation of the signals and connecting it to devices, which are called FES systems: functional electrical stimulation. And basically what that is, it's an electrical stimulator that can connect to your muscles or to the nerves that control your muscles. And that thing can make your muscles move. So now you take a brain signal and run it to this FES system and that'll make your arm move. And we've actually done that. We've had a person who is paralyzed think about moving his arm and through this FES system that was implanted in his arm, he picked up a drink and ate a sandwich and was able to do things. And the idea is to help restore the ability for people who can't move, they can't move their legs or their arms, or they can't speak, which is also produced by muscles. And you can replace those functions with the brain computer interface

Judith Warner: OK I'm sitting here trying to wrap my brain around this. No pun intended. That it's possible to look at a drink on the table in front of me and think about picking it up and bringing it to my mouth, and then those thoughts result in a prosthetic arm doing just that. The hardware that's required to make this happen, does it have to be implanted into your brain?

Dr. Donoghue: No, really you can divide the world up into what's called wearables, uh, and implantables. So the implantable devices are mostly, in fact entirely what I've worked with, the idea is there's a tiny sensor or sometimes a couple of them. It's about the size of a baby aspirin. It has a bunch of tiny hairthin probes on it — it has a hundred of them. The surgeon will open the skull, open a membrane called the dura, which are the thin membrane of the brain, put this device in, close it all up, and then that is a source of reading out the sort of high resolution signals from the brain that provides a lot of information about your motor command, what you want to do. And then the wearable devices, there’s been a recent surge in, uh, various kinds of little caps and bands and headbands and stickers and things you can put on your head.  Uh, if you've seen, what's called an EEG, uh, recording that there's kind of a hat that people wear studded with lots of little baby aspirin size discs, and those things pick up, uh, a signal from the brain. 

Judith Warner: And so what’s the difference in functionality between the wearable devices and the implants? 

Dr. Donoghue: A good analogy is, imagine you want to know what's going on in a football stadium and you're flying over it in the Goodyear blimp and EEG is kind of, you can, you can get the roar of the crowd. You can see, you know — did one side or the other, uh, just score a touchdown. So you get some information.  But if you really want to know, you know, what's the quarterback being told to do on the next play, you'd have to drop a microphone in — a headset — an implanted, uh, device that's in close to the source of the conversation. And that's what the implanted electrodes do for you. 

Judith Warner: That's a great analogy. And EEGs though, aren't particularly new, right? They've been around for quite a while. 

Dr. Donoghue: Oh, since the 1930s. And to be honest, I used to be quite cynical about whether there'd ever be anything ... you know, after almost a hundred years of trying to get something from an EEG, it’s except for the clinical cases I mentioned, there's not a whole lot, but I actually now think that there are a number of cases where some very serious scientists have applied some very clever tricks to get, uh, some idea of, you know, what's going on in the brain for different applications that we see now. Those will probably be a lot better. So clinical applications, I think will, will continue to emerge, uh, from this. And then there are claims of using them for various commercial, you know, like game playing and, you know, uh, meditation and things. And so I think that's a mixed bag of, of what they actually do.

Judith Warner: I'm interested, something you just said about how, you know, you used to be skeptical. These had been in use for a hundred years. Nothing had changed all that much, but then now things really have changed technologically so that there are great scientists who are already able to get more specific and useful, um, data this way. And it seems like with brain research, generally, this is true. Things were about the same for a hundred years. And then boom, just huge, huge, huge advances. What happened?

Dr. Donoghue: Well, first of all, it's science, right? I think that's the nature of science is, you know, there, there is some things where, you know, if you just grind away at it, you know, it'll work. And in science, there's just lots of questions where we don't know when we'll ever figure it out. Like, how does the brain work? Someday, if somebody says, I've, I've figured out the key of how your brain actually does all the cool things it does. I figured it out. We're done. You know, that, that will be probably one of the most transformational events in history, I think. Especially with brain research, I think there's a convergence. So one convergence is that neuroscience has come of age. It's actually got enough data that we know enough about in the brain to start saying, well, this is a disease state, this is a circuit gone wrong. And we know where that circuit is. Can we do something to that circuit? The other major issue has been technology advances to be able to make tiny devices that can be implanted in the brain, to be able to make electronics that can process huge amounts of information. You know, when I first started the idea of sensing one cell at a time in a conscious animal, you know, this was a really big deal. And all of a sudden, you know, between 1999 and 2004, we went from recording one cell to being able to record a hundred in humans. You know, so that was a, that was a huge shift in, in our capabilities.

Judith Warner: I heard you say once that, at some point in the nineties, you were sitting around with a bunch of your students and you all were kind of brainstorming sort of fantasizing about what you could do for people, you know, with the, with this work down the line, when you were talking about it, looking back on that you said “we've done all that”. And that just sort of like, you know, set a chill down my spine. And I, and I love thinking about both the moment and the fact that you're able to say that because how many people really can.

Dr. Donoghue: I mean, there were multiple moments like that, but I remember thinking, you know, these are things where we really could do something. You know, we were doing science up to that point, trying to understand how the brain worked, but we did realize that we had the tool, you know, this electrode array. We had an application where we understood enough about how the brain produced movement, that we could really help people who are paralyzed. You know, we could really do something meaningful.

Judith Warner: That must have been an incredible moment, and before you tell me more about it, could you explain what the "electrode array" is?

Dr. Donoghue: So the array is the interface with the brain. It's the signal detector. So neurons are extremely tiny little tree-like structures. Uh, they're so tiny that you could line up three cells next to a hair and they broadcast an electrical set of impulses, you know, just kind of a series of pops. And, uh, I give an analogy: The neuron puts out, you know, six pops called spikes. Uh, then that might mean that's my code for rightward. And then if I put out two pops, that's my code for left. And it's much more complicated than that, but that's not far off from where you start, you know. Each neuron has a kind of different message. And, but in order to pick up those kinds of electrical impulses, what you have to do is take basically it's, it's like a very tiny needle, you know, the size of a hair, it's all covered in plastic, except for the tip. So the tip is the sensing place and you place it into the brain. And when it nuzzles up close to a neuron, it picks up those pops. The problem with one electrode of course, is you've only got one little sample site and that's where the multi electrode array, and array just means a bunch of electrodes. So there's a hundred little sensors, a little, little prongs coming out. Those go into the brain. Then we learned that the brain tolerates that very well. And sometimes we put in multiple ones now and we can get a few hundred cells. Now that's a very tiny sample of the hundreds of millions that are in the vicinity, but but we still get enough of a sample to, to read out the brain.

Judith Warner:  You've been at this your entire career, for decades now. What have been some of the real eureka moments for you -- moments when you realized that this is real, we’re really gonna be able to transform lives?

Dr. Donoghue: Well, I think that the first one was of course, in a human who had been paralyzed. Matt Nagle was the first person implanted. He was, uh, severely injured with a spinal cord injury. Couldn't move at all from the neck down. We put an array in him. And of course, somebody who'd been paralyzed, there were people who said, well, maybe that part of the brain just shuts down altogether, or maybe it's taken over by some other, uh, function that it doesn't do anything related to the arm. So, uh, when we put the array in, that moment of turning it on and saying, wow, there's all kinds of cells here. They're very active. And actually they seem to be doing pretty much what we'd expect them to be doing if you were intact and didn't have a problem. So that was amazing. And then to work with Matt and have him, you know, controlling a cursor, and then we had this toy robotic arm that he controlled and, you know, little, little, we just tried everything to see what he could do. And it was really, you know, fun and, and, uh, and, and amazing to watch cuz we knew we were then on a path where a lot could be done. 

Judith Warner: How did he learn to use it? And how long did it take?

Dr. Donoghue: So what, what we said to him is: Imagine you’re controlling a mouse on a table, you know, you're moving a mouse around. And when he did that, the brain produced the activity as if he were really doing it. It's our problem to, to learn something. What we had to learn: what’s the  pattern of activity that's going on in Matt's brain. How does that relate to your hand moving in space like left, right forward and back. How do you make sense out of those signals? And then we could translate that into a command. Just like if you speak a foreign language, you know, you can take in, uh, Spanish and output English, you know, you're translating one thing into another. We had to put in an algorithm that said, okay, this cell is telling us that, this cell is telling us that; the two cells together are telling us something else. And we had to put all that together.

Judith Warner: In his case, was it a question of moving a robotic arm or was it a question of being able to move a cursor on a screen?

Dr. Donoghue: Most of the things we did with him involved, uh, moving a cursor on a screen. So he was playing a video game. He was spelling. You know, you could use all the usual, uh, computer stuff, like open your email. He moved his eyes to look at the screen, but the eyes didn't control anything. That was all done through his brain. 

Judith Warner: And it seems like a lot of this really exciting work in the past two decades has been around communication. And I'd love to hear more of that from you. Um, the landscape of what that looks like in helping people who are cut off from speech, let's say, being able to communicate.

Dr. Donoghue: Yeah. I think that, well, there have been two major thrusts recently. One of them is, you know: how well can you decode thinking about moving your hand to handwrite and turning it into typewritten language? And recently the, the BrainGate group, led by Leigh Hochberg at Stanford, they showed that if a person was imagining handwriting, they could decode letters at about 90 characters a minute. So this is amazing — a person who says, I can't use my hands, I can't type, but now I can communicate. I can type letters  just using the output of my brain. There was a decoding breakthrough and an understanding of how the brain characterized information.

Judith Warner: So previously, people were just sort of looking at a keyboard or imagining a keyboard and, and typing out the individual letters with their brains?

Dr. Donoghue: No, no. Well, so what Matthew would do is, uh, there's a keyboard already set up on most, uh, computer software interfaces that’s called point and dwell. So you move the cursor, which he could do with his brain over a letter. And if you dwell on that spot for a quarter of a second, it automatically types it. So I can, if I want to type “A N D,” I just move the cursor to “A,” wait a little bit. It clicks, go to “N,” wait a little bit. It clicks “D.” That gives you, um, maybe five or 10 characters per minute. I may be wrong with exact numbers, but now we're at 90 characters per minute.

Judith Warner: What about, I know that there you were involved in some very recent, uh, research findings regarding a man in Germany who had ALS, who had been completely locked in and who became able to communicate. Can you talk about that a bit?

Dr. Donoghue: Yeah. So that was the second study. So the first one was high speed communication with his handwriting. The second one was a question of when a person with ALS, uh, they will use any means of communication possible — their eyes, uh, sometimes a twitch of a face muscle. Any of those kinds of things can be used for some kind of communication at usually at exceptionally low rates. But eventually you become locked in and there's some debate about what the definitions mean, but locked in means you can't move anything really.  But, but I think, if you can move your eyes, I think you qualify as locked in, but not completely locked in, which is CLIS — completely locked in syndrome. Um, and again, you'll find people debate over the exact terms.

Judith Warner: The man in Germany, was he completely locked in?

Dr. Donoghue: He was completely locked in. Yes.

Judith Warner: So no eye movement even.

Dr. Donoghue: Nothing. There was no means of communication at all. Now, the question is what happens in your brain? Is there a conscious person in there that is living a conscious life, but can't have any output?  Dr. Birbaumer, who, uh, was working on that study was a lead investigator, uh, was very much interested in whether maybe the brain just shuts down, you know — that when you have no ability to act on anything, the brain shuts down, which in a way sounds kind of merciful. Then there was a patient, this patient in, in Germany, and he was completely locked in. We, uh, put this electrode array in his brain. Uh, there were actually two, I would say that it was probably not, um, as robust a signal as we were used to seeing. And he couldn't see well enough, he couldn't move his eyes. So he couldn't really look at an alphabet and move his eyes, you know, to say, I'm looking here or looking there.

So, uh, it was an unusual and, uh, ultimately a very slow pacem using a sound that represented just a couple of neurons firing. That meant, I'm selecting that letter. So it was kind of a yes/no. And, and the first thing you'd give is a series of letters: Is the letter you want in this block of letters? Yes. You know, it's like 20 questions: Now is it in the group of three? Is it this one? So with that kind of spelling, he was able to spell.

Judith Warner: How would he indicate to you though? How was, how was the yes or no coming to you?

Dr. Donoghue: Just thinking. That's … that's all he could do. There was a letter presented, and then if he raised the level of neuron activity so that it crossed a threshold with that sound feedback — if the tone changed sufficiently, it crossed the threshold — that meant “yes.” He was volitionally, voluntarily, consciously able to modulate his brain. And he would put out things like, “I love my son” or, you know, “adjust my pillow.” I think he had an expletive in there or, you know, so he was communicating in a very, you know, very, very, very slow rate, but he was communicating, demonstrating that there was a conscious being there able to express his wishes. He just had no way to express them, but now he did. That really brings tears to your eyes. It's, it's unbelievable that he could communicate again.

Judith Warner:  What did you learn about a person who is in that state? I mean, I think most of us from the outside would assume that that is just living torture, but it sounds like it wasn't necessarily for him.

Dr. Donoghue: No, no. Yeah. He seemed, I mean that if you look at the conversations that are published in the paper, that really kind of like family conversations, you know, was really wonderful to see that he was able to have an outlet to, to talk to his family and communicate with them. And actually the studies that have been done say that people who are at home with their families and have some minimal way of communicating actually have a reasonable quality of life, uh, even though they can't move.

Judith Warner: Really incredible though, and really leads into thinking about the future. And also, I guess, to scale up, to get more and more people to be able to have access to these devices eventually. How many people have ALS in the U.S. And, and I mean, I realize he was in Germany, but would be able to benefit?  Or people with paralysis?

Dr. Donoghue: For commercialization purposes, of course the, you know, commercial entities are interested in how many.  Fortunately or unfortunately, depending how you look at it, ALS is I think on the order of a, a few thousand, a year — 1500 or so, not very many people, but then on the other hand, hemiparetic strokes, which means a stroke in the brain that's on one side that leads to paralysis of the other side of the body,  there are, um, on the order of 700,000 people that happens to in the US. That's a lot of people. So there's a range of, you know, people who are paralyzed from ALS, which is a thankfully small number, but stroke is very big. 

Judith Warner: What do you think is next on the horizon? What's the next big thing do you think that we're gonna see?

Dr. Donoghue: Well, there's a lot of, a lot in that question. So in the realm of sensing the brain, I think, uh, you know, the ability for rapid communication for people who have lost the ability to speak is something that is already — I think we could, you know — we're gonna be able to do a good job of that. Um, restoring movement to people it's going to happen. Eventually. I think that right now the technology still needs a lot of work to be a really useful, affordable commercial device that is reliable for a long period of time.

Judith Warner:  As I understand that this has become, you know, a hot area for investors. And of course, you know, Elon Musk's Neuralink has brought a lot of attention, positive and negative. Do you think that's, where it will come from — Silicon Valley? Is it gonna come from individual billionaires, you know, bringing in the necessary money, or do, do we need something else?

Dr. Donoghue: You know, I think people like Musk demonstrate that you can really go fast if you have the resources. I suspect they run into issues where you need to bring clinicians and neuroscientists into — and they do have some working with them — but into close proximity. But I mean really day-to-day working with each other and really appreciating the complexity of the problem and the, you know, you have to deal with, you know, the brain as a, a clinical structure, you know, where things — you can get infections and you can get tissue reactions and things like that. You know, the engineering, it's not just, uh, producing a small device, but you can't, it can't get hot. You know, if something is hot up against your brain, you can have a seizure. So it's a, it's a very complex set of issues.

Judith Warner: It would be great to talk a little bit about the ethical issues that are coming into play now, as commercialization is happening with some talk of it, at least a lot of attention, a lot of money, you know, what do you see as the big ethical issues?

Dr. Donoghue: Well, in terms of brain computer interfaces, where you read out the brain, people are worried that you're going to, you know, read your innermost thoughts, um, and you know, know that you're going to commit a crime and the government will have access to your brain. And then you'll be, you know, you'll be arrested before you even commit the crime.

 The one that I'm more concerned about is the, the other end of neurotechnology is some kind of stimulation of the brain. You can modify brain circuits with electrical stimulation. You can introduce information into the brain with electrical stimulation. It's pretty crude, but it's been used for example, in Parkinson's disease where people are rigid and they shake, and you can put an electrical stimulating electrode, a small one about the size of a piece of linguini, deep into the brain and a little tablet that's about the size of a tic-tac candy. And miraculously, you can either stop or diminish all of that and people can, can move again. So it really is remarkable. What you're doing is changing a brain circuit. And so basically everything you are is a consequence of brain circuits, which means with electrical stimulation, you can change anybody into anything. There are people working, for example, in a, in a good sense of trying to overcome addiction. So addiction is a brain circuit that's gone awry. If you can bring it back into a non-addicted state, you wouldn't necessarily, you know, be addicted any longer to drugs. That could be a great thing.  But now imagine you're changing somebody. Well, let's say I could change your child's IQ and make them much smarter. Who gets that device, you know, and supposing it costs a hundred thousand dollars. Everybody's not gonna have access to it. 

So I can change your personality. I can change your intellect. I can change your memory capacity. Those things can't be done just yet, but there are things like that that are feasible with electrical stimulation. So there needs to be careful regulation.

Judith Warner: The idea of being able, I'll just say first, the, the reading out part of it, basically where you're able to get the signal coming out and being able to understand what the signal is saying. Are we actually close to being at risk of, let's say, um, I don't know, people being able to read your mind, uh, discern your intentions, or is that still sort of in sci-fi land?

Dr. Donoghue: Oh, no, that's what we do. When we do a brain computer interface, we're reading your intention to move. You want to move because you're not moving, right? Um, and you can, uh, you can tap into that. You can, you can tap into plans of what you want to do. Uh, that's been done in animals a lot, you know. We can, you know, see what you're planning to do in the future. Not, not the elaborate details, like, “I'm planning to take a vacation in the south of France,” that kind of thing. But, but generally, you know, “I'm, I'm planning to move my arm to the right and in a couple of seconds,” uh, those kinds of intentions can be read already. So there isn't necessarily a giant leap from that understanding to doing something more elaborate, except that it takes an implant in the brain to do it, which most people aren't gonna just say, I want that in my head.

Judith Warner:  And in terms of the issue of equitable access that you raised, I mean, living aside all together, the question of do we want to make smarter children? We have this brave new world that is completely unregulated where some of these big risks are kind of on the horizon. We don't know when they'll come. What model do we have for trying to ethically regulate something that does not yet exist?

Dr. Donoghue: Well, we have lots of examples of, uh, regulation in an ethics framework. You know, there's a whole set of standards of how we treat humans in clinical cases because of abuse in the past, there are already journals that deal with ethics in, in neuroscience. And neurotechnology, there's, uh, a group in New York that's getting together, a council to bring these ethical issues up. There’s already in Chile, there's already in their constitution, um, a data rights for, for your thoughts —  for your, for human brain rights. Um, so I think there, there is already activity of saying, you know, in the future, we'll be able to do these things, how we're going to protect people and protect their, their innermost thoughts and their brain, so that that's already ongoing. The one concern is that, you know, if, if it's out there, people are aware of it. People are gonna try to find ways of abusing this kind of technology.

Judith Warner: You know, is that something that is, I don't know, foremost among your worries?

Dr. Donoghue: It doesn't worry me that tomorrow we're going to have this kind of a problem, but I, it's not too far into the future. And if we do have a way to produce devices that are more sophisticated, smaller, faster, and we get them done in the near future, then that day will come sooner.

Judith Warner: Thank you so much. This has been wonderful. I mean, I could continue talking to you for hours about this and dissect every single answer, but, um, thank you. Thank you very much.

Dr. Donoghue: Oh, it was a great pleasure. It was really fun. Thanks Judith.

Judith Warner:  Thank you so much for listening to my conversation with Dr. John Donoghue. Join me next week as I speak with Dr. Rudolph Tanzi about how the aging brain can be its own worst enemy -- and what we can do about it. 

Dr. Tanzi: Right now, when you go to the doctor, there's no checkup from the neck up, right. They get above your neck and they look at the holes in your head, they look in your eyes and your ears and your nose and in your mouth. And it's like, Hey, you got three pounds of jelly back there. That kind of matters. Yeah. We don't look at that. But now we're getting to that point. We're getting to that point now where we can say, we can look at indicators of brain health, like we do for the heart with cholesterol and, and we can develop the equivalent of the stethoscope or a blood pressure cuff.

The World As You’ll Know It is brought to you by Aventine, a non-profit research institute creating and sharing work that explores how today’s decisions could affect the future. The views expressed don’t necessarily reflect those of Aventine, its employees or affiliates. 

For a transcript of the episode and/or more resources related to what you've heard in today's episode, please visit us at Aventine dot org slash podcast.

Danielle Mattoon is the editorial director of Aventine. 

The World As You’ll Know It is produced in partnership with Pineapple Street Studios.

Our Associate Producer is Yinka Rickford-Anguin. 

Our Producers are Sophia Steinert-Evoy and Stephen Key. 

Our Senior Editor is Joel Lovel.

This episode is mixed by Davy Sumner. 

And I’m your host, Judith Warner.

Original music by Hannis Brown.

Legal services for Pineapple Street by Bianca Grimshaw at Granderson Des Rochers. 

Our Executive Producers are Je-Anne Berry, Jenna Weiss-Berman and Max Linsky. 

The next episode will be out in a week. Make sure to listen on the Audacy app, or wherever you get your podcasts.