Bringing the blind back to light? Talk about Neuralink and the retinal brain-computer interface

Interview|Hongjun

Photo: Daisy

After drilling a hole in the skull and implanting a chip, you may have "superpowers" that you didn't have before, would you?

For ordinary people, brain-computer interface has always been a mysterious concept full of science fiction. In recent years, with the advancement of technology, the future of "human brain-machine interaction" is becoming a reality. Among them, the progress of Neuralink, a brain science company founded by Musk and a number of scientists, has attracted much attention.

Since 2019, Neuralink has successively announced its brain-computer interface practices in monkeys and pigs, and each time it has caused a huge sensation. And on January 29 this year, Musk announced that Neuralink had implanted a chip into the human brain for the first time. He also said that in the future, brain-computer interface technology will be applied to help paralyzed patients restore motor function, cure brain diseases such as Parkinson's disease and Alzheimer's disease, and help restore the vision of blind patients.

The picture comes from Musk's official X personal account

The human brain is one of the most advanced and complex organs known in the world, but with the efforts of generations of scientists, many mysteries about the brain are now being unraveled. In this episode, we invited Pumiao Yan, a Ph.D. candidate in electronics at Stanford who is engaged in the design and research of retinal brain-computer interface systems, and her co-mentor Krishna Shenoy personally helped a 69-year-old patient with high paraplegia to perform brain-computer interface surgery.

So, compared with the past brain-computer interface technology, what are the differences and progress of Neuralink, and how far is humanity from the real application of bringing the blind back to light? Next, let's go into the world of brain-computer interface to find out.

1 Neuralink and the evolution of Utah array technology: the ability to interact with more nerve cells and improve the security of packaging

2 There are three forms of brain-computer interfaces: invasive, semi-invasive, and non-invasive

3 Neuralink's equipment upgrade: a smaller, more precise, and suitable surgical robot for batch operation

4 Human Milestones: Look at Device Stability and Safety

5 介入式的Synchron:网状电极比Neuralink更少

6 The distance from the nerve cell varies greatly depending on the electrode technique

7 Most of today's brain-computer interfaces are implemented by controlling motor centers

8 Reading the mind is distant, not a functional area of the brain

【Retinal Brain-Computer Interface】

9 Artificial vision: The retina can survive within 48 hours of being out of the body, and the retina is more accurately understood

10 Retina Study: Not a Single, It's Conducting a Symphony Orchestra

11 Musk's three goals for Neuralink: typing, regaining athleticism, and regaining light

12 Neuralink Vision Group Resignation: Healthcare companies are vastly different from start-ups

13 Packaging technology is the most critical: 3-4 retina startups have gone out of business because of packaging problems

14 Electrical energy is the bottleneck of the long-term use of brain-computer interfaces

15 Neuralink Abuse of Monkeys?

【Stanford Brain-Computer Interface】

16 Krishna Shenoy早年参与脑机接口时，研究方向很难归类

17 Stanford Research on Brain-Computer Interfaces: Retinal Research, Language Center Research, Electrode Materials, Chip Design, Artificial Vision

18 "If We Really Knew the Principles of Neurology, the Book Would Not Be So Thick"

19 From the discovery of the brain system to the treatment of diseases, it takes one disease after another to conquer

20 Key Technology Breakthrough: How to Design a System to Collect Tens of Millions of Data

21 Are Brain-Computer Interfaces Suitable for Entrepreneurship – Biomedical devices are different from traditional entrepreneurship in Silicon Valley

Below are some of the highlights of the interview

01 Advances and differences in Neuralink technology

Silicon Valley 101: Research on brain-computer interfaces and brainwaves has been going on for many years, and your mentor, Krishna Shenoy, personally performed brain-computer interface surgery on a 69-year-old high paraplegic. We understand that the brain-computer interface in the past was mainly based on a technology called the Utah array, so compared with the previous technology, is the principle of chip implantation between Neuralink and the new generation of brain-computer interface companies different from the previous Utah array?

Pumiao Yan: It's true that research on brain-computer interfaces can be traced back to the 50s of the last century, and the earliest technology was dominated by Utah arrays. The Utah array is one of the few technologies that has been approved by the FDA (Food and Drug Administration) for implantation into the human body and allows implantation to be done for a long period of time or even more than 10 years. However, there are several problems with Utah array technology: First, the size of its electrodes is much larger than that of today's technologies, so it can cause many rejection reactions such as nerve scarring. In addition, the form of the Utah array cannot be fully implantable, which means that there will be an interface in the brain after implantation, and a wiring needs to be connected to it every time a large-scale data is read, and then it can be decoded.

Neuralink is currently moving forward in a few latitudes. First of all, they have enabled the implantation of neuromicrofibrils into the human brain on a large scale, and now they have achieved the scale of thousands of electrodes, so that they can obtain more array signals, interact with more nerve cells, and thus achieve more functions. In addition, from an industrial point of view, they can be fully encapsulated and fully implanted into the brain in the form of stable devices, so it also reduces a lot of safety risks, which is something that he has spent a lot of energy on.

"Silicon Valley 101": In other words, the signal of the previous brain-computer interface may not be stable enough, but the signal of the new generation of brain-computer interface technology is more stable and clear, and the security of the technology is better than that of the Utah array, right?

Pumiao Yan: At present, only from the perspective of rejection, the realization of complete encapsulation can solve some risks such as inflammation or infection, and it must reduce some of the safety risks compared to semi-invasive. However, there is no guarantee that the current Brain chip is completely safe, and whether the new technology will bring other problems, this may continue to be observed to see if it has a follow-up long-term experimental report.

Silicon Valley 101: Brain-computer interfaces currently include non-invasive, invasive, semi-invasive and other different modes. Non-invasive is well understood, that is, there is no wound, but by sticking 32-64 sensors to the brain. Invasive methods are also easy to understand, such as Neuralink's digging out a small piece of skull and burying it in a chip. So, what is semi-invasive?

The picture comes from the Internet, and the copyright belongs to the original author

Pumiao Yan: Semi-invasive means that you still need surgery, whether it's a minimally invasive surgery or a major craniotomy, but it doesn't require electrodes to go deep into your cerebral cortex, but it produces a kind of cortical electrode, which doesn't produce as much signal-to-noise ratio as compared to full invasiveness, and the signal quality is relatively weak. At present, there is also a relatively new semi-invasive technology called interventional brain-computer interface, which is to attach a mesh electrode around a large blood vessel to this blood vessel.

Silicon Valley 101: We know that Neuralink recently announced that it has achieved its first human experiment, so what kind of device is implanted in the human body this time? How is it better than the previous device? Because I know that you are doing retinal chips, and you are also doing hardware-related design.

Pumiao Yan: Actually, every time there is a good illustration on the Neuralink website, this time the implanted chip is called The Brainchip, and the whole size is about the same as a coin. The core of the device is its interface chip CMOS chip, which can support large-scale reading chips for signal acquisition, signal processing and transmission. Further up there is a battery that powers the entire system, and then packages them in their entirety. When implanting a brain-computer interface, you will cut a coin-sized hole in the skull of your brain that is close to the device, and then fix the device in it and connect it to its motor.

Image from Neuralink's official website

Compared to the previous equipment, they have made the equipment smaller, more compact and more precise. In addition, because they have spent a lot of effort to develop surgical robots, they are currently able to do large-scale thousands of electrodes in a very short time, possibly corresponding to dozens of microfibers for precise implantation, and can be completed in batches. In other words, with the help of surgical robots, this type of myocardial interface surgery can be performed in the future without relying entirely on an experienced doctor, but by a machine. This is actually something that startups will consider, because they have relatively high mobility, and if they want to operate on hundreds or thousands of patients, they can't just expect a single doctor to do everything all the time.

"Silicon Valley 101": What they want to achieve with this implant is also to let patients type with their minds?

Pumiao Yan: Their implantation this time is more about the stability after implantation, which is to prove that it can be used for human experiments. The first thing you may start with is whether you can read the signal and how stable the signal is, then see if the patient can do simple tasks such as screen typing or mouse control, and then slowly do more. The milestone of this human experiment lies more in its safety and the stability of its equipment.

"Silicon Valley 101": In addition to Neuralink, Synchron uses blood vessels to make brain-computer interfaces, and puts a device into blood vessels. How is this method different from Neuralink's brain interface?

Pumiao Yan: This is actually the kind of interventional device I just talked about, which is also semi-invasive. It is a minimally invasive procedure in which a mesh electrode is attached to a large blood vessel and then from the inside of the large blood vessel to the brain. However, the location of the implantation in this way will be controlled by the position of the blood vessels. It can only be along the blood vessels, but not more precisely to a certain functional area of the cerebral cortex, because there may only be small blood vessels in that place. In addition, the mesh electrode of Synchron currently has a smaller number of electrodes than Neuralink, and it can correspond to relatively few nearby cells.

02 How can brain-computer interfaces restore sight to blind people?

Silicon Valley 101: Your main research direction is the brain-computer interface of the retina, and the goal is to let some blind people see the light through brain-computer interface technology. Can you tell us where we are in research, and how far we are from regaining sight for blind people?

Pumiao Yan: My PhD subject is artificial vision. The principle is actually to implant an electrode similar to a brain-computer interface chip into the retina, which we call visual analysis, that is, the electrical signals generated by the cells that were previously light-sensitive are simulated by the brain-computer interface, and the corresponding electrical stimulation is collected on the retina and then transmitted to the brain.

One of the big reasons why I chose to do retinal research, including now that more startups are also starting to do retinal research, is that the retina can survive within 48 hours of being completely out of the body, and we have a lot of research that can only focus on retinal tissue to experiment, stimulate, and then see how it responds, so we have a more accurate understanding of the retina. Many people may simply think that our retina is similar to a camera, but no, our retina actually has different kinds of nerve cells. For example, some retinal cells only respond to the enhancement of light, some only respond when they are darkened, or only respond to certain actions. So when we want to replicate and simulate the signals of the retina, we need to study more about how to classify these visual cells and then stimulate them accordingly. It's like conducting a symphony orchestra. Now there are several groups in the world that have started to do human experiments.

"Silicon Valley 101": Does the brain-computer interface of the retina implant the chip directly into the eye?

Pumiao Yan: Actually, there are two different directions, and some groups will choose to directly stimulate the cerebral cortex of vision, and the effect may be more to be able to see light spots or light columns. But if you want to be more precise, for example, our group's definition of seeing again is to be able to see your loved one and partner again, so if you want this level of accuracy, with the current technology, you can only rely on this kind of brain-computer interface directly on the retina to achieve this level of accuracy. In fact, many human experiments have been achieved at the level of reading books and seeing letters.

"Silicon Valley 101": In fact, Musk also has several stages of vision for Neuralink. At first, he had the idea of starting this company because he thought that we could not type directly with our minds without the trouble of using mobile phones, so at the beginning, the direction of his brain-computer interface was to type with my mind, use my mind to control the robotic arm, or let the high paraplegic he be able to control his limbs. In addition to the brain-computer interface made on the skull, he seems to want to make a brain-computer interface on the spinal cord to restore human motor ability. But at a press conference at the end of November 2022, he mentioned that Neuralink's new direction is to make brain-computer interfaces for the retina. You just mentioned that there are several different directions in the current research on brain-computer interface in the retina, so what direction does Neuralink belong to?

Pumiao Yan: Neuralink belongs to the genus that hasn't started to do it yet. But some of Neuralink's original employees are now spinning out into a startup called Science Corp, and they're now more focused on making retinal chips. In fact, there is still a certain contradiction between the entrepreneurship itself and the entrepreneurship model. Many startups need to make a result in a short period of time after they get financing, but when it comes to medical experiments and hardware development, the pace is unlikely to be as fast as making an APP, especially the long-term work of phase III experiments. The medical direction is very rigorous, and it is necessary to verify its safety first, and even if it is verified to be safe, it needs to have an observation period of 1-2 years. So when I visited Science Corp before, they were much slower, and everybody's philosophy was different.

图片来自Science Corp官网

Silicon Valley 101: What are the materials of chip devices implanted in the human body, and are there any safety risks when embedded in the human body?

Pumiao Yan: All of these implantable devices, whether it's a cardiac stent or whatever, are more about the safety of the material. For example, the chip implanted by Neuralink this time is also fully encapsulated on the outside, so it only uses to ensure that the encapsulated material, that is, the material that really comes into contact with the human body, is completely safe, and there is no problem with FDA approval. Generally, the encapsulation materials they use have been approved by the FDA before. In this fully implanted human experiment, everyone is actually more concerned about its encapsulation effect. For example, the mobile phones and watches we use now involve a waterproof function, so the CMOS chip in the chip they implanted this time is actually very sensitive to water or humidity. In the past, many problems were caused by this, including the retinal chip we talked about earlier, in fact, there have been several retina-related brain-computer interface startups since 10 years, but in the end, all of them went out of business because of packaging problems. Because once it is not packaged, over time, water molecules will invade, or even more seriously, brain fluid will corrode it, and the chip will not work after infiltration. So at present, the most critical thing is actually packaging materials and packaging technology.

"Silicon Valley 101": The chip implanted in the brain this time is transmitted by Bluetooth, does it need to be charged, and how to charge it?

Pumiao Yan: Yes. According to Neuralink's current announcement, it uses a coin-sized battery, and the charging mode is a bit similar to that of Mega safe, which is to use microwave wireless energy transmission for charging. The battery lasts for about 11 hours before it needs to be recharged again. However, the lack of power does not have much effect, it just causes the brain-computer interface to be unusable, but assuming that the battery is broken, it needs to be taken out. In fact, the problem of electrical energy is also why many people have not tried to become a fully productized startup for so many years. Like most of the retinal chips choose wireless energy transmission, there is no way to put a battery in, because there is no guarantee of how long the battery can be used, repeated charging and discharging is no problem, if you want to change the battery, you must have another operation.

03 Security controversy over brain-computer interfaces

Silicon Valley 101: Speaking of security, Neuralink has actually had some negative news before. For example, some monkeys had strange reactions during monkey experiments, some monkeys would directly scratch their heads to show painful expressions, and it was reported that several monkeys quickly euthanized them. Is there any public material about their experiments that discloses the safety risks involved?

Pumiao Yan: So far, there hasn't been any public news. But even from the perspective of academic research, the first thing I can think of when doing monkey experiments is that the surgery itself will definitely have postoperative recovery pain, and there will be things like epilepsy reactions, for example. This has always been a problem that has not been completely solved by the brain-computer interface, and many people have been worried about whether the electrical stimulation generated by the implantation of this brain-computer interface will cause epilepsy and some rejection reactions. The exact cause of these problems in the Neuralink monkey experiment is still unclear, but many people still take a wait-and-see attitude towards this rush to meet the deadline, which may be a bit contradictory to the very rigorous approach of the academic community.

Image courtesy of Neuralink

Silicon Valley 101: How is the patient you did with your mentor recovering? Is he still using a brain-computer interface?

Pumiao Yan: The patient himself has been a high paraplegia for a long time and has been implanted for more than 10 years. At present, he is more likely to do some corresponding operations such as typing and mouse control under the conditions of an experimental design, but he may not have reached the situation we imagined in life, but as a volunteer of our experiments, he continues to help us do research.

Silicon Valley 101: Will a cutting-edge company like Neuralink have a large-scale and in-depth collaboration with university labs?

Pumiao Yan: There are some startups that professors want to do that are definitely very close to the school. However, many start-up companies come to university professors more out of mutual endorsement in the field, that is, there are experts who understand your technology and think that your technology is feasible. Because if you want to find VC financing, you still need a certain amount of recognition in the industry.

04 Stanford University's research on brain-computer interfaces

Silicon Valley 101: I'm sorry to hear that your mentor, Krishna Shenoy, has passed away from cancer. Krishna Shenoy has been one of the most authoritative experts in the field of brain-computer interfaces over the years, what are his main research directions and what contributions have he made to the field of brain science?

Pumiao Yan: My mentor joined Stanford around 2000, when brain-computer interfaces weren't really a hot topic. Even when he applied for a teaching position in our electronics department, the school was a little unsure whether it was neurology or electronics. Therefore, many of his later work revolved around building an experimental platform for brain-computer interfaces, and devoted himself to studying the control model of brain nerves over the motor center and the corresponding theoretical research and experiments. In recent years, he has begun to turn to the language center, and was the first researcher to start writing and decoding, and he basically led the large-scale experiments on the language center later.

"Silicon Valley 101": So at present, the classification of brain-computer interfaces is quite fine, including motor center, language center, visual center, etc. So what are the main sections of Stanford's current research on brain-computer interfaces?

Pumiao Yan: Because brain-computer interface is a field where there are many interdisciplinary disciplines, there are many sub-categories below. For example, my current co-supervisor, who used to do pure ophthalmology and only focused on retinal research, then turned to brain-computer interface for the retina. In the future, I will still graduate with a PhD in electronic engineering, but many of the collaborators I really do research on a daily basis are supervisors in neuroscience and ophthalmology.

But in fact, each group will have a more specialized direction. There are many groups that are biased towards theoretical research, such as monkey experiments, model studies, and some groups are more inclined to do new centers, such as Krishna, although he left, but the lab is still there, and after the technology began to support large-scale data collection in the past few years, many of them have been doing language centers and larger-scale cerebral cortex research. In addition, if you take apart the brain-computer interface system, there are more categories below. For example, many chemical engineering and materials majors are making pure photoelectrodes, mainly to study what kind of materials can be more minimally invasive, or what kind of materials can be softer, more able to reduce the production of scar tissue, and the signal can be more stable, including how to make the scale larger, or more accurate, etc. As far as I am concerned, our group is engaged in chip design, so my earliest starting point may be more to study systems with large brain-computer interfaces from the perspective of data acquisition and signal processing. In the future, encapsulation and implantation will involve separate research areas.

Silicon Valley 101: So it looks like an operation, but it can actually be across a lot of disciplines.

Pumiao Yan: Yes, the scope of research on brain-computer interfaces is very large, and the building across from the Stanford Department of Computer Science is basically working in this direction to a greater or lesser extent.

05 Cognitive progress of the brain in humans

Silicon Valley 101: One of the questions I'm curious about is how much we understand about the human brain so far.

Pumiao Yan: If you want to make brain-computer interfaces, you must have a basic understanding of neurology, and you must take some neuroscience courses. When you take out any textbook called Principles of Neurology or this type of textbook, it's a very thick book. At the time, Krishna joked that if we really knew the fundamentals of nerves, the book wouldn't be so thick.

In fact, the academic community now has a good understanding of a nerve cell at the most microscopic level and even its entire protein pathway, and it is also better understood about small-scale neural circuits, as well as a lot of understanding of large-scale partitions at the very macroscopic level. However, the middle layer, such as the correspondence between some nerve cells and the motor center, and the mechanism of each target, is still not very well understood. When Musk first wanted to pull financing, he said that what Neuralink wants to do is you be able to download a book directly into your brain. This actually involves the human memory system, or how the human learning function is formed. But this is actually a problem that the academic community is not so sure that it can understand what this process is like at the level of a cell, so this is why many people at that time felt that his vision was very ambitious.

Silicon Valley 101: The human brain has 86 billion nerve cells, but the electrodes that can be collected are only a few hundred or thousand. So what we know about nerve cells may only be a drop in the bucket.

Pumiao Yan: Yes. It's that I can look at one microscopically, or see who lights up in a piece, but the connection hasn't been established yet, let alone proving causality. So that's why I think science and technology and theoretical research are complementary to each other. If we want to go further and have a new understanding of the theory of the brain, we will definitely need technology that can take more data to help understand how the brain functions and how it works internally.

Silicon Valley 101: So how far has it been from now that we're starting to study brain-computer interfaces to uncovering some of the principles of the brain to actually treat diseases, such as Alzheimer's disease or depression?

Pumiao Yan: This may be divided into different types of diseases. For example, for epilepsy, there are a lot of startups that are actually doing quite well now. However, different types of mental illness have different causes and therefore different directions. For example, depression may be caused by a vitamin deficiency or a problem with the dopamine secretion center, and the brain-computer interface cannot solve all problems. However, I am still very optimistic about the application of brain-computer interfaces in this field. For example, in the last century, we still know very, very little about the nervous system, but scientists can make cochlear implant systems for hearing and help many deaf patients restore their hearing, so in fact, this process is also advancing little by little.

Silicon Valley 101: What's the one question you're most curious about the brain?

Pumiao Yan: I'm curious about how the brain works when we're doing some logical operations. The biggest difference between computers and people is that the logic of all digital circuits is very clear, but for example, when answering the question of how much is 25 * 25 equal? Whether the brain is a complete logical calculation, or how much is the memory function involved, these are still unknown questions.

Silicon Valley 101: What do you think are some of the most promising technological singularities in the future that are most likely to trigger the rapid development of brain science research in your research field or in these brain-related research areas that you know?

Pumiao Yan: I can talk a little bit about the field I'm studying right now. For the retina, if you want to do one-to-one stimulation with each nerve cell on the retina, it may take tens of thousands or even tens of millions of cells to achieve a good visual performance of a pixel, which involves a data collection problem, which is also the direction of my current research. Specifically, this requires us to plan the system and develop a collection strategy before we can design the hardware chip. If we can make a breakthrough in this direction, we will be able to truly collect and study thousands of cells at the same time.

But I think when it comes to hardware and biomedicine, it must be a long-term, high-investment thing, and this cycle may be more than 10 to 15 years. For example, we have been working on our current project for six or seven years, and now that the chip is working, we need to start thinking about how to conduct human experiments next. However, the requirements for manpower and resources in this process are actually very large, and it is difficult to rely solely on a scientific research project team of more than a dozen people in the academic community. That's why there's a lot of expectation for companies like Neuralink and Science Corp, and hopefully they'll be able to make a breakthrough when they're backed by enough resources.

Silicon Valley 101: So for startups, they need to solve the problem of how to raise money if the technology can't be commercialized in 10 to 15 years.

Pumiao Yan: Yes, that's a big problem, and that's where Neuralink's contradiction lies. Because from the perspective of the academic community, all invasive devices must at least conform to the basic principles of medicine, that is, they must have this need to perform this kind of harmful surgery. While Neuralink has a lot of big visions, it also has to prove why it's worth it because of the risks associated with the surgery. Therefore, many people thought that there was no market prospect in this field at the beginning, and the main reason was that it was a pure biomedical device direction, and its corresponding economic benefits and the economic effects of commercial products may not be a calculation method at all. So I'm also curious if it's because of Musk's personal appeal that I can invest so many resources into it, and I hope he can do a good job.

Silicon Valley 101: One of the things that I think Neuralink has done really well is that they didn't really think about how to do human experiments on just one person in the first place, but developed surgical robots, and wanted to use machines to make this thing mass-produced. In their experimental plan, they expect to perform 11 device implants in 2024, 27 in 2025, and then more than 22,000 by 2030.

Pumiao Yan: This is actually a direction that the entrepreneurship model will force you to explore. After all, in order to get so much financing, he must be able to solve the problem of scale.

Silicon Valley 101: What did your classmates do after they graduated?

Pumiao Yan: It depends on everyone's interests. There are many students who choose to start a startup on their own or join such a large startup to do project management. Some people who are only interested in theoretical or basic research choose to go to academia. Of course, many people will also choose to go to the industrial world, for example, although I am making this kind of large-scale brain-computer interface chip, but in fact, many signal processing and signal acquisition technologies can be applied to cameras, Lighter systems and other systems, but different sensors.

[Related Supplementary Information]

Utah Array: A high-channel-count microelectrode array invented by Richard Normann of the University of Utah for use in visual prostheses. Over time, it has been widely adopted by the neural engineering community. The array collects neural signals from up to 100 channels per device. It is the only brain-computer interface implant with long-term stability and safety in humans, and has been studied to date in a sample composed of dozens of patients for a total of more than 30,000 days. The Utah array has received commercial clearance from the FDA to monitor electrical activity in the brain for up to 30 days.

CMOS chip: It is a low-power, low-noise integrated circuit technology, commonly found in electronic devices such as digital circuits and microprocessors. CMOS chips have the advantages of low power consumption, low noise, good stability, and can also quickly switch the voltage state. In the field of mobile devices, CMOS is playing an increasingly important role. Mobile phone cameras and sensors, for example, are based on CMOS technology. Since CMOS has very low power consumption, it is easy to implement portable and mobile designs.

Synchron: Founded in 2012, one of the emerging BCI (Brain-Computer Interface) start-ups. Synchron implants BCIs through blood vessels, enabling patients who are physically paralyzed or have very limited mobility to utilize technologies such as brain-operated cursors and smart home devices, a new technology that has been used so far on three patients in the United States and four in Australia. In December 2022, Synchron announced a $75 million funding round that included funding from the investment firms of Microsoft founder Bill Gates and Amazon founder Jeff Bezos.

Science Corp: Founded in 2021, an emerging brain-computer interface startup, most of the founding team members are from Neuralink. His current research interests include retinal brain-computer interface, which received a seed round of funding in August 2022.