Reports from the Heart of the Machine
Machine Heart Editorial Department
For the past three years, DeepMind and EPFL, the Swiss Federal Institute of Technology in Lausanne, have been working on a mysterious project: using reinforcement learning to control overheated plasma in nuclear fusion reactors, and it has now declared a success.
DeepMind research scientist David Pfau lamented after the paper was published: "I have been waiting for this moment for a long time to share, and this is the first demonstration of deep reinforcement learning on a nuclear fusion research device!"
Controllable nuclear fusion, strong artificial intelligence, and brain-computer interfaces are some of the most important directions in the development of human technology, and when they can be realized, scientists will always say that it will be "still decades away" - there are too many challenges and limited methods at hand.
So is the use of artificial intelligence to control nuclear fusion a promising direction? This question may need to be answered by DeepMind, who came up with AlphaGo.
Recently, research by EPFL and DeepMind on the use of deep reinforcement learning to control plasma in tokamak devices has appeared in the journal Nature.
First, let's think about a question: Why use artificial intelligence to control nuclear fusion?
The tokamak is a ring-shaped vessel used to house nuclear fusion reactions, and its interior presents a special state of chaos. Hydrogen atoms are squeezed together at extremely high temperatures, creating a plasma that is hotter, spinning, tumbling than the sun's surface. Finding ways to control and limit plasma will be key to unlocking the potential of nuclear fusion, which is considered a source of clean energy for decades to come.
At this point, the scientific principles seem to make sense, and all that remains is the engineering challenge. Ambrogio Fasoli, director of the Swiss Plasma Centre (SPC), who was involved in the study, said: "We need to be able to heat this device and keep it for a long enough time for us to draw energy from it."
In stars also driven by fusion, gravitational mass alone is enough to pull hydrogen atoms together and overcome their opposite charges. On Earth, scientists instead use powerful magnetic coils to limit the fusion reaction, pushing it to the desired location. These coils must be carefully controlled to prevent the plasma from touching the vessel itself: this damages the vessel walls and slows down the fusion reaction.
But every time researchers want to change the configuration of the plasma and try different shapes to produce more energy or a purer plasma, a lot of engineering and design work is required. Traditional systems are computer-controlled, model-based and simulation-based, but Fasoli says traditional methods are "complex and not necessarily optimized."
Martin Riedmiller, head of DeepMind's control team, said: "Artificial intelligence, especially reinforcement learning, is particularly well suited to solve the complex problem of controlling plasma in tokamak." In his paper, DeepMind details the proposed AI that can autonomously control plasma.
Technical overview
The model architecture proposed by DeepMind is shown in the following figure, and the method has three stages:
Phase 1: The designer specifies the objectives for the experiment, which may be accompanied by control objectives that change over time;
The second stage: the deep RL algorithm interacts with the tokamak simulator to find a near-optimal control strategy to meet the specified objectives;
The third stage: The control strategy represented by the neural network runs directly on the tokamak hardware in real time (zero samples).
Figure 1: Schematic diagram of each component of the controller design architecture.
In the first phase, experimental goals are specified by a set of goals that contain different desired characteristics. The characteristic range includes the basic stabilization of position and plasma currents, as well as complex combinations of multiple time-varying targets. These goals are then combined into a reward function that assigns a scalar mass measure to the state at each time step. The reward function also penalizes the control policy so that it does not reach the terminal state. Crucially, well-designed reward functions will be minimally specified, providing the learning algorithm with maximum flexibility to achieve the desired result.
In the second phase, the high-performance RL algorithm collects data and finds a control strategy by interacting with the environment, as shown in Figures 1a, b. The simulator used in the study has enough physical fidelity to describe the evolution of plasma shape and current, while keeping the computational cost low enough to learn. Specifically, the study is based on a free-boundary plasma-evolution model that models the evolution of plasma states under the influence of polar field coil voltages.
The RL algorithm uses the collected simulator data to find the optimal strategy for the specified reward function. Due to the computational requirements of the evolutionary plasma state, the data rate of the simulator is significantly lower than that of a typical RL environment. This study overcomes data deficit problems through maximum posterior strategy optimization (MPO). MPO supports data collection across distributed parallel streams and learns in an efficient manner.
In the third phase, the control strategy is bound to an executable file with the associated experimental control objectives, using a tailor-made compiler (10 kHz real-time control) to minimize dependencies and eliminate unnecessary calculations. This executable is loaded by the Tokamak Configuration Variable (TCV) control framework (Figure 1d). Each experiment begins with a standard plasma-formation procedure, in which a conventional controller maintains the position and total current of the plasma. At a predetermined time, called a "handover", the control switches to the control strategy, and then the 19 TCV control coils are activated to convert the plasma shape and current into the desired target. There will be no further adjustment of the network weights after training is complete, in other words, zero sample migration from simulation to hardware.
Demonstration of basic functions
This study demonstrates the ability of the proposed architecture to control the objective in the TCV experiment. First they demonstrated precise control of the basic mass of the plasma balance. The control policy performance is shown in Figure 2. All tasks execute successfully and the trace accuracy is below the expected threshold. The results show that the RL architecture enables precise plasma control at all relevant stages of the discharge experiment.
Figure 2: Demonstration of plasma current, vertical stability, position and shape control.
Control the demo
Next, the study demonstrates the ability of the proposed architecture to generate complex configurations for scientific research. The result is shown in Figure 3:
Figure 3 Control Demo.
New multi-domain plasma demonstration
Finally, the architecture is showcasing the power of the architecture in exploring new plasma configurations. DeepMind tested the control of "droplets," a configuration in which two separate plasmas exist simultaneously inside a container. Using the proposed method, DeepMind simply adjusted the simulated switching state to account for different switching conditions from the uniaxial plasma, and defined a reward function to keep the position of each droplet component stable while increasing the domain plasma current.
Figure 4: Demonstration of continuous control of two independent droplets on the TCV throughout the 200 mm control window.
Future outlook
All in all, as fusion reactors get bigger, partnering with DeepMind may be the most critical. While physicists have a good grasp of how to control plasma in small tokamaks by traditional methods, the challenges will only increase as scientists try to make a nuclear power plant-scale version viable. Slow but steady progress is being made in this area.
Last week, the European Joint Ring Reactor (JET) project in Oxfordshire, England, broke new ground, setting a new record for extracting energy from fusion experiments, producing 59 megajoules of energy in 5 seconds. Meanwhile, the International Thermonuclear Fusion Experimental Reactor (ITER) international cooperation project in France is under construction, which is expected to start in 2025 and become the world's largest experimental fusion reactor.
Dmitri Orlov, associate research scientist at the San Diego Energy Research Center, said, "The more complex and performant the tokamak device, the more need to control more quantities with greater and greater reliability and accuracy." The AI-controlled tokamak device can be optimized to control the transfer of heat from the reaction to the vessel wall and prevent destructive "plasma instability". The reactor itself could be redesigned to take advantage of the tighter controls provided by reinforcement learning.
Ultimately, Ambrogio Fasoli believes that collaborating with DeepMind could allow researchers to push boundaries and accelerate the long journey to fusion energies. AI will empower us to explore things that humans can't explore, as we can reach our goals with control systems that we dare not take risks. "If we're sure we have a control system that gets us close to the limit but doesn't go beyond it, we can actually use it to explore possibilities that don't exist."
Reference Links:
https://www.wired.com/story/deepmind-ai-nuclear-fusion/