Over the past year, AI x Science has arguably been one of the most talked about areas. Previously, with the blessing of deep learning technology, AlphaFold2 from DeepMind greatly improved the prediction accuracy of protein structure, and this breakthrough showed the great potential of AI to solve problems in the field of science.
Artificial intelligence technology presents systemic opportunities in the field of science. In addition to biology, in many fields such as physics, chemistry, materials, geology, etc., AI represented by deep learning is combined with scientific computing to form a new computational method.
In order to better promote academic exchanges, especially peer exchanges between interdisciplinary and cutting-edge work, Heart of Machines and the Institute of Automation of the Chinese Academy of Sciences will jointly hold a series of technical forums to try to invite researchers to share recent work and discuss hot issues in a more relaxed and open communication atmosphere.
From 10:00 to 12:00 on April 13th, the first phase of the "Heart of Machine x Institute of Automation Technology Forum of the Chinese Academy of Sciences" was held online, with the theme of "Deep Learning Inspired by Physics", hosted by Huang Wenbing, assistant professor of the Intelligent Industry Research Institute (AIR) of Tsinghua University, and a number of experts in the field made technical sharing, focusing on deep learning inspired by physics, and the core audience was master's and doctoral students in the field of machine learning.
Introduction by the guest host
Huang Wenbing: Assistant Professor of The Intelligent Industry Research Institute (AIR) of Tsinghua University, ph.D. graduated from the Department of Computer Science of Tsinghua University, his main research interests are graph neural networks and graph model theoretical methods and their applications in the representation and decision-making of physical systems, intelligent chemical drug discovery and other tasks; he has published more than 30 papers in CCF-A international conferences or journals such as NeurIPS and CVPR; and has been selected as "Mizuki Scholar" of Tsinghua University, "Rhinoceros Visiting Scholar" of Tencent, and "Star Casting Program" of Microsoft Asian Research Institute He has won the international conference IROS Robot Competition Champion, Tencent Hornbill Bird Special Research Excellence Award, NeurIPS Outstanding Reviewer, AAAI Top SPC and other awards.
Special guests and keynote introductions
Share topic: Tensor networking and unsupervised learning
Introduction: Pan Zhang, Researcher and Doctoral Supervisor of the Institute of Theoretical Physics, Chinese Academy of Sciences, Main research direction is the intersection of statistical physics, machine learning and quantum computing, proposing unsupervised machine learning models based on tensor networks, statistical mechanics methods based on neural networks, and classical simulation new methods for quantum computers; in Physiological Review X, Physical Review Letters, Dozens of papers have been published in journals and conferences such as PNAS and NeurIPS.
Background: Unsupervised machine learning needs to express the joint distribution probability of data variables, and its representation spatial dimension increases exponentially with the increase of the number of variables under discrete variables. Tensor network methods developed in quantum physics and statistical physics are good at characterizing vectors in exponential linear spaces using low-rank structures, so they have the potential to be applied in unsupervised learning tasks.
Sharing Summary: Tensor networks are generally used in physics to describe high-dimensional quantum states. Naturally, it can also be used in machine learning to describe the joint probability distribution of discrete data variables. Pan Zhang will introduce the generation model based on the matrix product state and the tree tensor network, then discuss how to use the tensor network to understand the representational ability of the Boltzmann machine, and finally introduce the generation model that combines the autoregressive model with the tensor network.
Related Papers:
Zhao-Yu Han, Jun Wang, Heng Fan, Lei Wang, and Pan Zhang. Unsupervised Generative Modeling Using Matrix Product States. Phys. Rev. X 8, 031012,2018.
https://journals.aps.org/prx/abstract/10.1103/PhysRevX.8.031012
Sujie Li, Feng Pan, Pengfei Zhou, and Pan Zhang. Boltzmann machines as two-dimensional tensor networks. Phys. Rev. B 104, 075154.2021.
https://journals.aps.org/prb/abstract/10.1103/PhysRevB.104.075154
Share the topic: Principles of Statistical Physics in Artificial Neural Networks
Guest Profile: Wang Chuang, Associate Researcher and Master Supervisor of institute of automation, Chinese Academy of Sciences, ph.D. graduated from the Institute of Theoretical Physics of the Chinese Academy of Sciences in 2015, did postdoctoral research at Harvard University in 2015-19, and joined the Institute of Automation of the Chinese Academy of Sciences in the fall of 2019; the main research direction is the basic theory of deep learning, the learning of image feature representation, etc. At NeurIPS, Phys. Rev. B, JSTAT and other conferences and journals in the field of machine learning and statistical physics have published dozens of papers.
Sharing background: Artificial neural network models based on deep learning have developed rapidly in recent years and have been widely used in the fields of image, speech, natural language processing, etc., but people still lack a thorough understanding of their internal basic mathematical mechanisms. Looking back at history, statistical physics has been used to study machine learning, especially the research direction represented by neural networks, for nearly forty years. From the early Hopfield associative memory network and Boltzmann machine, to the copy and cavity methods developed by 2021 Nobel Laureate G. Parisi and his collaborators, scholars' understanding of the mechanism of artificial neural networks has been extremely influential. Nowadays, the typical performance analysis method based on statistical physics combined with high-dimensional statistics has developed rapidly in the principles of machine learning such as probability graph models, clustering, deep learning, training convergence and generalization error boundaries, and has also promoted people's understanding of the internal principles of deep neural network models.
Deep model training is a highly non-convex optimization problem, which is a process of interaction and evolution with the weights of millions of neurons. This process can be compared to the interaction process of a large number of physical molecules. Based on the stochastic differential modeling and stochastic matrix method of non-equilibrium statistical physics, we study the convergence process and optimization method of algorithms such as independent component analysis, generative adversarial model, and random graph matching graph network.
H. R. Tan, C. Wang, S. T. Wu, T. Q. Wang, X. Y. Zhang, C. L. Liu, Proxy Graph Matching with Proximal Matching Networks. AAAI,2021
https://ojs.aaai.org/index.php/AAAI/article/view/17179
C. Wang, H. Hu, Y. M. Lu., A Solvable High-Dimensional Model of GAN. NeurIPS ,2019
https://proceedings.neurips.cc/paper/2019/hash/6b3c49bdba5be0d322334e30c459f8bd-Abstract.html
C. Wang, Y. C. Eldar, Y. M. Lu, Subspace Estimation From Incomplete Observations: A High-Dimensional Analysis, IEEE Journal of Selected Topics in Signal Processing, vol. 12, no. 6, pp. 1240-1252, Dec. 2018
https://ieeexplore.ieee.org/abstract/document/8502097
Share topic: Leverage symmetry in deep dynamic models to improve model generalization performance
Guest Profile: Rose Yu, Assistant Professor in the Department of Computer Science and Engineering at the University of California, San Diego; Committed to advancing machine learning techniques for large-scale spatiotemporal data analysis and applying them to the fields of sustainability, health and physics; focusing on physics-guided AI, which aims to combine first-principles with data-driven models; and has been awarded to JPMorgan Chase, Facebook, Google, Amazon and Adobe presented the Faculty Research Award, several Best Paper Awards, and was named one of the "MIT EECS Rising Stars."
Background: Dynamic systems modeling is critical in many fields, including fluid dynamics, epidemiology, economics, and neuroscience. However, many dynamic systems consist of systems of nonlinear differential equations that make numerical simulations difficult. Therefore, accurate numerical calculations require manual tuning and take a long calculation time. There has been a lot of recent work using deep learning to speed up the solution of differential equations, but existing methods are difficult to generalize, and the fundamental reason is that physical data does not have a data specification and standard framework for reference. For example, we don't know how to rotate a sample of fluid flow to keep its orientation consistent. As a result, test data in the real world that lie outside the distribution is difficult to align with the training data.
Share summary: The current deep learning model used for spatiotemporal prediction is difficult to generalize. They are only suitable for specific domains, and prediction in systems with different parameters, external forces or boundary conditions will fail. In this talk, I will discuss how symmetry can be used as inductive bias to improve the generalization performance of deep dynamic models. I will also demonstrate the application of these models to challenging tasks such as predicting turbulence and real ocean data.
Related Links:
Wang, R., Walters, R., & Yu, R. Incorporating Symmetry into Deep Dynamics Models for Improved Generalization. In International Conference on Learning Representations (ICLR) 2021
https://openreview.net/pdf?id=wta_8Hx2KD
Wang, R., Walters, R., & Yu, R. . Meta-Learning Dynamics Forecasting Using Task Inference. arXiv preprint arXiv:2102.10271. 2021
https://arxiv.org/pdf/2102.10271.pdf
Dehmamy, N., Walters, R., Liu, Y., Wang, D., & Yu, R. (2021). Automatic Symmetry Discovery with Lie Algebra Convolutional Network. Advances in Neural Information Processing Systems (NeurIPS), 2021
https://proceedings.neurips.cc/paper/2021/file/148148d62be67e0916a833931bd32b26-Paper.pdf
Share the topic: Physics-inspired machine learning
Guest Profile: Lu Lu, an assistant professor in the Department of Chemical and Biomolecular Engineering at the University of Pennsylvania, also serves in the Institute of Computational Sciences and the Graduate Group of Applied Mathematics and Computational Sciences at the University, and as a lecturer in applied mathematics in the Department of Mathematics at MIT from 2020 to 2021. Land has interdisciplinary background research, including applied mathematics, physics, computational biology, and computer science; current research focuses on scientific machine learning, including theory, algorithms, and software and their applications to engineering, physical, and biological problems; and a Ph.D. in applied mathematics from Brown University in 2020, and a master's degree in engineering, applied mathematics, and computer science from Brown University, He graduated from Tsinghua University in 2013 with a double degree in thermal energy and economics, a minor in computer science.
Background: Although traditional numerical methods have made great progress in solving partial differential equations (PDE), there are still some limitations, such as the inability to seamlessly integrate additional data information into numerical algorithms, the difficulty of generating complex meshes, and the difficulty of solving high-dimensional PDEs. In addition, for the solution of PDE inverse problems, the algorithm implementation is complex, the computation volume is large, and the accuracy needs to be improved. Machine learning is a promising alternative, but training deep neural networks often requires large amounts of data; in many scientific problems, data is difficult to obtain. We can embed physical knowledge (such as physical laws, PDEs, or simplified mathematical models) into neural networks to design better machine learning models. These models can automatically satisfy some physical invariants and can be trained faster to achieve better accuracy.
In this talk, I will review some of the popular trends in embedding physics knowledge in machine learning and introduce some common approaches, including physics-informed neural networks (PINNs), multi-fidelity neural networks (MFNNs), and deep operator networks. DeepONet）。 I will also discuss some applications of physics-inspired learning algorithms in forward and reverse multiphysics problems and multiscale problems.
![Nature sub-journal: Untrained neural networks can also perform face detection[fig]](data:image/gif;base64,R0lGODlhAQABAIAAAP///wAAACwAAAAAAQABAAACAkQBADs=)