Recently, sponsored by the Chinese Society of Image and Graphics, organized by the Intelligent Graphics Committee of the Chinese Society of Image and Graphics, Lingyun Light Technology Co., Ltd., and Beijing Yuanke Ark Technology Co., Ltd., and supported by the Beijing Light Field Generic Technology Platform, the "Research and Application of 3D Gauss and Light Field Technology" seminar was successfully held in Beijing Shougang Park.
Liu Yongjin, director of the Intelligent Graphics Special Committee of the Chinese Society of Image and Graphics, Yu Fei, science and technology project manager of the Beijing Municipal Science and Technology Commission and Zhongguancun Management Committee, and Yang Yi, senior vice president of Lingyunguang, delivered speeches respectively.
As one of the organizers of the conference and the operating unit of the light field generic technology platform, Ms. Lingyun Guangyang Yi initiated the initiative of establishing the "Light Field Generic Technology Platform Youth Expert Group" at the meeting, which will continue to focus on the theoretical research and application of 3D Gaussian and light field technology, cooperate with major scientific research institutions to jointly carry out technical research and subject research, promote the deep integration of academic research and industrial practice, and build a complete technological innovation and application ecosystem.
As an important branch of graphics, 3D Gaussian and light field technology is of great significance for improving visual perception, augmented reality, virtual reality, medical imaging and other fields. The symposium invited relevant experts and scholars from the Chinese Society of Image and Graphics, Tsinghua University, Chinese Academy of Sciences, Beijing Institute of Technology, Chinese Academy of Sciences and other units to jointly introduce in detail the mechanism, potential, optimization, application direction and major achievements of 3D Gaussian Splatting (3DGS) technology.
Topic 1
A review of the basic theory and research of 3D Gaussian
Yu Tao, an associate researcher at the National Center for Information Science and Technology, Tsinghua University, shared the unique application fields and advantages of NeRF and 3DGS with the theme of "Neural Rendering: NeRF and 3DGS". NeRF excels at creating photorealistic visuals and handling complex scenes, while 3DGS offers more advantages in real-time, flexibility, and intuitiveness.
Gao Lin, researcher and doctoral supervisor of the Institute of Computing Technology, Chinese Academy of Sciences, shared new geometric representations such as Gaussian splash and neural radiation field with the theme of "Research Progress on Gaussian Splash and Neural Radiation Field", and discussed the advantages and disadvantages of these geometric representations.
Topic 2
3D Gaussian in-depth research and evolution
Liu Yongjin, a tenured professor at Tsinghua University, shared the theme report of "Real-time New Perspective Rendering in Low-light Environments Based on 3DGS". Images taken in low-light environments will show considerable brightness variation and multi-view inconsistency, so a set of anisotropic 3D Gaussian functions can be used to represent the consistent radiation field of the physical world, a camera response module can be designed to compensate for the inconsistency of multiple views, and then a gradient scaling strategy with gradual attenuation can constrain Gaussian near the camera to reduce artifacts. This use of a consistent multi-view image as input, with Gaussian-DK for high-quality rendering, eliminates ghosting and floater artifacts to complete the task of compositing new views.
Liu Yebin, a tenured professor at Tsinghua University, shared a series of his work on digital avatar generation with the theme of "3D Gaussian Reconstruction and Generation of Dynamic Scenes". He first introduced the related work based on NeRF, and introduced a new method of 3D Gaussian point cloud rendering with higher rendering quality and faster rendering speed. Focusing on the advantages of Gaussian Spending in 3D scene expression and rendering, Professor Liu Yebin focused on his recent research work on real-time reconstruction and viewpoint drawing of dynamic scenes, 3D portrait video generation technology based on image large models, and holographic digital human reconstruction and generation technology.
Yao Yao, associate professor and doctoral supervisor of Nanjing University, shared the theme report of "High Realistic Point Cloud Renderable Algorithm and Framework". He said that traditional 3D reconstruction methods typically include photogrammetry, camera pose reconstruction, point cloud generation, and mesh construction, but with the advent of neural radiance field technology, multi-view images can be directly used to optimize and generate high-quality 3D models, greatly simplifying the entire process. Despite the rapid development of neural radiance field technology, traditional methods still have significant advantages in terms of geometric accuracy.
Topic 3
Combination and application of light field data and 3D Gaussian
Liu Bowen, PDT Manager of Yuanke Light Field, shared the theme report of "The Past, Present and Future of Light Field Product Line and 3DGS". In the past, the company cooperated with Tsinghua University, Nanjing University and other well-known universities to successfully develop a number of sets of multi-view stereo vision systems (MVS), which laid the foundation for the development of light field technology. Now, relying on the Beijing light field common technology platform, we have developed a number of photometric stereo vision systems (Photometric Stereo), which have obvious advantages in data solution dimensions and accuracy, taking the large-scale 7D light field acquisition system as an example, the system has reached the world's leading level in spherical LED density and illuminance index. In the future, the company plans to deeply combine light field data with 3DGS technology to promote commercial applications in the fields of digital human, scene reconstruction and rendering, and form a feasible solution.
Large-scale 7D light field acquisition system
Roundtable discussion
The roundtable forum was presided over by Du Hua, deputy general manager and director of the R&D center of Yuanke, and invited Liu Yebin, professor of Tsinghua University, Cao Xun, professor of Nanjing University, Weng Dongdong, researcher of the School of Optoelectronics of Beijing Institute of Technology, Gao Lin, researcher of the Institute of Computing Technology, Chinese Academy of Sciences, Yu Tao, associate researcher of the National Research Center for Information of Tsinghua University, and Yao Yao, associate professor of Nanjing University, to discuss the "3D Gaussian research and application direction" and form the following views:
As one of the three troikas of AI, 3D data and multi-dimensional light field data are far from enough in terms of data volume and openness, and need to be created by academia and industry.
The optimization of quality, speed, and editability is the direction of 3DGS's continuous research, and how to decouple geometry, texture, and lighting from multi-view images is a key problem.
The 3DGS rendering results are "very good", but the geometric measurement accuracy is not higher than that of the traditional MVS method, so the traditional method will always exist, but if the advantages of 3DGS are applied to the traditional MVS method, it may be possible to erase the new spark.
The rendering performance and expression of human hair in 3DGS have advantages over traditional methods, and it can be accessed without much modification of the traditional CG pipeline, which is expected to greatly promote the application of 3D digital humans and spatial video.
For the C-end 3D display ecology, one of the major challenges and uncertainties currently faced is that stereoscopic video has not been popularized on a large scale so far, and how far is it for AR/VR and other devices that need 6Dof display to be popularized?
If 3DGS is a dragon slayer in the field of graphics, then the application is a whetstone, if there is no application polishing, then 3DGS cannot become an industrial "weapon".
Wang Yuwang, assistant researcher of the National Research Center for Information of Tsinghua University, Wen Yuhui, Ph.D. and senior associate professor of Beijing Jiaotong University, Zuo Zhiwen, associate researcher of Zhejiang Gongshang University, Ruan Zhengsen, lecturer of Zhejiang Gongshang University, and Xiong Wei, general manager of Yuanke, attended the meeting.