Text | Zhou Xinyu
Edit | Tang Wing-yee
36Kr learned that the American AI 3D startup "Luma AI" recently completed a $43 million Series B financing, which was exclusively invested by A16Z.
Previously, Luma AI has completed a $20 million Series A funding round from investors including Amplify Partners, NVIDIA (NVentures), and General Catalyst.
Founded in 2021, Luma AI is a technology company focused on 3D content generation, providing 3D content generation and 3D content reconstruction technology solutions.
Up to now, Luma AI has launched features such as shooting and capturing 3D pictures and videos, Image to 3D, Video to 3D, and Text to 3D. Based on Luma AI's technology, users can generate the required 3D models in a short time and at a low cost with just one mobile phone.
The core team of Luma AI has a resume of overseas leading manufacturers and universities. Founder and CEO Amit Jain comes from Apple's AR/VR division and has extensive experience in 3D computer vision and deep technology products. Founder and CTO Alex Yu graduated from UC Berkeley and worked with Angjoo Kanazawa, a professor at the Artificial Intelligence Research Lab, on 3D computer vision related to NeRF (Neural Radiance Fields).
This year, Jiaming Song from Nvidia and Stanford University, and Matt Tancik from UC Berkeley's AI Lab and one of the authors of the NeRF paper, joined Luma AI as Chief Scientist and Head of the Applied Research Team, respectively.
With the open-source technology and the influx of more players, the 3D content generation industry is becoming more and more competitive. For the 3D industry, the real "moat" of enterprises is in 3D data.
Compared to text and 2D image data, there is not much 3D data available for training, and annotated and semantically matched 3D data is even more scarce. At present, the common solution is to use the 2D diffusion generation model to guide the optimization of the 3D representation, so as to achieve 3D content generation.
The so-called NeRF technology is at the heart of Luma AI's 3D content. Originally released by UC Berkeley and Google, this deep learning system can model the geometry and appearance of a 3D scene based on a small number of 2D images.
Barkley Dai, head of growth at Luma AI, told 36Kr that based on NeRF and Gaussian Splatting 3D reconstruction technology, Luma AI can quickly build high-quality 3D data based on 2D image data for machine learning and algorithm training.
Reducing the computing resources and time required to generate high-quality 3D content is the key to the implementation of AI 3D generation technology. Since the beginning of this year, the time taken for Text to 3D has shrunk from hours to minutes.
In November 2023, Luma AI released its latest Text to 3D tool, Genie, on Discord, which reduces the time it takes to Text to 3D to seconds. Based on the deep neural network training based on a large number of 3D shape, structure and scene data, Genie has established the ability to understand the geometric correspondence between semantics and 3D space.
Genie 3D内容生成效果。
Genie recreates the effect of a 3D model of a shoe.
Based on the input natural language prompts, Genie generates the corresponding 3D model in about 10 seconds and provides downloadable standardized file formats for 3D data such as GLB, OBJ and FBX. Barkley told 36Kr that there are very few manufacturers on the market that can achieve a 3D content generation time of 10 seconds.
Genie uses the page.
In Jain's view, the reasoning and generation process of all visual generative models should be based on the understanding of 3D realistic scenes, and the visual presentation of the Internet is the expression result of dimensionality reduction to 2D. With the development of content carriers such as XR, the scene of 3D content generation and landing is far more than games, film and television, but will become a consumer-level content production and expression form.
Welcome to the conversation!