Just one photo can generate a dance video, and another large model application is out of the circle!
Starting from the first working day of 2024, the videos of the Terracotta Warriors, Musk and netizens from all over the world dancing subject three and Internet celebrity dances have been swiped on domestic social media and circle of friends. These videos of about 10 seconds or so are not real people, they are all generated by large models, and this low-threshold "dancing" method has triggered a wide range of experiences among netizens and set off a wave of fighting and dancing. Netizens from all walks of life have frequent golden sentences: AI cured my limbs of incoordination, the wind of subject three blew to the archaeological community, and I no longer have to worry about skipping subject three......
This is a free feature of the Alibaba Cloud Tongyi Qianwen APP, and you can enter the experience page after entering the passwords such as "Tongyi Dance King" and "National Dance King" in the Tongyi Qianwen APP. After the user uploads the photo according to the prompts, a dance video with both god and shape can be generated in ten minutes, and the generated video can better retain the facial expressions, body proportions, clothing and background characteristics of the original image. At present, Tongyi Qianwen is the first batch of users to provide users with 12 popular dance templates such as subject 3, Mongolian dance, paddle step, and ghost step dance.
It is reported that the algorithm behind this function is Animate Anyone, a self-developed video generation model developed by Alibaba Tongyi Laboratory. As early as the end of November, the study exploded on overseas social media platforms such as Twitter and Youtube, with more than 100 million views of related videos, and the project's Star on Github exceeded 10,000 in just a few days.
In addition to the amazing generation effect, the technical route of the algorithm has also received extensive attention. Video generation is one of the most popular research directions in the field of large models, and foreign technology companies such as Google, Meta, and Runway are actively deploying, but for a long time, the video generation of character images has faced many technical challenges, such as character action videos with consistent character images, smooth and controllable movements, and flawless timing.
According to the published paper, Animate Anyone integrates a number of innovative technologies, including the introduction of ReferenceNet, which is used to capture and retain the original image information, which can highly restore the details of characters, expressions and clothing, in addition, the algorithm uses an efficient Pose Guider to ensure accurate and controllable movements, and in addition, through the timing generation module, it effectively ensures the coherence and fluency between video frames. Under the test of the same dataset, the performance of Animate Anyone is significantly better than that of similar models at home and abroad.
In September this year, Tongyi Qianwen became the first batch of large models in China to pass the record, and the functions of Tongyi Qianwen APP continued to upgrade after it was launched, and it can currently provide dozens of functions such as text dialogue, voice dialogue, translation, PPT outline assistant, Xiaohongshu copywriting, and video generation.