HumanNeRF: Free-of-the-view rendering of moving characters in monocular video

We introduce a free-viewing rendering method – HumanNeRF – that works for a given monocular video in which the person is engaged in complex body movements, such as a video on YouTube. Our method can pause the video at any frame and render the subject from any new camera viewpoint or even the full 360-degree camera path of that particular frame and body pose. This task is particularly challenging because it requires synthesizing realistic details of the body, which may not be present in the input video from different camera angles, while also synthesizing fine details such as the folds of cloth and facial expressions. Our method optimizes the volume representation of the person in a typical T-pose and works with the playing field to map the estimated typical representation to each frame of the video by backward distortion. The playground is broken down into rigid and non-rigid movements of the bones, produced by deep networks. We show significant performance improvements over previous work, as well as convincing examples of rendering humans from a free perspective in one-eyed video in challenging, non-controlled capture scenes.

《HumanNeRF: Free-viewpoint Rendering of Moving People from Monocular Video》

Address of the paper: http://arxiv.org/abs/2201.04127v1