Wisdom Stuff (Public Number: Zhidxcom)
Author | ZeR0
Edit | Shadow of indifference
Microsoft and Meta are like two unstoppable trains, moving forward at breakneck speed.
Zhidong reported on April 14 that after releasing a series of large model open source tricks, Meta AI's basic artificial intelligence research (FAIR) team released the first AI animation drawings open source project, using AI technology to easily make various character graffiti into animation.
Meta founder and CEO Mark Zuckerberg posted on Instagram a GIF of his daughter's little man moving.
▲The little man drawn by Zuckerberg's daughter moved (Source: Zuckerberg)
Meta AI has released animated code and a new dataset of nearly 180,000 annotated amateur drawings to help AI researchers and creators innovate further. To Meta's knowledge, this is the first annotated dataset featuring this type of art.
To make it easier for people to explore the open-source animation drawing project, Meta has also released an intuitive how-to video.
The project's thesis, titled "A Method for Animating Children's Drawings of the Human Figure", has been accepted by Transactions on Graphics, a top international journal in the field of computer graphics, and will be presented at SIGGRAPH 2023, the top international computer graphics conference.
Project website: http://www.fairanimateddrawings.com
Thesis direction: https://arxiv.org/abs/2303.12741
Code direction: https://github.com/facebookresearch/AnimatedDrawings
Dataset direction: https://github.com/facebookresearch/AnimatedDrawings#amateur-drawings-dataset
First, open source sketches to animation code, so that AI can understand human imagination
Meta's animation mapping project began in 2021 when FAIR researchers wanted to make the latest advances in computer vision more intuitive, animating humanoid figures in character drawings.
The human imagination is so extensive, and the characters drawn are strange, which may be very abstract, or they may create some magical painting styles because they are "handicapped parties". For humans, it is not so difficult to understand other people's random graffiti, but let AI models understand some unique and even bizarre character paintings, and the pressure is a bit great.
In this regard, the FAIR researchers envision using object detection models, pose estimation models, and image processing-based segmentation methods to quickly create digital versions of the drawing. It can then be morphed and animated using traditional computer graphics techniques.
But there's a catch: it's hard to get a picture of people at the scale needed to train a computer vision model. And these paintings also need to be marked with bounding boxes, segmentation masks, joint positions and other information.
Another method is to create drawings synthetically, which is also problematic: the generation method requires a lot of sample data to learn, while the style transfer method may not capture all the nuances of the drawing and the photograph, and may not do some of the changes that would occur in the actual drawing, such as paper creases, erased lines, bright lights, and shadows.
To this end, the meta-researchers built a series of subtasks such as human figure detection, segmentation, pose estimation, and animation from a single graph generation task to a series of subtasks, and created an animated drawing demonstration.
Creators can publicly access this animated drawing demo site on their browser, upload their drawings, view/correct some annotation predictions, and receive animations of humanoid characters in the drawings – all in less than 1 minute.
Parents can choose to allow or not allow Meta to retain images and annotations for future research, and whatever they choose will not affect the use of the tool. The researchers hope that by publishing the demo, they will eventually collect 10,000 drawings to improve the performance of the model.
As a result, users reacted positively to animated graphic presentations, uploading and agreeing to use more than 1.6 million images in the first few months, and many of the images uploaded were not amateur drawings at all, but pictures of company logos, stuffed animals, anime characters, pets, action figures, and all sorts of other things people wanted to animate.
Although the demo made it clear in the description that the persona was necessary, the user uploaded some quadrupeds, birds, fish, and many other forms. Users also expressed their expectations for more comprehensive tools, such as transparent backgrounds, support for different bone types, multiple interactive characters, sound effects, background scenery, and text overlays.
After feeling everyone's enthusiasm for turning drawings into animations, Meta decided to release open source versions of the models and code used in animated drawing demonstrations to inspire more developers to try and experience.
Second, four simple steps to make the hand-drawn character move
If you want to try to animate your own characters, but don't want to deal with downloading code and using the command line, you can go to the Animated Drawings website in your browser.
Website address: https://sketch.metademolab.com/
After uploading the drawing, the user has the option to adjust the detected bounding box, split mask, and joint position, and select an action to animate.
Its system incorporates redesigned computer vision models trained on photos of real-world objects. Since there are significant differences in the field of painting in appearance styles, Meta fine-tuned the model using the amateur painting dataset.
The first step is to upload a drawing of a humanoid character, noting that the character is drawn on a white piece of paper with no lines or folds, make sure that the shooting is well lit, that the arms and legs are not stacked on the body, and that it does not contain any infringing information.
In the second step, resize the box around the character to make sure it is exactly the frame character.
In the third step, separate the character from the background and highlight it. If the character's body part is not highlighted, you can use brushes and eraser tools to fix it; If the arms or legs are glued together, they can be separated with an eraser tool.
The fourth step is to check the character's joints. If your character doesn't have any arms, drag the elbow and wrist joints far away from the character, it can still be animated.
The next step is to use the segmentation mask and these joint positions to animate your uploaded character with motion capture data. You can choose the action you want this character to make.
If you want to download and run the code yourself, you can go to the GitHub project and follow the guided steps to try it.
Code direction: https://github.com/facebookresearch/AnimatedDrawings
Meta's animated drawing project has been tested on macOS Ventura 13.2.1 and Ubuntu 18.04. If you install on a different operating system, you may experience problems. Meta recommends activating the Python virtual environment, such as Conda's Miniconda, and then following the steps below to download and install it.
First, run the following command.
Once set up, you can animate. If everything is installed correctly, an interactive window will appear on the screen. (Use the space bar to pause/unpause the scene, the arrow keys to move back and forth in time, and the q key to turn off the screen.) ）
Behind this, characters, actions, scenes, etc. are controlled by configuration files. You can change the configuration file to export MP4 videos, GIFs and other files in different formats.
Meta trained a plotted humanoid pattern detector and pose estimator and provided a script to automatically generate label files from model predictions. In order for it to work, you need to set up a Docker container running TorchServe, which is detailed on GitHub.
Once set up, just enter a single line of command and instantly convert the image to animation.
For example, enter a picture of an onion head man drawn on paper.
After a short wait, the AI model analyzes, detects, segments, manipulates the input hand-drawn onion head man, and animates it using BVH (Human Motion Capture Format) motion data from the human performer, and then saves the generated animation as a GIF file.
You can also add multiple characters to your scene, add hand-drawn background images, or use BVH files with different skeletons.
▲Hand drawn background image
Detailed steps to create your own BVH file are also provided in the GitHub project. For example, you can record a video of yourself dancing with your phone's camera, then export BVH with Rokoko, create a new motion profile, and reposition the profile to fit the skeleton exported by Rokoko, and then you can make an animated version of the humanoid character.
Meta's amateur plot dataset strategy: open website, censorship filtering, external sharing
To teach the AI to read various hand-drawn character works, it needs to learn a large number of sketch datasets.
With the new dataset that Meta shared today (described in detail in related research papers), researchers and practitioners can build tools to more easily and accurately analyze amateur plotting content, unlocking new hybrid digital-physical experiences.
Previously, more than 3.2 million people around the world visited Meta's animated drawing demo website, which was released at the end of 2021, uploading a total of 6.7 million images. Human reviewers then filtered the amateur plots that participants chose to share with Meta's research team, performed multiple levels of filtering to ensure high quality, and implemented privacy safeguards to minimize the potential for misuse of the data.
The specific improvements are divided into two steps: first, self-supervised clustering methods are used to identify and filter out-of-domain images, such as real photos; Second, a contracted agency manually reviewed the remaining images to ensure they met the standards. Reviewers were asked to check if the images were hand-drawn drawings on paper with at least one full-body humanoid; It is also checked to ensure that the image does not contain characters protected by intellectual property or any private or vulgar content. Because reviewers are primarily English speakers, images containing non-English words are excluded to avoid that they may contain inappropriate content.
While Meta's demo can only do a limited set of actions, many users of the animated drawing demo have provided feedback that they would like more features like multiple characters, extra actions, smiles, winks, and gaze cues. GIFs with dancing figures are examples of extending open-source code and datasets for other creative and educational purposes.
With these resources, other researchers can add to the methods of meta-analysis and augmentation of amateur plotting to extend the original presentation capabilities.
This dataset reflects real-world conditions such as blurry, hard shadows, wrinkled surfaces, and background elements that are not present in digital mapping and high-resolution scanning. The dataset also includes bounding boxes, segmentation masks, and annotations for joint positions — features that can provide the model with more ways to identify or animate the drawn figures, which are valuable to researchers.
Conclusion: Open source stimulates AI technology exploration and adds an engine to human creativity
Painting is a natural and expressive way where everyone can draw their own work. In particular, children's paintings are always imaginative and imaginative, and using AI technology to convert these static pictures into animations in seconds will further open the door to imagination.
Meta is using the code and amateur drawing dataset of the open-source animation painting project to lower the development threshold for more researchers and creators interested in participating in such research and experience, making it easier for more people to explore the use of AI technology to supplement human creativity.