Van Gogh's masterpieces can also be reverse-cracked by AI?!
Feed an original picture directly, and the AI can incarnate Van Gogh · Bengo to redraw the entire picture little by little (the original picture can be arbitrary).
Carefully comparing the two sides, the AI almost achieved a 1:1 replica, and also clearly showed the construction process of the entire picture for us novices.
The magic comes from the University of Washington in a project called Inverse Painting, which has been selected for SIGGRAPH Asia 2024, and two of the authors are alumni of Northeastern University (Bowei Chen) and ShanghaiTech University (Yifan Wang).
As soon as the project was released, it caused heated discussions on Reddit, and the highest praise was for the artist group (doge).
Artists may be outraged by this
However, some people say that it is very helpful for people to learn how to draw.
It can even be used to "decipher" some of the master's relics and extract hidden or lost techniques.
Based on a diffusion model
From the original image → the time-lapse video of the painting process, Inverse Painting uses a diffusion-based reverse painting method.
By learning from a real artist's painting videos, combining text and area understanding to define painting "instructions", and updating the canvas with a diffusion-based renderer, resulting in a video that resembles the painting process of a human artist.
Further disassembly, this process mainly consists of the following steps:
- Learn the artist's painting process. A total of 294 videos of the acrylic landscape painting process (with an average duration of 9 minutes) were collected, and preprocessing operations such as video frame segmentation, cropping and screening were carried out to allow the model to learn the painting process.
- Define painting instructions. Based on the understanding of the different elements of the painting (such as the sky, trees, people, etc.) and the relationship between the elements, the model generates a set of "drawing instructions" that tell the model what to draw first and what to draw later.
- Use the diffusion model. Generate realistic images with diffusion models and gradually add details to the canvas, just like when an artist paints;
- Textual and regional understanding. The model also generates text directives that tell the model what to draw and a region mask that specifies the exact location of the content;
- Step-by-step rendering. Start with a blank canvas and gradually generate a complete painting process;
- Time control. To simulate the artist's drawing speed in the real world, the model also takes into account the time interval between each step;
These steps as a whole can be grouped into two phases of training + testing, as follows:
第一阶段：指令生成（Instruction Generation）
这个阶段主要生成两个关键指令集，文本指令生成器（Text Instruction Generator）和区域掩码生成器（Mask Instruction Generator）。
The former compares the state of the target painting with the current canvas to generate a short text instruction, such as "Draw the sky" or "Add flowers", to determine which elements should be added next.
The latter is a binary image that indicates which parts of the canvas should be updated.
Together, the two ensure that the model is drawn only in the appropriate area of the canvas.
△ Generated text instructions
Stage 2: Canvas Rendering
Next, use the instructions generated in the first phase to update the canvas.
Using text directives and region masks, as well as the current canvas's image and target painting, a diffusion-based renderer is used to update the canvas.
This renderer employs a technique called the "Denoised Diffusion Probabilistic Model", which progressively removes noise from a noisy image to produce continuous image frames.
In summary, the renderer takes into account a variety of conditional signals when updating the canvas, including text commands, region masks, time intervals, and the characteristics of the target and current paintings.
These conditional signals help the renderer more accurately simulate the drawing style and process of a human artist.
测试（Test-Time Generation）
During the testing phase, the model uses a trained pipeline to generate a painting from scratch.
There are two main keys here:
- Autoregressive process. Each step relies on the results generated by the previous step, thus generating a coherent painting process;
- Fixed time intervals. A fixed time is used between each step to update the canvas, simulating the passage of time during a real painting;
最终，与三个基线方法（Timecraft、Paint Transformer、Stable Video Diffusion）相比，其生成结果明显更优。
At the same time, SOTA results have also been achieved for the relevant variants.
Two of the authors are Chinese
Briefly introduce the authors of the project, a total of 5, two of whom are Chinese.
Bowei Chen received his bachelor's degree in software engineering from Northeastern University in China (admitted in 2016), and later pursued his master's and doctorate degrees at the Robotics Institute of Carnegie Mellon University and the University of Washington.
His research interests are at the intersection of computer vision and graphics, with an emphasis on image and video generation.
Yifan Wang, who majored in computer science from ShanghaiTech University, just received his Ph.D. in CS from the University of Washington this year.
The profile shows multiple work experiences (including bytes, Google, Adobe, etc.), and he is currently a research scientist at Meta Reality Labs Research.
In fact, the AI reverse painting carried out by this team has also been done before, and netizens also mentioned the project Paints Undo.
At that time, it was mainly used in the animation field, and in just 3 months, it had won 3.3K stars on GitHub.
There have been mixed reviews about this, with the biggest controversy being the fear that AI could be used to falsely claim authorship.
After all, explaining artistic ideas is one of the means for the author to prove the originality of the work.
Even for the fact that it can teach people to learn painting, some netizens refuted it with their own experience:
At present, it does not quite correspond to the real painting process
So, are there any children's shoes that understand painting?
— END —
QubitAI · 头条号签约
Follow us and be the first to know about cutting-edge technology trends