PIKA Officially Launches Lip Sync - Injecting New "Imagination" into AI Video

I was going to sleep again... But it's so good that I can't die, before going to bed, I took a look at the update dynamics of several AI products...

And then...

After 3 months of silence, PIKA has finally been renewed...

Instead of de-rolling the model, without de-rolling control, they continued to work towards the goals they set at the beginning of 1.0:

Dialogue.

One of the things that sets PIKA's model apart from all other AI video products is that.

When you generate a character, there is a high probability that it will speak, that is, the lip shape will move, simulating the feeling that the character is having a conversation.

PIKA believes that this kind of effect can be more in line with the real "short film" effect, after all, a short film, if you want to look good, the dialogue between people is essential.

Only in dialogue can there be drama, and only in conflict can there be tension.

In the case of a traditional runway, etc., the shots are all empty shots, and the dialogue can only be done in the form of narration in many cases. In that way, when the audience watches the film, the sense of immersion will be greatly reduced.

But when PIKA1.0 was launched last year, it didn't actually get through to voice, which means that you could only run out of a mouth that was moving blindly, but there was no sound.

You need to go to 11Labs or the Magic Sound Workshop by yourself. Then use Jianying or something, and put together the voice and AI clips of your run.

But there is a problem: the shape of the lips and the pronunciation do not match.

So it's absolutely reasonable for PIKA to do lip sync, they will do it, and they will do it.

No, here it is. Let's take a look at PIKA's new trailer first.

However, PIKA's trailer ... You know.

You'll have to try it yourself.

It just so happens that this lip sync is also open to super collaborators, so I'll test it.

After entering the PIKA homepage, upload a video or a picture, you can see such a function:

This is the lip drive.

When you click on it, you will see the speech production area.

PIKA has also cooperated with 11Labs, and has taken 11Labs' TTS, and the upper area can select a specific voice talent, and then enter the text and turn it into speech.

Of course, you can also upload your own audio.

I'm still used to using 11Labs or Magic Sound Workshop to run the audio myself, after all, those two things are a little more usable.

Once it's uploaded, you can start crafting.

I have probably run dozens of cases, from 1/2 front, to 1/5 front, from front face to side face, from realistic to 2D, from picture to video... I also counted and tested it all over again

Let's take a look at a few examples that I think work well:

But there are many more, and it is a bad case.

On the whole, after my experiment, I recommend that I use the text/image to generate the video first, and then use the video to perform lip synchronization, rather than directly using the image to lip sync.

Because the effect of the two is very similar, and if you use video to drive, you can also achieve some large camera movements or changing backgrounds.

PIKA's upper lip is synchronized, and the imagination must be very strong, because whether it is Heygen or Wonder Yuan, all they can do is a static photo to speak, and in terms of effect, there are certain limitations, because the background will not move.

For example, I used Marvel Meta to get this Trump a long time ago.

PIKA's own AI video coupled with lip sync will make some effects that were very complex but dramatic to achieve before, which can bring some new imagination to AI videos.

But in terms of lip synchronisation, frankly speaking, there is still a certain distance to go compared to Heygen and Marvel Yuan.

For example, only the front face is supported, and the side face will collapse.

For example, when there are some occlusions on the face, the recognition is wrong.

For example, when there are some background faces, they will be recognized together.

For example, the lips are often shaking, and they are not stable enough.

For example, sometimes the lips are mushy.

Wait a minute.

But after all, PIKA's update this time is still the Test version, and it has not yet been released to the public.

There's still a lot of room for optimization.

想想MJ的V1时刻，对吧。

I'm looking forward to PIKA's follow-up optimization of lip sync to inject some new vitality into AI videos.

How do I have a feeling, though.

11Labs, the one who does AI dubbing, is the biggest winner...