Meta’s new AI system “Movie Gen” can deepfake videos from a single photo

On Friday, Meta announced a preview of Movie Gen, a new suite of AI models for creating and editing video, audio and images, including creating a realistic video from a single photo of a person. The company claims that the models outperform other video synthesis models when evaluated by humans, bringing us closer to a future where anyone can synthesize a complete video on any topic, on demand.

The company has no plans yet for when or how it will make these features available to the public, but Meta says Movie Gen is a tool that could allow people to “enhance their inherent creativity” rather than replacing human artists and animators . The company envisions future applications such as easily creating and editing “day in the life” videos for social media platforms or creating personalized animated birthday greetings.

Movie Gen builds on Meta's previous work in video synthesis and follows 2022's Make-A-Scene video generator and Emu image synthesis model. Using text prompts for guidance, this latest system is the first to be able to create custom videos with sounds, edit existing videos and insert changes, and convert images of people into realistic personalized videos.

When it comes to AI video synthesis, Meta isn't alone. Google introduced a new model called “Veo” in May, and Meta says its Movie Gen editions outperform Sora, Runway Gen-3 and OpenAI's Chinese video model Kling in human preference tests.

Movie Gen's video generation model can create high-resolution 1080p videos up to 16 seconds long at 16 frames per second from text descriptions or image input. Meta claims that the model can handle complex concepts such as object movement, subject-object interactions and camera movement.

Still, as we've seen with previous AI video generators, Movie Gen's ability to generate coherent scenes around a given topic most likely depends on the concepts found in sample videos that Meta used to train its video synthesis model. It should be borne in mind that selected results from video generators often differ significantly from typical results and obtaining a coherent result may require a lot of trial and error.