Metas Movie Gen seems like a big step forward for AI videos (but you can't use it yet)

At this point, you probably either like the idea of ​​creating realistic videos with generative AI, or you believe it's a morally bankrupt endeavor that devalues ​​artists and will usher in a catastrophic era of deepfakes that we'll never escape. It's hard to find a middle ground. Meta isn't about to change its mind with Movie Gen, its latest AI model for video creation, but no matter what you think of AI media creation, it could end up being a significant milestone for the industry.

Movie Gen can produce realistic videos at 16 fps or 24 fps at up to 1080p (upscaled from 768 x 768 pixels), in addition to music and sound effects. If you upload a photo, it can also be used to generate personalized videos. And most importantly, it seems easy to edit videos with simple text commands. In particular, it can also be used to edit normal non-AI videos with text. It's easy to imagine how useful this could be for cleaning up something you shot on your phone for Instagram. For now, Movie Gen is just pure research – Meta won't be releasing it to the public, so we have some time to think about what it all means.

The company describes Movie Gen as its “third wave” of generative AI research, following its early media creation tools like Make-A-Scene as well as newer offerings that use its Llama AI model. It is powered by a 30 billion parameter transformer model that can create 16-second videos at 16 frames per second or 10-second videos at 24 frames per second. It also features a 13 billion parameter audio model that can sync 45 seconds of 48kHz content such as “ambient sounds, sound effects (Foley), and instrumental background music” to video. “Due to our design decisions,” there is no synchronized language support yet, the Movie Gen team wrote in its research paper.

According to Meta, Movie Gen was initially trained on “a combination of licensed and publicly available datasets,” including around 100 million videos, a billion images and a million hours of audio. The company's language is a bit unclear when it comes to sourcing – Meta has already admitted to training its AI models on data from every Australian user's accounts, and it's even less clear what the company uses outside of its own products .

As far as the actual videos go, Movie Gen definitely looks impressive at first glance. Meta says that in its own A/B testing, people generally preferred the results to OpenAI's Sora model and Runway's Gen3 model. Movie Gen's AI humans look surprisingly realistic, without many of the gross telltale signs of AI videos (particularly distracting eyes and fingers).

“While there are many exciting use cases for these basic models, it is important to note that generative AI is not a replacement for the work of artists and animators,” the Movie Gen team wrote in a blog post. “We share this research because we believe in the power of this technology to help people express themselves in new ways and give people opportunities they might not otherwise have.”

However, it is still unclear what mainstream users will do with generative AI videos. Will we fill our feeds with AI videos instead of taking our own photos and videos? Or will Movie Gen be broken down into individual tools that can help sharpen our own content? We can already easily remove objects from the background of photos on smartphones and computers, but more sophisticated AI video editing seems to be the next logical step.