D-ID launches AI video translation tool that includes voice cloning and lip syncing

AI video creation platform D-ID is the latest company to launch a tool to translate videos into other languages ​​using AI technologies. However, in this case, D-ID also clones the speaker's voice and changes their lip movements to match the translated words as part of the AI ​​editing process.

The technology stems from D-ID's previous work – you may remember the viral trend a few years ago where users animated their older family photos and could later have those photos speak. On the back of that success, the startup closed a $25 million Series B funding round in 2022 to serve its growing number of enterprise customers in the US who used its technology to create AI-powered videos.

The company's now-launched AI Video Translate technology, currently offered free to D-ID subscribers, allows creators to have their videos automatically translated into other languages ​​to increase their reach. A total of 30 languages ​​are currently available, including Arabic, Mandarin, Japanese, Hindi, Spanish, and French, among others. A D-ID subscription starts at $56 per year for the cheapest plan and the fewest number of credits that can be used for AI features, then goes up to $1,293 per year before moving to enterprise pricing.

D-ID suggests that the new AI video technology could help clients save on localization costs when targeting their campaigns to global audiences in areas such as marketing, entertainment and social media. The technology will compete with other solutions for both dubbing and AI video.

For years, dubbing technologies have made it easier for video viewers to hear audio in their own language, but have often been inaccessible to smaller producers. That's changing now as companies have increased access to technology. For example, YouTube has introduced a multilingual audio feature designed to help its producers reach a wider audience by translating their videos into other languages. Popular YouTuber MrBeast (Jimmy Donaldson) was one of the early adopters and has used the technology to bring several of his popular videos into 11 additional languages.

With AI, the ability to create, translate, or clone voices is also expanding. Microsoft announced this year that it would use AI to translate and dub YouTube and other videos as you watch them. In July, creator platform Vimeo introduced tools that can translate audio and subtitles by mimicking the speaker's voice using AI technology. Numerous companies also offer voice cloning or AI translation tools (or sometimes both), including Descript, ElevenLabs, Speechify, Veed, Camb.ai, Captions.ai, and Akool, to name a few, as well as tools that let you create videos with AI avatars that can speak dozens of languages, like those from HeyGen, Deepbrain AI, and others.

AI libraries for dubbing and lip-syncing, such as Wav2lip, have also made it easier for startups to develop such tools, while also conveying to developers that they make using AI technology easier and perhaps more cost-effective.