Hollywood has already used computer-aided visual effects to make veteran actors look decades younger, enable action stars such as Arnold Schwarzenegge

AI Modifies Actor Performances for Flawless Dubbing

submited by

Style Pass

2021-05-28 00:30:07

Hollywood has already used computer-aided visual effects to make veteran actors look decades younger, enable action stars such as Arnold Schwarzenegger and Will Smith to battle digital doubles of themselves, and virtually resurrect dead actors for new Star Wars films or TV commercials. Now AI that can learn the idiosyncrasies of each actor’s voice and facial expressions is making it easier to dub films and TV shows in new languages while preserving the acting nuances and voices of the original-language performances.

The London-based startup Flawless AI has partnered with researchers at the Max Planck Institute for Informatics in Germany to commercialize technology that can digitally capture actors’ performances from 2D film or TV image frames and transform those into a 3D computer model. After training AI to learn specific actors’ vocal and facial performances, the startup can generate modified versions of the original performance that change the actor’s voice and facial expression to fit an entirely different language.

“It's actually a pixel perfect 3D representation of the head of each of the actors,” says Nick Lynes, co-CEO and founder of Flawless AI. “And because of that pixel perfect frame, it represents every single phenomena and idiosyncratic style possible that the actor does, because it doesn't take much before the AI has understood all of the idiosyncrasies.”