From animated portraits to faces that can be manipulated in real-time
The first attempt to animate photos was with deepfakes but technology evolved quickly and new algorithms don’t need so much training data to animate photos. Deepfakes were used to replace original faces with others in videos but some companies tried to do the same with more advanced technology.
D-ID for example is a market leader in deep learning, computer vision, and image processing technologies that partners with MyHeritage, an app that helps to trace your family tree starting with a DNA kit. As well as this service, they include a Photo Enhancer that allows restoring old family photos through AI. In addition, by partnering with D-ID, MyHeritage introduced Deep Nostalgia, a tool that allows photos to be animated with amazing and also creepy results. However, D-ID took another step forward and developed a new version of this tool called Speaking Portrait able to animate people from photographs and make them say whatever the user wants resulting incredibly realistic.
Deep Nostalgia, although amazing, had its limits. After a still photo of a person was uploaded, their orientation in the shot was analyzed to determine which direction their head and eyes were facing for reference to create the AI-generated movements but the users had no control over the movements in the generated video without considering speaking.
Speaking Portraits instead, allows anyone to generate a full HD video from a source image and can combine that animation with either recorded speech or typed text and animate it in real time. It may resemble a deepfake, but the technology underlying a Speaking Portraits is said to be entirely different. D-ID intends to launch the product with support for only three languages: English, Spanish, and Japanese but will add more if demand arises.
Speaking Portrait requires the submission of a 10-minute training video of the requested character that must coincide with guidelines provided by D-ID. And it also supports the ability to swap out the backgrounds of the animated face.
Here’s an example.
However, the impressive features of Deep Nostalgia which comes from the D-ID’s Live portrait technology are still available as a separate tool. It can be applied to any still photo and will animate the head while other parts stay static and can’t have a replaced background. Still, the result below shows how movements and facial expressions performed by the real person are seamlessly added to a still photograph. So you can act as a sort of puppeteer of the still photo image.
D-ID’s co-founder and CEO admit that this type of technology has risks, and he and his company are “keen to make sure it’s used for good, not bad,” and they plan to collaborate with others in the field to avoid abuse.
We had already discussed how this technology could lead to impersonating someone else’s and ruin their reputation, or deceive others. We could never know who’s behind a face. And although this could be a great technology for movies or to bring to life our memories, it’s still dangerous.
Source petapixel.com
