Facebook’s PIFuHD tech creates 3D models from photos

Another step for creating perfect avatars of ourselves

Facebook is developing a new technology named PIFuHD that can create 3D human models out of just one photo. The algorithm uses a deep neural network to detect the subject of the photo and isolate it from the background but it can also recreate the back of the human subject.

PIFuHD stands for Pixel-Aligned Implicit Function for High-Resolution 3D Human Digitization. PIFuHD is based on the previous Pixel-Aligned Implicit Function (PIFu) method and a hierarchical layered neural network. The neural network takes into account the global and local contexts of the image, which allows to achieve high accuracy of the final 3D model.

The architecture consists of 2 levels of PIFu modules:

1. A basic level that focuses on extracting global features from an image, this is similar to PIFu;
2. A refinement level that focuses on extracting local context information and adding precise detail to the 3D model.

Facebook first showed off its interest in digitally recreating humans in March 2019, with ‘Codec Avatars’. This project focused specifically on the head and face, where the avatar generation required an expensive scan of the user’s head with 132 cameras. Using groundbreaking 3D capture technology and AI systems, Codec Avatars could let people in the future create lifelike virtual avatars of themselves quickly and easily, helping social connections in virtual reality become as natural and common as those in the real world.

In May 2019, the company showed off real-time markerless body tracking with unprecedented fidelity, using a model that takes into account the human muscular and skeletal systems.

Normally, the tracking technique used for movies (motion capture), for example, needs some marks as reference points which will then used in 3D applications to create a model with such movements, but here the A.I. doesn’t need any reference marks.

Avatar body generation is another step on the path to their end goal allowing users to exist as their real physical self in virtual environments.

This latest machine learning system, instead, is the result of these previous systems and it allows tracking the human body, extracting it from just one photo, and recreating its backside.

So, once this technology will be available for everybody, its use will not be limited to avatars for games but can open to new horizons for movies where VFX applications will allow us to maybe use an avatar of a famous actor for our movies (like in S1m0ne) or simply we’ll be able to make our movie alone playing different characters in an easy way.

Adobe should take the opportunity in the future and integrate this system for their applications, such as Adobe After Effects.

However, this also has negative implications such as people that will pretend to be another person via social and/or via video calls, for example, or worse, that someone can steal our identity. So, this technology could become another means for a fake society.