Scanning a photorealistic avatar for virtual reality (VR) needed more than 100 cameras, but now Meta researchers have revealed that it can only be done with an iPhone.
Facebook first announced “Codec Avatars” in 2019, which are a class of photorealistic and learned face models that accurately represent a person’s geometry and texture in 3D and therefore in virtual reality (VR) . Avatars are almost indistinguishable from video, and the company expects real avatars to be important in the future, especially considering their metavers ambitions.
Meta had been using a specialized platform to generate codec avatars called MUGSY, which contained 171 high-resolution cameras, as shown below. But, according to Upload, it has recently made progress and been able to avoid MUGSY by scanning a smartphone with a front-end depth sensor, such as an iPhone with FaceID.
To capture the codec avatar, the phone must scroll to a neutral face, and then again while copying a series of 65 facial expressions. Researchers say the scanning process takes three and a half minutes on average, although users will have to wait six hours for a machine with four high-end GPUs to represent the avatar in detail. It is likely that if this product is made available to the general public, the rendering will occur on a GPU in the cloud and not on the user’s device.
The iPhone’s scanning system uses the “Prior Network” of the Universal Prior Model (UPM), a neural network that helps generate the person-specific codec avatar. This UPM hypernet is trained by scanning the faces of 255 diverse individuals using an advanced capture platform, similar to MUGSY but with 90 cameras.
Other researchers have already shown the generation of avatars from a smartphone scan, but Meta claims that the quality of its results is the latest generation. However, the system is limited to scanning only one person’s head and is unable to interpret glasses or long hair.
Codec avatars are still in time for release
Meta Avatars currently have a basic cartoon style, so this kind of high fidelity still seems a long way off. In this note, Meta avatars appearing in products like Oculus Quest 2 have diminished in realism over time to better adapt to complex environments in applications such as Horizon Worlds.
Zuckerberg’s Avatar
In February, Mark Zuckerberg said in an interview with Lex Fridman that in the future, virtual reality users can use an “expressionist” avatar in casual games and a “realistic” avatar in business meetings.
Facebook has declined to say when codec avatars may be available to general users, but the company is confident it is approaching.
Image Credits: Photos Using Authentic Volumetric Avatars from a Photo Browsing, Meta Reality Labs