The latest Lex Fridman podcast is causing a stir, mainly for its ultra-realistic avatars. But there’s more to the interview than meets the eye once you start reading between the lines.
Recently, Lex Fridman – a popular podcaster and computer scientist – conducted an hour-long interview with Meta’s CEO Mark Zuckerberg. This was their third podcast together, but it made the rounds for being the first of its kind: it was recorded entirely inside VR with the help of the proprietary Codec Avatar model, Meta’s tool that allows for ultra-realistic digital representations of ourselves.
While Codec Avatars have been around for a while (first shown four years ago during Connect 2019), this marks their first official foray into the mainstream. The reception so far has been overwhelmingly positive. Pundits and viewers were left completely wowed, and the podcast comment section was quickly filled almost entirely with expressions of disbelief and amazement. As a result, the interview became more than just a showcase of the Codec Avatars. It became a bit of a pivotal moment, with many users feeling they were witnessing something historic and consequential in the making.
These reactions were the exact polar opposite of what happened in August 2022 when Mark Zuckerberg posted a screenshot of himself from inside Horizon Worlds (Meta’s main Metaverse platform). Avatars used in Horizon are much more stylized, cartoony …and, well, basic. The reaction was a never-ending stream of mockery and ridicule.
It’s obvious that Zuckerberg’s post was meant to be directed toward existing Horizon users, but it quickly started making the rounds outside those circles. Memes kept pouring in, the media joined in with a bunch of snide articles and before long, the now infamous screenshot came to represent everything wrong with Facebook’s Metaverse endeavor. The backlash was so bad that Zuckerberg was forced to issue a correction, claiming the photo was “taken very quickly” and that big updates were coming soon.
Given all of that, it’s genuinely quite impressive to see Meta scoring such a big PR win just over a year later, especially one in the very same department. It’s as if not just the avatars, but the entire idea of Metaverse had found its redemption arc with users actively discussing the possibilities, perils, and ramifications and generally being more accepting of the prospect.
“Maybe fundamentally we like photorealistic things,” said Mark Zuckerberg on the podcast.
If his Codec Avatar seemed slightly surprised or disappointed when he said that, that’s because Mark has historically never felt convinced that VR needs to be super realistic to deliver immersion or presence. In 2021, he made it clear that the goal was to have both stylized, cartoony avatars and realistic ones. In the same vein, he stressed that the feeling of social presence is not specifically dependent on aesthetics or avatar fidelity. We can feel as if we’re there with others regardless, he claimed.
So is this moment of glory somewhat bittersweet for Mark? Well, yes and no.
Photorealism in XR
During the podcast, Zuckerberg seemed to have no choice but to recognize the appeal of photorealism, especially since Lex Fridman himself looked totally captivated by the Codec Avatars.
“It’s going to be a lot easier for people to kind of take some of these experiences seriously with the photorealistic avatars to start. And then I’m actually really curious to see where it goes longer term.”
Though it might seem like a small off-the-cuff remark, it represents quite a significant shift. In short, photorealistic Codec Avatars seem to be taking the front seat, perhaps going as far as becoming the default avatar model in future metaverse iterations. In principle, this should also give Yaser Sheikh and Micheal Abrash additional leverage in any internal debates regarding Codec Avatars. Zuckerberg however, made it clear that he’s far from giving up on stylized or cartoony representation.
“Once people get used to the photorealistic avatars, people actually prefer being able to express themselves in ways that aren’t so tied to their physical reality.”
Later in the podcast, Mark stated even more explicitly that he believes photorealistic avatars will play a key role in helping users “transition” from a strictly physical world to a world that’s a mix of both digital and physical.
The robo-avatars from Connect 2021, therefore, are not going away. What did change, however, is how Mark Zuckerberg plans to get there. The roadmap seems to have shifted toward capitalizing on the wow factor of Codec Avatars and then gradually expanding these models with a variety of customizable options.
There’s already been some criticism and contrarian opinions around Meta’s ultra-realistic avatars being dystopian, stifling, or unnecessary, so I feel it’s important to stress that some of the criticism might be a bit unfair given Zuckerberg’s general stance on the issue.
Codec Avatars potentially running on standalone Quest Pro
During the podcast, we can see both Lex and Mark using Quest Pro headsets together with Beyerdynamic DT 990 PRO headphones. Both headsets are tethered, though we don’t explicitly see if they’re tethered to a PC or just power. Considering these are still early days for Codec Avatars, it would be reasonable to assume it’s all driven by a powerful computer but Mark remained quite vague in terms of explaining the exact details and focused instead of steering the discussion towards the various challenges related to capturing and creating personal Codecs or creating Codecs at scale.
This happened twice and looked like a deliberate maneuver. Reluctance to discuss hardware aspects indicates that Codec Avatars might already be fully capable of running on standalones, or at the very least will roll out as a standalone feature when ready. As Zuckerberg put it, Codec avatars are perfectly attainable.
“(…) with today’s technology. You’re using a Meta headset that exists. It’s a product that’s for sale now. You can drive these with that, but the production of these scans in a very efficient way is one of the last pieces that we still need to really nail.”
Given the level of graphical fidelity, this might sound too good to be true but could possibly be done with pre-loaded assets (in this specific case, Mark’s and Lex’s super realistic Codecs would have been uploaded to both devices in advance). This might be what Mark was hinting at, adding that: “In addition to being photorealistic, it’s also actually much more bandwidth efficient.”
Moreover, having a completely black environment, which is how the podcast was done, had the added benefit of reducing the compute workload (though all Codec showcases are always done on black backgrounds, so this could be done to reduce distractions or for other reasons).
There are also a bunch of other smaller hints, but suffice it to say that Codec Avatars are not going to be made public until they can be run untethered. Since Meta Quests do not include eye and face tracking, that would mean headsets available in the Pro line (such as the rumored La Jolla headset).
Mixed reality up for grabs
On the podcast, Lex and Mark also discussed AI and the newly released Quest 3. Large parts of the conversation focused on technical and hardware aspects, passthrough cameras, depth sensors, hand tracking, and similar. A large emphasis was placed on mixed reality as being the next breakthrough.
It’s interesting to look at those statements in the context of recent comments from John Carmack, who’s happy to see passthrough and AR improvements but remains skeptical about whether mixed reality can deliver a true paradigm shift. This mirrors Carmack’s previous opinions about Meta’s Metaverse push, where the idea of the Metaverse became overshadowed by the marketing agenda and big buzzwords leading ultimately to overinflated expectations and disappointment.
On the podcast, Zuckerberg mentioned the ability to crouch behind your real-life couch to duck incoming fire. This indeed seems like a pretty isolated use case, something Carmack refers to as “technology looking for justification.” But Zuckerberg also sees a more general use case for MR: as a way to ease people into VR and immersive environments and help remove the anxiety that comes from being blocked from the outside world.
“By being anchored to and being able to see the physical world around you, it just feels so much safer and more secure.”
Judging from the podcast alone, there’s no one clear direction or scenario detailing how Mixed Reality will roll out or how important it will be in terms of the entire Meta ecosystem. Down-to-earth, low-hanging fruit use cases are thrown together with more speculative one-shot examples. Behind the big marketing push, Meta seems to be taking an observe-and-adapt type of approach.
“It’s only when you have millions of people using something that you start getting the whole developer community really starting to experiment and build stuff.”
As long as developers do their work, Meta is ready to embrace whatever sticks and whatever the outcome might be. From such a perspective, mixed reality can be considered a win either way. Even if adoption is not going to be as spectacular as ads and promotional materials might suggest, big wheels are turning, and for developers, this seems like the best time to dip their toes in and experiment with MR.
Closing thoughts, is AI immersive?
In terms of AI and AI assistants, Zuckerberg sets forth a pretty interesting vision where instead of having one AGI (Artificial General Intelligence), users instead interact with various specific, purpose-built AIs.
From a technical standpoint, the difference is not all that big. All of Meta’s AI personas are and will be based on their massive pre-trained language model called LLaMa, but it does mean a distinctly different user experience from what other companies seem to be aiming for. Instead of talking to an omnipotent Echo, Alexa, Google or Cortana, users will instead interact with various purpose-built personas that can then appear in XR as embodied avatars together with their own realistic Codecs.
“Maybe you’re sitting there physically and then you have a couple of other people who are there as holograms, and then you have Bob, the AI, who’s an engineer on your team who’s helping with things, and he can now be embodied as a realistic avatar as well.”
AI developments are still fresh, but it is interesting to see how Meta will try to tie their AI research with the existing XR branch. As for AR, neither Lex nor Mark mentioned the new Ray-Ban glasses, which is not surprising considering that the entire conversation revolved mainly around Codec Avatars.
Despite the rather narrow scope of discussion, the interview as a whole was pleasantly candid and yielded a lot of useful tidbits about Meta’s general plans for XR and the Metaverse. The future of Codec Avatars both in XR and in traditional content creation seems bright indeed.
Mat Pawluczuk is an XR/VR writer and content creator. As with all AR Insider contributors, his opinions are his own.
More from AR Insider…
