Facebook’s XR Road Map

VR enthusiasts got upset at Facebook’s recent announcement that their headsets will begin to test ads. One VR developer who had originally signed up to include ads in their game reversed their decision due to the controversy.

VR enthusiasts were also shocked last year when Facebook announced their consumer headsets will require real-name Facebook accounts, or risk a ban.

These moves were all expected. Let’s stop being surprised by Facebook’s ad-driven business model and try to understand it better. Like all business models, it selects for and rewards decisions that increase revenues. In this case, employees and features that increase ad revenues or user retention will get rewarded more. People who fight for greater civic responsibility at the expense of ad revenue or retention will tend to get frustrated and leave, over time. We always presume good intentions and can find multiple exceptions to these forces. But we don’t need an oracle to see what wins out, long-term.

So let’s map out the company’s next five to ten years of XR development based on everything we know from public information.

Why Should You Believe Me?

Cassandra, of Greek mythology, had a gift to see the future, but was cursed to be forever disbelieved. She foresaw the original Trojan horse attack, where Greek soldiers hid inside a large gifted statue to invade Troy in its sleep. Even if she had saved the city, Trojans would have dismissed her as “alarmist.”

But if Cassandra had been a prototyper, her best tactic would have been to build a cheap wooden horse, good enough to pop out of and surprise downtown Troy. “Imagine I’m a Greek soldier and you’re all asleep!” Trojans might have developed an instinct to “beware of Greeks bearing gifts” before their city got sacked.

I rely on prototyping in my work because you can’t tell people the future. You can only show them possibilities and let them feel it for themselves.

So please don’t believe me. Try this on and decide for yourselves. [If you really need to see my bona fides, skip to the appendix.]

Let’s start with the recent past and then chart our way into the unknown.

2020: Get The “Real Name” Thing Out of The Way

Facebook’s business model only really cares about what you might like or buy in real life, at least until their advertisers care about selling products for your alternate personas.

Facebook reportedly buys data from other companies in places it can’t collect. It counts on its network to report on us across web, mobile and beyond.

Real IDs are the best way to tie it all together (best, as in worst, in my opinion).

2021: Get any Advertising Objections Flushed Out Early

There’s not much money to be made in VR ads today. But Facebook needs to normalize them to minimize future friction. Ads are going to be big money.

Remember your indoctrination lessons (drone voice):

Ads are a normal part of life. They’re the only way to compete against the other, more ruthless, ad-driven companies. People just won’t pay for stuff anymore.

But where do you think the money for these ads comes from? It’s built into the prices of the branded products you buy. But it doesn’t have to be this way. There are other and even better ways to fund the content we crave.

2021+: Drive VR Costs Below All Competitors

Here’s a deeper analysis I published recently on how much Facebook likely spends and expects to make from XR, given such a low entry price for Quest2.

Bottom line: earning a percentage of game sales won’t be enough, unless the headsets also earn a meaningful profit per unit. Facebook will eventually want XR revenue to be at least as profitable as their web and mobile ad-driven business is today.

Keep in mind the people leading XR now came from ads….they love ads. So what do you think they’re betting on?

On the other hand, if (and I’m not saying they are) they are intentionally taking losses to drive competitors out of business, saturate the market, lock-up suppliers or lock-in customers, then I’d focus more on antitrust probes.

2022+: Establish a Growing Ecosystem

Facebook needs developers […developers, developers…] to make the content that attracts and retains customers. They reportedly haven’t treated some VR developers very well so far, including strong-arming, competing with, and sometimes blocking eager developers from the platform.

By offering extra money, say from early VR ads or Quest-exclusives, or even buying some studios whole, they may hope to smooth over any hurt feelings with money. For some, money can buy love. And many of the company’s defenders seem to have a financial stake in the outcome.

Most importantly though: developers need scale, meaning a large base of customers to attract big enough paying audiences. Small studios may take bigger risks, but bigger studios will tend to invest first on platforms where they can make the most money.

The critical threshold is upwards of 10 million headsets sold. At 100M or so, they’ll attract the AAA studios — the ones that routinely make blockbusters. Facebook just needs one Halo-like hit to sell a ton more units.

2022+: Drive VR Adoption into Everyday Activities, Including Work

If we all meet as avatars in VR, FB has many more opportunities for mediating our conversations, adding (and extracting) value from more of our lives than just our political debates and weekend barbeque content.

They already offer a “Quest for business” version that costs 3x more and requires a yearly support fee. But it promises not to spy or serve ads.

Great! Why don’t they make this available for consumers, perhaps without the annual fees? Is it because their consumer product wouldn’t compete as well?

Many of us don’t want to be tracked or monetized especially while exercising, at school, exploring the world, or talking to friends and loved ones. How about you?

2022+: Avatars for Everyone

Facebook will likely follow its real names policy with requirements that your avatar generally look like you. They’ll eventually automate that process from one or more photos, with room for customizations, like new clothing and accessories, which they’ll also be happy to sell.

There are some social benefits to “veracity” in virtual communication (i.e., WYSIWYG for people), assuming the emotional content is conveyed correctly. But it’s not like video chat, which is also imperfect. Machine learning algorithms for avatars will use tracking data from our eyes, mouths, and bodies. This can sometimes portray us incorrectly.

There are benefits at other end of the spectrum too: full freedom to choose. It allows more people to be themselves, beyond everyday RL constraints. But it can also chase some marginalized people off the platform, due to culturally offensive or hateful expression by others. See “HotTub Twitch” or “furries” for examples of how some people may choose to express themselves. See modern Nazis for better examples of universally offensive and hateful self-expression.

Expect to see some fusion of Portal and VR with video and avatars mixed.

2022+: Work with Leading Fashion Brands

Facebook reportedly has a deal with Ray-Ban, which is part of eye-wear giant Luxottica, which also owns LensCrafters, Sunglass Hut, prescription lens makers and many of your favorite brands. It’s basically the Facebook of eyewear.

The question is: What will they ship? [FWIW, Google had also partnered with Luxottica but never shipped a more fashionable version of Glass.]

I don’t expect these first Ray-Bans will have AR displays at all. But the recent announcement of Snap Spectacles with modest AR displays (plus spending $500M acquiring the waveguide maker) could change that, causing a re-think and a delay to better compete head to head with consumer Spectacles.

The logic is: if you’re only doing voice assistants, like Alexa or “M”, you can do well enough with earbuds and existing phones. Glasses, however, would give Facebook the ideal platform for world-facing (i.e., Instagram-ready) cameras, and possibly eye-tracking too. These sensor types could help understand how we each interact with the world. They could capture us more spontaneously and routinely. Automatic electric tinting sunglasses is another desirable feature for sunglasses, but the options are limited by technology. Microphones and private-audio speakers or transparent earbuds are basic requirements.

Working with major brands helps Facebook establish smart-glasses as fashionable (also hopefully well-designed) and create distribution outlets beyond the usual “big box” stores.

Expect Facebook to announce even more brand partnerships for things like virtual clothing and accessories, or hybrid items that exist in both VR and RL.

2022+: Aria Grows

With Project Aria, they’re already testing AR tech out in the world, in advance of the ideal display technology. What they need to learn is how to make the eventual consumer glasses cheaper and more power-efficient for all-day use.

And that means they’ll collect a ton of user data now, using machine learning to inform more optimized hardware for actual consumers later.

Facebook can learn where people go, what products they like, who they’re attracted to or angry at, and even for which body parts do they steal glances. They’ll learn when people break the law, how hard they actually work, and how often they lie. The information is very personal.

I’m not sure any employee knowingly signed up for their bosses or co-workers knowing these things. But the data is invaluable for coming up with new algorithms to answer these same questions more efficiently than today.

2022+: Privacy Controls

Facebook will add privacy controls for any glasses they ship with cameras or automatic audio recording. Otherwise, consumers would push back on things like recording in the locker room or bathroom. Businesses will be concerned that employees might wear these consumer devices into meeting rooms, factory floors, or other proprietary areas and stream content to the cloud.

If these are your prescription glasses, you may not be able to simply “put them away” in sensitive locations, as you might do with cell phones. You might be functionally blind without them. So the glasses need a way to prevent recording and/or make it clear to everyone else when they are.

We saw this concern with cell phones early on. Illicit recording was only partially addressed by adding hard-wired recording lights and sounds. Facebook may add “no spy” zones that disable recording features in some locations. They’ll call this “privacy-first design.”

But to shut the glasses off entirely? Not likely. These glasses give Facebook back some of the ad-tech superpowers they recently lost, like the ability to track us through unique IDs and time-stamped locations. The glasses will still be reporting on us, no doubt, potentially bypassing our phone’s privacy settings via encrypted streams of data.

They’ll map the world and our personalities to make better predictions about us, which means their advertisers can make smarter and more lucrative bets on us.

2023+: Ads Evolve

At first, ads might only appear as 2D billboards inside games, perhaps simply advertising other VR games. It’s all fairly innocuous, right?

Keep in mind, advertising is a lot like Vegas. The goal is not to overly control us against our will. The goal is to keep us stuck in place, addicted, so advertisers can place more and more bets about us. The House always wins, long-term. So they just need us to keep playing long enough.

The winning ads won’t be your typical infomercials or blaring and repetitive radio ads that just hope for something, anything, to stick. The winning XR ads will get and keep our attention very well. They’ll be more story-based, more character-driven, more emotionally rich.

Think about why some people will watch the Superbowl every year, just for the ads — whatever works to hold our attention longer or, ironically, grab us emotionally, in a way we don’t realize we’re being grabbed.

2023+: Eye Tracking and beyond

If the first glasses don’t have eye-tracking, expect it to follow within a year or two. With eye-tracking (ET) and a few well-placed sensors, they’ll learn our unconscious reactions to anything we see. This is immediately helpful to learn which ads work for each of us, so we get more of the stuff that works.

But virtual/augmented content can change dramatically whenever we blink or look away. It can change without us even noticing (see the article above). This means our devices can iterate on many different product options in a very short time. They can then focus on what we like most and give us more of that.

Other companies will use ET to simply force us to pay attention to ads before advancing to something else we really want. But Facebook won’t be so brute-force about it, because they’re playing the long game with your attention.

Expect our personalized ads to keep changing until we like them.

2024+: More BCI

Eye-tracking is one kind of Brain-Computer-Interface (BCI). It’s currently the most powerful way (IMO) to unobtrusively infer what we’re focusing on and how we feel about it (e.g., excited vs. bored).

Other more directly-sampled BCIs will be helpful in place of controllers, buttons, and so on. FB is thinking about sensing our muscle movements to match our intentions in VR without actually moving our hands or arms.

BCI can’t yet read our conscious thoughts, like some ticker-tape. But as FB introduces new ways to type with our brains, this gets closer to recorded stream of consciousness. The tipping point for mental privacy comes when we can start typing with FB’s technology and then realize what we just accidentally typed our innermost thoughts. Oops. Delete.

2024+: Experiential Advertising: Sell-atars

The next level of advertising is experiential. At first (2022+), it will use simple 3D product placements. Think about seeing some interesting sneakers being worn in your favorite VR game, but you’re able to click on them to find out where to buy. By 2023 or so, AI agents (aka NPCs) will act like embedded salespeople, in some cases even using celebrities, domain experts, other digital influencers.

The key is that it feels less like an intrusion into your chosen experience and more of a woven element, demonstrated to reduce consumer objections. These ads will be selected more automatically than product placements in movies today because they can all be changed at runtime — dynamically bid on by advertisers— during the experience, without you even noticing.

Today we might record a living celebrity in a studio, or occasionally digitally reconstruct a dead one. By this point, it will be common for virtual copies of trusted and influential people to monetize multiple products, TBD at runtime.

Eventually, expect to see branded avatars telling us how to approach lifestyle, money, political concerns the way YouTube or TikTok videos work now, but in interactive 3D. You can see this beginning with virtual 3D influencers already.

2025+: Custom Casting

Why stop at celebrities and influencers when they also have our friends and family — the people we really trust — readily digitized as “avatars” for chat purposes?

At first, the benefit for average AR/VR consumers is that they can automate more aspects of their digital lives. With an “AI agent” that learns your personality as well as your gestures, you can be in two or more places at once, surfing your attention back and forth as desired. You might automate some chores and interactions, or even mildly entertain friends while you’re busy with something else.

I’m sure there will be some kind of ‘opt-out’ at least. Maybe we can limit in which contexts we’re “used” autonomously. There may even be compensation, bribes essentially, to grease the skids on this.

But the default terms of service will allow our biometrics, appearance and behavior to also be rented out to persuade our social circles of whatever the highest bidder is selling. Given the expected pushback, this could be kept subtle — digital agents who only vaguely remind us of people we know. Facebook is not known for subtlety.

2025+: Synergies for Some

Once FB has a full range of useful every-day applications for AR and VR — essentially an operating system plus many conformant apps — they can build more synergies across the platform. In anti-trust terms, that potentially means more lock-in. But the point is actually to keep you within their data ecosystem.

For example: once they figure out you like a certain car (from experiments they run with eye tracking and more) your AI playlist might play more songs about cars. Your banking app might suggest this preferred car is actually affordable to you. And they can get you a special deal today.

That synergy is benefiting someone, probably not us, because we’re not the ones directly paying for it. This can happen sooner than 2025, but I think it will wait until XR has hundreds-of-thousands to millions of ready users.

2026+: AR Glasses

It took a lot of R&D and likely several big acquisitions, but Facebook is finally able to combine light-weight prescription glasses with a decent AR display and an all-day battery life.

What Facebook learned from several years of non-display glasses, like the Ray-Bans or Aria sensor glasses, is invaluable to making these consumer glasses something that people actually want to wear for 18 hours at a time.

Except there’s a real pushback from consumers about whether they want Facebook gathering so much data on them and using it for marketing.

I’ll hesitantly predict that by this time, Facebook has finally accepted the need for a more consumer-friendly business model. The anti-trust investigations will probably help, since FB may be legally barred from using our personal data to advertise to us without our expressed consent. And no one really wants to be the target of advertising. Much better to sign up for an unbiased service that helps us find the products we really want.

With a combination of micro-transactions, crypto, and well-loved 3D content, Facebook will begin to take a small percentage of every transaction running through their networks: 0.1% for starters, but climbing over time. A small percent of trillions of dollars of commerce is quite substantial.

2026+: Growth

AR Glasses will take off like smartphones did in the early millennium. Despite the stories, smartphones weren’t an overnight success. It took wave after wave; generations of hardware improvements gated on new technology and manufacturing processes, big software improvements, wider consumer acceptance, low switching cost, and some paid contract plan “subsidies” to get to one billion smartphones. All in all, it took about a decade.

2030+: Beyond the Event Horizon

It’s hard to see beyond this time. Facebook probably doesn’t even know its own end-game yet. It just believes it has to win at all costs, because to lose is to be dependent on other companies’ platforms and to be left behind.

I think it’s a near-certainty that hundreds of millions of XR devices are out in the market by 2030. The state of society and our own self-governance is what’s most unknown. I worry that this “event horizon” idea could be a bit too literal, a wall of massive change is blocking us from seeing past.

We’re at a critical inflection point. This ultra high-tech ad-driven machine, epitomized by Facebook, represents a new kind of Trojan horse that we’ve only just recently welcomed into our world. Night is falling and the sacking has barely begun. In the future, we run the risk of being so entranced by our shiny new gifts that we lose the will to fight entirely.

On the other hand, if Facebook and others really become true privacy-first, consumer-first companies with business models that reinforce it, if they listen to criticism and welcome independent oversight, they could still become the core companies we depend on, with enough meaningful introspective data and the right protections to actually make our lives better than today.

This may be one of the most important questions of the decade.

Appendix: The Role Not Travelled

Early in my career, I wrote speculative fiction in addition to building VR experiences. I got a huge boost when Walt Disney Imagineering built a time machine in 1994 — no wormholes or magic wands required. We spent $20 million buying 3D graphics supercomputers from SGI, inventing new VR headsets, designing new experiences along with Disney artists and animators. It was something that might run on a Quest today. But it happened 25 years early.

In 2010, I helped convince my bosses that AR will be the next big paradigm shift. They put billions into HoloLens. A few years later, I lead the early UX for Amazon’s Echo Frames, designed as fashionable glasses and made with practical technology. From 2016 to 2019, I helped Apple figure out whatever it might be doing by prototyping experiences and technologies from the future.

I mention those roles to highlight the most important job I didn’t do. A few months before Apple, I was invited to Facebook to interview for a position leading Facebook’s new AR program. Everyone else was still focused on VR. This was a growth opportunity with perfect timing to do the kind of things I generally love to do: figure out the future.

But I had a few concerns about the company, so I asked questions like:

If we figured out a new business model that didn’t involve selling ads but was just as lucrative, would you switch?

The general response I got was: We’d do both. They see nothing wrong with their current business model. I’m not spilling any secrets here. Their CEO proudly told Congress, “Senator, we sell ads.”

This wasn’t surprising for a company that started life by rating college women on their looks and now makes its money figuratively booking bets among advertisers about the future behavior of its most loyal users.

Cambridge Analyica wasn’t surprising to me either. I worked at Microsoft from 2008-2012. I noticed that some nearby employees in Bing had access to Facebook’s (i.e., your) social content, on-demand.

For those and other reasons, I never worked at Facebook. But you can think of this article written as if my evil twin had actually taken that job, here’s what I would have done, or at least what I can share.

Editor’s note: A version of this post originally appeared on the author’s Medium, contributed here with permission. We’ll be back in Part II of this series to extrapolate Facebook’s R&D spending and current XR financial snapshot.

Avi Bar-Zeev is a design and technology leader with a background that includes Microsoft, Apple, Google Earth, Second Life, and Disney. His consulting company is RealityPrime. Follow him on Twitter at @avibarzeev and on Medium for more insightful content.