by 
XR Talks is a series that features the best presentations and educational videos from the XR universe. It includes embedded video, as well as narrative analysis and top takeaways. Speakers’ opinions are their own.
2016 visions of near-term AR glasses ubiquity hit a brick wall — mostly due to technical and cultural realities. Cultural factors are prominent (a different article), but technical issues arguably loom larger. That has a lot to do with limitations in converging digital and optical technologies.
As background, digital technologies — such as silicon chip-based computing and displays — are advantaged by the forward march of Moore’s Law. This defines the shrinking size and cost of chip components — the reason your flat screen TV is cheaper now than when you bought it last year.
But optical technologies like lenses follow different laws. They’re bound by physics, such as the way light moves. This is one reason AR glasses are such a challenge — particularly waveguide-based systems that are the go-to format in headsets like Hololens 2 and Magic Leap One.
“I can tell you this is a hard problem,” said AR optics Guru Mark Spitzer at ARiA. “If you use waveguides, like with Hololens, they’re very inefficient. It needs a lot of power, and it has a limited field of view. When looking through the combiner, acuity of the real world is somewhat diminished.”
As further background, waveguides employ optical components like mirrors and lasers. These could always be limited by the ways they project light over the real world. Light signals can’t render the color black for example, so waveguides can’t get the “true blacks” that your TV does.
Speaking of your TV, the question is if digital displays could eventually leapfrog optical waveguide systems in AR glasses. In other words, rather than light projected on glass in front of your eye (or into your eye), could tiny digital displays in front of your eye be advantageous?
“For 28 years, the displays were never good enough. That’s coming to an end. And when you have that, you can solve some interesting problems like see-through optics,” said Spitzer, who formerly ran GoogleX. “What this means is an end to displays limiting visual performance,”
Think of this like your smartphone, whose AR is done through video passthrough. In other words, you don’t see the real world… you see a passthrough image from the camera that’s rendered by pixels on your screen. There’s no see-through glass, nor light projected on it… it’s all pixels.
This approach could apply to AR glasses in that the thing in front of your eye is a lens and display screen that renders a passthrough image via camera, along with AR graphics. The benefit is sidestepping optical challenges and going right to a Moore’s-law advantaged all-digital approach.
“Because cameras present to your eyes in the right format, your eyes and brain can absorb them and you can use a digital processor,” said Kopin CEO John Fan at ARiA. “Optical see-through is OK for information snacking. Once you get more complex data, video see-through is going to win.”
The model here is what already happens in VR headsets that have passthrough cameras to bring in the real world. This is sort of the opposite of AR — instead of projecting graphics on the real world, the latter is brought in via passthrough cameras and added to a pixel-dominant experience.
This could be where Apple is headed. Following the money, as we like to do, a few of the acquisitions in its AR buying spree point to passthrough video-based AR. That includes VRVana, a Montreal Startup that has VR/AR hybrid glasses built around a video-passthrough approach.
Oculus Chief Scientist Michael Abrash is also big on video passthrough due to greater UX control. In outlining projections of AR and VR convergence, he believes that the best AR (what he refers to as MR) will happen in VR headsets — via passthrough video — before it happens in AR glasses.
“Mixed reality in VR is inherently more powerful than AR because there’s full control of every pixel rather than additive blending,” he said at OC5. “The truth is that VR is not only where mixed reality will first be genuinely useful, it will also be the best-mixed reality for a long time.”
But here lies a key challenge for passthrough video as a core AR modality: it requires a VR-like headset. Tiny passthrough video displays are unrealistic for AR glasses due to not only bulkiness, but latency issues, near-eye focal dynamics and eye strain from illuminated pixels.
Though Moore’s Law is at play, it could take several cycles until we see digital displays that are compact enough for stylistically-sound glasses. And multiply size, weight and wiring by two, given that an outward-facing display would be required to show one’s eyes for social interaction.
On the bright side (excuse the pun) near-eye digital displays would have relatively low processing and resolution requirements. That’s because a smaller display area combined with foveated rendering would enable an inherently smaller and more-efficient data processing payload.
“The fovea is only about two degrees, so in pixels that’s 120×120,” said Spitzer. “So data rates for just serving the fovea are pretty low… and the optic nerve is only a 10-megabyte-per-second channel so displays don’t have to be fed more than [that] — low data rates by today’s standards.”
Moreover, Spitzer reminds us that eye tracking can be applied in order to digitally control things like focus, using natural fovea movements. Taking that a step further, AR glasses with video-passthrough could enhance vision through things like telephoto zoom or corrective features.
“Since you’re using the GPU and system-on-a-chip to mediate, you can enhance vision,” said Spitzer. “There’s a long list of things that that can be done to give you better than human vision… That itself is a compelling AR use case and we’re going to be able to do it in the next few years.”
See Spitzer’s full presentation below.
For deeper XR data and intelligence, join ARtillery PRO and subscribe to the free AR Insider Weekly newsletter.
Disclosure: AR Insider has no financial stake in the companies mentioned in this post, nor received payment for its production. Disclosure and ethics policy can be seen here.
Header image credit: AR in Action, YouTube