Can AR Unlock TV's Next Era?

If you think back to the launch of BBC iPlayer in 2007, the Apple iPhone a year later, and even OpenAI’s release of Chat GPT in 2022, history remembers these as category-defining events when technology moved sharply from margins to mainstream.

These are moments when big global companies made bold moves on emerging technology and made it real. Last month, Sky TV – a pay TV business that operates in six global markets – launched Sky Live. It’s a camera that sits on top of its Sky Glass TV set and brings interactive and immersive features to the TV screen. These include AR filters, virtual product try-ons, and participatory AR experiences with your favorite TV characters (more on those in a bit).

Living room TV webcams are, of course, not new. Xbox Kinect enabled gesture-based gaming fifteen years ago, while Facebook made games core to its portal launch in 2018. Legacy cable firms, software giants, and device manufacturers have been jostling for position in ‘battle for the living room’ for many years, but no one has yet become the category king.

At launch, Sky focused on four main features; video calling, watch together, fitness, and games. Media coverage focused on these features in positive ways, even though Sky Live is limited (at least for now) to its Sky Glass TV devices. We’re not talking about a device that will have consumers queuing around the block – not initially, at least.

Room-Scale AR

Backing up, the launch of Apple’s very early-stage Vision Pro dominated the AR headlines in early June. More B2B showcase than Apple’s typical B2C product launch, this was still a high-profile event for what’s currently a high-cost, niche AR product with little content. But you can bet Apple has ambitions to make Vision Pro and its successors mainstream. If Apple is making the first big moves to popularise AR in a headset device, perhaps Sky has made what will be seen as a similarly significant move to popularise AR in a TV device.

There were two things not really picked up with the launch of Sky Live, but offer big hints about how TV could change over the next few years; Sky’s broader platform strategy, and the introduction of room-scale AR technology.

For context, Sky Live has been designed to work on a platform called Entertainment OS, which delivers a consistent, integrated Sky TV experience on a range of devices including the recently-launched Sky Stream, a smaller, lower-cost device much like a Roku, Chromecast, or Amazon Fire Stick.

Entertainment OS is also potentially interoperable with any device supported by its parent company Comcast. And at the launch, Sky has confirmed plans to launch an app store and developer platform – which clearly suggests scale and global ambition.

Hard Wired

Every day, hundreds of millions of people worldwide experience AR on mobile phones for fun and functionality. Fun face filters, initially a category-defining feature for Snapchat, have since spread across multiple platforms. It’s now easy to reuse 3D digital assets developed for proprietary formats across the web in HTML5. AR is being democratized.

This bodes well for the future of AR on TV because AR technology has been hard-wired into Sky’s platform (Disclosure – my company DeepAR, has been working with Sky over the last 18 months on this). While I am not privy to the company’s long-term plans, this suggests that there is a lot more to come.

Sky Live has embedded people tracking and auto-framing at the camera operating system level, which allows any application running on entertainmentOS to track people in the frame and get a perfectly-framed image – an integral enabler for two of its core features, video calling (launched with Zoom) and Watch Together.

Network Effects

It will take time for Watch Together to gain critical mass as a feature, as it needs everyone to be connected. Like sales of the telephone, network effects take time, but I think shared watching could become popular around ‘appointment to view’ TV watercooler moments – such as the World Cup, Election Night, Eurovision, or The Oscars.

AR filters aren’t yet available in Watch Together, but the underlying technology is there so we can expect to see more innovation as the platform scales. Meanwhile, person-centric AR face and body filters can be found in the platform’s Video Booth app, a feature that wasn’t profiled strongly in the launch. It works a little bit like Snap in that the viewer opens the app, chooses a filter, and records a video, which can then be downloaded or shared via an on-screen QR code.

There are around 40 filters in Video Booth at launch – a mix of novelty face filters as well as character, program, and brand-based filters. These include PawPatrol, Trolls, and Matrix. Other underlying technologies such as background segmentation and replacement filters allow the platform to put the viewer into a fully-branded environment – such as the Sky News studio.

From Viewer to Participant

Finally, video insertion technology offers the audience Hollywood-level green-screen technology from the webcam, putting the viewer into a movie. This is the feature that allows the Pawsome Fitness app (based on IP from Paw Patrol) to work, so the viewer can appear alongside the character on the screen in their own episode. It does feel like a shift from viewer to participant.

AR brings experiences to life in a completely new way, to create new immersive TV experiences around characters, content brands, and advertisers. And with commercial use cases for brands – such as virtual product try-ons and interactive advertising – it’s easy to see the commercial appeal to invest in room-scale AR.

It’s both expected by consumers and totally inevitable that this uniquely immersive and commercially-valuable technology would find its way to the living room. That’s why we see the arrival of AR on the big screen as the biggest innovation in TV since video-on-demand.

George Berkowski is founder of DeepAR.
.