by 
In April, I was invited to speak at a conference in Shanghai. The conference was all about AI and the Apple Vision Pro (AVP)… and I had no clue about either. It turned out that the event was simply inviting last year’s Apple Design Award winners and hoping for the best.
So I held a talk – about neither AI nor AVP – and got to know several XR people with fantastic and inspiring ideas. This was also the first time I had a Vision Pro on my head. The most striking first impression was its difference from the VR headset that I own, an Oculus Quest.
The second impression was more subtle: a sense of deja vu. I started making mobile games in the iPhone 4 era. Back then, mobile games were just small-scale console games. What followed was an industrywide phase of experimentation and innovation, and of course, also a lot of errors. I felt and foresaw the same spirit when I tried on Vision Pro – the potential to find new and sensible interactions, and thus the potential to shape the standards of a new medium. It felt like the gold rush of mobile gaming. And wanted to be a part of it.
Phasing In
Upon returning from the conference, the first thing I did was to go out and buy a Vision Pro. The second thing was to get a new MacBook, as I soon learned that I needed a silicon chip for developing. And so began my mission to find the essence of XR.
This mission culminated in the game Cueva De Los Manos. But along the way, I learned several lessons. And in the communal spirit of the XR community that welcomed me, I’d like to share those lessons and experiences. Retrospectively, I would divide the process from start to finish into 3 phases: How, what, and where. Let’s take those one at a time…
1. How Does This Work?
I watched all videos about Vision Pro development that I could find… which ended in ultimate confusion. I had to decide if I wanted to use Swift or Unity first, and ChatGPT decided on Unity. I should add that my coding skills are poor but manageable, I have a quite “artistic“ approach to that. So, without having the AVP yet – these initial steps happened without the device itself – I tried to get my MacBook ready for development by installing plugins and packages whose purpose, to this day, I still don’t know. Diving into a new development environment can be scary, especially if it’s so unexplored, but this is what I was here for after all.
2. What Do I Have?
As I was still in the “guessing phase” without actually having a Vision Pro, I tried to estimate which data I get from the device to play around with. I had a few concepts that heavily involved the environment, with out-of-body experiences or trippy environment bending. I knew that I would get a 3D environment mesh, but what I didn’t know was that it’s not textured. And I forgot that blind angles exist. So when I got my Vision Pro, I had to rethink the environment – the 3D mesh was just not detailed enough. What worked very well, on the other hand, was the hand tracking. It was quite perfect and without any lag, so I wanted to do something with that. And I was a bit disappointed to find out that I could not get the face tracking data. I know it’s there, since Apple uses it to create a fantastic persona reproduction, and I would have loved to play around with it. Later, you can read why.
3. Where Can I Go?
Comparing the age of the Vision Pro to that of other media, I knew that we should not take anything for granted yet. Will look and pinch always be the interaction standard? Do virtual representations of keyboards, windows and buttons work? I wanted to make a game with as many proposals as possible – something that you want to show to your friends when they try your Vision Pro. Therefore, it needed to create moments of amazement as early as possible in the game.
Form & Function
Sticking with the concept of moments of amazements, one of those moments in the game is to transform your environment into a Stone Age cave. I chose that because the real-time environment mesh is a bit clunky. So I made a shader that causes environmental surfaces to randomly swell. This creates the illusion of a narrow cave, and also avoids running into objects. The user “enters” the cave not by pushing a button, but by walking around a boulder in front of them. This small hike seems like something shamanic, as if you really need to “go somewhere.“ But this UX element also has a practical purpose – to scan your environment from every angle. Combining form and function in such a way is a key tactic that XR developers quickly learn.
Another key moment in the gameplay is seeing the shadow of your hand on the boulder in front of you. This UX dynamic causes the player to intuitively see and understand – without an interuptive tutorial – that these are the controls. No menus, buttons, nor pinches. Users at this point see a still cave painting with hungry/threatened/hunting people, and have to use their hand’s shadow to help. Keep your hand calm and the scene comes to life, while your shadow manifests into a solid object, working like a lever, catapult, or hook. Both hands together can form one big object or two smaller ones.
Setting the Scene
Since I already had the cave, the presentation as Stone Age art felt natural. The clumsy character of the physics puzzles, the wordless presentation, the abstract shapes, and hand prints thematically align with Lascaux-like cave depictions. Speaking of handprints, because Stone Age people blew stone dust around their hands to trace their outlines, it would have been nice to use face tracking as a “blow recognition.” But again, this data isn’t available.
Meanwhile, since Lascaux does not have such hand prints, I did some research and found the “Cueva de las Manos“ in Argentina. It is not only full of such hand prints, their “art style” nicely matched that of the game: clumsy, geometrical, and mysterious. And with that, the game’s setting was established.
10,000 Year-Old Loop
In the end, I learned that the North Star that I had been chasing was to make the experience a dedication to the cave. I put a learning section behind the boulder, copied paintings 1:1 and put them on the virtual cave walls (I figure the copyright has expired by now 😉). Altogether I closed a 10,000-year-old loop: Their paintings were the very first human attempts at abstraction – something that is genuinely human, and marks the beginning of our pictorial culture.
And while they were using stone dust at the time of creation, today we use pixels to construct the same images. This both symbolizes human advancement and underscores universal and biological constants. And through this creative and technological process, one thing stuck with me: our opportunity as XR developers is to create experiences that not only imitate but extend and elevate reality. The most successful XR experiences will accomplish that goal in their own unique ways.
Philipp Stollenmayer
Game Designer