The Road to Project Swan: Behind Pico’s AI Play

---

Pico had a noticeable presence at this year’s Augmented World Expo. It sponsored an entire Gaming and LBE discussion track and participated in a number of panels. Among many appearances, one talk, “Road to Project Swan” by Ke Jing and Nan Ye, was especially interesting, as it contained new information about the upcoming Pico OS 6 that had not been available before.

Recap of what we already know

The first part of the talk was a reworked version of what Pico had shown at the Pico Developer Event in March and to the GDC audience in April. These slides did not contain any new information, but just to recap, the upcoming headset features a “Dual-Eye 4K Experience,” which translates to a 4000 PPI (Pixels Per Inch) display. Inside it is powered by a dual-chip architecture combining a flagship SoC, most likely Qualcomm’s Reality Elite, with proprietary silicon. The proprietary custom silicon includes specialized hardware-level blocks, and delivers low mixed reality latency.

The general capabilities of Pico OS 6 include its Spatial Engine, which allows spatial rendering to happen at the OS level. This should make multitasking easier, with 2D windows and 3D assets able to share the same OS environment. WebSpatial is an open web framework that allows you to spatialize 2D assets (making them appear to pop out in plain terms). WebSpatial uses standard web languages such as CSS and is supposed to work across all XR systems including visionOS. Again, this was known before, but it was good to have it all reaffirmed at AWE.

New June AWE Scoops: The AI-First Development Paradigm and Enterprise Robotics

The presentation featured some new slides I had not personally seen before. First of all, Pico OS will offer Model Context Protocol (MCP) integration. In previous demonstrations in March and April, Pico’s toolchain was entirely manual. At GDC, David Campos had to go into Android Studio and type out the code blocks himself. The June announcement of the Spatial Editor MCP Server and its direct integration into Unity means developers can now use tools like Claude Code, Cursor, or Trae to automatically manipulate the 3D viewport, generate game logic constraints, and procedurally build scenes using natural language. Pico’s slides also reveal structured llms.txt files, which means the team is explicitly making their SDKs machine-readable for AI agents.

While the GDC talk discussed the difficulty of asset conversion layers and fine-tuning particles, the AWE presentation introduced a completely automated background generative pipeline. The demonstration showcased an active sequence calling an LLM over and over to procedurally texturize and construct a full 3D courtyard scene autonomously.

At previous presentations, Pico warned about the “decent overhead” of running Unity and the Pico Spatial Engine side by side, noting it could complicate optimization. At AWE, the team unveiled a new Performance Analysis tool to address this. It leverages AI to trace code lines, analyze frame drops, and automatically fix performance issues.

Toward the end of the presentation, the team showcased how all of these tools can work together to create a fully functional XR game using nothing but prompts and natural language. The video documenting the process was sped up, but Ke Jing and Nan Ye assured the audience that it took no more than 30 minutes to obtain the final result: a Space Pirate Trainer-type shooting game where the objective is to hit toy airplanes as they fly around. I found that part of the talk quite impressive. If everything works as smoothly as in the demonstration, it would definitely put Pico OS 6 at the frontier of AI-first development frameworks.

Separately, the presentation introduced platform-level support for physical robot teleoperation and AI video training models. Perhaps this had already been announced by Pico elsewhere, but this was the first time I had seen telerobotics explicitly mentioned.

There you have it. Information about Pico OS 6 is being released iteratively, and this AWE presentation was no exception. The AI-first development paradigm is becoming the norm, with Google introducing XR Blocks, Snap Inc. introducing CLAD (Closed Loop Agentic Development), and other major XR players also looking to embrace “vibe coding.” From what I saw, Pico is leading the way.