“Vantage” is AR Insider’s editorial contributor program. It enlists spatial computing executives and innovators for first-hand strategic insights. Find out more or contact us to participate here. Authors’ opinions are their own.  

Toyota makes mixed reality magic with Unity and Microsoft HoloLens 2

by Nick Davis

One of the core principles of Toyota Motor Corporation is Kaizen, which translates to continuous improvement. In both production equipment and work procedures, Kaizen seeks to drive maximum quality, efficiency gains, and elimination of waste.

Toyota often turns to technology to deliver these improvements, which is why the automaker was an early adopter of 3D data for digital engineering and later embraced real-time 3D technology. This includes the use of Unity’s real-time 3D development platform across its automotive lifecycle.

Its virtual pipeline starts by importing vehicle data into Unity using Pixyz. This process quickly converts Toyota’s large computer-aided design (CAD) assemblies into lightweight content suitable for real-time 3D.

The company then develops applications tailored to its needs and deploys them to various platforms. That can include training sessions in VR, creating realistic car configurators for its luxury Lexus brand, or condensing inspection workflows from days to hours with HoloLens 2.

Driving continuous improvement 

Toyota has used Unity to create and deploy mixed reality applications to Microsoft’s HoloLens2 across its automotive production process. Naturally, its team was eager to expand their mixed reality capabilities with Microsoft’s revolutionary wearable holographic computer.

Watch the talk below from Koichi Kayano, the project leader of mixed reality for automotive digital engineering at Toyota, which introduces several proofs of concept progress. He explains how mixed reality helps Toyota achieve Kaizen in several aspects of design, manufacturing, and field service.

How mixed reality delivers Kaizen at Toyota

Drilling down, here are some of the many ways Toyota is saving time, reducing costs and driving efficiencies with mixed reality.

Improving design reviews for computational fluid dynamics (CFD) analysis

Previously an arduous task, CFD analysis is now made simpler with the assistance of mixed reality. Toyota uses Unity and HoloLens 2 to capture and display CFD analysis on a vehicle in real-time to streamline the design review process.

Navigating around a stationary vehicle, the user can simulate and analyze how its design affects aerodynamics. And using multiple HoloLens 2 devices, Toyota’s team can share their view with one another to better communicate and collaborate during a review process.

Increasing understanding of vehicle functionality

Once a vehicle is assembled, it’s challenging to explain the functionality of hidden mechanisms within the vehicle. This is made more difficult when the function requires the vehicle to be in motion.

Using HoloLens 2 and Unity, users can now move around and inspect the inner workings of a “moving” vehicle – making a task that was once impossible not only doable, but easy and safe.

Reducing human error during inspections using machine learning

The possibility of human error is always present, even among expert technicians. Simple errors such as a loose coolant cap carry consequences if not detected and corrected immediately.

With the help of machine learning, Unity, and HoloLens 2, engineers are being guided to recognize and remedy inconsistencies that are easily missed by ordinary inspection.

Trying to reduce human error previously required a lot of human effort, however. To train machine learning models on Microsoft Azure to recognize these simple mistakes, Toyota’s team needed to take 20,000 photos and spend 200 hours annotating the photos manually.

Using Unity, Toyota created 3D models of the vehicle and the body parts under the hood, varied the model’s position in 3D space, and automatically captured a large volume of labeled images to train its machine learning models.

Compared to the traditional workflow that took 200 hours, this approach generated the needed amount of auto-labeled images in just 30 minutes—a 400X improvement in speed. This synthetic data also proved just as effective at training the machine learning models as the manually annotated photographs. For Toyota, this kind of time and cost savings is an ideal example of Kaizen in action.

Reducing mistakes in field service

Proper electrical wiring configuration is crucial for ensuring vehicles operate as intended. In a finished vehicle, however, inspection of connector positions and pin assignments is a serious challenge.

Instead of relying on 2D diagrams, Toyota’s team now has the ability to visualize the entire three-dimensional electrical wiring diagram inside the engine, doors, dashboard, or any other part of the car they require (see image above). This allows Toyota’s field service engineers to gain contextual understanding and to visualize the location of the wiring systems without the labor and time needed to remove physical parts.

Enabling remote assistance in field service

As a global company, Toyota needs to connect field support engineers and experts across various locations. This presents numerous challenges and hinders effective collaboration and training.

With HoloLens 2 and Dynamics 365 Remote Assist, Microsoft’s mixed reality distance collaboration tool, two or more participants can share the same view, communicate, and collaborate no matter their location. This solution allows remote staff to inspect work, educate, and train field engineers, while saving considerable time and money and improve overall results.

Making it easier and faster to create field guidance applications

Step-by-step tutorials are critical to the ability of Toyota’s service engineers to make repairs effectively, but creating these manuals is time- and resource-intensive. In the past, a digital version of a work procedure app would take up to ten days and require an on-site computer graphics engineer.

With Dynamics 365 Guides, Microsoft’s virtual training, performance, and instruction solution, the same task now takes 90 percent less time—just one day. This automated process allows anyone with basic training to create necessary applications through detailed instructions and guidance, freeing up programmers to focus on other tasks.

Nick Davis is Senior Content Marketing Manager for Automotive, Transportation & Manufacturing at Unity Technologies. A version of this article originally appeared on the Unity Blog, published here with permission.  

More from AR Insider…