Building Babylon Native for the Apple Vision Pro

Having a new platform to develop for is always exciting. The last one from Apple I remember is the introduction of arm64 for macOS.

Vision Pro coverage https://www.bloomberg.com/news/newsletters/2023-06-25/apple-2023-and-2024-road-map-iphone-15-m3-macs-ipad-air-vision-pro-headset-ljbftjwx

With the Apple Vision Pro, we have a new OS, a new SDK, a new simulator, a new input and a totally new UX.

Getting the same experience with same rendering code on the Web and Native is the purpose of Babylon Native.

The Keynote and a few tweets had me wonder if the API would be so much high-level that it would be close to impossible to use something other than Swift.

Now Xcode 15 and vision SDK are available, it’s the moment to take a quick look at what is available.

Xcode beta 2 available at https://developer.apple.com/download/all/?q=Xcode

Please note that the SDK is Beta. Things might change. This post is about exploration, testing, experimenting and getting more familiar with what Apple provided.

Toolchain and Xcode project

C Headers are available for XROS

First things first, let’s have a look at what is provided.

• C/C++ Headers are there. The bare minimum should be available to build our C++ codebase.
• https://github.com/leetal/ios-cmake is a toolchain file for ios. Basically, a set of variables used by CMake to properly generate Xcode project for the intended device.
• Vision Pro is not yet supported and needs CMake 3.28. https://github.com/leetal/ios-cmake/pull/175
• As I’m writing this post, 3.27 is not yet released and there is approximately 4 months between 2 CMake major releases.

So, nothing before end of year to get a valid toolchain. This doesn’t prevent us to try with some manual steps.

Building Babylon Native

I generated the project using iOS toolchain. The iOS playground is already using Swift, with an Objective C++ bridge with C++. It’s not possible to call C++ from Swift directly.

Then inside Xcode, I changed the complete project to visionPro and verified the deployment target. There is more information on this page: https://developer.apple.com/documentation/visionos/bringing-your-app-to-visionos#Add-visionOS-as-a-supported-destination-for-your-app

Make sure to manually set visionOS as the SDK and platform

Now, we are entering the compilation process, I faced 3 issues.

Sometimes, ignoring issues is the best to get things moving.

• There is no preprocessor for visionOS and our base library ( https://github.com/bkaradzic/bx ), so it gets a little bit mixed. I had a few #includes and commented some code I know is unused in Babylon Native.
• Same for Metal renderer in bgfx ( https://github.com/bkaradzic/bgfx ). The check is basically iOS or macOS. The change is tiny.
• MTKView is not available! It’s part of the Metal Tool Kit (hence its name) and it’s a helper class that creates and handles the device. You just need to add delegates for rendering and resizing the view. I commented the code for now.

After a few more seconds of build, I had an App ready!

Running App on Simulator

I now have a nice icon in the dashboard! Victory! Well, not so fast.

It's loading!

App is there, I can launch it, but I get nothing except a black screen and no breakpoint reached in Xcode.

No luck, black screen.

It can be frustrating but I’m actually pretty happy. Now, it’s only a matter of getting the correct type of window and a valid Metal device. The road is still long but I can remove Xcode/toolchain/CMake from the equation. More to come!