Babylon React Native: Bringing 3D and XR to React Native Applications
Posts from early 2019 and early 2020 already started telling the story of Babylon Native and hinted at integration with React Native. In this post, I will provide an overview of the progress that has been made on this integration as well as the next steps.
My name is Ryan Tremblay, and while I’m not on the Babylon team, we’ve had a great partnership in standing up Babylon React Native. About a year ago, a group within Microsoft began work on Mixed Reality features for Power Apps. After surveying the current landscape of AR development on iOS and Android, we decided to accelerate our time to market by having two separate native implementations built on top of SceneKit (iOS) and SceneForm (Android). This allowed us to start getting feedback quickly, but we knew this short term solution wouldn’t scale. We wanted to bring our existing features to additional platforms (including web browsers), and we wanted to continue to add more features.
When considering our options for a longer term solution, it quickly became apparent that Babylon.js and Babylon Native were a great match for these goals. Since Power Apps Mobile is already based on React Native, integration with this framework was critical for us, as was supporting the XR features of Babylon.js natively on iOS and Android. To help accelerate this work, my colleague Alex Tran and I jumped in and got to work with lots of help from the Babylon team along the way.
What is Babylon React Native?
- Provides a simple solution for combining 3D rendering with platform native screen space UI via platform agnostic code.
This has a couple notable implications:
How is Babylon React Native used?
An alpha version of Babylon React Native can be installed into a React Native application today via the NPM package. There are a few additional configuration steps required, so if you want to try this out, be sure to take a quick look through the README.
From an API perspective, there are really only two integration points with React Native.
The first is the useEngine custom React hook, which manages the lifetime of a Babylon Native engine instance. Since Babylon React Native relies on a custom hook, it currently assumes the use of React function components rather than class components.
The second is the EngineView React component, which is bound to a Babylon camera and presents a scene.
A simple example of using these two constructs together would look like this:
Of course EngineView can be combined with React Native UI components.
What about XR (augmented/mixed/virtual reality)?
I mentioned earlier in this post that XR support was important for Power Apps Mixed Reality features. While not tied to the React Native integration, XR support for iOS (via ARKit) and Android (via ARCore) was added as part of the same overall effort. Since Babylon Native already had experimental support for HoloLens (via OpenXR), much of the required XR infrastructure was already in place.
From a usage standpoint, app code can simply use the existing Babylon.js XR related APIs. In a browser, these are backed by WebXR. In a native application, these are backed by ARKit/ARCore/OpenXR (depending on the platform). If in the future an OpenXR implementation on top of ARKit and/or ARCore becomes available, we expect we could leverage it directly and reduce the amount of iOS/Android specific XR support code.
WebXR, ARKit, ARCore, and OpenXR all have a common set of XR features, many of which are already supported across the board in Babylon.js and Babylon Native, including plane detection, hit testing, and anchors. Other features are only supported by a subset of platforms, such as direct access to the visual tracking feature points. In these cases, it is possible to query at runtime whether the feature is supported. We plan to continue to follow this pattern and add additional XR features in the future such as lighting estimation.
Check out the video below created by Alex Tran for a quick demo of the Babylon Native XR features, as well as the benefits of combining Babylon Native with React Native such as fast refresh and easily being able to combine platform native UI with Babylon’s 3D rendering.
Much of the functionality to support both React Native integration and XR (iOS and Android) has been implemented and is available today in the published NPM package. Various stability, functionality, and performance improvements are planned over the coming months.
Support for Windows and HoloLens is still in early stages of development and is ongoing.
We have done some prototyping around providing the same integration points (useEngine and EngineView) for ReactJS and hope to dive deeper into this to ensure we have a consistent programming model any time Babylon is combined with a React variant.
We also have some aspirations around declarative scene definitions using JSX (similar to https://github.com/brianzinn/react-babylonjs) as React Native has some great fast refresh optimizations when only JSX is changed, but no concrete plans here yet.
What else is needed for folks to adopt Babylon React Native? What other XR features are needed? I’d love to hear thoughts from the rest of the community on this!
To learn more about Babylon Native and the integration with React Native, check out these resources.