Babylon Native: Continuing the Journey
Time flies! Almost two years have passed since we announced the Babylon Native project and it has been steadily gaining attention while we flush out all the capabilities.
In the last update, Ryan Tremblay from the Microsoft Power Apps team wrote about Babylon React Native which brings 3D and XR to React Native. As we continue on this journey to support developers to write once and run everywhere, more and more teams are joining the effort with the core Babylon Native team to assist on the various issues and features to complete the story. Read on to find out more about what’s been happening.
Who is using Babylon Native?
We now have multiple teams working with us on the Babylon Native effort. An additional team from Power Apps has started working with the core team who have, for instance, contributed code to flush out AR feature points for iOS. A team from HoloLens has also joined the effort to bring Babylon Native to the HoloLens 2 device which has resulted in contributions like Babylon React Native for Windows and V8 debugging support. Some members of the community are trying to make games using Babylon React Native. We have even added Linux support as requested by the community a while back.
We are thrilled to have partner teams engage with the core team and appreciate the willingness of the open source community to hop on board to help. These contributions are likely just the tip of the iceberg. There will certainly be more exciting partnerships and contributions in the future.
Foundation Improvements
There have also been a number of foundational advancements since the start of the project. The core team has been iterating and learning while trying to solidify the foundations.
For example, after many iterations, we eventually landed on separating the the library into separate sections: Core, Plugins, and Polyfills. We have also created patterns for the build system as well as some automated testing. These infrastructure pieces help the consumers of this project extend the library enabling them to add new and exciting capabilities though additional plugins or polyfills.
Rendering Updates
Babylon Native rendering at its core is all about translating the Babylon.js engine calls into the native rendering API calls for the target device or platform. Getting this right has been challenging to say the least. To understand how all of this works, we need to understand Babylon.js, WebGL, WebXR, shader transpilation, bgfx, and how rendering works on browsers.
We have learned a ton through the development of this project and I am sure we still have more to learn. Getting into the details of how this all works is perhaps a topic for another time, but for now, I will highlight a couple of things. First, we now have a core Graphics object which is responsible, in part, for the determining when to initialize and shutdown bgfx as well as when to render. In progress at the time of this writing, the Graphics object is also getting support for hardware scaling, multi-threaded rendering (or two-threaded rendering as I like to call it), and better frame buffer management. These updates should make the rendering more stable with a cleaner code base as well as easier integration with existing systems.
Performance Challenges
Performance can mean many different things. Some of the main ones are runtime FPS, memory usage, disk footprint, and power usage. For Babylon Native, we strive to achieve the best case for all of these as much as we can, but some parts have been challenging to overcome.
One example relates to a feature in JavaScript engines called Just-In-Time (JIT) compiling which is a way to optimize JavaScript execution. JIT may not be available in certain situations such as when using Hermes in React Native or when using JavaScriptCore on iOS. Without JIT, the runtime performance of Babylon Native can suffer greatly if there are many scene objects as traversing the scene graph with JavaScript uses much of the frame time.
Another area is the N-API/JSI adapter in Babylon React Native. Babylon Native uses N-API from node.js to communicate to and from JavaScript, whereas React Native uses a different but similar mechanism called JSI. This coupled with Hermes slows down the JavaScript execution more than we would like. We will investigate options to improve this in the future.
One final topic is disk footprint. App developers often want a small download size, especially for mobile devices. The binary size of the Babylon Native library needs to be small. We did some initial analysis of our binaries and it turns out that including the two libraries for shader transpilation, glslang and SPIRV-Cross, produces a much larger binary. Perhaps there are build options we can set to make the compiled size smaller. Another train of thought is that shaders can be precompiled in specific situations, in which case we can avoid including glslang and SPIRV-Cross completely.
See this issue for the latest updates on performance.
Upcoming Features
There are a handful of the important features that we want to implement soon. First is camera and video texture support, a requirement for many scenarios. There is a work-in-progress pull request to explore the options and to help determine the right way to implement it. Another important feature is text rendering. So far, we have gotten away without implementing this feature by using mesh writer, but this is not the best way to render text in many situations. A proof of concept implementation can be found here. Finally, we need to support audio. Many scenarios will not be complete without audio and is especially true for XR where spatial audio is an essential part of the experience.
Conclusion
The Babylon Native project is full steam ahead as more teams join the party. We are closer than ever to realizing the dream of writing code once and run everywhere, on any device, on every platform. I hope this article shows a glimpse of the kinds of things the team thinks about on a daily basis. Check out the repos here or here and post any questions on the forum!
Gary Hsu — Babylon.js Team
