Providing Input on Input… Two Years Later
Almost exactly two years ago, to the day, I wrote an article about my plans for an upcoming change to device input in Babylon.js. This change would take the underlying system and replace it with one that could accept both Web-based and Native input. So, I’m happy to say that the current foundation for the system has been integrated into the upcoming version of Babylon.js 5.0 but getting to this point has been quite an adventure. I figure that maybe I should talk a bit about what’s new and different, as well as some of the challenges faced in its inception.
The Plan is (Not) Simple
Device input is kind of like an iceberg. It looks simple but as you flesh things out and account for very specific edge cases, when you should and should not listen for input, and even for how to allow the user to temporarily disable the system, you realize just how deep the rabbit hole goes.
On top of that, you have to account for the various idiosyncrasies found in all kinds of browsers (both on mobile and desktop). Long story short, there’s a lot of variables.
So I went about figuring out how we got our input and what the events looked like. For the most part, most browsers adhere to a set of general standards and format their input events the same. This makes life easier and makes coding simpler. And then there’s Safari. As far as browsers go for day to day browsing, Safari works fine but how they handle input events is drastically different from every other browser. When I had started my work, I originally didn’t realize this. Fortunately for me, I had some amazing community members and teammates who were able help me to identify issues and I was able to shore up any problems. With this, the DeviceSourceManager (and its underlying DeviceInputSystem) was created. While the DeviceSourceManager was available in version 4.2, it didn’t actually talk to Babylon.js’ primary input system, the InputManager. The InputManager took input from browser input events and would send them to their relevant functions/observables/etc.
For 5.0, we wanted to reconcile these two system so that they worked together, rather than as two redundant system. To accomplish this, we replaced all of the event logic in the InputManager and routed everything through the DeviceSourceManager. Having this system in place, handling the input for the InputManager would enable us to persue our next step, getting the InputManager talking with BabylonNative.
Talking with Native
I just want to preface this by saying that the work on BabylonNative’s input talking with Babylon.js’ InputManager is still ongoing, as plans for multi-touch support and keyboard support are currently in development.
For BabylonNative, and by extension BabylonReactNative, the next step was to get the input to actually work with the InputManager. Because the bulk of the work for input was already done using the DeviceSourceManager for BabylonNative, getting it to talk to the InputManager wasn’t too difficult. Getting the InputManager (and all connected subsystems) to listen was a different story.
Originally, the InputManager was built to interact directly with DOM events. With respect to browsers, it was pretty “close to the metal”. While this works fine with web, Native has no idea what a DOM Event is or even what an HTMLElement is. Because of this, the input path had to be reworked a bit. Going into this, we knew that two things had to be taken into consideration for our rework. First, any DOM specific code (eg. checking for an HTML Canvas or using an event listener) had to be removed. Second, an “event” still had to be sent up since our users can use the direct event data. We were able to accomplish this by continuing to forward the event data on web (with a bit of context) but also taking our Native input data and creating a compatible “event”. With some minor tweaks, we were not only able to get Native talking with the InputManager, we also got communication with our camera and input observables (eg. onPointerChangedObservable).
Future Plans
I’m not done with the Babylon.js’ input system, not even close. I’ve got all kinds of ideas that I want to integrate into the system. For example, I want to make the input system easily extensible. I’d love it if the system could be set up in such a way that our community could make extensions to support all kinds of devices. I’m not just talking about different gamepads but creative and weird ideas, like a network-based controller to integrate interactive scenes into your OBS scene or an audio-based controller to control a scene with a guitar.
I’m also tooling around with the idea of making an Input Mapper so that you can use abstracted functions/objects instead of having to manually account for each and every button on all the devices that you want to support. It could be as simple as just using “Fire” instead of writing code for each right trigger for every gamepad.
A phrase that I find echoes in my head is “No plan survives first contact”. It basically means expect plans to change. I know that the things I have planned and when they’ll be implemented may change. On top of that, I’m not the only one who’s adding to this plan. I’ve had many discussions with the team and even with the community about where we should go with input.
While I’m not 100% sure what Babylon.js’ input system will look like in two years, but I can assure you that it will be improved upon and I’m looking forward to telling you all about the journey.
