Moving the ball forward on BabylonNative audio
I love audio, so during hack-week this year I wanted to see if I could implement a subset of the WebAudio API in BabylonNative and get a demo working with it. Nick Barlow had done some research a while back on adding WebAudio API nodes to BabylonNative using LabSound that he wrote about here, so I decided to see if I could build on his findings and get LabSound working in BabylonNative on all the platforms we support (UWP, Win32, macOS, Android, iOS and Linux).
TL;DR I got a cool little Theremin synthesizer audio demo working in BabylonNative!
Play with the browser version here: https://playground.babylonjs.com/#VZ99Q8#12.
… and if your C++ kung-fu is legit, build the BabylonNative version: https://github.com/docEdub/BabylonNative/commits/zzz/230911-fhl-audio
Read on for the deets on getting LabSound working in BabylonNative …
— -
Any time you go into a hack-week with a project like this, it’s super-exciting because you have no idea if it’ll even work when you start out. All you have is your wits and your tools to beat the clock, and hopefully luck is on your side. It’s absolutely thrilling if everything works in the end, but you’re venturing into the unknown, which makes it a high-risk adventure.
For this project, the risk was mitigated by the prior work done by Nick. He looked at several different options for implementing audio in BabylonNative and found LabSound to be the most viable. Using his findings as a starting point, I was fairly confident I could get LabSound working in BabylonNative, and after a couple days of hacking I already had the demo working on Windows and macOS. As I moved on to iOS, I was riding high and starting to think it would be a cakewalk. That’s when I ran into my first show-stopping issue; LabSound’s miniaudio backend wasn’t working on iOS!
Red alert!!!
This actually turned out to be not that big a deal. The problem was a small bug that was easily diagnosed with a bit of stepping through the code in the debugger. I got it fixed fairly quickly and voilà! Just like that the demo was working on iOS!
UWP was an entirely different story.
The first UWP issue I ran into with the miniaudio backend used by LabSound was the `ActivateAudioInterfaceAsync` function being dynamically linked at runtime with the assumption that the mmdevapi.dll would be available. This was failing hard on my machine because the .dll was not being found and as a result, no audio device was getting connected up. It took me a while to figure out what the problem was due to the entire miniaudio implementation being done in a single giant ~92,000-line header file, but I diagnosed it eventually and was able to get it working by linking statically to mmdevapi.lib, instead. Probably not the best solution, but good enough for a hack prototype.
I was still not getting a valid audio device connection, though. Further investigation led to the discovery of a second set of issues on UWP that required a dive into the LabSound CMake configuration to force the miniaudio build to use the correct code path, and then I needed to sift through the giant miniaudio header file again to hard code the audio device GUID, forcing it to use the default system device. Clearly not the best solutions, but this was prototype/hack code, so it’s all good. I burned most of the day on it, but it was alive!
Time was starting to run out, though, and I still had to tackle Android and Linux.
Luck was with me on the Android build. The miniaudio backend worked out of the box.
Unfortunately, I ran out of time on Linux. It should not be difficult to get working, later, but I had hoped to get the demo working on all platforms before the week was over. So close! Maybe next time.
Regardless, important lessons were learned …
1) We now know that LabSound can be made to work on all the non-Linux platforms BabylonNative supports, and …
2) I learned a lot about how to setup JavaScript class hierarchies using NAPI to implement the WebAudio API in BabylonNative, which was a whole other rabbit hole that I’ll write about later if people are interested.
Cheers! Thanks for reading and stay tuned for more to come!
Andy Fillebrown — BabylonNative Team
