Light estimation in Babylon.js
One of the most used features in AR is placing a virtual object in a real environment. Whether using hit-testing or the experimental plane detection, WebXR already offers you the ability to do that. The best examples are always furniture shops placing models of their furniture in your environment. We have a great getting-started guide for that right at https://babylonjs.medium.com/webxr-ar-and-e-commerce-a-guide-for-beginners-2b9ae1d9e796.
When we are placing a virtual object in the real world, our goal is to make sure the object looks as “realistic” as possible. This usually revolves around shadows and lighting. Adding the right lights and adding shadows to a model in AR can make a huge difference. But what is the best way of achieving that?
Fake it till you make it
The first GL experience I have ever made was a basketball trainer as a student project (https://github.com/RaananW/BasketBall-trainer-openGL. The compiled version still works, surprisingly). During development I have noticed that since there are no shadows, and the shading was… bad (or horrible, to be honest) it is very hard to know where the ball was. Being the innovative student I was back then, I researched “how to add shadows to your game” and then gave up completely. What I ended up doing was add a flat disc on the floor right under the ball. Simple — a flat disk in ball-position and set the y axis to be 0.01 (I hope you know why 😊). That was a very simple solution, and it worked well!!
A few things we can learn from that –
- I am not very good at my own game
- I totally ignored the lights and their direction
- This fake shadow, though simple and obviously fake, adds a nice effect to the game.
Why am I telling you this? Because this is exactly what most AR applications do nowadays. The first hit-test demo I ever tried (https://developers.google.com/web/updates/2018/06/ar-for-the-web) did exactly that:
It added a fake shadow under the flower. Nothing more than a “darkened” disc.
If you are using a framework that supports shadows out of the box it is much easier to add a “real” fake shadow that is based on the mesh’s structure. This is usually what XR apps nowadays are doing to make the object look more realistic. Apple’s AR demo does that:
We can play “find the fake sofa”, but you can clearly see that the shadow on the added piece of furniture is right under the object, while the real shadows are stretched to the left. Nonetheless, is still looks more convincing than without a shadow.
So, we solved the shadow issue (kind of?) but we still have the lighting. When adding a 3D mesh to a (Babylon or any other engine’s) scene the mesh will be influenced by the light defined in your project. The model’s appearance depends very much on a pre-configured light source. And they need to fit every scenario. Just like the shadows, we “fake” lights that need to be as simple as possible. Of course, our configured lights will not influence the real world.
Another factor that adds to realism (and in my opinion the most important one) is a decent environment map. Reflections are hard to calculate with your eyes and having “something” reflect off objects makes them look much more realistic. A global environment map will provide this realism to most imported objects. You can read about it here — https://doc.babylonjs.com/divingDeeper/materials/using/HDREnvironment
And again, in case of an AR application it is very hard to generate a proper environment map for your imported objects. Especially since the application will run in different real-world environments. You won’t scan them all and provide environment maps for everyone, and storing one might cause the reflections to look unrealistic (because eventually we DO know if there is a window behind us or not…).
Let’s summarize. In order to “realistically” display an object we need to fake:
- Lights
- Shadows
- Environment map
- Happiness
WebXR to the rescue!
WebXR offers a simple solution to the “faking it” problem. WebXR light estimation module, officially defined here: https://www.w3.org/TR/2021/WD-webxr-lighting-estimation-1-20210909/. This feature had been around for a while and has been recently added to Babylon as a WebXR feature — https://github.com/BabylonJS/Babylon.js/pull/11117.
The idea is that the underlying system provides us with a lot of details that allow us to “match” the object we are placing with the real world. Light estimation can provide us:
- Light color (and intensity)
- Light direction
- Reflection cubemap (environment)
- Spherical harmonics coefficients
- Happiness
Notice I am saying “can” — the data is being polled and not necessarily provided on each frame. You are in charge of defining what information you wish to receive and how often. And you can also decide how to use it in your scene.
As always we did our best to provide the developer with as many possibilities as possible but still have a default experience that “just works”. When enabling the feature you can fully configure it using the options object documented here — https://doc.babylonjs.com/typedoc/interfaces/babylon.iwebxrlightestimationoptions.
The default behavior will generate the cube texture every time the system updates it and will update the light detail (direction and intensity) practically on each frame. Babylon will not actively change your scene, until you set it. Here is a default scene (Babylon.js Playground basic light estimation):
Isn’t that boring… Let’s start with adding some light. The options allow us to automatically create a light source from the data provided (Babylon.js Playground Light estimation with light):
Now, that’s more like it. We don’t need to fake a light source, it will be generated and updated for us. The light direction and color will automatically update whenever it has new data.
And since we have a directional light, we can add a shadow caster that will generate “real” shadows based on the light’s direction (Babylon.js Playground light estimation with shadows):
Oh, we’re halfway there…
We have lights, we have shadows, now comes the fun part — the environment texture. Babylon allows you to set it to be the global environment texture using a simple flag (Babylon.js Playground light estimation with shadows and environment):
See? It is that simple!
Of course, if you don’t want any of those or want the update interval to be a little lower than “every time you have one”, you can set it in the options using “cubeMapPollInterval” and “lightEstimationPollInterval”. And since reflection cubes might have some performance hit on some older devices, you can turn it off completely, if you only want your lights to affect the model.
Now it’s up to you
So there you have it! Now it is up to you to decide how to use it. Light estimation is turned on per default in your latest chrome for android (no need for flags) so you can already use it in production. Just like hit test and anchors.
The demo I have shown here simply places a few spheres on the ground, but you can have a lot of fun with real models and hit tests. Use hit test to place objects on top of other surfaces:
And try finding what model was added here 😊
I can’t wait to see what you can come up with! The class and its options are documented here — https://doc.babylonjs.com/typedoc/classes/babylon.webxrlightestimation, and we will soon have a dedicated page in the documentation with a lot of other examples.
You can contact me on our forum or twitter. My alias is always RaananW
