Using High-Contrast, Image-Based Lighting in Babylon.js
The lighting in your 3D environment can mean be the difference between a beautiful scene and one that feels broken or forgettable to your users. With that in mind, we are always looking for ways to improve the rendering in Babylon.js so the pathway to creating beautiful 3D experiences is easy and obvious. Recently, we encountered an issue with some of the high dynamic range images (HDRI) that were being used for image-based lighting (IBL) in the engine. We often reference ray tracers as our ground truth for rendering and actively make course corrections when we find discrepancies. To illustrate the problem, consider the comparison render below.
When we designed our lighting system in Babylon.js we looked at many types of HDRI sources and designed our lighting calculations to work well with a range of image types. What we didn’t anticipate was the types of images we are seeing from sources like HDRI Haven where the dynamic range of their images regularly covers 22 exposure values (EV) for an external, sunny scene. An exposure value is either double the amount of light from the EV below it and half the amount of light from the EV above it so you can see that the range of values in an image like this is enormous. The reason is to accurately capture the amount of light coming from the sun while still exposing shadow detail in the scene correctly.
A typical 8-bit image that will be rendered on a standard, non-HDR monitor has a color range of black (0.0) to white (1.0) per channel where a 32-bit HDRI can have values that reach far beyond a value of 1.0 per channel. We want those high values in the IBL so the renderer knows how bright each light source is. When ray tracing, we can look at each visible pixel of the IBL to calculate the amount of light that is hitting any part of our mesh, which is what you see in the Arnold render above. These renders can take several minutes at minimum, depending on scene complexity, so we have to come up with an approximation system to render in real-time at 60 frames per second.
This is where spherical harmonics come into play for real-time rendering. The simplest definition for spherical harmonics is that it is a small set of color data that shows the color contribution from the environment depending on the direction a surface is facing. So if you think of a desert environment on a cloudless, sunny day the spherical harmonics would have brown tones in the directions pointing at sand and blue tones in the directions pointing at the sky. But since the data set is very small, usually nine colors to calculate the environment contribution, an IBL with areas of high contrast will cause the spherical harmonics to be biased toward severe differences in the colors in the data set and will cause harsh transitions or banding in the final render.
The banding that is seen in the render above is a by product of a spherical harmonics data set that has too much contrast in it. This is because the original HDRI has maximum values well over 360,000 on a 0.0–1.0 scale. While the only pixels in the HDRI to reach these values are those in the sun, the contrast of a 22 EV image is so large the impact of those pixels spreads around the spherical harmonics.
Seen another way the image set below is the original image, the irradiance of the image influenced by the high contrast in the sun, and the resulting nine colors that make up the spherical harmonics. Note that in the spherical harmonics, seven of the nine colors are white which accounts for the banding that is seen on the spheres in Babylon.js.
To fix the issue, the first thing we did was to introduce a clamp in our lighting calculation, preventing the maximum values present in the IBL from exceeding an amount that still worked for our lighting, even if not ideal. This new clamped version is seen below, but when compared to the original, ray-traced version we can see that there is a loss of subtle color and the intensity of the main light source is diminished. There is also a harsh edge to the highlight on the sphere which makes the render look unnatural. While this is a good step to minimize the original Babylon.js render issue which was unacceptable, there is still more we can do to bring back the subtle qualities of the ray-traced render.
The next step in our process is a manual step, but from an art standpoint taking extra care in creating your assets will pay off in the end. Open the image in your favorite image editor and make a duplicate layer of the original, high-contrast image.
Reduce the exposure of this new layer until the upper limits of the image are compressed to a more reasonable level. You will need to experiment with the amount of compression for the value range in the image to obtain the results that work best for your scene. In this case I changed the exposure until the central pixels in the sun, which started over 360,000 in each channel, were reduced to around 1,500 in each channel. That is still a lot of contrast on a 0.0–1.0 scale, but largely compressed from the original and will prevent the spherical harmonics from being too heavily influenced by extreme values.
Lastly, create a mask that shows only the brightest pixels from the exposure adjusted layer to replace the extreme-contrast pixels from the original HDRI. Now when we look at the edited image, the irradiance of the edited image, and the spherical harmonics generated from the edited image, you can see that the errors have been removed.
However, there is still one last consideration to make. When we look at a render of the edited HDRI used as the IBL for the sphere, we can see that we have returned the color contribution from the environment but have lost all the intensity from the main light in the IBL.
To combat the loss of the main light intensity, we will add a directional light and position it to align with the main light direction in the IBL. We can set the power of this light to return the intensity that we lost while still retaining all of the subtle color contribution of the IBL. This is a method is commonly used in real-time rendering to avoid needlessly biasing the spherical harmonics.
You can see that we can get very close to the ray-traced results using a little extra effort for real-time rendering. While the original image works well with a ray-traced render, we do need to compensate for the shortcuts needed to get to rendering 60 frames per second. It is worth taking the time to note with your HDRI files what the upper limits of the range happens to be. This is easily done with a color picker in your favorite image editing tool and a quick reduction of extreme values will greatly help your rendering when relying on spherical harmonics.
And if you would like to experiment with your own HDRI environments, you can quickly get up and running using this playground that I created for our testing. You can convert your chosen HDRI into your own IBL environment using a few different tools like Knald Lys or IBL Baker. Once you have the environment file created, simply point this playground to your file. Happy lighting!
Patrick Ryan — Babylon.js Team
