Messed up 3D Meshes: A case study in the limits of integer floating point

On top: The original mesh when exported. On Bottom: the mesh when scaled to 0.001 before export

3D math can be tough. In my experience tooling away at our humble glTF export plugins for 3dsMax and Maya there was a specific issue that stuck with me, always reminding me of how mindful one should be when working with floating point precision.

Let’s set the stage; a user notices that some of their meshes export with strange geometry depending on its initial scale:

Source scene configuration
The source scene when exported.

Strange, right? The initial suspicion is that some of these vertices are being optimized out for one reason or another. Looking at the scene, we can see that the particular mesh is configured to have its vertices optimized. When disabling this, we get much of the same strangeness in our output scene, just at a higher polycount:


Ok, no noticeable visual change. We can reasonably rule out the optimization algorithm causing this change in geometry…well somewhat at least. However, the fact that we have the same general shape at different polycounts is still incredibly suspicious. Looking at the optimization logic we can then get the general gist of what it does:

Our unoptimized geometry optimizer

Essentially, we search all other vertices that we have exported to the current vertex that we are extracting from Max, and replace it if we already have an equivalent vertex with matching position, UV coordinates, and other relevant attributes:

Now why is all this relevant? It will all make sense soon, I promise! The key here is in the datatypes:

See Autodesk’s documentation for the IPoint3 type. Key point being that the coordinates of these points for our vertices are stored as integers, rather than the IEEE-754 floating point while we are working with them. The key benefit of using integer floating point opposed to the standard floating point format is that its error is consistent within its range, as opposed to floating point precision loss, where one encounters greater loss in precision the farther from 0 your value is:

Imagine this square as the set of all real numbers, while the intersection points are the ones that we can represent as IEEE-754 floating point. (Shamelessly borrowed from:
The relative precision of an integer floating point space

As you can see, integer floating point is great for representing incredibly large values, not so great for representing incredibly small values compared to your range.

Again, why is this relevant? If we go back to our original issue, I originally hinted at one crucial detail!:

The answer was in front of us the whole time

The affected meshes are only the ones scaled to be virtually invisible before export! This is a common technique for animating visibility within glTF’s current capabilities, and as a result, it exposed some flaw with how we were retrieving geometry:

Raise your hand when you see it…

We were retrieving the mesh geometry in world space, then converting the geometry to local space… Yikes.

Luckily enough the fix was simple enough, and the investigation was very fun, at least when it didn’t nearly bring me to doubt my understanding of digital math. Instead of retrieving all our geometry at 0.001x scale, then scaling it up to local space, we instead should have grabbed it at the appropriate frame of reference:

Use local space, dummy.

You can always check out the latest and greatest version of the exporters here:

Nicholas Barlow — Babylon.js Team

Babylon.js: Powerful, Beautiful, Simple, Open — Web-Based 3D At Its Best.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store