Messed up 3D Meshes: A case study in the limits of integer floating point
3D math can be tough. In my experience tooling away at our humble glTF export plugins for 3dsMax and Maya there was a specific issue that stuck with me, always reminding me of how mindful one should be when working with floating point precision.
Let’s set the stage; a user notices that some of their meshes export with strange geometry depending on its initial scale:
Strange, right? The initial suspicion is that some of these vertices are being optimized out for one reason or another. Looking at the scene, we can see that the particular mesh is configured to have its vertices optimized. When disabling this, we get much of the same strangeness in our output scene, just at a higher polycount:
Ok, no noticeable visual change. We can reasonably rule out the optimization algorithm causing this change in geometry…well somewhat at least. However, the fact that we have the same general shape at different polycounts is still incredibly suspicious. Looking at the optimization logic we can then get the general gist of what it does:
Essentially, we search all other vertices that we have exported to the current vertex that we are extracting from Max, and replace it if we already have an equivalent vertex with matching position, UV coordinates, and other relevant attributes:
Now why is all this relevant? It will all make sense soon, I promise! The key here is in the datatypes:
See Autodesk’s documentation for the IPoint3 type. Key point being that the coordinates of these points for our vertices are stored as integers, rather than the IEEE-754 floating point while we are working with them. The key benefit of using integer floating point opposed to the standard floating point format is that its error is consistent within its range, as opposed to floating point precision loss, where one encounters greater loss in precision the farther from 0 your value is:
As you can see, integer floating point is great for representing incredibly large values, not so great for representing incredibly small values compared to your range.
Again, why is this relevant? If we go back to our original issue, I originally hinted at one crucial detail!:
The affected meshes are only the ones scaled to be virtually invisible before export! This is a common technique for animating visibility within glTF’s current capabilities, and as a result, it exposed some flaw with how we were retrieving geometry:
We were retrieving the mesh geometry in world space, then converting the geometry to local space… Yikes.
Luckily enough the fix was simple enough, and the investigation was very fun, at least when it didn’t nearly bring me to doubt my understanding of digital math. Instead of retrieving all our geometry at 0.001x scale, then scaling it up to local space, we instead should have grabbed it at the appropriate frame of reference:
You can always check out the latest and greatest version of the exporters here:
https://github.com/BabylonJS/Exporters/releases
Nicholas Barlow — Babylon.js Team
