Our Journey into the Module World.

In the Babylon.js team we are developing an open source WebGL based 3D engine empowering our users to easily create amazing 3D web experiences that you can discover on our website.

As we have started during the early days of TypeScript, we chose to rely on Internal Modules now called Namespaces and we were quite happy until a recent conversation:

User: Guyz, I am using npm like everybody these days and you only have a CDN, could you create a npm package?

Me: Thank you for your message, we have just published our package.

User: I cannot import from “babylonjs”… It seems that your lib does not support modules?

Me: Right, we should have thought about it, we are extremely sorry, you’ll have it in 2 hours.

Me: (1 hour later): Here you go, we deployed a UMD module. Now, everybody can use it, thanks for the valuable feedback.

User: Thx Bro, you rock :-)

Seems perfect isn’t it? but another thread started:

User: Can we take advantage of modern workflows and tree shaking? It would be great to have an ES6 build.

Me (this was 2 months ago): No problem, I have read a blog about it and I’ll just change our code a bit to create a new target for ES6 in our build.

Keeping in mind our Motto in the team is to Never Break Backward Compatibility (except for Bugs), we needed a transition path where we could still provide our ES5 UMD npm package and a new npm ES6 package for modern development workflows.

2 months later, I am still working on the latest details of the migration and wanted to highlight the difficulties we encountered along the migration journey. I am pretty sure I missed some of the available tools or techniques as it sounds overly complex for a common task, so please, do not hesitate to shout out if we could have made it easier. The code is currently available on the libToModules branch under heavy testing and should be available soon on npm.

Let’s start the journey with our first stop being the required code changes.

Stop 1: From Namespace to Module in ts files

As we have about 400 TS files and we are pragmatic developers, it sounds possible to automate that task. We created a tool to remove the declare Namespace BABYLON { header in each file and the corresponding }. Then, we went with some friends from the community over all the files to resolve the missing import in VSCode (Big thanks to them).

Everything looked OK so far except for one of the pattern we were using quite often: Interface Augmentation of classes. Relying on the power of TypeScript namespaces, we were able to define a class like Scene and used inside it all the other available types from the project like EffectLayer for instance. This is the “beauty and the trap” cause as soon as you want a more modular and decoupled framework, this implies that Scene should not know anything about EffectLayer and not run extra code unless you actually useEffectLayer. As code speaks more than words (at least mine);

From embedded functionalities in Scene.ts file:

We migrated all the effect layer related code to a new EffectLayerComponent file:

We know it might sound over kill at first, but we have been running this for 6 months without drawback. We have achieved our backward compatibility goals (same public APIs) and introduced lose coupling of our main components.

Unfortunately, this pattern does not build anymore after the migration.

Stop 2: Module Augmentation

Thanks to TypeScript and their documentation, it looked like we simply needed to convert our interface augmentation to module augmentation.

We then replaced the previous code in EffectLayerComponent.ts:

This was straightforward and soon, our code was building again with the tsc command line. So, it was time to use our gulp build task which sadly failed on first launch as we could not anymore create a UMD bundle. Indeed, it is not possible to create a UMD bundle (with global variable fallback) directly from TypeScript. Only amd and system support the outFile (closest flavor from bundle) option but unfortunately, we need a UMD bundle to not break backward compatibility. It seems that we have to rely on Extra Tooling as mentioned here...

Stop 3: Adding More tooling and tooling for the tooling

We were definitely in need of a bundler and we quickly figured webpack would be the most frequently used solution in the community (at the time of this post). This was enough to push us to use it to ensure we rely on the same tools than the community. So, here we were, modifying our gulp process to bundle everything in webpack to generate a UMD lib. We could still rely on tsc directly to create our ES6 target for most of the modules as they purely are TypeScript.

After lots of fiddling with paths and plugins we ended up with webpack-stream for the gulp integration, ts-loader to support TypeScript in Webpack and hard-source-webpack-plugin to improve our build time which went up from 30 seconds to 2 minutes after the switch (“It is more modern they said… It is a lot more work I thought…”)

Great, it was time to test again and Whhhhhhhhaaaaaattttttttt ??? the build process exploded the default Node.js memory limit of 1.7Gb on a 64-bit install. Reading about nodejs related memory issues it seems a common practice is to increase the memory limit with the --max-old-space-size option but is that all we can do?

Stop 4: Memory Budget

Looking at the Github issues around Webpack + TsLoader it sounds like we are not the only ones with those kinds of issues, so I was willing to give it a shot to help the community. I began to check by profiling our node process if this was easily solvable but unfortunately this was related to some ts-loader caching mechanism and several webpack processes we run during the build.

Due to time pressure (do not forget we thought it was a short migration task), I then tried another famous TypeScript plugin for Webpack: awesome-TypeScript-loader and it worked directly after switching again a couple of paths here and there. Time to try again, it only failed at run time which was a huge progress. Unfortunately, we forgot to migrate our Shader custom build process. Our shaders are stored in files who have the .fx extension and contain the GLSL code as well as custom #include directives to modularize them. They have not been handled by the bundler so far and they were “gulped” manually in our js before.

Digging a bit, I found a couple npm packages dealing with the bundling of WebGL shaders (obviously there are packages for everything on npm) but they do not work well for us as they would have forced our ES6 package users to take a dependency on this tooling as well and it was not necessarily available for all major bundlers.

Stop 5: Shaders

Simple, let’s create a quick node script transforming our .fx in .ts by wrapping the glsl code in string and allowing import: ProcessShaders. The custom script was easy enough to hack together. We even converted our shader #include mechanism in TypeScript import statement to allow better granularity in the tree shaking. This is the Grail of tree shaking for our shaders and everything is now done… we thought.

I ran again and have a straight runtime failure again undefined is not having the property... What the heck is wrong with me ???

After disabling the JavaScript source map support in the browser and debugging the webpack loader generated code, it looked like our objects were not defined.

Stop 6: Circular Dependencies

Despite ES6 support for circular dependencies, it is not the case in most bundlers and our community favorite one: Webpack. Off we went, installing another tool to help listing all of the project circular dependencies: DependencyCruiser. First run worked (Hooray !!!) and we figured we have about 700 circular dependencies in the code base (Boooooooo !!!). Great I thought, there should be some tools available to help fixing this. After surfing the web for hours always ending up on cute cat videos, I contacted the TypeScript team. We understood quickly that we simply had to reorganize some code. I then went manually fixing our dependencies by relying on mainly 3 techniques:

· Extracting code in separate files.

· Changing the import statement in import type to help ensure the build process does not emit unnecessary js imports (not sure why I had to do this in some cases as it should be the default case with TS elision but it was anyway helping me to quickly catch type vs runtime dependency by simply looking at the import block).

· Adding some “sort of” Dependency Injection here and there.

The code finally ran !!!

But yes, there is always one but, after a couple of tests I figured that we had lost backward compatibility with our previous Namespace based build (and I could be murdered for this). Actually, we previously relied on nested namespace in a couple of places and the Webpack generated UMD root is now BABYLON so all the framework exports are available under BABYLON but nested namespaces like BABYLON.DEBUG are gone.

Stop 7: Nested Namespace

We chose here to follow one approach made by a community member where basically we have a legacy entry point module which is re-exposing all the index members in the Namespace they belonged before. It is a bit ugly but pragmatic, back compatible, it works and as we have a lot of code to migrate, maybe nobody will ever notice.

All good now, even if our UMD bundle is bigger than before due to all the extra module added code (funny cause we are trying to make it slim again). <mode sarcasm="on">Keep in mind it is more modern.</mode> I was able to finally fire a new test and it looked all green but unfortunately we completely forgot our declaration files. As we need 2 flavors (UMD and ES6) we thought that we could use TypeScript to generate our bundled declaration as well as ES6 declaration. It works amazingly well for our unbundled ES6 package but as soon as we dug in how to create the declaration files for the bundle it does not sound obvious anymore.

Stop 8: Declaration Files

At this point I started to cry as much as I wanted to make the modules cry but I thought I couldn’t simply give up on that. I am currently watching Vikings and I have got too much honor and pride… or I simply love my job and want to keep it. As TypeScript does not want to deal with bundle, we found a lot of npm packages doing just that (How weird is that? there are packages for this and I even found one making nyancat). Looking at their code they all semt to be derived from dts-bundle where the main readme highlights that the developer was in the same situation than us but it is unmaintained, experimental, and to use with care… As we enjoy the risk, we tried and it failed due to only one unsupported feature: Module Augmentation. Unfortunately, we are relying on this to split some part of the framework and allow tree shaking. The solution again: create our own tool.

Figuring TSC was the best placed to emit the declarations, we used an AMD build of only the declaration that we manually process (regex FTW) to create all of our DTS flavors as we want them… Funny side story gulp-TypeScript does not support the --emitDelclarationOnly tag so we had to rely on a shell command here. This is a part I would like to change to hook in the tsServer and emit the .d.ts from the generated AST instead but I am waiting for this process to fail in order to do it (I’ll invest only if really needed). Happy to be finally in good shape with all of our own process: Shaders, Declaration, Build, Stitching I decided to start a new build. This time everything was green on Windows so I pushed to my branch of the babylon repo.

Travis Failed but Whhhhhhhhhhhhhhyyyyyyyyyyyyyyyyyyyy ?

Stop 9: Cross Plat Build

Travis builds run on Linux, I work on Windows, so I ran the build on MacOS (it usually is pretty close from linux and we already had issues with case sensitivity of paths before). Fixing a couple of issues in the new build process, I was confident, but Travis was still failing. After 2 hours of fail and retry with a reference not found TypeScript exception on travis only, I had to install Linux to give it a try. It all ended up being Module Augmentation which is case sensitive on Linux but not MacOS nor Windows. This sounds simple enough but was still an interesting waste of my weekend.

All Fixed, I now finally had a green Travis Build for the UMD package. I started working on the ES6 version. I even thought I could use it in modern browsers during development, but I guess you are now used to the following: It miserably failed…

Stop 10: ES6 Flavor

Native Browser support for ES6 is strict in terms of import resolution and unfortunately TypeScript transpiled JavaScript is more “node friendly”. The funny part is TypeScript, Whatwg and Node would need to find an agreement and a strong norm to ensure full compliance. But in the end, this is the users like us who need to find workarounds. So, that was it, I began to know the solution, I created my own tool to adapt the generated import path to be browser and bundler compatibles.

Finally, I have almost all what I need and the rest of the migration is simply some rewiring in our external libraries. This is probably one of the weirdest tasks I’ve had to do in my career as every single step sounds easy but fails with a new unexpected issue forcing us to create or add new tools.

So with a build time increased by 100%, a way more complex build system and a lot of new tools to maintain we have finally reached our destination and can start testing on real world cases our new build.

As this sounds overly complex, I am sure I have missed some part of the picture or some magic tools (I tried npm install magictools but this package does not exist) which could solve all of our issues. Finally, my open questions are:

Did I completely miss the point?

How are others doing it (in small teams)?

Are people allowing huge breaking changes in order to migrate to ES6?

What I wish for Christmas is an easy to follow process to create js framework for browsers (targeting different flavors UMD, ES6, Native Browser ES6) with their attached artifacts (.map, .d.ts) directly from TypeScript.