We recently started using Azure Pipelines to enable continuous integration builds on the Babylon.js repo. For those not familiar, continuous integration is the practice of merging developer code into the main branch on a daily basis. For Babylon.js, every pull request made to the master branch is automatically built and tested. Once all tests pass, the pull request can be safely merged into the master branch. Several pull requests from various developers get merged into Babylon.js master on a daily basis. A fast and reliable continuous integration process is an absolute necessity to ensure pull requests are merged within an acceptable time frame while maintaining good code quality in the master branch.
Here are our experiences with using Azure pipelines for continuous integration so far:
- It just works! (reliability is 100% so far)
- It is fast! (~25 min thanks to 10 parallel jobs available for free to any OSS project)
- It is more productive! (important build information directly embedded in the GitHub repo)
Per job status on pull requests
The per job status on any pull request makes it easy to identify what went wrong with a build.
Build Summary and Logs
Checking details on a failed task takes you to the build status on Azure DevOps. The Logs page on Azure DevOps provides a good overview letting you narrow down the issue at a glance.
In addition the build summary page provides a good overview for the build without having to go back and forth between Azure DevOps and GitHub.
The test results page provides a global summary of all tests without the need to drill down into thousands of lines of logs.
And of course you can always drill down into individual failing tests.
Reliable WebGL testing
The best part about using Azure pipelines is the rock solid Xvfb support. The WebGL1 and WebGL2 render tests no longer need to rely on Browser Stack making them much more deterministic. If a CI fails on one of these tests, more often than not we can safely assume it is code related rather than an infrastructure issue.
Overall, Azure pipelines has integrated really well into the team’s workflows. The only downside so far is that the overall UI isn’t very mobile friendly, so it is hard to do a quick check from a phone.
Many thanks to @PatriceVB who convinced us to try out Azure pipelines!
And of course @sebavanjs did all the work integrating the Azure pipelines infrastructure. I simply get to be the messenger chronicling our experiences with it.
Saurabh Bhatia — Babylon.js Team