Grim Grinning Textures
Welcome Foolish Mortals!
If you’ve been to a Disney Park somewhere in the world, then you’ve likely encountered one of the greatest attractions The Walt Disney Company has ever created. Though it goes by different names, Haunted Mansion, Phantom Manor, Mystic Manor, and has different cultural flavors, it has captivated the imaginations of billions of people across the globe…and I am happy to count myself among them. But every great narrative begins with inspirational back story…
Yes I’ll admit it out loud…I am a Disney Fanatic! In my home, we live and breathe all things Disney. My Disney story is not unlike many other stories. When I first went to a Disney Theme Park it was a life-altering experience to say the least. My first trip was to Disneyland in Anaheim California. It was my first exposure to the tantalizing world that exists at the intersection of art and technology. In this intersection I found robotics dancing with captivating animation, beautiful sculpting blended with puppetry, engineering intertwined with creative expression. I was hooked!
This intersection of art and technology became an obsession. I am, and have always been, drawn to it like gravity. Through my life, I have danced like a fool in this intersection.
I have boogied with electronics, waltzed with sculpting, salsa’ed with graphics, and country line danced with creation. This intersection is the most beautiful place in existence and I will live the rest of my life in it.
I am forever grateful to the Walt Disney Company and the countless men and women who have worked there, for showing me the intersection. I am captivated by their work and inspired by their creations.
So here we find ourselves…in present day…on the proverbial ‘eve’ of Halloween, a time where many of us celebrate a fascination with the dark, the scary, and the macabre. With this moment in mind, several weeks back, I set out on a journey to do 2 things:
1 - Celebrate and honor the inspiration that The Walt Disney Company has given to the world.
2- Do so, by creating something at the intersection of art and technology.
Without any further ado, I present a Halloween experience that I affectionately refer to as “Grim Grinning Textures.”
The idea was simple, and stems from one of my earliest childhood creative ponderings - I wonder if I could put MY face on one of the singing busts at Disneyland.
From here on out, we’re going to dance together in the intersection, so put on your dancing shoes!
To pull this together, I leaned on Babylon.js, one of the world’s most powerful, beautiful, simple, and open web rendering engines.
With the concept in mind, I dissected it into 3 distinctly different explorations.
1- Can I pass a video texture into a graphics shader?
2- Can I pass a Web Cam texture into a graphics shader?
3- Can I pass a GUI texture into a graphics shader?
When exploring large creations like this, I like to start with the most simple case and build from there. The first question I asked myself was can I load a video file as a Babylon.js texture, and then pass that into a shader (a program that runs on the GPU). Armed with a video file of the projected singing busts (thank you internet), I set off to find out.
It turns out that Babylon.js makes this incredibly simple. There’s already a built in construct for a “video texture.” You can read more about it here: https://doc.babylonjs.com/divingDeeper/materials/using/videoTexture
In the experience itself, check out line 150 of the code. This simple, single, wonderful line of code allows you to specify a video file and will load that video file into a texture in the Babylon scene.
Now this is where things get REALLY interesting. Babylon has a very advanced tool called the Node Material Editor.
This incredible tool allows you to connect nodes together to create advanced custom shaders. It’s like writing a program visually, and it is absolutely incredible! Oh and guess what…one of the nodes available in this tool is the “Texture” node. Do you see where we’re headed?
After some truly enjoyable experimentation and creation in the Node Material Editor, I emerged from my dance with a very specific Node Material (shader) and a delightfully inviting texture node. It really is as simple as taking the video texture and passing it right into that lovely texture node. You can find the code for this on line 138.
So after sculpting a couple of busts in Blender and carefully laying out the face UVs to match the video source, I brought them into Babylon and assigned the Node Material to them (complete with magic video texture piped into the texture node)!
So at this point, I now have the ability to display the singing bust faces on the actual geometry. Pretty cool right! But we can’t stop at this faithful digital recreation of the famous singing busts scene, I must push forward and continue the dance!
From here, it’s probably not a big mental leap to imagine that we can swap any video texture into that same material input. Right you are!
Babylon.js is a very advanced platform with an incredibly intuitive API. So it makes complete sense, that there would be easy to use functions to access common browser capabilities such as requesting access to a devices web cam!
Take a look at line 318 in the code and you can see just how easy it is to access the Web Cam. What’s REALLY awesome though, is that just like a video file, Babylon gives you the option to load the Web Cam stream as a texture in the Babylon scene. You know what this means right? Once you select the “haunt” button below one of the singing busts, the input for the shader powering that bust’s rendering is swapped from the original video source to the newly created Web Cam texture!
At this point. a LARGE smile crept across my face. While dancing in the intersection, I realized I had digitally fulfilled a dream of having my own face on one of the famous singing busts from the Haunted Mansion at Disneyland.
I wonder just how far we can take textures in Babylon.js? Could I create an intro experience that helps capture that dark and playful spirit of the ride itself?
This part of our intersection dance gets a little more complicated. Babylon.js is a web rendering engine, so you might think that it would take advantage of using commonly accepted web components. Like using HTML to create GUI elements for example. This is where you’d be pleasantly surprised. Babylon actually has it’s own GUI system, and the reason for this is very simple, unlike HMTL, in Babylon a GUI can be projected into the 3D scene itself. This concept kind of blew my mind when I first heard about it.
Armed with this knowledge I decided to create a GUI and project it onto a headstone that captures the “essence” of the ride.
By “essence” I mean some Haunted Mansion inspired text and a start button that collectively eerily disappears and reappears on the headstone. To pull this off I knew I was going to need to leverage shaders again and I tuned back to the Node Material Editor. But this time, I thought I was in trouble. How do I project a GUI onto a 3D object in the scene AND to a shader at the same time?
Well like most of the discovery in this project, it turns out that Babylon once again makes it VERY easy.
To create a GUI in Babylon, you start by specifying if you want it to be full-screen or projected onto a mesh. This was step one. If you look at line 105 in the code you can see the constructor used to make this happen. Babylon treats all GUIs as a very special kind of texture called the Advanced Dynamic Texture. To describe it simply, it is an HTML canvas that you can “draw” on, that’s loaded into the scene as a texture. Pretty brilliant right?
I bet you probably see a pattern here right? I created a new shader using the Node Material Editor that once again takes a texture as an input. Simply assigning the Advanced Dynamic GUI Texture to that input unlocked a world of possibilities, particularly the ability to add procedural noise and fade the elements of the GUI in a way that made them feel mysterious and thematic.
How’s that for magical?
Ok, one last BONUS item to cover before we part ways. Given that there’s a shader controlling the rendering of the GUI that’s projected onto the headstone. How incredible would it be if we allowed the user to “hover” over the headstone and illuminate the GUI in places where it has disappeared.
What I’m discovering is that just about anything seems possible in this incredible engine. In lines 120–126 we capture the location of the mouse, project it into the Babylon scene, figure out the relative UV location when it’s over the headstone mesh, and then pass that UV coordinate info to the shader when the mouse moves. (Sorry Mobile folks :))
And with that our journey comes to an end.
I again want to take a moment to honor the incredible influence that the Walt Disney Company has had on my life. Thank you for the spirit of innovation and imagination you have evoked in creators across the globe. In particular, I also want to take a moment to thank 3 incredible Disney Legends. If it wasn’t for the creative brilliance of Yale Gracey, Rolly Crump, and Harper Goff, the world would never have known the Haunted Mansion. The work of these magnificent creators inspires me. It has shown me the intersection of art and technology and I’ll dance there for the rest of my life.
