Fantasia: An interactive real-time generative AI installation

For the FTI Open Technology and Innovation Days hosted by Howest, we developed an interactive real-time generative AI installation.

The installation concept was inspired by the 1940s animation movie Fantasia by Walt Disney, which was based on Goethe’s The Sorcerer's Apprentice.

This poem begins with an old sorcerer departing his workshop, leaving his apprentice with chores to perform. After a while, the apprentice is tired of fetching water by hands, and he enchants a broom to do the work for him, with magic that he does not fully control. Soon, the space fills with water, and the apprentice realises that he can not stop the broom, as he is not fully capabable of controlling the magic. At the end, the old sorcerer returns and breaks the spell. The poem ends with a statement that only a master should invoke powerful spirits.

This story can be a beautiful allegory of how people are feeling with the current innovations and debate in generative AI. We translated this ‘power’ of controlling water in the Fantasia installation.

We came across this amazing showcase of real-time diffusion in Touchdesigner, using StreamdiffusionTD. Inspired by this showcase, I approached my colleague Wouter Verweirder with the following brief:

Fantasia: Conduct the Water Dance

Dive into a magical world and conduct your own Disney-inspired adventure with interactive visuals that are developed live by Generative Artificial Intelligence. Be inspired by the timeless classic Fantasia from 1940 and use your movements to control the water to the rhythm of the music. Watch as your magical powers come to life in the world around you!

To create this installation, Wouter Verweirder explored Touchdesigner, building a project flow that takes webcam input to steer the generation thropugh body tracking. This allowed us to ‘control’ the waves within the project.

While this worked well, it was not perfect to experience that you, as a person, were actually controlling waves. By including a second step, a liquid resolver, it felt that one had much more control over steering the waves. For the technical deepdive, take a look at Wouter’s blog post.

The showcase of the installation was a success. The installation was geared to a younger audience to discover and play with technology. Children interacted with the installation and were captivated for longer amounts of time, which is always a good sign. A specific highlight that kept the attention was the ability to use different prompts and allow people to steer a variety of landscapes, ranging from lego, to a flower field as if painted by van Gogh.

In the workflow, it was not possible to work with negative prompts, so it was important that inputted prompts were vetted before using - as there was no filter.

We’re excited how this can be developed further, and to explore which use-cases can employ this live visualisation technique. If you are interested in testing, the tech behind it, or developing a case study together — drop us a message.