An experiment on projection mapping with AI-generated content
In the heart of Bruges, for the 2023 winter festival 'Wintergloed', CREATE.eu was responsible for setting up Aurora, an unique traveling light experience that featured 10 unique light installations. The installation 'Waves', being a standout attraction, marked our first journey into the realm of generative AI, an ambitious experiment to blend cutting-edge technology with artistic expression. The abstract essence of 'Waves' served as an ideal canvas for this digital adventure.
Check out our concise step-by-step breakdown of how we experimented with AI in this installation.
Proof of concept
Our first step was to develop a proof of concept, making sure our approach to this installation would have potential. We used Stable Diffusion’s “AnimateDiff” motion models for its flexibility in generating animated content. The process began with a simple yet effective workflow: using ControlNet to guide animations along the contours of the building designated for the Waves-projection. By tracing the building's facade in black and white, we created a line art guide for ControlNet.
This initial test was promising, suggesting that our approach had potential.
Harmonizing the animation with the architecture
From this first test, an early observation arose. The generated animation had the tendency to extend beyond the projection surface. To address this, we created a ComfyUI workflow, which enabled us to selectively mask the areas where the diffusion process operated. This adjustment ensured that the animations remained confined within the desired projection boundaries — to the building's exact shape.
Creating a workflow for the projection mapping
Masking the projection area
Influencing animation effects
As some first boundaries were set, making sure the animations were limited to the projection area, it was time to take a closer look within this area. While the abstract movements that were initially generated were captivating, we felt like we could take it to the next level. We looked for a way to enrich these animations, making sure the end result would be up to standards.
By rendering trails along the building’s contours and using these as inputs for a second ControlNet, we added an extra dynamic layer to the projection, giving it more depth and elevating the overall impact of the projection mapping.
Mass Generation of Content
As the workflow was established, we started experimenting. Exploring various prompts and scene durations to generate a diverse array of intriguing visuals for the final display.
Weaving a Seamless Narrative
As we wanted to show a coherent narrative, our next challenge was to seamlessly connect these different scenes. Using ComfyUI, we developed a new workflow centered around the “FILM” interpolation model.
This allowed us to smoothly transition from one scene to the next, using the FILM-generated morphing animation as a base to guide the AnimateDiff process. This hides the interpolating frames by replacing them with newly generated ones.
Final Result
In bringing our vision to life, we aimed to gradually intensify the animation. Beginning with gentle, abstract water waves transitioning into a series of winter scenes before returning to the serene start.
To complement this visual journey, we meticulously edited the timings of an ambient song, synchronizing it with key moments in the visuals to create a display of light and sound that captivated our audience.
From concept to execution, it's been a collaborative journey of experimentation and learning. One of the primary challenges lies in skillfully directing, enhancing and leveraging AI to enrich the artistic project. The iterative process of refining our approach throughout the project, allowed us to push the boundaries of what was possible. Harnessing the potential of AI-generated content demands a delicate balance of creativity and technical proficiency to make sure it integrates seamlessly with our vision.
As we conclude our exploration of projection mapping with AI-generated content, 'Waves' stands as a testament to innovation and creativity.
