One of the main challenges faced during the early conception of this project was the potential for latency to ruin the user experience. Fortunately, the slight delay from the physical hand input to the audible result becomes largely irrelevant once the user practices and finds the rhythm. However, while directly synthesizing sounds via math achieved superb results in terms of responsiveness, it sometimes lacked perceived authenticity and organic realism.
To bridge this gap, I initiated experiments using the Realtime Audio Variational autoEncoder, a generative artificial intelligence model designed for real-time, high-quality neural audio synthesis. This model relies on a clever two-stage training architecture that compresses complex audio into low-dimensional latent spaces, allowing it to generate stunning acoustic results with incredibly low CPU usage. These learned representations can be adjusted instantaneously to manipulate the audio signal on the fly.
Functionality of RAVE model. Caillon and Esling 2021, 2
I tested this by loading a pre-trained neural percussion model directly into the audio pipeline using a dedicated neural network module. After synthesizing the footsteps in the texture generator, the signal feeds into the encoder, gets manipulated inside the latent space, and flows back out through the decoder to produce the final audio signal.
Patch with nn~ module and percussion model
The initial results have been incredibly successful, proving that manipulating parameters in the latent spaces via gestural control feels highly intuitive and delivers pristine audio quality. Moving into the next semester, I plan to thoroughly explore the idea of training custom models on specific ground surface textures or even raw footstep recordings to completely replace the synthesized patches with genuine AI-driven acoustic realism.
Caillon, Antoine, and Philippe Esling. arxiv.org. December 15, 2021. https://doi.org/10.48550/arXiv.2111.05011 (accessed June 22, 2026).
