Author: philipp.fasching

MuGeVI: A Multi-Functional Gesture-Controlled Virtual Instrument

When browsing through the project and paper titles, this one instantly took my attention, as it involves a similar concept to my own Virtual Foley Stage project, in terms of interaction. Therefore this review comes in handy to investigate the approach used by the authors and compare with my own ideas and processes.

The paper proposes an interactive multi-functional virtual instrument based on computer vision. Gestural Hand recognition via the MediaPipe Hand Landmarker will ensure real time music creation in Max/Msp. There are 4 different modes to manipulate the audio signal, which enables use in either standalone or accompanying applications.

In the first stage, it uses the OpenCV library to display real-time video content from the player. This will then be monitored and analyzed by the Hand Landmarker system to gather gestural movement data, which will be mapped to different functions and tasks within the audio environment. Essentially, this process is the same approach I want to accomplish in my project and seems to be ideal.

The next step includes sending the acquired information from the MediaPipe Pipeline to the audio environment via OSC messages. While the paper’s chosen audio pipeline is Max/Msp, my own project will work within plugdata, to keep all angles of the project accessible and open-source.

From an audio perspective, the reviewed project is focused on creating an interface to play and compose music. This can be done through playing specific notes or chords, while the gesture movement gets mapped to specific musical expressions. As my project pursues more a sound effect and sound design goal, these mapping strategies do not seem to be as useful. I will have to implement a system with more control in the time domain, rather than within the pitch, as this is the most crucial part in foley work.

Even if the paper has not brought a lot of new insights to me regarding some details in my own project, it is good to see other work with a similar idea turning out well. Ultimately, my main concerns about the playability and latency issues, do not seem to have been experienced or considered in the paper. This leaves me on a positive note and anticipation for my own device.

– First ideas and concepts

The research of project ideas for my personal project took a little longer than expected. For this project I was planning to learn new skills, as well as create something valuable to present and use for my career aspirations after the masters course. As one of my main interests lies within sound design and foley for linear and interactive media, the project should be within that area.

When thinking about foley work, it seems to me that the actual foley recordings have not really changed a lot since its invention in the 1920s. Of course equipment and technology shaped its development, but in essence the performance and recording of probs is still the same. This sparked an idea of bringing the foley performance into the digital realm, while still keeping the haptic feeling and body performance in the real world.

I came across an interesting project, an application to trigger sounds with a leap motion controller, which seems to have died off at some point and is not supported anymore. I was not really aware of the concept of leap motion to trigger sounds, but I already stumbled upon some gesture control via gloves controllers. A well known example would be the MiMU Gloves demonstrated by Imogen Heap in her NPR Music Tiny Desk Concert from 2019. In that performance she controls parameters of a sampler and multi effect application with hand gestures live while singing on top of it. The main function of the gloves in this performance is to create sound scapes, pads and rhythm. It seems to be used more like a backing instrument in combination with the artist’s voice.

As there are already applications to control foley and sound effects playback via audio input, for example human voice, I had an idea to use some kind of sensors to trigger and control foley sounds via motions and gestures. This could happen through some kind of gloves or motion capture via a camera, but is not only bound to that. Bringing this idea into my project, there could be an option to control the amplitude, pitch and playback speed of multiple samples simultaneously, as well as seamlessly blend between them. That would be one way to get different textures from multiple recordings and create something new out of it. As an option there could also be a granular synthesis option to get different sounds, depending on the situation and sonic vision. With the motion and gesture control this seems like an interesting and interactive way to play a sampler like a physical instrument.

For my project, the main goal would be to develop a software which could take gesture or motion control inputs and control certain playback parameters and effects to have a virtual foley stage controlled via physical movement. For the controller input I will research available controllers at FH Joanneum or IEM, as developing and building such a unit would go beyond the scope of this project. I still need to figure out many key factors involved in this project, but this seems like the direction I want to head towards.

The next steps would involve research on the development platform to allow this as well as how to incorporate the data output of the motion tool to realise the controls in the application. It would be also necessary to determine the scope of the application and what the prototype should be able to perform. As foley and sound effects cover a broad sonic landscape, concentrating on one specific type would make the definition of goals more approachable and lead to a quicker implementation of specific targets.

– A sound design journey

My musical education started with my drum practice lessons when I was eight years old. Prior to that I had a little obsession with the handheld tape recorder, where I remember running through the house and recording everything I came across. At some point during my teens I swapped out my drumsticks with two record players and I started to mix vinyl records. This newfound artform of combining and merging tracks together to create a sonic experience and adventure has completely taken over most of my attention for years to come. During that time I also had my first ventures into producing music digitally via Ableton Live. This experience was at some point a little dampened by my then lack of technical abilities and expertise in the mixing and mastering field. This led to a newborn curiosity in the more technical areas of music production, especially in the art of disc mastering and lacquer cutting. It all started with the question ‘Why does this record sound so different compared to xyz’. To deepen my knowledge and expertise I invested in a desktop cutting lathe system for one-off and small run vinyl record production. This was followed by years of learning all the ins and outs of this cutting system, groove geometry and disc mastering in general. And finally ended up taking over the cutting room in a vinyl record pressing plant just outside of London, UK for the last 3 years.

Music and sound, to me, induces the strongest feelings and excitement out of any sensoric system in the body. There is nothing as beautiful as the sensation that can be experienced in music and sound from my perspective. As a drummer from heart, rhythm and groove is something that needs to be felt and experienced, rather than explained and understood. Nothing can connect and unite people across the planet and various backgrounds like music does. It is such a vital tool to society and has fuelled various movements and revolutions alike. The inspiration one can get through music is to me personally unique compared to other art forms. I also appreciate the story aspect behind a full album much more than just listening to single tracks and songs. The same is going for DJ sets and the intentional compilation of songs to create experiences, which was the reason I was so captivated when starting out playing vinyl records. 

My prior interest in sound design has always been more from a sole music perspective. It was only about a couple years ago when I discovered some field recordings and the artist speaking about their craft. The way this topic was presented was so captivating, I immediately knew that I wanted to get involved as well. Since then I have experimented with different types of field recorders and microphones, from shotgun, ultrasonics and contact microphones. The idea of capturing natural sounds and mangling them to create fascinating soundscapes was always appealing to me. I always experimented with sounds and sketched ideas, more than actually arranging these into more finalized works. Therefore I am really interested in collaborating with people from different disciplines and creating interesting projects throughout my studies. When making the decision to further deepen my knowledge in the sound design field, but moreover get a better understanding of all the opportunities and challenges, it was soon clear to me that the course at FH & KUG would be a great fit.

For my creative work I would envision sound design to be the main anchor, as a supporting creative outlet for multi disciplinary media or in a standalone position. The main role should always be to contribute to a joint, well balanced experience. Without sound design I feel things lack emotion and a certain connection to the media. This is why it should always encourage a bigger goal and support the cause. Sound design should therefore be a well planned and calculated stage at the project, with input and feedback from most stakeholders. It will probably not be the easiest task to convince people about the importance of a well structured approach, but in my work and work environment I want to enforce this as best as possible. 

Within the next two years I want to try out everything I can and see what kind of area is the most interesting to specialize and focus on in my future. I am very open to dive into the fields of interactive and linear media, but also have a hunger to learn more about the actual sound design process of a physical product and some material science behind it. In the craft itself I am trying to learn and soak up all the skills that would need to be necessary to achieve my goals. I am looking forward to learning more about storytelling and dramaturgy in general, which will strengthen my understanding of the topic, as well as the ability to work alongside a team and the internal communication that comes with that. My mission is to fully commit to every project with my passion for sound design by creating meaningful experiences. I want to find my niche, where I can then develop my personal style and skills to be sought after.

Specifically interesting would be to work in audio post production for a short movie or animation film. I want to learn the basics of mixing sound for media in dolby atmos or other surround formats. Within these projects I also want to focus on my foley skills and work. This would be both a creative and technical thrilling experience. Besides that I am also looking forward to designing audio assets for interactive media, like a video game. I do like the approach and process to this, especially with all the variations one might do on similar sounds. Another fascinating skill to learn would be the implementation of these sounds to create a realistic sonic environment in an animated world. The last major topic I want to engage in would be product sound design. I am inspired by how one can shape materials and form an audibly pleasing experience that creates a stronger emotional connection to an object.

Ultimately I am super excited to have taken this path and direction, within a field where my creative needs and my technical interests are both met complementary. I cannot wait to create holistic and immersive experiences to enrich and inspire people around the world. Sound is both my language and tool.