MuGeVI: A Multi-Functional Gesture-Controlled Virtual Instrument
When browsing through the project and paper titles, this one instantly took my attention, as it involves a similar concept to my own Virtual Foley Stage project, in terms of interaction. Therefore this review comes in handy to investigate the approach used by the authors and compare with my own ideas and processes.
The paper proposes an interactive multi-functional virtual instrument based on computer vision. Gestural Hand recognition via the MediaPipe Hand Landmarker will ensure real time music creation in Max/Msp. There are 4 different modes to manipulate the audio signal, which enables use in either standalone or accompanying applications.
In the first stage, it uses the OpenCV library to display real-time video content from the player. This will then be monitored and analyzed by the Hand Landmarker system to gather gestural movement data, which will be mapped to different functions and tasks within the audio environment. Essentially, this process is the same approach I want to accomplish in my project and seems to be ideal.
The next step includes sending the acquired information from the MediaPipe Pipeline to the audio environment via OSC messages. While the paper’s chosen audio pipeline is Max/Msp, my own project will work within plugdata, to keep all angles of the project accessible and open-source.
From an audio perspective, the reviewed project is focused on creating an interface to play and compose music. This can be done through playing specific notes or chords, while the gesture movement gets mapped to specific musical expressions. As my project pursues more a sound effect and sound design goal, these mapping strategies do not seem to be as useful. I will have to implement a system with more control in the time domain, rather than within the pitch, as this is the most crucial part in foley work.
Even if the paper has not brought a lot of new insights to me regarding some details in my own project, it is good to see other work with a similar idea turning out well. Ultimately, my main concerns about the playability and latency issues, do not seem to have been experienced or considered in the paper. This leaves me on a positive note and anticipation for my own device.
