by philipp.fasching -

MuGeVI: A Multi-Functional Gesture-Controlled Virtual Instrument – NIME Paper Review

MuGeVI: A Multi-Functional Gesture-Controlled Virtual Instrument

When browsing through the project and paper titles, this one instantly took my attention, as it involves a similar concept to my own Virtual Foley Stage project, in terms of interaction. Therefore this review comes in handy to investigate the approach used by the authors and compare with my own ideas and processes.

The paper proposes an interactive multi-functional virtual instrument based on computer vision. Gestural Hand recognition via the MediaPipe Hand Landmarker will ensure real time music creation in Max/Msp. There are 4 different modes to manipulate the audio signal, which enables use in either standalone or accompanying applications.

In the first stage, it uses the OpenCV library to display real-time video content from the player. This will then be monitored and analyzed by the Hand Landmarker system to gather gestural movement data, which will be mapped to different functions and tasks within the audio environment. Essentially, this process is the same approach I want to accomplish in my project and seems to be ideal.

The next step includes sending the acquired information from the MediaPipe Pipeline to the audio environment via OSC messages. While the paper’s chosen audio pipeline is Max/Msp, my own project will work within plugdata, to keep all angles of the project accessible and open-source.

From an audio perspective, the reviewed project is focused on creating an interface to play and compose music. This can be done through playing specific notes or chords, while the gesture movement gets mapped to specific musical expressions. As my project pursues more a sound effect and sound design goal, these mapping strategies do not seem to be as useful. I will have to implement a system with more control in the time domain, rather than within the pitch, as this is the most crucial part in foley work.

Even if the paper has not brought a lot of new insights to me regarding some details in my own project, it is good to see other work with a similar idea turning out well. Ultimately, my main concerns about the playability and latency issues, do not seem to have been experienced or considered in the paper. This leaves me on a positive note and anticipation for my own device.

by julia.trummer -

Interaction in Sounddesign

For this first blog post, I had to step a bit outside my comfort zone because we’ve started collaborating with sound designers on a music interface. That alone is already an interesting project. For the first blog post we had to research on nime.com and I came across a study about agency and creativity in musical interaction for people living with dementia and cognitive decline. I find this topic really interesting, especially since it connects in some ways to what I’m considering exploring in my master’s thesis.

Bild von jotoya auf Pixabay

Agency and Creativity in Musical Interaction for those living with Dementia and Cognitive Decline

Dementia is an umbrella term for a range of progressive conditions that affect the brain. These conditions can cause challenges with memory, problem solving, cognitive function and decision making. For people living with dementia, musical interventions have been shown to support important aspects of life, such as the sense of self. Sustained engagement with music can have a positive impact despite these challenges.

In this context, agency refers to the sense of control or ownership an individual feels over their actions and the resulting consequences. It describes the experience of being the initiator of one’s actions rather than just responding to external direction.

For people living with dementia, agency is often considered to be diminished. In research, dementia is frequently approached through a biomedical deficit model that focuses primarily on the skills and memories a person has lost. This perspective can lead to the assumption that because language and memory are impaired, agency must also be lost. However, this connection is often taken for granted rather than critically examined.

As a consequence, people living with dementia are frequently viewed as passive participants in therapeutic activities and are often expected to engage only in relatively basic tasks. In the study referenced, for example, participants were limited to playing simple instruments such as percussion while following the lead of experts. This setup reflects and reinforces the assumption that their role is primarily responsive rather than self-directed.

Biomedical deficit model

The biomedical deficit model is a framework commonly used in dementia research that focuses primarily on the skills lost by individuals and the tasks they are no longer able to achieve. This model prioritizes the identification of cognitive impairments, such as challenges with memory, language and problem-solving.

This paper proposed and tested a procedural music platform called the “SliderBox”, which was specifically created for people living with dementia. The goal of the project was to allow people with dementia to go beyond basic interactions to create sound and provide tools that facilitate unguided musical experiences and enabling them to actively participate music activities.

Source: J. Pigrem, J. Christensen, A. McPherson, R. Timmers, L. de Witte, and J. MacRitchie

The Hardware: The SliderBox is an accessible MIDI controller made of wood, with eight analogue sliders and eight push-buttons. It provides multi-modal feedback through LED light strips and buttons to help guide the user.

Conclusion

Some participants were struggling when there were more then two possible actions. This also directly related to the engagement, whereas less people would engage with the prototype, when it had to many possible actions.

The researchers also observed that the SliderBox had a high engagement and lack of negative behaviors, showing the potential for those platforms.

In this experiment concludes that it is absolutely possible to facilitate engaging musical interactions that also foster agency and creativity for those with cognitive decline.

Sources

[1] J. Pigrem, J. Christensen, A. McPherson, R. Timmers, L. de Witte, and J. MacRitchie, ‘Agency and Creativity in Musical Interaction for those living with Dementia and Cognitive Decline’, in Proceedings of the International Conference on New Interfaces for Musical Expression, 2024, pp. 315–323.

by meric.oeztuerk -

Crossing the Bridges

“Crossing the bridges” can refer to a specific film, a musical album, a metaphor for spiritual or professional journeys, or a proverb about not worrying about future problems. The meaning depends entirely on the context, such as a 1992 drama film, a 2013 film about returning to a village, a 2005 documentary about Istanbul, a musical album, or the idiom “don’t cross that bridge until you come to it”

 My name is Meriç, and I’m a sound designer, music producer, and electronic dance music performer  known as numeric.

 I started making electronic music about eight-ten years ago, and over time it turned into something much deeper a lifelong search for sound, emotion, and space.  

Music and sound have always been central to my life. This passion led me to explore questions like how to create sound and arrange electronic music compositions. This curiosity drove me to research more, investigate deeply, and develop a passion for music and sound. I started my journey as a DJ in 2014, performing at various venues. Through this experience, I realized the need to create compositions that could convey my stories through sound. Over the years, I have immersed myself in music composition and production, constantly striving to enhance my sound and musical quality. I have learned sound design, production, effect techniques, mixing techniques, arranging, and experimented with various hardware/software synths.

My passion for music, especially electronic music composition, led me to pursue a Master’s degree in Music at Istanbul Technical University . During my studies, I honed my skills in sound design, mixing, and mastering techniques, solidifying my desire to build a career in the sound design area. My time at ITU provided a rich exploration of the audio realm. I deepened my understanding of musical structure and harmony through music theory courses and explored sound manipulation through audio programming . I gained insights into contemporary music trends and techniques through my coursework. I also had to meet with great professors, artists and well minded fellow students.And I think most important I discovered how interesting and at the same time beautiful to research what you are passionate about .

 Sound design, for me, is both a craft and an inquiry. I spend countless hours refining my techniques, creating unique pieces and constructing custom systems for performance and media art. This personal research runs parallel to my artistic output, which lies at the intersection of electronic music, experimental audio, and immersive experience.I’m fascinated by how sound behaves in space, how it moves around the listener, and how technology can make those experiences more alive.Studying sound design gives me the tools to connect artistic ideas with technical reality.It helps me understand how sound works, but also how to use it to create emotion, meaning, and atmosphere.In my creative work, sound design is not separate from music Its the biggest part of the music.It’s the texture, the emotion, a tool which shapes a raw material.I see it as a way to build worlds, whether in a live performance, movie, theatre  installation, or interactive setting or advertisement. In the future, I want to continue my journey in sound design on both an academic and an artistic level, developing my practice professionally through research, creative projects, and collaborations.In the next two semesters, I would like to focus on the intersection of creative sound design, spatial audio, and intelligent systems .During my previous master’s studies, I worked on projects related to immersive and 3D sound, and that’s where my real passion for this field began to grow.

That experience deeply shaped my artistic direction and was one of the main reasons I decided to join the Sound Design program at IEM, a place that, in my opinion, holds a tremendous position in the field of immersive sound.

Now, I want to take this interest to a much higher level, both creatively and conceptually.

 In the coming semesters, I plan to explore how immersive sound can be used to create emotionally powerful and spatially engaging experiences, bridging artistic expression and technological research.

by philipp.fasching -

Motion/Gesture tool for virtual foley and sound design

– First ideas and concepts

The research of project ideas for my personal project took a little longer than expected. For this project I was planning to learn new skills, as well as create something valuable to present and use for my career aspirations after the masters course. As one of my main interests lies within sound design and foley for linear and interactive media, the project should be within that area.

When thinking about foley work, it seems to me that the actual foley recordings have not really changed a lot since its invention in the 1920s. Of course equipment and technology shaped its development, but in essence the performance and recording of probs is still the same. This sparked an idea of bringing the foley performance into the digital realm, while still keeping the haptic feeling and body performance in the real world.

I came across an interesting project, an application to trigger sounds with a leap motion controller, which seems to have died off at some point and is not supported anymore. I was not really aware of the concept of leap motion to trigger sounds, but I already stumbled upon some gesture control via gloves controllers. A well known example would be the MiMU Gloves demonstrated by Imogen Heap in her NPR Music Tiny Desk Concert from 2019. In that performance she controls parameters of a sampler and multi effect application with hand gestures live while singing on top of it. The main function of the gloves in this performance is to create sound scapes, pads and rhythm. It seems to be used more like a backing instrument in combination with the artist’s voice.

As there are already applications to control foley and sound effects playback via audio input, for example human voice, I had an idea to use some kind of sensors to trigger and control foley sounds via motions and gestures. This could happen through some kind of gloves or motion capture via a camera, but is not only bound to that. Bringing this idea into my project, there could be an option to control the amplitude, pitch and playback speed of multiple samples simultaneously, as well as seamlessly blend between them. That would be one way to get different textures from multiple recordings and create something new out of it. As an option there could also be a granular synthesis option to get different sounds, depending on the situation and sonic vision. With the motion and gesture control this seems like an interesting and interactive way to play a sampler like a physical instrument.

For my project, the main goal would be to develop a software which could take gesture or motion control inputs and control certain playback parameters and effects to have a virtual foley stage controlled via physical movement. For the controller input I will research available controllers at FH Joanneum or IEM, as developing and building such a unit would go beyond the scope of this project. I still need to figure out many key factors involved in this project, but this seems like the direction I want to head towards.

The next steps would involve research on the development platform to allow this as well as how to incorporate the data output of the motion tool to realise the controls in the application. It would be also necessary to determine the scope of the application and what the prototype should be able to perform. As foley and sound effects cover a broad sonic landscape, concentrating on one specific type would make the definition of goals more approachable and lead to a quicker implementation of specific targets.

by philipp.fasching -

Sound is both my language and tool

– A sound design journey

My musical education started with my drum practice lessons when I was eight years old. Prior to that I had a little obsession with the handheld tape recorder, where I remember running through the house and recording everything I came across. At some point during my teens I swapped out my drumsticks with two record players and I started to mix vinyl records. This newfound artform of combining and merging tracks together to create a sonic experience and adventure has completely taken over most of my attention for years to come. During that time I also had my first ventures into producing music digitally via Ableton Live. This experience was at some point a little dampened by my then lack of technical abilities and expertise in the mixing and mastering field. This led to a newborn curiosity in the more technical areas of music production, especially in the art of disc mastering and lacquer cutting. It all started with the question ‘Why does this record sound so different compared to xyz’. To deepen my knowledge and expertise I invested in a desktop cutting lathe system for one-off and small run vinyl record production. This was followed by years of learning all the ins and outs of this cutting system, groove geometry and disc mastering in general. And finally ended up taking over the cutting room in a vinyl record pressing plant just outside of London, UK for the last 3 years.

Music and sound, to me, induces the strongest feelings and excitement out of any sensoric system in the body. There is nothing as beautiful as the sensation that can be experienced in music and sound from my perspective. As a drummer from heart, rhythm and groove is something that needs to be felt and experienced, rather than explained and understood. Nothing can connect and unite people across the planet and various backgrounds like music does. It is such a vital tool to society and has fuelled various movements and revolutions alike. The inspiration one can get through music is to me personally unique compared to other art forms. I also appreciate the story aspect behind a full album much more than just listening to single tracks and songs. The same is going for DJ sets and the intentional compilation of songs to create experiences, which was the reason I was so captivated when starting out playing vinyl records. 

My prior interest in sound design has always been more from a sole music perspective. It was only about a couple years ago when I discovered some field recordings and the artist speaking about their craft. The way this topic was presented was so captivating, I immediately knew that I wanted to get involved as well. Since then I have experimented with different types of field recorders and microphones, from shotgun, ultrasonics and contact microphones. The idea of capturing natural sounds and mangling them to create fascinating soundscapes was always appealing to me. I always experimented with sounds and sketched ideas, more than actually arranging these into more finalized works. Therefore I am really interested in collaborating with people from different disciplines and creating interesting projects throughout my studies. When making the decision to further deepen my knowledge in the sound design field, but moreover get a better understanding of all the opportunities and challenges, it was soon clear to me that the course at FH & KUG would be a great fit.

For my creative work I would envision sound design to be the main anchor, as a supporting creative outlet for multi disciplinary media or in a standalone position. The main role should always be to contribute to a joint, well balanced experience. Without sound design I feel things lack emotion and a certain connection to the media. This is why it should always encourage a bigger goal and support the cause. Sound design should therefore be a well planned and calculated stage at the project, with input and feedback from most stakeholders. It will probably not be the easiest task to convince people about the importance of a well structured approach, but in my work and work environment I want to enforce this as best as possible. 

Within the next two years I want to try out everything I can and see what kind of area is the most interesting to specialize and focus on in my future. I am very open to dive into the fields of interactive and linear media, but also have a hunger to learn more about the actual sound design process of a physical product and some material science behind it. In the craft itself I am trying to learn and soak up all the skills that would need to be necessary to achieve my goals. I am looking forward to learning more about storytelling and dramaturgy in general, which will strengthen my understanding of the topic, as well as the ability to work alongside a team and the internal communication that comes with that. My mission is to fully commit to every project with my passion for sound design by creating meaningful experiences. I want to find my niche, where I can then develop my personal style and skills to be sought after.

Specifically interesting would be to work in audio post production for a short movie or animation film. I want to learn the basics of mixing sound for media in dolby atmos or other surround formats. Within these projects I also want to focus on my foley skills and work. This would be both a creative and technical thrilling experience. Besides that I am also looking forward to designing audio assets for interactive media, like a video game. I do like the approach and process to this, especially with all the variations one might do on similar sounds. Another fascinating skill to learn would be the implementation of these sounds to create a realistic sonic environment in an animated world. The last major topic I want to engage in would be product sound design. I am inspired by how one can shape materials and form an audibly pleasing experience that creates a stronger emotional connection to an object.

Ultimately I am super excited to have taken this path and direction, within a field where my creative needs and my technical interests are both met complementary. I cannot wait to create holistic and immersive experiences to enrich and inspire people around the world. Sound is both my language and tool.