Category: Uncategorized

In the second part of my blog series from the IRCAM Forum, I will summarize 2 workshops and performances, where I found similarities and approaches that could be interesting for my own research project.

Concrete Motion

Concrete Motion is an experimental tool designed for educational settings that facilitates the study and creation of sound-based music through physical movement. The system integrates the flexible audio processing of Max/MSP combined with Google MediaPipe body-tracking, within the TouchDesigner environment. By leveraging these technologies, it establishes an interactive digital space where listening and electroacoustic analysis are mediated through the user’s bodily gestures. This approach aims to bridge the gap between abstract musical concepts and tangible, physical interaction for learners and creators alike. While the objectives in this project’s research lies in educational ambitions, the fact it is using gestural interaction to create an accessible environment, can be directly translated into my own idea of an accessible interface for my project. A main difference in the technical construction would be, that the MediaPipe Hand Landmarker is running directly through Touchdesigner. I am not sure if there is an additional latency involved or if this could also be an idea for my project. Especially if I plan to get some additional visualisation on the gestures and processed audio.

Liminal

Liminal is an interactive installation that moves beyond traditional control-based models to explore a “liminal” space where agency is shared between humans and AI. In this environment, human gestures do not function as direct commands but instead serve as contextual information that influences the system’s evolving behavior over time. The architecture uses real-time computer vision and a Python-based decision layer to ensure that audiovisual changes emerge through gradual modulation and probabilistic weighting rather than immediate cause-and-effect. By distributing authorship across both the participant and the machine, the work transforms interaction into a sustained, meditative dialogue shaped by accumulated history and continuous negotiation. I visited the performance and workshop, as this is as well a similar approach from a gestural interaction to audio creation philosophy. In comparison to other projects of this theme, I found the bidirectional interaction and decision making between the human input and the models system. Maybe this could also be seen as an anecdote to how future work will, if not already, operate in all daily activities. On a technical level, the computer vision was also implemented through Touchdesigner, as in the other project mentioned earlier. 

Both projects incorporate most technicalities I strive to achieve in my own project, hence why it was interesting listening to the approaches and talking about the ideas involved. While I will head towards a different end goal with my product, they are still good examples to compare the technical feasibility and workflow. All in all, our days at IRCAM Forum brought me exciting insights and takeaways from diverse fields of audio focussed research.

Last week we attended the IRCAM Forum in Paris as part of our semester excursion. Overall it was an interesting couple of days, where we had the chance to experience a lot of the latest research and techniques in various fields of audio and sound. Within this post I will try to summarise my main takeaways on some chosen talks, workshops and performances I have attended.

Partiels & ASAP Plugins

I visited a talk and workshop by Pierre Guillot, which showed updates and news from the tools Partiels and ASAP. Partiels is a software dedicated to analysing audio signals and retrieving useful data for signal processing and sound design applications. One of the new interesting developments involved a direct python integration to access analysed data directly. The ASAP tools are more a direct creative solution to manipulate audio in various applications. The three major tools mentioned would be the Psycho Filter – allowing to apply spectral filters directly to a source, the Pitches Brew – an advanced pitch and formant manipulation via interactive frequency curve editing, and Stretch Life – a time manipulation tool for compressing and stretching sound dynamically. Further notable mentions have been the Spectral Morphing and Spectral Crossing tool, which allows to combine and ‘morph’ two audio sources on the spectral domain. All together these seem to be interesting tools, cleverly designed and quite accessible for most users I would imagine.

GRM Tools – Atelier

Another interesting workshop was the presentation of the GRM Tools Atelier software. It is a sound processing and synthesis environment working for real-time and multichannel productions, both in a live or studio application. I really liked the modular approach and the quick and intuitive randomisation capabilities, which allows for a fast agitation of multiple parameters at once. This can be an interesting choice for sound artists wanting to work with only one standalone software and dealing with more graphically intuitive controls than for example puredata or Max/Msp. As I already own some similar synthesizer software of a similar modular system, without the multichannel capabilities, I will for now stick to these though.

VASE

I also experienced a performance of VASE by composer Yuval Seeberger. The device is a motorized music-box installation that performs a 12-minute, semi-algorithmic composition by integrating a physical punched paper score with advanced digital processing. The system utilizes a specialized “ensemble” consisting of an acoustic-mechanical music-box, an analog motor, Max/MSP synthesis, and RAVE neural audio models to create a rich, layered sonic environment. While the structure follows a formal organization, the irregular communication between the computer and the mechanical hardware ensures that each loop cycle contains subtle, unpredictable variations. Through the use of Piezo and magnetic rail-coil pickups, the piece effectively bridges the gap between tactile mechanical movement and real-time AI-driven sound generation. I was quite fascinated by the various soundscapes it was able to produce and also how the random interaction by the composer influenced the installation.

A product or business idea is a structured proposal that identifies a specific problem, outlines a solution, and defines how value is created for users and stakeholders. In design-driven innovation, such ideas are grounded in real user needs and aim to create both functional and experiential improvements.

Understanding the underlying idea of a product is the first and most important step in its development. For the idea of a guiding system at German train stations the exact paraments for the final product are not yet defined. But a closer look at the product idea is still a valuable step towards more clarity and understanding.

The core problem lies in the current experience of train platforms, which are often perceived as stressful, unorganized, and confusing environments. Boarding and exiting trains can be physically demanding, especially during peak times or for individuals with limited mobility. This creates friction in the interaction between passengers, trains, and the platform itself, ultimately reducing the overall quality of the travel experience.

Addressing this issue matters because improving the usability and comfort of train travel can make it a more attractive mode of transportation. A better experience could encourage more people to choose trains over cars, contributing to reduced traffic congestion and lower environmental impact.

The proposed solution is a physical guiding system integrated directly into train platforms. While still in development, the current idea is the use of light-based elements, such as illuminated pathways, signals, or dynamic indicators, to guide passengers intuitively. This system would enhance orientation, communicate real-time information, and support smoother boarding and alighting processes without adding visual clutter.

The target audience includes all users of the train system, with a primary focus on passengers. At the same time, organizations like Deutsche Bahn act as key stakeholders and customers, investing in and maintaining the system. The expected impact includes improved navigation, more efficient passenger flow, and a more structured and user-friendly platform environment.

From a business perspective, the model could involve an initial infrastructure investment by railway operators, followed by ongoing maintenance.

Ultimately, the idea combines user-centered design with systemic impact, aiming to transform train platforms into more intuitive, accessible, and enjoyable spaces.

The Value Proposition Canvas is a strategic tool used in design and innovation to ensure that a product or service aligns closely with user needs. It consists of two main components: the Customer Profile and the Value Map. The Customer Profile focuses on understanding the user by identifying their jobs (what they want to achieve), pains (challenges or frustrations), and gains (desired outcomes or benefits). The Value Map, on the other hand, outlines how a product or service responds to these needs through products and services, pain relievers, and gain creators. Together, these tools help designers create solutions that are both relevant and impactful. (Strategyzer, 2026)

To get a better understanding of the anticipated product and its purpose for the user, two canvases were produced for two different players. The first one focuses on the train passenger as an end user. Their Customer Profile emphasizes practical goals such as arriving on time, navigating platforms easily, and boarding trains without stress. Gains include comfort, clarity, and reliability, while pains involve confusion, overcrowding, physical strain, and lack of accessible information. The Value Map responds with a physical support and guidance system, clearer information structures, and inclusive design features to accommodate diverse user needs.

The second example represents the Deutsche Bahn (DB) as a customer. Here, the Customer Profile highlights organizational goals such as transporting passengers efficiently from A to B, ensuring smooth system operations, and maintaining profitability. The identified gains include improved punctuality, enhanced public image, and increased customer satisfaction. However, DB also faces significant pains, such as technical failures, delays, and negative public perception. The corresponding Value Map proposes solutions like improved guidance systems, better information displays, and more structured platforms, all aimed at reducing inefficiencies and enhancing the overall service experience.

Overall, these two profiles demonstrate how the Value Proposition Canvas can bridge organizational objectives and user experiences, enabling more targeted and user-centered design solutions.

The “Harmonix Series” by Wing Hei Cheryl Hui and Patrick Hartono represents a visionary bridge between human-computer interaction and therapeutic art, moving beyond the technical novelty of Digital Musical Instruments to address a profound need for the democratization of creativity. What I find most compelling about this work is its commitment to radical accessibility; by shifting the focus from mastering a complex tool to simply experiencing a soundscape, the authors empower users of all physical and cognitive abilities to become creators. This is further elevated by the intentional integration of mindfulness into the interface design, which transforms the act of music making into a meditative process for emotional regulation rather than just a performance. The synergy between robust technical implementation and a sophisticated aesthetic sensibility is palpable, resulting in an instrument that doesn’t just function, but truly resonates on a human level. Ultimately, this project serves as a vital reminder that the future of music technology should prioritize deep human impact and digital wellbeing, treating the user as a whole person seeking connection and calm.

I chose this article because the title immediately caught my attention. I was curious what “Tiny Touch Instruments” actually are and what kinds of decisions and thoughts go into programming such instruments.

The author, Rebecca Abraham, is a researcher and composer working in the area of digital and collaborative music-making. In the paper, Abraham describes a project centered on Tiny Touch Instruments (TTIs), a set of mobile, web-based musical instruments that are played through touch gestures on a smartphone or similar device. The project is situated within a broader context of mobile music ensembles, such as the Stanford Mobile Orchestra, which explore how mobile technology can support collective music-making. You can access the Tiny Touch Instruments here.

As part of this research, Abraham composed two pieces titled Skating and Skipping. Both works are performed using the TTIs that the author programmed. The instruments run on a webpage and are controlled using gestures such as tapping, swiping, or holding a finger on the screen. These interactions generate sound while also producing visual feedback, allowing performers to see and hear the effects of their gestures. One important aspect of the project is accessibility: the pieces are designed so that they can be performed without prior rehearsal and even by people without formal musical training.

The two compositions use different approaches to notation and performance. In Skating, performers follow a graphic score that includes visual shapes and brief text instructions. Participants draw certain gestures on their screens or interact with others in the group, for example by imitating nearby performers or responding to sounds they hear in the room. The focus of the piece is less on precise melodies and more on shared sonic textures that emerge through group interaction.

Skipping uses a different format. Instead of a static graphic score, performers follow an animated score projected on a large screen. This score combines graphics, animations, and text instructions that guide the performers’ actions over time—for example, indicating where on the phone screen to interact or encouraging them to increase the frequency of tapping. The piece gradually shifts from simple exploration of the instruments toward more intentional interaction between performers.

Through observations, interviews, and surveys with participants across several performances, Abraham analyzed how people experienced these pieces. One key finding was that performing without rehearsal encouraged exploration and experimentation. At the same time, performers gradually became more comfortable with the instruments as the piece progressed. Another important result concerns notation: a multimodal approach that combines graphics, animation, and text proved particularly effective.

Visual elements helped performers understand the relationship between their gestures and the sounds produced by the instruments.
An especially interesting observation was how the performances changed participants’ perception of their smartphones. During the performance, the phone was no longer experienced primarily as a device for communication or distraction, but rather as a creative musical tool that enabled collective expression.

These ideas resonate strongly with my own design interests. In my research, I am exploring the concept of “ear candy” and interactive sound design. Inspired by this article, I am considering developing my own small touch-based digital instruments that people could access online. My goal would be to design them in a way that is not only playful and engaging, but also educational, allowing users to learn something about sound or interaction through experimentation.

Recently, I came across the research “Entangling with Light and Shadow: Layers of Interaction with the Pattern Organ” by Jasmine Butt, Nathan Renney, Benedict Gaster, and Maisie Palmer, developed within the Expressive Computer Interaction Research Group at UWE Bristol.

RESEARCH OBJECTIVE

This research explores the design and use of a camera-based digital musical instrument called the Pattern Organ. This visual-audio synthesis artifact investigates new ways of interacting through light and shadow.

Users can modify a waveform by placing their hands or objects in front of the instrument’s camera, creating shadows and patterns. Through this interaction, they can perceive how both the environment and the sound change in real time.

The project initially started as a digital tool to represent the process of optical sound technology. However, during the workshop sessions, this idea evolved further. The focus shifted from a purely visual-audio synthesis system to a more open, participatory, and exploratory process.

THEORETICAL BACKGROUND

A matter of finding the grain of the world

Bruna Goveia Da Rocha and Kristina Andersen [2]

Figure 2: Design of the original Instrument

Drawing from analogue optical sound technologies used in early cinema, the research reinterprets these practices through a post-human perspective. Two main theoretical perspectives are considered.
The first is from N. Katherine Hayles, who describes the world as a complex and highly interconnected syste[3]m. In this view, cognition is not limited to humans but moves dynamically across humans, animals, and technological systems. The second perspective is Karen Barad’s Agential Realism [4]. This theory describes reality as something continuously shaped by the interaction between material and meaning. Matter and information are not separate but constantly influence each other.

A strong emphasis is also placed on material thinking and hands-on experimentation with different materials.

CONSIDERATIONS

Throughout this process, there are two aspects that I personally find particularly interesting.

From a theoretical perspective, I find the idea of an entangled and participatory workshop very powerful. In this context, three elements—human, machine, and materials—are in constant dialogue. They continuously influence each other during the creative process. This approach is very effective in stimulating critical thinking, both in design, where each input can generate new ideas or solutions, and in educational contexts.

Figure 3: Sperimentation using a rotating can

From a practical perspective, I was particularly interested in the use of raw data. This concept influenced the method of sonification used in the project. Rawness can be understood as a choice to avoid interpreting or transforming the data through complex digital processing. Instead, the data produced by the system is used more directly, without adding layers of interpretation.

This does not necessarily mean that raw data is more accurate or more realistic. Rather, it means that the measurements are not modified or filtered, allowing a more immediate connection with the original signal.

In the case of camera-based sonification, two main approaches can be identified:

1. Extracting features from image data to control or modulate sound
2. Using a more direct method, where pixel brightness values are translated into sound signals with minimal processing

CONCLUSION

This research opens important questions about how data should be treated and interpreted. It challenges the idea that data always needs to be processed, optimized, or controlled.

It also highlights the role of human intervention and how our decisions shape the way systems behave. At the same time, it shows how the physical and material nature of interaction—light, shadow, objects—can influence digital processes in meaningful ways.

More broadly, it invites us to rethink the relationship between humans, technology, and the material world. Instead of separating them, this work suggests that meaningful interaction emerges from their continuous entanglement.

REFERENCES

[1] J. Butt, N. Renney, B. Gaster, and M. Palmer, “Entangling with light and shadow: Layers of interaction with the pattern organ,” in Proceedings of the International Conference on New Interfaces for Musical Expression (NIME ’25), Canberra, Australia, June 24–27, 2025.

[2] Bruna Goveia Da Rocha and Kristina Andersen. 2020. Becoming Travelers:
Enabling the Material Drift. In Companion Publication of the 2020 ACM Design
ing Interactive Systems Conference. ACM, Eindhoven Netherlands, 215–219.
https://doi.org/10.1145/3393914.3395881

[3] Katherine Hayles. 2006. Unfinished Work: From Cyborg to Cognisphere.
Theory, Culture Society 23 (2006), 159–166. https://doi.org/10.1177/0263276406069229

[4] Karen Barad. 2007. Meeting the universe halfway: quantum physics and the
entanglement of matter and meaning. Duke University Press, Durham London.

While reading about PlaySoundGround, I was struck by how a seemingly simple idea—combining sound and play—can completely transform the playground experience. The project turns familiar playground equipment into interactive musical instruments, where physical movement directly produces sound .

At first glance, this idea feels almost obvious. It made me question why more playgrounds are not designed in this way. If play is already based on movement, rhythm, and interaction, connecting it to sound seems like a natural extension. Yet, in most traditional playgrounds, this potential remains unexplored.

What I found particularly interesting is how the project reveals the relationship between playing and play. As the authors describe, both involve creative interaction within physical and social constraints . By making this connection explicit, the playground becomes more than just a physical space—it becomes an interactive, expressive environment.

Another aspect that stood out to me was that the playground was scaled for adults. This shift challenges the common assumption that playgrounds are only for children. Extending such experiences to adults opens up new possibilities for interaction, creativity, and social engagement. It suggests that play is not limited by age, but rather by how spaces are designed.

Overall, this project made me reflect on how small design interventions can unlock entirely new experiences. Even a simple addition like sound can make playgrounds more engaging, interactive, and meaningful. It also reinforces my interest in designing participatory and playful systems that invite users to actively shape their own experiences.

https://zenodo.org/records/1177685