Exploring Interaction Through Light and Shadow: The Case of the Pattern Organ
Recently, I came across the research “Entangling with Light and Shadow: Layers of Interaction with the Pattern Organ” by Jasmine Butt, Nathan Renney, Benedict Gaster, and Maisie Palmer, developed within the Expressive Computer Interaction Research Group at UWE Bristol.
RESEARCH OBJECTIVE
Figure 1: Illustration of an interface pattern
This research explores the design and use of a camera-based digital musical instrument called the Pattern Organ. This visual-audio synthesis artifact investigates new ways of interacting through light and shadow.
Users can modify a waveform by placing their hands or objects in front of the instrument’s camera, creating shadows and patterns. Through this interaction, they can perceive how both the environment and the sound change in real time.
The project initially started as a digital tool to represent the process of optical sound technology. However, during the workshop sessions, this idea evolved further. The focus shifted from a purely visual-audio synthesis system to a more open, participatory, and exploratory process.
THEORETICAL BACKGROUND
A matter of finding the grain of the world
Bruna Goveia Da Rocha and Kristina Andersen [2]
Figure 2: Design of the original Instrument
Drawing from analogue optical sound technologies used in early cinema, the research reinterprets these practices through a post-human perspective. Two main theoretical perspectives are considered. The first is from N. Katherine Hayles, who describes the world as a complex and highly interconnected syste[3]m. In this view, cognition is not limited to humans but moves dynamically across humans, animals, and technological systems. The second perspective is Karen Barad’s Agential Realism [4]. This theory describes reality as something continuously shaped by the interaction between material and meaning. Matter and information are not separate but constantly influence each other.
A strong emphasis is also placed on material thinking and hands-on experimentation with different materials.
CONSIDERATIONS
Figure 3: Images from the first workshop’s exploration
Throughout this process, there are two aspects that I personally find particularly interesting.
From a theoretical perspective, I find the idea of an entangled and participatory workshop very powerful. In this context, three elements—human, machine, and materials—are in constant dialogue. They continuously influence each other during the creative process. This approach is very effective in stimulating critical thinking, both in design, where each input can generate new ideas or solutions, and in educational contexts.
Figure 3: Sperimentation using a rotating can
From a practical perspective, I was particularly interested in the use of raw data. This concept influenced the method of sonification used in the project. Rawness can be understood as a choice to avoid interpreting or transforming the data through complex digital processing. Instead, the data produced by the system is used more directly, without adding layers of interpretation.
This does not necessarily mean that raw data is more accurate or more realistic. Rather, it means that the measurements are not modified or filtered, allowing a more immediate connection with the original signal.
In the case of camera-based sonification, two main approaches can be identified:
Extracting features from image data to control or modulate sound
Using a more direct method, where pixel brightness values are translated into sound signals with minimal processing
CONCLUSION
This research opens important questions about how data should be treated and interpreted. It challenges the idea that data always needs to be processed, optimized, or controlled.
It also highlights the role of human intervention and how our decisions shape the way systems behave. At the same time, it shows how the physical and material nature of interaction—light, shadow, objects—can influence digital processes in meaningful ways.
More broadly, it invites us to rethink the relationship between humans, technology, and the material world. Instead of separating them, this work suggests that meaningful interaction emerges from their continuous entanglement.
REFERENCES
[1] J. Butt, N. Renney, B. Gaster, and M. Palmer, “Entangling with light and shadow: Layers of interaction with the pattern organ,” in Proceedings of the International Conference on New Interfaces for Musical Expression (NIME ’25), Canberra, Australia, June 24–27, 2025.
[2] Bruna Goveia Da Rocha and Kristina Andersen. 2020. Becoming Travelers: Enabling the Material Drift. In Companion Publication of the 2020 ACM Design ing Interactive Systems Conference. ACM, Eindhoven Netherlands, 215–219. https://doi.org/10.1145/3393914.3395881
[3] Katherine Hayles. 2006. Unfinished Work: From Cyborg to Cognisphere. Theory, Culture Society 23 (2006), 159–166. https://doi.org/10.1177/0263276406069229
[4] Karen Barad. 2007. Meeting the universe halfway: quantum physics and the entanglement of matter and meaning. Duke University Press, Durham London.
