This blog is a summary and review of Kevin Blackistone’s NIME paper, “Ultrasound Probe as Tool for Tangible Sound Performance Using Physically Sculpted Phantoms”. I wasn’t entirely sure what to expect, as my first association with the word “phantom” was more in line with some type of ghosting effect (a visual trailing artifact behind moving objects). What I was met with instead was a deep dive into medical imaging techniques, material exploration, and its sonification.
Core Concept: Medical Imaging Techniques
This paper documents a project centered around a standard handheld medical ultrasound probe. While I did get an ultrasound in the past myself, I didn’t know the technology behind it. After some research on the side, I have learned that this device is a type of transducer, transducers being devices that convert energy from one form to another (a category in which microphones and speakers also reside). It relies on the piezoelectric effect, which allows the probe’s crystals to vibrate and emit high-frequency sound waves, also known as ultrasound. As these sound waves travel or vibrate into the body, they reflect back to the device as echoes, which are then used to create a real-time visual interface. The NIME paper documents various movements when operating the probe and how they affect the resulting image.
While these devices are typically used on human tissue, medical training often relies on “phantoms,” which are opaque, dyed gelatin molds designed to mimic organic tissue, with the aim of helping students practice handling the probe and other sub-dermal (just beneath the skin) tools. Blackistone uses the ultrasonic medical probe in combination with DIY phantoms to create a physically expressive and sonic performance.
On DIY Phantom Design
The DIY phantoms used were composed of everyday food items such as tofu, grapes, orange peels, mushrooms, and even coffee grounds, with the aim of creating different membranes, materials, and surfaces, each of which reflects ultrasound differently, producing unique spectral characteristics once converted into sound via an additive synthesis approach. It is also a positive ethical choice to use organic matter, as it minimizes waste.
In my opinion, this approach of using DIY phantoms instead of human tissue is a clever choice, as it creates this playful and intuitive way of sculpting a soundscape, shaping the end results before they even hit an electronic translation process. It also shifts the focus to a more post-human-centered approach, in which the story isn’t as much on an individual as it is on Frankenstein-like phantoms.
If we were thinking in terms of musical instruments, the phantom in this case acts as the body of the instrument, and the probe as a tool for playing it.
On the Synthesis Approach
The paper outlines a sound synthesis framework that translates real-time ultrasound imagery into multi-spectral audio by mapping object depth (the distance from the probe’s surface) to frequency and pixel brightness (corresponding to material density) to amplitude. From my understanding, this is done via Inverse Fast Fourier Transform (IFFT) by sampling single or multiple vertical lines across the video stream. Each line is then mapped to a bank of individual frequency bins, without phase reconstruction or more advanced spectral processing. The result is a frame-by-frame additive synthesis. This approach is somewhat limiting, something the author openly acknowledges, as some of the phantom designs end up producing relatively similar soundscapes.
My first thought after reading up on ultrasound probes was to make a more straightforward synthesis approach of taking the ultrasound echo, shifting it to the audible range, applying an amplitude curve accounting for human hearing, and listening to the results. Then, I would probably build some type of detection tool for different shapes to trigger different types of sound events. From there, it could be combined with the already established idea of taking the depth and density values for some other sound parameters.
This focus on mapping dynamic frequency curves brought me back to the author’s brief, separate quick mention of the app SPEAR (an application for audio analysis, editing, and synthesis), which I would like to quickly mention as I tried using it in the past. My idea was to analyze audio samples via the software and reconstruct them via additive synthesis in SuperCollider (a platform and programming language for audio synthesis). Such an approach, however, proved more finicky than I initially expected, with the number of various sinusoidal frequencies needed to get a hint of the original sample being larger than anticipated. If I were to try again, I would also take into consideration the varying amplitude of the frequencies over time, which would be achieved by putting the needed values in arrays read as values for ADSR (attack, delay, sustain, release) envelopes.
Final thoughts
I enjoyed reading about this project. I was particularly drawn to the sculptural aspect and design choices which were taken when tackling the DIY phantoms. The project works well for a performance, as it has an anchoring point most of us would be able to understand, that being the live visual representation of the scan in combination with the exploration and manipulation of the chosen phantoms. If this were developed further, I would be interested in what other synthesis methods they would come up with, or in what way they could enrich the possibilities of the current setup.
Sources:
