Incorporating Andy Farnell’s procedural architecture provided a sophisticated solution to many of the project’s challenges and served as an excellent starting point. Building on top of this existing structure proved incredibly valuable, allowing me to adopt key functional elements while heavily modifying the critical signal flow to suit my specific needs. The core idea in this current version is to generate the footsteps using gestural computer vision input running through a Python script.
To achieve this, the data coming from the MediaPipe hand-tracking model had to be optimized. Instead of tracking the hand’s absolute position on the camera screen, the most intuitive and ergonomic mapping measures the relative distance between the tip of the thumb and the other four fingers on one hand. This approach turns the hand into a fully closed control system, regardless of its position in the camera frame. The algorithm automatically calculates the maximum distance and scales the values seamlessly between zero and one, transmitting them to plugdata via the OSC protocol.
Current state of the project in plugdata
In the updated patch, closing the distance between the fingertips decreases the values, while opening the palm increases them. Under the current mapping scheme, the right hand triggers the foot phase generator, while the left hand controls the individual parameters of the heel, roll, and ball phases. This allows the user to perform the rhythm of the footsteps single-handedly while manipulating the intricate acoustic nuances with the unoccupied hand.
To improve interactive capabilities, the continuous walking loop from the original patch was removed. Instead, an opening and closing threshold mechanism was introduced. Whenever the distance between the right thumb and pinky crosses a specific numeric threshold, a digital impulse activates the first part of the foot phase, and crossing it again triggers the rest of the roll-off. When this gets performed in a continuous rhythm, the resulting synthesized footstep signal offers highly refined, expressive control over all temporal parameters.
