When starting this project, there was only a vague idea what sounds the system should reproduce. To make those decisions, it is incredibly helpful to look at how professional motion picture sound teams operate. Motion picture sound is generally split into four distinct categories, each with a specialized team. On one side, you have the production dialog recorded on set and the automated dialogue replacement used to fix missing or inaudible lines.
On the other side, you have the sound effects department and the Foley crew. The sound effects editors primarily cut and edit pre-recorded sounds from massive audio libraries to match the picture. They handle all the sounds a scene needs that do not directly revolve around the actors’ physical performances. In contrast, Foley artists perform sounds live to the picture.
Motion Picture Sound Editing Teams. Yewdall 2012, 293
Their main task is to give subtle nuances and characterization to an actor by performing their footsteps, movements, and the handling of everyday objects. They deliver a sonic performance to support the acting, whether through a naturalistic style or a hyper-realistic cinematic approach.
Reflecting on the sounds created in the Foley booth, this project’s desired audio output narrows down to footsteps, body movements, and object handling. Because handling objects encompasses such a broad variation of acoustic attributes, I decided to focus even further. While watching Foley artists perform footsteps, I noticed that the delicate movements of the hand captured via computer vision directly mimic the visual similarities of moving foot muscles. These intricate hand gestures can likely be directly translated to solve the physical challenges of synthesizing footsteps.
Yewdall, David Lewis. Practical Art of Motion Picture Sound 4th ed. Oxford: Focal Press, 2012.
