Category: Uncategorized

3.1 Overview

Seed-VC [5] is a diffusion-based zero-shot voice conversion model developed by ByteDance’s Seed Team. With 552

million parameters, it achieves state-of-the-art speaker similarity (SECS: 0.867 vs. 0.754 for OpenVoice and 0.844 forCosyVoice) while preserving linguistic content. The model requires only 1–30 seconds of reference audio and

performs conversion without any fine-tuning.

The training process involves an External Timbre Shifter (ETS) that applies random timbre perturbations to the

source audio before content encoding. This forces the content encoder to discard speaker-specific information,

producing speaker-agnostic tokens. The style encoder (CAM++) operates on the original, unperturbed source to

extract the ground-truth speaker embedding. The model is trained to reconstruct the original source from the perturbed

content tokens and the ground-truth style embedding, thereby learning to combine arbitrary content with arbitrary

style.

Key public benchmarks for Seed-VC V2: – SECS (Speaker Embedding Cosine Similarity): 0.867 – WER (Word Error

Rate, content preservation): 8.2% – MOS (Mean Opinion Score): 4.12/5.0 – Inference speed: ~0.5× real-time on A100

GPU

3.2 Content Encoder

The content encoder employs a two-level discrete representation to address the timbre leakage problem that plagues

earlier voice conversion systems:

•

Narrow level (32-token codebook): HuBERT-Large features from layer 18 are processed by a ConvNeXt

encoder and quantized via Binary Spherical Quantization. This produces a compact token sequence that captures

prosody and temporal structure, used by the AR model for duration-aware generation.

•

Wide level (2048-token codebook): A Whisper-small tokenizer on the same backbone produces a richer token

representation for fine content fidelity. These tokens condition the CFM diffusion process.

The two-level design ensures that the AR model has precise prosodic alignment while the diffusion model has

sufficient content detail for high-fidelity reconstruction.

3.3 AR Transformer

The autoregressive transformer generates content tokens conditioned on source content and target style. Architecture

details: – 12 transformer layers, 768 hidden dimension, 12 attention heads – Rotary Position Embedding (RoPE) for

sequence position encoding – KV-cache pre-allocation for efficient inference

The AR model takes the source narrow tokens and target style embedding as input, and generates target narrow tokens

autoregressively.

3.4 CFM / Diffusion Transformer

The core generation engine is a Conditional Flow Matching (CFM) [6] model built on a Diffusion Transformer

(DiT) backbone:

•

13-layer DiT with 512 hidden dimension

•

Euler ODE solver with 30 integration steps

•

Multi-condition Classifier-Free Guidance (CFG) on both content and style embeddings

•

Flow matching replaces traditional diffusion with a straight-line ODE path from noise to data, enabling faster

inference with fewer steps

3.5 BigVGAN Vocoder

The vocoder converts mel-spectrograms to waveforms using NVIDIA’s BigVGAN v2 [7], a GAN-based neural

vocoder: – 22 kHz output sampling rate – 80-band mel-spectrogram input (1024 FFT, 256 hop, 1024 window) – 256×

upsampling via transposed convolutions – HiFi-GAN discriminator for adversarial training

The vocoder is pretrained and used without fine-tuning.

2.1 Air vs. Bone Conduction

The human auditory system perceives sound through two distinct pathways [1]:

1. Air conduction: Sound waves travel through the external ear canal, vibrate the tympanic membrane, and are

transmitted via the ossicular chain (malleus, incus, stapes) to the oval window of the cochlea. This pathway

preserves the full audible frequency spectrum (20 Hz – 20 kHz).

2. Bone conduction: Vibrations of the skull bones are transmitted directly to the cochlea, bypassing the outer and

middle ear. This pathway acts as a mechanical lowpass filter with a cutoff at approximately 4 kHz [1].

The internal voice perception is the sum of both pathways, while a recording captures only the air-conducted

component. This difference explains why recorded voice sounds unfamiliar — it lacks the bone-conducted low-

frequency energy and the occlusion-effect mid-frequency boost.

2.2 The Occlusion Effect

When the ear canal is blocked (as occurs naturally when the vocal tract radiates sound toward the skull), the bone-

conducted signal is enhanced in the low-to-mid frequency range. This is known as the occlusion effect [2]. The effect

peaks at approximately 2.8 kHz with a gain of 10–15 dB relative to open-ear conditions [3].

The occlusion effect has three contributing mechanisms: – Osseotympanic: Skull vibrations radiate into the ear canal

and pressurize the occluded volume – Inertial: The ossicular chain’s mass resists skull vibration, creating relative

motion at the oval window – Compressional: The cochlea is compressed and expanded by skull vibration, creating a

traveling wave

2.3 Skull Resonance

Bone conduction transfer functions exhibit a resonance in the 600–1000 Hz range, corresponding to the mechanical

resonance of the middle ear structures [4]. Stenfelt and colleagues measured bone conduction sensitivity and

demonstrated that:

• Below 500 Hz, bone conduction is approximately 10 dB less sensitive than air conduction

• Between 500 Hz and 4 kHz, the sensitivity gap narrows
• Above 4 kHz, bone conduction sensitivity drops sharply

2.4 From Psychoacoustics to Computation

Our goal is to model this perceptual transformation computationally. By designing a digital filter that approximates

the bone conduction transfer function, we can simulate the internal voice from an external recording, then compare

the two using speaker embedding similarity

Abstract

This Blog presents “Discover Your Inner Voice,” a method for modeling the perceptual difference between how a

speaker hears their own voice (via bone conduction) and how it is heard by others (via air conduction). Using CAM++

speaker embeddings from the Seed-VC diffusion-based voice conversion pipeline, we extract a 192-dimensional style

delta vector ΔV = V_perceived − V_raw, where V_raw encodes the externally recorded voice and V_perceived

encodes the voice after psychoacoustic skull-resonance filtering. Our key finding — ‖ΔV‖ = 1.191 with cosine

similarity 0.9967 — reveals that CAM++ is inherently invariant to spectral filtering, a desirable property for speaker

verification but a limitation for perceptual style extraction. We discuss the implications, propose a filtered reference

override solution, and outline a roadmap toward clinical applications in laryngeal cancer rehabilitation.

1. Introduction

The question is as old as recorded audio: “Why does my voice sound different on recording?” Every speaker

experiences the disconnect between their internal self-perception and the external recording. This phenomenon arises

from a fundamental psychoacoustic difference: we hear our own voice through two simultaneous pathways — air

conduction through the ear canal and bone conduction through the skull.

While this discrepancy is a familiar curiosity, its computational modeling has remained unexplored. Modern voice

conversion systems can clone a speaker’s voice from a few seconds of reference audio, but they operate on the

external voice — the voice as heard by others. The internal voice — the voice as perceived by the speaker — is lost.

This project introduces the Inner Voice framework: a method to capture the transformation between a speaker’s

external and internal voice using speaker embeddings from a state-of-the-art voice conversion pipeline. We pose the

following research questions:

1. Can a psychoacoustic skull-resonance filter produce a measurable shift in speaker embedding space?

2.Can this shift be encoded as a style delta vector (ΔV) for use in voice conversion?

3.Does the ΔV carry perceptually meaningful information about the internal voice?

The scope of this work is a proof-of-concept on a single subject, with a full implementation pipeline including a

Gradio web interface, real-time metrics, and a clinical application framework for laryngeal cancer rehabilitation.

Contributions: – A psychoacoustic skull-resonance filter modeling bone conduction effects – The ΔV = V_perceived

− V_raw style delta extraction pipeline – An override_style parameter for the Seed-VC voice conversion system – A

three-tab Gradio interface with real-time visualization – The experimental finding that CAM++ is filter-invariant to

spectral coloration, quantifying this with ‖ΔV‖ = 1.191 – Open-source release of all code, vectors, and documentation

The remainder of this paper is organized as follows. Section 2 reviews bone conduction psychoacoustics. Section 3

describes the Seed-VC architecture. Section 4 covers the CAM++ speaker encoder. Section 5 presents the proposed

ΔV extraction method. Section 6 details the system implementation. Section 7 reports results and analysis. Section 8

discusses a clinical application. Section 9 concludes with future work.

1.1 Related Work

Voice conversion has evolved through several generations. Early systems used Gaussian mixture models (GMMs) to

map source-to-target spectral envelopes [11]. Parallel data was required — the same utterance from both speakers.

The introduction of cycle-consistent adversarial networks (CycleGAN-VC) [12] enabled non-parallel training,

followed by variational autoencoders (VAEs) and autoencoder-based approaches that disentangled content from

speaker identity.The current state-of-the-art uses diffusion models and self-supervised learning. DiffWave [13], SpecDiff [14], and

VoiceBox [15] demonstrated that diffusion models produce high-fidelity speech. Seed-VC [5] achieved the first truly

zero-shot voice conversion competitive with speaker verification systems, reaching a speaker embedding cosine

similarity (SECS) of 0.867.

Concurrent work in speaker embedding analysis includes studies on embedding invariance [16], which showed that

verification embeddings are robust to channel effects. Our work extends this investigation to the specific case of bone-

conduction modeling.

No prior work has attempted to model the bone-conduction vs. air-conduction perceptual difference using speaker

embeddings. The Inner Voice concept is, to our knowledge, the first attempt to computationally capture the internal

voice perception.

8. Feedback, Corrections and a More Honest Framing

The second presentation and professor feedback were very useful because they exposed weak points in how I was explaining the project. Some parts were technically working, but the wording was not always careful enough. This was especially true around terms like simulation, room model, coordinate convention and perception.

One important correction was the word simulation. It is too strong for the current system. The prototype does not simulate a complete room. It does not include reflections, reverberation, diffraction, occlusion, source directivity or measured room response. A better term is direct-sound auralisation or controlled reference model. That makes the project more honest.

Another correction was the phrase room model. It can sound like the system models the room acoustics. It does not. It calculates source-listener geometry and direct-sound behaviour. So it is better to say Python-based direct-sound calculation script or calculation model, not full room model.

The coordinate drawing also needed correction. The angle mapping used in the system should not be presented as a universal coordinate convention. It is a plug-in angle mapping used to send direction values to the IEM plug-ins. In this mapping, front is 0 degrees, left is positive 90 degrees, right is negative 90 degrees and back is plus or minus 180 degrees. That is fine as an implementation mapping, but it should not be overgeneralised.

Some theoretical claims also had to be removed or softened. For example, distance-dependent HRTF changes and air absorption were not strong enough to present as central elements of the current prototype. Temperature-based sound speed also should not be mentioned casually unless the exact formula and assumptions are shown. These details can easily create questions that distract from the actual working system.

This feedback improved the project. It forced me to separate what is technically implemented from what could become future work. That is uncomfortable sometimes, but it is good for the thesis. The project becomes stronger when its limits are clear.

The current framing is now much better: the system is a working direct-sound auralisation baseline. It is controlled, testable and ready for comparison. It does not pretend to be the real room. Instead, it prepares a clean reference condition so that the real room can later be studied more clearly.

7. REAPER, mcfx and IEM Ambisonics

The REAPER signal chain is where the calculated values become audible. Each source track has a practical plug-in chain. The first mcfx gain-delay instance generates the sine tone and applies the first gain stage. Then a custom JSFX called TEZ_Smooth_Propagation_Delay applies the time-of-flight delay in a smoothed way. After that, a second mcfx gain-delay instance applies extra gain reduction if it is needed.

The second mcfx instance exists because one gain stage has a limited controllable range in my OSC/REAPER setup. For example, the model might calculate that a source at 20 metres should be about minus 26 dB relative to 1 metre. If one mcfx stage can safely cover about minus 18 dB, the remaining minus 8 dB is applied by the second stage. The acoustic model still calculates one value. The two-stage split is only an implementation detail.

This was confusing at first, but the logic is simple: the model says how much quieter the sound should be. REAPER then needs to apply that level reduction. If one gain control cannot apply the full value, the reduction is split across two serial gain controls. In dB, serial gain values add together, so minus 18 dB plus minus 8 dB becomes minus 26 dB.

The delay side had another technical issue. If delay values are changed too abruptly while the listener moves, clicks or crackling can happen. That is why the custom smooth delay FX exists. It receives the calculated time-of-flight delay and changes it more smoothly, so movement sounds more stable.

After the source tracks, each signal is routed into the Ambisonic bus. The IEM MultiEncoder receives the source direction values. The IEM SceneRotator receives the listener head rotation. The EnergyVisualizer helps check the spatial energy visually, and the BinauralDecoder converts the Ambisonic scene into headphone playback.

This means the current prototype is headphone-based. The binaural decoder assumes that the left signal reaches only the left ear and the right signal reaches only the right ear. If the output is played over two loudspeakers, some things like level changes and beating can still be heard, but the binaural head-tracking result is not reliable because of acoustic cross-talk.

So the REAPER chain is not only playback. It is the part where the mathematical model becomes a listening experience.

6. The Direct-Sound Calculation Model

The calculation part of the project is intentionally simple. It does not try to model a complete room. It calculates the direct path between each source and the listener. This is important because it keeps the system testable and avoids making claims that are too large.

The model calculates three main things: distance, level and time of flight. First, it calculates the difference between source and listener position on the x, y and z axes. These differences are dx, dy and dz. From these values, it calculates the straight-line distance between source and listener. This distance is called r.

Second, the model calculates level attenuation. The system uses 1 metre as the reference distance. At 1 metre, the level change is 0 dB. As the source gets farther away, the level decreases according to free-field pressure attenuation. This follows the relationship p proportional to 1/r. In dB, every doubling of distance gives about minus 6.02 dB. So 1 metre is 0 dB, 2 metres is about minus 6 dB, 4 metres is about minus 12 dB, and so on.

This is sometimes easy to misunderstand. It is not minus 6 dB per metre. It is minus 6 dB per distance doubling. That is why a 20 metre test position gives about minus 26 dB, not minus 120 dB.

Third, the model calculates time of flight. Sound does not arrive instantly. If a source is farther away, the sound arrives later. The delay is calculated as distance divided by the speed of sound. For example, at 20 metres, the time of flight is about 58 milliseconds.

There is also a practical limitation: the 1 metre reference clamp. Below 1 metre, the system does not increase the level further. This prevents unrealistic gain close to an idealised point source. However, this also means that if the goal were to simulate a very small studio or very close source-listener distances, this part would need to be reconsidered.

The direction calculation is also part of the model. The source position is converted into azimuth and elevation. Listener yaw is not added to the source azimuth. Instead, source direction goes to the MultiEncoder, and head rotation goes separately to the SceneRotator. This separation prevents double rotation.

For me, this is the core of the project: a simple model that is honest about what it does.

5. From First Prototype to GUI Version 2

After the first working version, the next important feedback was about control and usability. My professor suggested that instead of playing sound files from REAPER tracks, it would be better to use plug-ins that generate the sound directly. This made sense for the project because the sound sources are sine tones. If the tones are generated inside the plug-in chain, the system becomes cleaner and more repeatable.

He also asked for more detailed and semi-controllable parameters in the GUI. This led to the second GUI version. I think this version is much stronger than the first one. It has room selection, source layouts, custom source count, editable source positions, presets, and listener movement. It feels more like a small research tool than only a technical test.

In the setup screen, the user can choose between different room scales and source layouts such as mono, stereo, quad and octagon. There is also a custom mode where the number of sources can be changed. This is important because the system should not only demonstrate one fixed setup. It should allow repeatable variations.

The GUI also includes a physical direct-sound mode. This is the mode used for the project results. It follows the free-field pressure relationship where each doubling of distance gives about minus 6.02 dB. Earlier versions had an exaggerated mode for demonstration, but after feedback it became clear that this should not be presented as scientifically valid. The scientific part of the project should remain with the physical direct-sound calculation.

This second interface version also made the system easier to explain. The user first configures the room and sources, then enters the listening mode. That structure matches the project logic: define the experiment, then listen by moving. It also makes the system more useful for future comparisons, because a setup can be repeated instead of improvised each time.

The GUI was a big part of the project for me personally. It was where the technical system became something visible and usable. It also gave me a reason to learn more Python and PyGame. I started with only basic Python knowledge, and the project pushed me to build something I would not have built otherwise.

At this stage, the project became much more coherent. It had a clear interface, a calculation model, a REAPER chain, and a future research direction.

4. The First Working PyGame Prototype

The first version that really worked was made with PyGame. It was still simple, but it was the first time the system became interactive in the way I wanted. The GUI sent x and y values for the listener position. These values went into the calculation script, where the relation between source and listener was calculated. Then the resulting parameters were sent to REAPER and the IEM plug-ins.

This was an important moment because the project stopped being only a plan. I could move something in the interface and hear a change in the audio system. Even if the first version was rough, that connection was exciting. It showed that the basic chain was possible: GUI movement, Python calculation, OSC messages, REAPER processing, and spatial output.

At this stage, the interface was not yet the final one. It was more like a proof of concept. But it already contained the essential idea. The listener position mattered. Source distance could be calculated. Direction could be calculated. Values could leave Python and arrive inside the audio engine. That was the first working skeleton of the project.

Technically, this also helped me understand the different roles of the scripts. The GUI was not the model. It was the interface. The calculation part was the direct-sound model or direct-sound calculation script. The OSC sender was the communication layer. REAPER was the audio engine. This separation later became important for explaining the project clearly.

The first PyGame version also made me think more seriously about movement. I did not want the listener to just drag a dot and read numbers. I wanted the interaction to feel a little more embodied. Even though the system is headphone-based, movement is still central to the experience. The listener moves through a virtual source layout and hears changes that relate to position.

This is also where the project started touching game-like thinking. The controller, the movement, and the view direction reminded me of old first-person games. That connection was not planned, but it became part of the development story. It helped me think about how spatial audio tools could feel less like lab software and more like something you can inhabit.

The first PyGame prototype was not polished, but it was the first real success. It proved the system could exist.

3. First Interface Attempts: Pure Data and Max/MSP

My first instinct was to try Pure Data. It felt natural because it is close to sound-design workflows and patch-based thinking. I thought it might be a good place to create a simple interface that sends position values to the audio system. In theory, that made sense. But in practice, I quickly ran into a limitation: I wanted a visual and movement-based interface, and Pure Data did not feel comfortable enough for that.

The problem was not that Pure Data cannot send values or control sound. It can. The problem was the kind of experience I wanted. I wanted the listener to move, see sources, understand room geometry, and feel the relation between position and sound. The visual and animation side became important very early. My Pure Data attempts did not give me that feeling, so I stopped trying to force the GUI into a tool that did not suit the direction anymore.

After that, I tried Max/MSP. Max felt better for visual work and interaction. It gave me more control and felt more flexible than Pure Data for this particular task. I could imagine a more useful interface there. But even with Max, the result did not quite match what I wanted personally. It still felt like I was fighting the environment a little bit. I wanted something that behaved more like a small interactive application than a patch.

This part of the process was a bit frustrating, but it was useful. It showed me what the interface needed to do. It was not enough to have sliders or number boxes. The interface needed to show a room, show sources, allow movement, and later allow custom layouts and presets. It had to be a small spatial tool, not only a control panel.

In my second meeting with my professor, Python came up as a better option. I already had some basic Python knowledge, and the project became a good reason to build on it. This was slightly intimidating, because I had not used Python in this way before. But it also felt like the right challenge. If I could build the interface myself, I could make it behave exactly the way the project needed.

So Pure Data and Max were not failures in the end. They were part of the filtering process. They helped me understand that the core of this prototype needed to be a custom interface and calculation system, not just an audio patch.

Over the last months, this project focused on researching the user experience at German train stations and identifying opportunities for improvement. The process combined research, design, prototyping, and user testing in order to better understand the needs, frustrations, and expectations of travelers.

A major part of the project were two prototypes, that were developed to explore possible solutions for improving the train station experience. They helped translate theoretical insights into practical concepts and allowed ideas to be tested in a more tangible way. Through the design process, it became clear how important accessibility, clarity, and intuitive navigation are within busy public transport environments. The prototypes were both tested with users to validate them. These sessions provided valuable feedback and revealed both strengths and weaknesses of the project vision. Observing users interact with the concepts helped identify usability issues and highlighted areas that required further refinement. The testing phase showed how essential direct user involvement is when designing for public spaces and services.

The final task of this research phase was the creation of a video documenting the progress made throughout the project. The video brought together the different stages of the process, from early research and observations to prototype development and user testing. It served not only as a summary of the work completed, but also as a reflection on the learning experience and the development of the project over time.

Overall, this part of the project provided valuable insights into user-centered design and the complexity of improving public transport experiences. It demonstrated the importance of continuous research, iteration, and testing in creating meaningful and practical design solutions.

References

Videos used from https://www.pexels.com/de-de/