The math of listening

Samples: what sound is to a computer

A microphone's converter measures air pressure R times a second. Meatball records at 8 or 16 kHz. Each measurement is a 16-bit signed integer (s16le), a number between −32768 and +32767. The first thing we ever do is normalise it to a float in [−1, 1]: x = int16 / 32768.

Two facts decide everything downstream. Nyquist: a sample rate R can only represent frequencies up to R/2. At 8 kHz that's a 4 kHz ceiling, fine since speech lives ~300 to 3400 Hz. And the DC-offset trap: a cheap converter adds a constant bias. We measured −531 counts on one dongle, a pure 0 Hz component reading as a healthy −35 dBFS "signal," which fooled me for an hour into thinking a dead mic was alive. So we always subtract the mean (x −= x.mean()); the true floor underneath was −76 dBFS, 40 dB lower. Subtract the mean, always.

From time to frequency: the DFT, and why the FFT

Fourier's idea: any signal is a sum of sine waves. The Discrete Fourier Transform asks "how much of each frequency is in these N samples?"

X[k] = Σ  x[n] · e^(−2πi·kn/N)
       n = 0 … N−1

Each X[k] is a complex number: its magnitude is how much of that frequency is present, its phase is where the wave sits in time. Bin k maps to frequency f = k·R/N, so frequency resolution is R/N: bigger N means finer bins but a longer frame (more latency). At N=1024, 8 kHz:

frequency resolution = R/N = 7.812 Hz per bin
frame length         = N/R = 128.0 ms

The DFT as written is O(N²). The FFT (Cooley–Tukey, 1965) computes the exact same numbers in O(N log N)by recursively splitting the sum into even and odd samples, for N=1024 about 100× fewer operations. That's the difference between real-time and not.

Which FFT: the real one

Audio is real-valued. The DFT of a real signal is Hermitian-symmetric (X[N−k] = conj(X[k])), so the top half of the spectrum is a mirror image, pure redundancy. We use the real FFT (numpy.fft.rfft): it returns only the unique N/2 + 1 bins (0 Hz to Nyquist), about 2× faster and half the memory. Every script in the rig uses it.

X     = np.fft.rfft(frame * window)   # 513 complex bins for N=1024
freqs = np.fft.rfftfreq(N, 1/R)       # the Hz value of each bin

The magic dust: windowing

The FFT secretly assumes your N samples repeat forever. A real chunk doesn't loop cleanly, so the jump at the wrap-around edge smears one tone's energy across many bins: spectral leakage. The fix: multiply the frame by a window that tapers to zero at both ends. We use the Hann window. Live proof, a 440 + 1200 Hz test signal:

no window:    6.07 %   energy smeared across the spectrum
Hann window:  0.03 %   ← 200× cleaner

Cost: the main peak gets a hair wider. Worth it every time. (A footnote that teaches a lot: the 440 Hz tone landed in the 437.5 Hz bin: 440 isn't an exact multiple of 7.8125 Hz, so it sits between bins and spreads to its neighbours. Bins are discrete; the world isn't.)

Reading the bins: magnitude, power, dBFS

Magnitude |X[k]| is the amplitude at that frequency; power is its square. We report in dBFS (decibels relative to full scale): 20·log10(amp), where 0 is the max and everything else is negative. The DC-blocked floor sits near −76 dBFS; speech towers ~40 dB above it. And RMS(the loudness of a frame) equals the total spectral energy (Parseval's theorem), so the gate can watch a cheap time-domain number and "see" exactly what the FFT shows.

Special sauce: spectral subtraction

Sample the quiet room for ~15 s and average |X[k]| over every frame: that's a noise fingerprint. Boll's 1979 idea: every live frame is speech plus that same noise, so subtract the noise's magnitude and keep the original phase:

X   = rfft(frame · hann)          # complex spectrum
|S| = max(|X| − α·|N|,  β·|X|)    # subtract noise mag, floored
S   = |S| · e^(i·phase(X))        # reattach the original phase
y   = irfft(S);  overlap-add      # back to the time domain

α is how hard you scrub; β is a spectral floor that stops musical noise(the warble of bins flickering on and off). Here's the part nobody tells you: run live on the hissy QuickCam, target "the quick brown fox":

RAW                    floor −22 dBFS   "the quick brown BOX …"   ✗
α=1.0 β=0.15  gentle    floor −32 dBFS   "the quick brown FOX …"   ✓ fixed it
α=1.5 β=0.08           floor −39 dBFS   "the quick brown BOX …"   ✗
α=2.5 β=0.02  greedy    floor −55 dBFS   "(nothing)"               ✗ destroyed

It's a U-curve. The objective was never minimum noise floor: crush it 33 dB and you also crush the speech into artifacts the recogniser can't read. The sweet spot shaves ~10 dB, just enough to flip box → fox, and stops. The best-sounding result (lowest floor) was the worst transcript. Optimise the right metric.

Profit: the voice gate

Now the box can decide on its own when someone's talking. All time-domain: measure each ~30 ms frame's RMS in dBFS, smooth it with a one-pole filter, and run a tiny state machine with two thresholds:

db  = 20·log10( rms(DC-blocked frame) )
ema = a·db + (1−a)·ema                 # smooth, a ≈ 0.4

OPEN   when ema > open_thresh          # a turn begins → record → transcribe
CLOSE  when ema < close_thresh for ~0.5 s   # hangover: pauses don't cut you off

Two thresholds (hysteresis) so a level hovering at the edge doesn't chatter the gate; a hangoverso a breath mid-sentence doesn't end your turn. Both come straight from the fingerprint: open = floor + 12, close = floor + 6dBFS. On Meatball's best ear that's open −28, close −34.

And that's the whole arc: samples became a spectrum, the spectrum became a fingerprint, the fingerprint set the gate, and a junk-pile in a garage can hear you coming.