VOX
INANIS

What if you could embed a hidden message into an audio file? One that's completely inaudible to human ears, but that an AI speech-to-text model like OpenAI's Whisper would confidently transcribe.

And what if you could flip that around? Add a subtle layer of noise to your voice recordings that makes them untranscribable, effectively anonymizing your speech against automated surveillance.

That's Vox Inanis (Latin for "Voice of the Void"). A research toolset exploring both sides of adversarial audio: injection (forcing a model to hear a target phrase) and disruption (preventing a model from transcribing anything useful).

Vox Inanis uses gradient-based adversarial attacks against OpenAI's Whisper model. It takes any piece of audio (speech, music, ambient noise) and computes a tiny perturbation that changes what Whisper transcribes when added to the original signal.

The perturbation is bounded. Each audio sample can change by at most a fraction of a percent, controlled by an L∞ budget (typically 3-5% of full scale). To human ears, the audio sounds identical. To Whisper, it says something completely different.

Both modes produce audio that sounds identical to the original when played back through speakers. The changes exist purely in the statistical patterns that neural networks rely on.

Adversarial attacks on neural networks have been studied extensively in the image domain since Szegedy et al. (2013) and Goodfellow et al.'s Fast Gradient Sign Method (2014). Vox Inanis builds on several lines of prior work in audio:

To my knowledge, Vox Inanis is the first open implementation that combines MI-FGSM optimization, STFT-domain perceptual masking, and codec-in-the-loop training with a Straight-Through Estimator in a single pipeline targeting Whisper's encoder-decoder architecture.

DIFFERENTIABLE WHISPER

Whisper's model.transcribe() API is not differentiable. It performs autoregressive beam search with discrete token selection. I bypass it and call the encoder and decoder directly:

audio (16 kHz mono, len ≤ 30s)
  → whisper.audio.pad_or_trim
  → whisper.audio.log_mel_spectrogram   // differentiable STFT + mel + log
     shape: (1, n_mels, 3000)
  → model.encoder                       // frozen transformer
     shape: (1, 1500, n_audio_state)
  → model.decoder(input_ids, audio_features)
  → logits, shape (1, S, vocab)
  → cross-entropy loss vs. target tokens
  → backpropagate gradients to raw audio samples

The log_mel_spectrogram function is pure PyTorch ops (STFT, matmul, log, clamp), so gradients flow through it. The encoder and decoder are frozen (requires_grad=False on every parameter). Only the input audio waveform accumulates gradient. I'm not training the model. I'm training the input.

FORCED-PREFIX DECODING

Whisper's decoder is conditioned on a prefix of special tokens that select language and task: [<|startoftranscript|>, <|language|>, <|transcribe|>, <|notimestamps|>]. I feed the full sequence [prefix, target_tokens] to the decoder and compute cross-entropy only on the content positions. Standard teacher-forcing loss restricted to the positions that matter.

Language sensitivity is critical. The attack optimizes against a specific language prefix token. If validation uses Whisper's auto-language-detection and the detected language differs from the attack language, the attack fails silently. Whisper decodes with a different prefix and never enters the high-probability region the optimizer found.

MI-FGSM OPTIMIZER

I use MI-FGSM (Momentum Iterative Fast Gradient Sign Method) with L∞ projection:

// Normalize current gradient by L1 mean (stabilizes across steps)
grad_norm = grad / (grad.abs().mean() + 1e-12)
// Accumulate with momentum (decay = 0.9)
grad_momentum = 0.9 * grad_momentum + grad_norm
// Sign-step with cosine-scheduled learning rate
delta = delta - effective_lr * grad_momentum.sign()
// Project back into L-inf epsilon-ball
delta = delta.clamp(-eps, eps)

The cosine learning rate schedule decays from lr_max to lr_min over the run. Large steps early for exploration, small steps late for refinement. The perturbation is initialized with uniform random noise within the epsilon-ball rather than at zero, which empirically improves convergence by avoiding symmetric gradient cancellation.

I also apply logit temperature scaling (T = 0.8) to sharpen the decoder's output distribution before computing cross-entropy. This provides stronger gradients early in optimization when the model is still uncertain.

PERCEPTUAL MASKING LOSS

The L∞ constraint bounds peak amplitude but says nothing about audibility. A perturbation spread uniformly across all samples is far more audible than the same energy concentrated where the cover audio is already loud. I enforce perceptual hiding with an STFT-domain masking loss:

cover_mag    = |STFT(x)|
delta_mag    = |STFT(delta)|
threshold    = cover_mag * 10^(mask_db / 20)
masking_loss = mean(ReLU(delta_mag - threshold)^2)

L_total = L_CE + lambda * L_mask

The STFT uses a Hann window with n_fft = 1024 and hop length = 256 at 16 kHz, giving 8 ms hop and 64 ms window. At mask_db = -25, the perturbation may be at most ~1/18 of the local cover amplitude in each time-frequency bin. Bins exceeding this threshold contribute quadratic penalty. Bins below contribute zero.

The tradeoff: lambda = 0 gives a pure CE attack (converges fastest, fully audible). Lambda in [10, 200] is the practical range. Lambda = 50 works well for covers with high-entropy texture (noise, music, ambient). Speech with quiet pauses needs higher lambda to keep the silent gaps clean.

CODEC-IN-THE-LOOP (STE)

A perturbation optimized against raw audio will not survive AAC compression at 128 kbps. To produce codec-robust attacks, I route the perturbed audio through ffmpeg inside the optimization loop. ffmpeg has no usable Jacobian, so I wrap it in a Straight-Through Estimator: the forward pass runs the real ffmpeg pipeline (resample to 44.1 kHz, AAC encode, decode, resample back to 16 kHz), the backward pass treats the codec as the identity function.

The STE is mathematically loose (it ignores the codec's actual local behavior) but empirically effective. To make this fast enough for iterative use, I replaced the file-based ffmpeg pipeline with a pipe-based version that streams audio through stdin/stdout, eliminating all disk I/O.

GRADIENT CHECKPOINTING

PGD requires the full backward pass through encoder and decoder. Storing all intermediate activations approximately doubles peak memory vs. inference-only. To run Whisper small (460 MB) on GPUs with limited VRAM, I wrap each transformer block in torch.utils.checkpoint.checkpoint(use_reentrant=False). Activations inside each block are discarded after the forward pass and recomputed during the backward pass. Trades ~1.4x compute for ~3x lower peak memory.

RUN 24EC741A // TARGETED INJECTION ON REAL SPEECH

Cover audio: ElevenLabs-generated speech, 22.52 seconds, 360,281 samples at 16 kHz mono. A mock job interview in German. The speaker introduces themselves, mentions a preference for strong coffee, and describes their approach to complex systems. Fluent, natural German speech with varied prosody and natural pauses.

Target text: "hello world! how are you" (English)

HYPERPARAMETERS

LISTEN FOR YOURSELF

TRANSCRIPTION ACROSS STAGES

The clean transcription is accurate and complete. Whisper small has no difficulty with the ElevenLabs voice. After injection, the original 58-word transcript is replaced entirely by the two-word target phrase. After the YouTube codec round-trip, the injected payload degrades to garbled fragments, but the original speech is not recovered. Whisper produces only 6 tokens of nonsensical output.

Current state: The injection works, but it's audible. You can clearly hear crackling and a white noise layer on the perturbed files. The next goal is to push the perturbation further below the audibility threshold and see how invisible I can actually make this.

CONVERGENCE

SIGNAL METRICS

DIAGNOSTIC VISUALIZATIONS

KEY FINDING: DISRUPTION SURVIVES THE CODEC

The target phrase "hello world! how are you" did not survive the YouTube codec. The perturbed-stage transcription hits the target, but the after-YouTube transcription produces "Yes, my lord, ollh good!" instead.

However, the original speech was not recovered either. The 128 kbps AAC round-trip destroyed the precise adversarial pattern that encoded the target phrase, but it also destroyed the cover audio's original statistical features that Whisper relied on for accurate transcription. The result is complete transcript annihilation: neither the target phrase nor the original speech survives.

This has direct implications for the defensive use case. An adversarial perturbation optimized for injection also functions as a disruption tool, even when the injection itself fails. The codec strips the payload but leaves the damage.

Every voice recording you make (video calls, voice messages, podcast appearances, ambient audio from smart devices) can be fed into STT models and transcribed without your knowledge or consent. The resulting text is searchable, indexable, and permanent. Your spoken words become data.

Run 24ec741a was designed as a targeted injection experiment, but its after-codec result demonstrates the anonymization principle empirically: the perturbation completely prevented Whisper from recovering the original 58-word interview transcript, even after lossy compression stripped the injected payload. What remains is 6 tokens of garbled nonsense.

Any responsible publication of attack capabilities must be paired with defense analysis. The following defenses are under investigation:

Vox Inanis ships with an interactive web interface for experimenting with attacks in real-time. Built on FastAPI with server-sent events for live progress streaming.

Upload any audio clip or generate synthetic pink noise. Configure all attack parameters (epsilon, optimizer, masking depth, codec simulation). Watch the PGD loss converge in real-time. Compare clean vs. perturbed vs. post-codec audio side by side. Inspect 8 diagnostic visualizations per run. Runs on Apple Silicon MPS or CUDA, completing in about 5 minutes for a 22-second clip at 300 steps.

Building Vox Inanis taught me that the boundary between "what humans hear" and "what machines hear" is surprisingly thin, and exploitable in both directions. A perturbation smaller than background noise can completely rewrite a transcript. The same technique that enables injection also enables defense.

The adversarial ML literature is full of image-domain attacks, but audio is a different beast. The signal is one-dimensional, the relevant features live in the frequency domain, and real-world deployment means surviving lossy compression. Making all of this differentiable (including the codec) was the core engineering challenge.

The most surprising finding from this phase was not that the injection worked. It was that the disruption survived the codec even when the injection didn't. That accident points toward the most impactful application: not putting words in Whisper's mouth, but taking them away.