Noise Suppression in VoiceBlender: From a WASM MVP to a Native-Rate Go RNNoise

VoiceBlender can now remove background noise from call audio. Add "filters": [{"type": "denoise"}] to a leg and the audio from that leg is cleaned before anything else hears it: the room mixer, recordings, speech-to-text, speech detection and answering-machine detection.

Under the hood is rnnoise-go, a pure-Go port of Xiph’s RNNoise. We wrote it after a first version built on WebAssembly worked, but cost too much for telephony. This post explains why we needed denoise, how we tested it, and what went into each version.

Why noise suppression matters on a phone call

Many calls do not come from a quiet room. People call from a busy street, a car, a café, or an open-plan office where twenty other agents are talking. The mobile network carries all of it, and a VoIP platform gets whatever the microphone picked up.

A person on the other end can mostly ignore that noise. Software handles it much worse, and more and more of what VoiceBlender does with call audio is done by software:

• Speech-to-text and voice agents. Background noise causes transcription errors, and a voice agent may take a bus going past as the caller starting to talk.
• Speech detection. VoiceBlender’s speaking.started / speaking.stopped events come from an energy-based voice activity detector. In our tests, with noise alone at three times the VAD threshold, 250 of 250 frames counted as speech before denoising, and 0 of 250 after.
• Answering-machine detection. This is where noise hurt most. More on it below.

Why it has to happen on the server

In WebRTC, noise suppression mostly runs on the user’s device, before the audio is encoded. Browsers have a built-in noiseSuppression option in getUserMedia. Apps that want better results add their own in-browser processing, such as Krisp or longpipe (more on longpipe below). Either way, you ship the code that runs on the device, so you choose the denoiser, and the server gets audio that is already clean.

Telephony doesn’t give you that choice. On a SIP call, the other end is a mobile or landline handset you don’t control. The phone may do some noise reduction of its own, but how much depends on the phone and the network, and you can’t change or even see it. Narrowband codecs like G.711 and AMR-NB then encode the noise along with the voice. What reaches your SIP trunk is whatever the phone and the network passed on.

So the first place you can clean the audio is the media server, which receives it before anything else processes it. Denoising it there means recordings, speech-to-text, speech detection and AMD all get the cleaned audio. And it works the same way whether the caller is using a new smartphone or a twenty-year-old desk phone.

Why noise breaks AMD

VoiceBlender’s AMD, like most AMD, is a state machine that measures energy and time. A person answers with something short, like “Hello?”, and then stops to wait for a reply. A voicemail plays a long greeting without stopping. AMD measures how long the first burst of speech lasts and waits for the silence after it.

Steady background noise means that silence never comes. The street or the office fills the pause, the “greeting” appears to go on and on, and AMD decides it is talking to a machine. On a noisy mobile call, AMD will hang up on real people or pass them to a voicemail flow. For an outbound campaign, those are the calls you most wanted to connect.

How we tested it

AMD already had a test corpus of real voicemail greetings from three public datasets, plus a set of human greetings. That corpus shows AMD works on clean audio. It cannot show whether denoise helps, because clean audio has no noise to remove. So we built noisy versions of it.

Speech. 46 short human greetings (“Hello?”, “Yes, who’s calling?” and so on, in several languages), generated with ElevenLabs TTS (eleven_multilingual_v2). TTS gives us a clean, repeatable speech signal that we can regenerate, and that we can publish without recording real people.

Noise. Two real-world ambient recordings from Pixabay, used under the Pixabay Content License:

• Busy city: 57 seconds of busy-street background noise, 48 kHz stereo.
• Office: 29 seconds of office background noise, 44.1 kHz stereo.

Both are in the repository under tests/data/noise/, so anyone can rebuild the corpus. We chose background noise on purpose, not other people talking nearby. Separating one voice from another is a much harder problem, and not one RNNoise tries to solve.

Mixing. A small tool, cmd/gen-noisy-greetings, decodes each noise file, averages the stereo channels to mono and resamples to 16 kHz. It then mixes each greeting with a random segment of noise at a set signal-to-noise ratio. Some details matter:

• The SNR is measured against the speech’s active level, not the whole file. Otherwise the silence before and after the greeting would lower the speech level, and every mix would come out noisier than intended.
• Segments are drawn from the middle of each recording, skipping a margin at each end, because recordings often start and end quietly. A segment much quieter than the recording’s average lands in a quiet patch, so the tool draws another one instead of passing off near-silence as noise.
• The random seed is fixed, so the same command always produces the same corpus.

We made two SNR levels, 15 dB (noise well below the speech, a fairly ordinary noisy call) and 5 dB (noise almost as loud as the speech). Two noise types at two levels, times 46 greetings, gives 184 test files. Every file is a human answering, so a correct AMD result is always “human”.

The integration test TestAMD_NoisyAccuracy_Denoised runs every file twice: once as-is, and once through the same denoise chain the live AMD feed uses. It reports the numbers instead of asserting a threshold, so it does not lock in today’s figures as a requirement.

Two SNR levels show that denoise helps, but not where AMD starts to fail. For that, a second test, TestAMD_SNRSweep, runs the same 46 greetings at 14 SNR levels from 30 dB down to −10 dB, again with and without denoise.

The first version: RNNoise as WebAssembly

The starting point was a talk at RTC.ON by Sam Bhattacharyya, “Building a Video Effects SDK in the WebGPU and GenAI era”. The talk covered his open-source browser effects SDK, longpipe, which runs virtual backgrounds, reframing and microphone noise removal in the browser. The noise removal uses RNNoise compiled to WebAssembly, running in an AudioWorklet.

A second talk at the same conference, “Hear No Evil: Real-Time AI Noise Suppression on the BEAM” by Feliks Pobiedziński (Software Mansion), came at the problem from the server side. It covered real-time AI noise suppression running on the Erlang VM, in media pipelines rather than in the browser. It convinced me that denoising on the server, per call and in real time, was practical and not only something done on the user’s device.

RNNoise suited us well. It is a small recurrent network (int8-quantised GRUs) that works on 10 ms frames, adds almost no latency, and is BSD-licensed. It has run in production in many places for years. A WASM build also has an advantage for a Go server: we can run it under wazero, a pure-Go WebAssembly runtime, so we keep our CGO_ENABLED=0 static build.

So the MVP embedded the 135 KB rnnoise.wasm module, compiled it once at startup, and kept a pool of instances with 24 denoiser states in each. It worked. With it, AMD accuracy on the noisy corpus went from 1.6% to 98.4%.

Why the WASM version was not good enough

The problem was sample rate. RNNoise, in C and therefore in WASM, is fixed at 48 kHz: it takes 480 samples per 10 ms frame and nothing else. Telephony mostly runs at 8 kHz (G.711, AMR-NB) and 16 kHz (G.722, AMR-WB, Opus wideband). To denoise a narrowband call, the filter chain had to:

1. resample the leg’s 8 or 16 kHz audio up to 48 kHz,
2. run RNNoise,
3. resample back down to the room’s rate.

Each step costs something:

• CPU. RNNoise cost the full 48 kHz price of about 205 µs per frame under wazero, plus two anti-aliasing resamplers. The extra work does not improve anything: an 8 kHz call has nothing above 4 kHz to recover.
• Latency. Each polyphase resampler adds group delay, on audio where every millisecond counts.
• Complexity. Turning denoise on or off in the middle of a call changed the chain’s working rate between 16 kHz and 48 kHz. That meant rebuilding the resamplers and the block size behind a fade.
• Concurrency. States in one WASM instance share a lock, so streams in the same instance run one after another. Under load this was the biggest cost (see the benchmarks below).

RNNoise has no deep reason to need 48 kHz. It measures energy in bands of the spectrum and predicts a gain for each band. The bands are defined in absolute frequencies, and upstream even trains on audio with the top of the spectrum cut off at random. The 48 kHz limit is in the code, not in the model.

rnnoise-go: RNNoise at the call’s own sample rate

So we ported RNNoise to Go: github.com/VoiceBlender/rnnoise-go, BSD-3-Clause like upstream. It is based on the current upstream model (two causal convolutions feeding three GRUs with skip connections, int8 weights), not the older network in most WASM builds.

Bit-exact at 48 kHz first. Before we changed anything else, the port had to match upstream C exactly at every stage: features, pitch, network and output, on silence, voiced speech, noise and a full-scale transient. make diff-vs-c checks this against a C build, and a golden-file test keeps checking it in CI without a C toolchain. That gave us a fixed reference before we started changing sample rates.

Then any sample rate from 8 kHz up. All the rate-dependent values are computed from the sample rate:

• The frame is always 10 ms, so the FFT bins are always 50 Hz apart. The band edges therefore land on the same frequencies at any rate.
• The model’s gains apply only to the bands that fit below the rate’s Nyquist frequency. At 8 kHz that is the bands up to 4 kHz.
• The pitch search runs on native samples, and the detected period is scaled to the 48 kHz sample count the model was trained on.
• A Bluestein FFT covers frame sizes the mixed-radix FFT does not handle.

The lower limit of 8 kHz also comes from upstream training. Upstream cuts off the training audio at a random point between about 3 kHz and 24 kHz. At 8 kHz the input is cut off at 4 kHz, which is in the 7th percentile of what the model saw in training. That is unusual but still within its training data. Below that we would be guessing.

Does it work as well? On real speech, native operation measures within 0.1 dB of resampling to 48 kHz and back, at every rate. The one cost is lower pitch resolution at low rates. It shows up only on strongly periodic signals, where resampling to 48 kHz is still slightly better below 16 kHz. make nativerate-report reproduces the comparison.

Performance work. The network is dominated by int8 matrix-vector products. We wrote AVX2 and NEON assembly kernels for them, bit-identical to the portable Go they replace. The GRU gate kernel went from 256 µs to 5.7 µs (45×), after which it is limited by memory bandwidth. Version 0.3.0 added more SIMD and a dedicated radix-5 FFT butterfly, since a 10 ms frame at 48 kHz is 480 samples (2⁵ × 3 × 5). That took another 5–7% off every sample rate while keeping the output bit-exact. The denoiser allocates nothing after construction. A single read-only Model is shared by every stream, and each stream keeps only its own small recurrent state. There are no instance pools and no locks.

In VoiceBlender, the denoise filter now runs at the rate the leg and room already agreed on, whether 8, 16 or 48 kHz. It never resamples just for denoise, and turning it on or off mid-call no longer changes the working rate.

Results

AMD accuracy on noisy human greetings

Same 184 files and same AMD parameters, scored with no filter, with the WASM MVP and with rnnoise-go:

Without denoise, AMD already fails at 15 dB SNR, where the noise is clearly quieter than the voice. Both denoisers fix that. The native-rate port gets every file right, and it doesn’t resample the AMD feed.

Where AMD breaks: the SNR sweep

The sweep shows where AMD starts to fail, 46 greetings per point:

Without denoise, AMD is fine down to about 28 dB. It starts getting calls wrong around 26–24 dB, gets about half of them wrong at 20 dB, and gets almost all of them wrong from 15 dB down. That is a narrow range, and 20 dB of SNR is not a particularly noisy call.

When it’s wrong, AMD doesn’t just return “not sure”. Almost every error is a machine verdict, so a real person is treated as voicemail and hung up on or sent to a voicemail flow.

With denoise, busy-street noise is handled perfectly down to 0 dB, where the noise is as loud as the voice. Office noise stays above 90% down to 0 dB. Denoise only really gives way below that. In practice this moves AMD’s breaking point about 20–25 dB further into the noise.

The sweep and the 5 dB corpus above draw different random noise segments, so the office result at 5 dB differs slightly between them (95.7% here, 100% above). With 46 files per point, one file is 2.2%, so small differences between neighbouring rows are sampling noise.

Cost per stream

All figures in this section are from rnnoise-go v0.3.0 on an AMD Ryzen 9 7900 (AVX2).

One 10 ms frame, single stream, measured inside VoiceBlender’s filter:

For a wideband call the port costs about a third of what the WASM version did, and that WASM figure does not include the two resamplers it also needed. None of the rnnoise-go paths allocate.

Under concurrency

This is where the WASM design hurt most. With 24 or 48 streams denoising a 48 kHz frame at the same time, on 24 hardware threads:

A round has to finish within the 10 ms frame. At 24 streams the WASM version already used half of that time, because streams in the same instance were waiting on a lock. rnnoise-go streams share only read-only weights, so the cost grows with the number of cores, not the number of streams.

Against the C original

The full BENCHMARKS.md has the method. The rnnoise-go rows below are from our own runs of its capacity benchmark on v0.3.0. They are best of several runs, on voiced speech that never triggers the silence gate, so they are a worst case.

“Sessions per core” is measured with a stream on every hardware thread and all of them busy (12 cores, 24 threads), then divided by the 12 physical cores. That gives lower numbers than dividing 10 ms by the single-stream cost, which suggests about 158 per core at 16 kHz. With the whole chip busy, the clock drops from about 5.5 GHz to 4.2 GHz, each pair of threads shares one core, and every stream reads the model weights through the same L3 cache. Plan with the loaded figure.

At 48 kHz, Go is about 1.4× slower than C. That is the cost of staying bit-exact against the C scalar build: a non-saturating multiply-accumulate, a real divide in the activations, and no fused multiply-add. But telephony does not run at 48 kHz. For a 16 kHz call, rnnoise-go is about 1.65× cheaper than native C, because C pays the full 48 kHz cost plus resampling in both directions. At the highest Speex resampling quality, the resampling alone costs more than the whole Go denoiser at 16 kHz.

For capacity planning, count about 130 denoised wideband calls per core on this machine, or about 1,550 across all 12 cores. Real calls have pauses, and the silence gate skips the network on those frames, so actual cost is lower.

Using it

Per leg:

curl -X POST http://localhost:8080/v1/legs \
  -H 'Content-Type: application/json' \
  -d '{
    "type": "sip",
    "to": "sip:+15551234567@carrier.example.com",
    "amd": {},
    "filters": [{"type": "denoise"}]
  }'

Server-wide, for every new leg:

AUDIO_FILTERS=denoise

It can also be turned on or off during a call with PUT /v1/legs/{id}/filters, for example once an agent notices the caller is on a busy street.

AMD uses the leg’s filter configuration. It never enables denoise itself, but if the leg has denoise, AMD analyses the denoised audio. That applies from ringing onwards, before the leg has joined a room. Based on the numbers above, if you run AMD on calls from mobile users or noisy environments, enable denoise on those legs.

Two things to know about RNNoise:

• It needs about a second to settle at the start of a call while it learns the noise.
• It is a speech enhancer. Put it before voice effects like robotic or pitch in a chain, not after.

rnnoise-go is a standalone library with no VoiceBlender dependencies. It has no cgo and one dependency (golang.org/x/sys). The 3.4 MB model ships with the library, so a single go get github.com/VoiceBlender/rnnoise-go is all you need. You can also load the weights from a file instead. It comes with a small CLI, so you can try it on your own recordings without writing any code:

go install github.com/VoiceBlender/rnnoise-go/cmd/rnnoise@latest
rnnoise noisy.wav clean.wav

The library has a few options that VoiceBlender doesn’t use yet. By default it behaves exactly like upstream, and these change that:

• GainFloorDB limits how much any band can be turned down, for example -20 for at most 20 dB. That leaves some noise in, but avoids dropouts on quiet speech and the “musical noise” effect of strong suppression.
• Aggressiveness raises each band’s gain to a power: below 1 it suppresses less, above 1 more.
• Fast switches to a vectorised pitch search that is no longer bit-exact with C. It saves about 8% at 48 kHz and under 1% at 16 kHz, and in testing the output doesn’t change.

The CLI has -floor and -aggressiveness flags for the first two, so you can hear the difference on a recording before choosing values.

If you need noise suppression at telephony sample rates in Go, you can use it directly.

Thanks to Jean-Marc Valin and the RNNoise contributors for the algorithm and the trained weights, to Sam Bhattacharyya, whose RTC.ON talk and longpipe got this started, and to Feliks Pobiedziński, whose RTC.ON talk made the case for doing it on the server.