Walking through a latent room impulse response

A room impulse response autoencoder, frozen and never fine-tuned, turns out to organise its latent space by microphone coordinate. Move the listener and the response moves with it — continuously, including through positions that were never measured.

MeshRIR · 441 mics · 5 cm grid FLAC VAE · pretrained, frozen latent 32 × 10 = 320 numbers

Move the microphone

Drag anywhere on the map. Small dots are the 441 real measurement positions; every other pixel is a decoded impulse response at a coordinate nobody measured.

Continuous: defined at every coordinate, including the three quarters of the map that were never measured.

—position
—to source
—arrival
—DRR
—C50

loading field…

What “moving in the latent space” actually means

Each measured impulse response h(x) is pushed through the encoder to a point z(x) — 320 numbers. Do that for all 441 microphones and you get 441 points. They are not scattered: they form a smooth two-dimensional sheet inside that 320-dimensional space, indexed by where the microphone was standing.

So “moving” means: pick any coordinate, including one between microphones, read the corresponding point off that sheet by interpolating its neighbours, and run it through the decoder. What comes back is a 10 240-sample impulse response at 22.05 kHz — a real waveform you can listen to, for a position nobody ever measured.

Every position on the map above is one such decode. At the 441 measured spots it approximates the real recording; everywhere in between it is invented, and the point of the demo is that it is invented smoothly.

Where the position information lives

The latent is 32 channels × 10 time frames, each frame covering 46 ms of the response. Position is not spread evenly across it: it sits almost entirely in the first two frames, the direct sound and the early reflections. The remaining eight, the reverberant tail, carry very little.

That makes sense. This is one room, so the late field is diffuse and decays more or less the same way wherever you stand; there is no position information in it to find.

Use only those first two frames and the leading direction of the latent correlates 0.988 with the direct sound's time of arrival. Can we say: a frozen autoencoder, trained on a different dataset entirely, spends its main axis on time of flight?

Six panels analysing the latent field: position information per latent frame, the physical grid drawn in latent space using all frames versus the first two, latent PCA coloured by source distance, PC1 painted onto the mic plane, and latent distance against physical distance.
Top middle versus top right is the whole argument. The same 441 microphones, each joined to its four grid neighbours and drawn in latent space: a tangle when you use all ten latent frames, a mesh when you use only the first two.

Not good for:

Accuracy. Predicting a held-out microphone from its neighbours, simply copying the nearest measured response wins. Microphones 5 cm apart produce responses that differ by less than the autoencoder's own reconstruction error, so the model has nothing to add. Only when the measurements thin out to roughly half a metre does latent interpolation start to win.

With no training, no fine-tuning, and 320 numbers per response, copying the nearest measured RIR is more accurate than anything we decode. But a lookup table only answers where you measured. We can decode continuously inside the microphone grid.