Stem Separation for Visuals

Stem separation (also called source separation or demixing) is the process of isolating individual instruments or vocal tracks from a mixed audio recording. A typical separation model splits a song into four stems: vocals, drums, bass, and other (which captures everything else like guitars, synths, and keys). This technology has advanced rapidly thanks to deep learning, with models like Demucs, Open-Unmix, and Spleeter achieving impressive separation quality. For music visualization, stem separation is transformative because it allows each musical element to drive different visual layers independently. Instead of mapping a single mixed signal to all visuals, you can have the kick drum control one effect, the vocals control another, and the bass drive a third.

Modern stem separation models are neural networks trained on thousands of songs where the individual stems are available (typically from multitrack recording sessions). The most common architecture is a U-Net, a type of convolutional neural network originally designed for image segmentation. The model takes the mixed audio spectrogram as input and predicts a mask for each stem. Multiplying the original spectrogram by each mask isolates that stem. Newer models like Hybrid Transformer Demucs combine spectral and waveform domains, processing the audio in both time and frequency representations simultaneously. Training requires paired data (mix plus individual stems), which is why large labeled datasets from professional recordings are essential. The models learn to recognize the spectral fingerprints of different instrument types.

Without stem separation, a visualizer has access only to the mixed signal. This means a loud kick drum and a loud vocal note look identical to the analysis pipeline: both are just spikes in energy. With separated stems, each instrument can be analyzed independently and mapped to distinct visual behaviors. The vocal stem might drive lyric-synced text animations or smooth color shifts. The drum stem provides precise onset information for rhythmic flashes and camera shakes. The bass stem controls low-frequency visual pulsing and bloom intensity. The "other" stem, often containing harmonic content like pads and guitars, can drive ambient textures and background motion. This multi-stem approach creates richer, more musical visualizations that feel intentionally choreographed.

No separation model is perfect. Common artifacts include bleed (remnants of other instruments appearing in a stem), phasing (subtle tonal changes caused by imperfect reconstruction), and temporal smearing (transients losing their sharpness). These artifacts matter less for visualization than for audio production, because the stems are used for analysis rather than playback. Even a vocal stem with some drum bleed still provides a much cleaner signal for driving vocal-specific visuals than the raw mix. However, separation quality does affect the precision of onset detection and energy tracking on individual stems. Higher-quality models produce cleaner analysis data. It's worth noting that separation quality varies by genre: pop and electronic music with clear mix separation tends to yield better results than dense orchestral arrangements.

Stem separation is computationally expensive. Running a full Demucs model on a five-minute song can take 30 seconds to several minutes depending on hardware. GPU acceleration significantly speeds up processing. For web-based tools, separation is typically performed server-side or via WebAssembly ports of lighter models. Zeltor Labs does not run a separation model today. Its analysis estimates energy in frequency bands (low end, mids, vocal range, highs) from the mixed audio, in your browser, which is fast and keeps your audio on your device but cannot fully isolate instruments that share a range.