Stem Separation: How It Works

Stem separation is the task of decomposing a mixed audio recording into its constituent sources. Given a finished song where vocals, drums, bass, guitars, synths, and other instruments have been mixed together into a single stereo track, a stem separator attempts to reconstruct approximate versions of each source as if they had been recorded individually. This is fundamentally an "impossible" problem from a signal processing perspective. Mixing is a lossy process: when two signals are added together, information about how to separate them is lost. It is like trying to un-bake a cake to recover the eggs, flour, and butter. Yet modern machine learning approaches achieve surprisingly good results by learning statistical patterns from large datasets of multi-track recordings where the individual stems are known. The applications of stem separation are numerous. DJs use it to create acapellas and instrumentals for remixing. Music educators use it to isolate specific instruments for students to study. Karaoke services use it to remove vocals from songs. Producers use it to sample and remix existing recordings. And music visualizers use it to analyze each instrument independently, creating richer and more nuanced visual responses.

Most stem separation systems produce four or six stems. The four-stem configuration separates a mix into vocals, drums, bass, and "other" (everything else, including guitars, synths, strings, and effects). This is the standard configuration for Spleeter and Demucs and provides the best quality because each stem has a relatively distinct spectral and temporal profile. Six-stem models (Demucs offers an experimental one) further split the "other" category into guitar and piano, leaving a smaller residual "other" stem. Quality tends to decrease with more stems because the task becomes more ambiguous: distinguishing a synth pad from a guitar pad, for example, is harder than distinguishing either from vocals or drums. For specialized applications, models have been trained for specific separations: vocals only, drums only, or dialogue vs. music vs. effects (for film/TV). These single-target models often achieve better quality than multi-stem models for their specific source, since they can dedicate the full network capacity to a single separation task.

Stem separation quality varies significantly depending on the characteristics of the input music. Several factors determine how clean the resulting stems will be. Genre and arrangement complexity are the biggest factors. Sparse arrangements with distinct instruments (acoustic singer-songwriter, jazz trio, classical chamber music) separate much better than dense, heavily layered productions (modern pop, EDM, metal). When instruments occupy distinct frequency ranges and are clearly delineated in the mix, the model has more information to work with. Mixing and mastering choices also matter. Heavily compressed masters with limited dynamic range are harder to separate because the dynamic cues that distinguish sources are diminished. Extreme stereo processing, reverb, and effects can smear source boundaries and reduce separation quality. Conversely, clean, well-separated mixes where each instrument has its own frequency space and stereo position produce the best results. Audio quality affects results as well. Lossless audio (WAV, FLAC) produces better separations than low-bitrate lossy formats (128 kbps MP3), because lossy compression introduces artifacts that the model may interpret as source content. However, high-bitrate lossy formats (256+ kbps) produce results very close to lossless. Vocals generally separate best because the human voice has a highly distinctive spectral profile that the models learn to recognize accurately. Drums separate well because of their percussive transient profile. Bass separates well in isolation but can bleed into the drums stem when kick drum and bass guitar occupy similar frequency ranges. The "other" stem is typically the weakest because it is a catch-all for everything the model could not classify.

Stem separation transforms music visualization from a single-channel response to a multi-dimensional experience. Instead of mapping one audio signal to all visual parameters, a stem-aware visualizer can assign each stem to a dedicated visual layer, creating a richer and more musically meaningful result. The vocal stem can drive lyric-synced text effects, face or figure animations, and color temperature shifts (warm during vocals, cool during instrumental breaks). The drum stem can drive beat impacts, particle bursts, camera shake, and rhythmic visual patterns that respond to individual drum hits rather than the overall energy envelope. The bass stem can drive large-scale visual motion: global color pulses, waveform distortions, and depth-of-field changes that give the visual a visceral, physical quality. The "other" stem (harmonics, melody instruments) can drive harmonic color, geometric complexity, and atmospheric effects. This multi-stem approach produces visualizations that feel more "intelligent" because different visual elements respond to different musical elements, just as a human VJ would operate different visual layers in response to different aspects of the music. The result is a visualization that a viewer can "read" musically: they can see when the bass enters, when the vocal melody rises, and when the drums pull back. Zeltor does not separate stems. Its free Stem Analyzer estimates, from the full mix, how the energy is balanced between vocals, drums, bass and other instruments; the audio stays in your browser.

Several tools are available for stem separation, ranging from free open-source software to commercial services. Demucs (by Meta/Facebook Research) is the current open-source state of the art. It runs locally and is free to use; a GPU makes it much faster, and it also runs, more slowly, on a CPU. The model supports 4-stem and 6-stem separation and offers several model variants optimized for different quality/speed tradeoffs. Spleeter (by Deezer) was the first widely-available deep learning separator and remains popular for its simplicity and speed. While superseded in quality by Demucs, Spleeter is lighter and faster, making it suitable for batch processing and embedded applications. LALAL.AI, iZotope RX, and Moises.ai are commercial services that offer high-quality stem separation through cloud APIs or desktop applications. For music visualization, the quality bar is lower than for professional remixing. Stems used for visual control signals do not need to be artifact-free because listeners are watching the visuals, not isolated stems. A drum stem with minor bleed from other instruments still provides a much better beat-reactive signal than the full mix, and a vocal stem with some background artifacts still clearly indicates when vocals are present.

Stem separation technology is improving rapidly. Each year brings new models with higher SDR scores, faster processing times, and more flexible stem configurations. Several trends are shaping the future of the field. Real-time separation is becoming practical as models become more efficient and hardware becomes more powerful. While current state-of-the-art models require seconds to minutes to process a song, lightweight model variants can run in near-real-time on modern GPUs, enabling live stem separation for DJ performances and real-time visualization. Fine-grained separation beyond the standard four or six stems is an active research area. The goal is to separate individual instruments from a dense mix — isolating the piano from the guitar from the synth pad — which requires models that understand timbral identity at a much finer granularity. This would enable even richer multi-layer visualization. Combining stem separation with other music understanding tasks (chord recognition, melody extraction, beat tracking) into unified models is another trend. Rather than running separate models for each task, a single model could simultaneously separate stems and analyze each one, producing a comprehensive musical description in a single pass.