How MusicMind Analyzes Music

Every audio analysis pipeline begins with the same fundamental operation: transforming the time-domain signal into the frequency domain using the Fast Fourier Transform (FFT). A raw audio signal is a sequence of amplitude samples — 44,100 per second for CD-quality audio. The FFT decomposes a window of these samples into frequency components, revealing what notes, harmonics, and noise are present at each moment.

In practice, we use the Short-Time Fourier Transform (STFT): slide a window across the signal (typically 2048 samples with 50% overlap), apply a Hann window function to reduce spectral leakage, then compute the FFT of each windowed frame. The result is a spectrogram — a 2D matrix where rows are frequency bins, columns are time frames, and values are magnitudes.

// Web Audio API FFT setup
const analyser = audioContext.createAnalyser();
analyser.fftSize = 2048;  // 1024 frequency bins
const freqData = new Uint8Array(analyser.frequencyBinCount);

// Each bin represents ~21.5Hz (44100 / 2048)
// Bin 0:   0-21.5Hz    (sub-bass)
// Bin 10:  215-236Hz   (bass)
// Bin 50:  1075-1097Hz (mid-range)
// Bin 200: 4300-4322Hz (presence)

analyser.getByteFrequencyData(freqData);

The FFT gives us the raw material. Every subsequent analysis — key detection, onset detection, spectral features, stem estimation — operates on this frequency-domain representation. The choice of window size creates a fundamental trade-off: larger windows give better frequency resolution (narrower bins) but worse time resolution (smeared transients). 2048 samples at 44.1kHz gives ~46ms frames — a reasonable compromise for music analysis.

Musical key detection is one of the most valuable features for visualization — it determines color palette, emotional tone, and harmonic context. The standard approach uses Harmonic Pitch Class Profile (HPCP) features.

Step 1: Spectral peaks. From the FFT magnitude spectrum, identify local maxima above a noise threshold. These peaks correspond to the fundamental frequencies and harmonics of the notes being played.

Step 2: Pitch class mapping. Map each spectral peak to one of 12 pitch classes (C, C#, D, ... B) using the formula: pitchClass = 12 * log2(freq / referenceFreq) mod 12. This "folds" all octaves together — a 440Hz A and an 880Hz A both contribute to the A pitch class.

Step 3: HPCP vector. Sum the magnitudes for each pitch class across all peaks, weighted by a cosine window centered on the exact pitch class frequency. The result is a 12-dimensional vector representing the energy in each pitch class — the chroma feature.

Step 4: Key profile matching. Compare the averaged HPCP vector against 24 key profiles (12 major + 12 minor) using cosine similarity. The Krumhansl-Kessler key profiles, derived from perceptual experiments, provide the reference. The best-matching profile gives the detected key and a confidence score.

// MusicMind key detection: chroma → KK profile correlation
const chroma = computeChromaFromBuffer(samples, sampleRate);
const keyResult = detectKeyOffline(chroma);

// Returns:
// keyResult.key       = "A"      (tonic)
// keyResult.scale     = "minor"  (mode)
// keyResult.strength  = 0.82     (confidence 0-1)

Accuracy on well-produced commercial music is typically 80-90%. Common failure modes: highly chromatic jazz (ambiguous tonal center), key modulations within a song (the algorithm returns the dominant key), and heavily distorted guitar (harmonic content obscures the fundamental).

Onset detection identifies the moments when new musical events begin — drum hits, note attacks, vocal entrances. For visualization, onsets are the primary triggers for visual events: flashes, shockwaves, glitch effects, and transitions.

The SuperFlux algorithm (used in our pipeline) improves on simple spectral flux by applying a maximum filter across frequency bins before computing the difference. This suppresses spectral modulations from vibrato and tremolo while preserving true onset energy:

// SuperFlux onset detection (conceptual)
for each frame t:
  // Standard spectral flux
  flux[t] = sum(max(0, |X[t]| - |X[t-1]|))

  // SuperFlux: apply max-filter first
  X_max[t] = maxFilter(|X[t-1]|, width=3)
  superflux[t] = sum(max(0, |X[t]| - X_max[t]))

// Peak-pick onsets from superflux curve
onsets = peakPick(superflux, threshold, minDistance)

Transient classification goes a step further: once an onset is detected, we classify it as "hard" (percussive — drums, plucks, clicks) or "soft" (tonal — swells, pads, sustained notes). The classification uses the spectral flatness at the onset frame: high flatness (noise-like spectrum) indicates a hard transient; low flatness (harmonic peaks) indicates a soft onset.

In our visualization pipeline, hard onsets trigger immediate visual responses — flashes, particle explosions, camera jolts. Soft onsets trigger gradual morphs — color shifts, bloom increases, slow camera movements. This distinction is what makes the visualization feel "musical" rather than simply "reactive."

Tempo estimation finds the dominant periodicity in the onset/energy signal. MusicMind's BPM estimator computes the autocorrelation of the onset detection function, then identifies the lag corresponding to the strongest periodic pattern in the 60-200 BPM range.

Beat tracking extends this by placing individual beat positions on a grid. The algorithm:

1. Estimates tempo from autocorrelation
2. Places candidate beat positions at the estimated inter-beat interval
3. Adjusts positions using dynamic programming to maximize alignment with detected onsets while maintaining tempo consistency
4. Handles tempo changes by allowing the inter-beat interval to vary within a constrained range

For visualization, the beat grid is critical for deterministic synchronization. Every animation, transition, and camera movement snaps to the beat grid. This ensures that the same audio input always produces identical visual output — a requirement for video export, where frames must be reproducible.

Common challenges: polyrhythmic music (competing periodicities), rubato passages (tempo fluctuations), and electronic music with intentional off-grid elements. The beat tracker's confidence score helps the visualization system decide whether to strictly quantize or allow more fluid timing.

Beyond key and tempo, a rich set of spectral features captures the timbral character of the audio — the "texture" that makes a track sound bright or dark, smooth or harsh, simple or complex.

These features are computed per-frame and smoothed over short windows (50-200ms) to reduce noise. The combination creates a multi-dimensional "audio fingerprint" that evolves over time — each moment of the song has a unique spectral character that drives a unique visual treatment.

MusicMind is a 14-layer musical cognition engine built entirely on native Web Audio APIs and custom signal processing algorithms. Rather than depending on external WASM binaries or third-party libraries, every algorithm — from FFT-based chroma extraction to Krumhansl-Kessler key profile correlation — is implemented in pure JavaScript, optimized for the specific analyses that music visualization requires.

Architecture: The analysis pipeline runs synchronously on the main thread in under 200 milliseconds for a typical 4-minute song. No WebWorker overhead, no WASM compilation step, no 2MB binary download. Analysis begins the instant the audio buffer is decoded — results are available before the first visual frame renders.

// MusicMind native analysis pipeline
const samples = buffer.getChannelData(0);
const sampleRate = buffer.sampleRate;

// Key detection: FFT → chroma vector → KK profile correlation
const chroma = computeChromaFromBuffer(samples, sampleRate);
const key = detectKeyOffline(chroma);  // Pearson correlation vs 24 profiles

// BPM: autocorrelation of onset strength function
const { bpm, confidence } = BeatTempoEngine.estimateOffline(buffer);

// Energy, complexity, loudness: windowed analysis
const energy = computeEnergy(samples);
const { dynamicComplexity, loudness } = computeDynamicComplexity(samples, sampleRate);

// All results available synchronously — no worker, no async

Why native beats WASM for this use case: The conventional wisdom is that WASM is always faster than JavaScript for compute-heavy tasks. For a general-purpose library running 200+ algorithms, that's true. But MusicMind runs exactly 5 targeted analyses on the audio buffer. Modern JavaScript engines (V8, SpiderMonkey) JIT-compile tight numerical loops to near-native speed. The overhead of downloading, compiling, and instantiating a 2.5MB WASM binary — plus the cost of copying data across the WASM boundary — exceeds the computation time for our focused analysis set.

The result: analysis completes in ~200ms vs. 3-5 seconds with WASM-based approaches. No unsafe-eval CSP requirement. No binary downloads. Zero external dependencies. The entire analysis engine ships as part of the application bundle at negligible size.

Raw audio features are numerical signals. The creative challenge is mapping these signals to artistic decisions: which visual effect to use, when to transition, how the camera should move, what post-processing creates the right emotional tone.

Our Auto-Director operates in two phases:

Phase 1: Offline screenplay generation. Before rendering begins, the director analyzes the full audio signal and generates a "screenplay" — a timeline of visual events. This includes: effect assignments for each section, transition points aligned to the beat grid, camera movement curves, post-processing parameter envelopes (bloom, chromatic aberration, grain), and color palette derivation from the detected key.

Phase 2: Real-time rendering with micro-adjustments. During playback, the screenplay provides the macro structure, but frame-level parameters are driven by real-time audio features. Bass energy modulates particle density on every frame. Onset detection triggers per-beat visual events. Spectral centroid adjusts color temperature continuously. The screenplay is the plan; the real-time features are the improvisation.

The director's creative personality is defined by a "vibe profile" — a parameter set that controls preferences for effect types, transition frequency, camera movement intensity, post-processing character, and color temperature. Six profiles (Cinematic, Hyperpop, Ambient, Aggressive, Minimal, and Auto) provide distinct visual personalities. Auto mode selects a profile based on the detected audio characteristics — a 140 BPM minor-key track with high spectral flux gets the Aggressive profile; a 72 BPM major-key track with low energy gets Ambient.

The technical pipeline described here — FFT → chroma → key detection, FFT → onset → beat tracking, FFT → spectral features → timbral analysis — transforms a stream of audio samples into a rich, multi-dimensional representation of musical content. Each dimension (key, tempo, onsets, spectral character, stems) provides a different "handle" for visual expression.

The shift from volume-reactive visualizers to feature-rich, intelligently directed systems represents a fundamental change in what music visualization can be. It's no longer "make things bounce when the music is loud." It's "understand what the music is doing and create an appropriate visual interpretation." The platform is universal (Web Audio API runs everywhere), the analysis is instant (under 200ms on a laptop), and no external dependencies are required.

The remaining frontier is creative, not technical. The algorithms can extract what the music is doing. The open question is: what should the visuals do in response? That's where human taste, artistic intuition, and intelligent creative direction converge — and where the most interesting work is happening now.