Temporal-Modal Replay for Continual
Audio-Conditioned 3D Motion Generation

Temporal-Modal Replay (TMR) advances continual learning for audio-conditioned 3D motion generation through: (a) a unified framework covering three fundamentally different generator families, including flow matching, autoregressive VQ-VAE, and diffusion, across two qualitatively distinct audio conditioning types, including speech and spatial binaural audio; (b) the first continual learning benchmark for audio-conditioned 3D motion generation, evaluated against twelve baselines spanning all four CL paradigms, including memory-based, regularization-based, optimization-based, and architecture-based methods; (c) a theoretically grounded temporal feature operator whose principal eigenspace uniquely minimizes replay-induced forgetting among all equal-rank projections; (d) up to 75% reduction in catastrophic forgetting, lossless retention with 25% fewer replay samples, and 59% improvement in feature-space coverage over competing replay methods; (e) near-flat long-horizon retention across nine sequential sessions where vanilla fine-tuning catastrophically diverges, using only a 15% replay buffer; and (f) zero additional trainable parameters, zero inference overhead, and negligible replay construction cost, working across generator architectures with no auxiliary model or architectural modification.

BEAT2 (Speech conditioned full body 3D gestures). The goal is to generate natural, coherent, and synchronized motion of the lips, facial expressions, hands, legs, and full body, which TMR is able to retain even after 9 long sessions while maintaining low FGD.

MOSPA (moving towards sound source). The goal is to generate coherent 3D motion toward the sound source, which is clearly captured by TMR after the second session for the Sensitive genre.