RFWave: Multi-band Rectified Flow for Audio Waveform Reconstruction
Abstract
Recent advancements in generative modeling have significantly enhanced the reconstruction of audio waveforms from various representations. While diffusion models are adept at this task, they are hindered by latency issues due to their operation at the individual sample point level and the need for numerous sampling steps. In this study, we introduce RFWave, a cutting-edge multi-band Rectified Flow approach designed to reconstruct high-fidelity audio waveforms from Mel-spectrograms or discrete tokens. RFWave uniquely generates complex spectrograms and operates at the frame level, processing all subbands simultaneously to boost efficiency. Leveraging Rectified Flow, which targets a flat transport trajectory, RFWave achieves reconstruction with just 10 sampling steps. Our empirical evaluations show that RFWave not only provides outstanding reconstruction quality but also offers vastly superior computational efficiency, enabling audio generation at speeds up to 97 times faster than real-time on a GPU.
Configuration Definitions
+CFG2 : Classifier-free guidance with a guidance coefficient of 2.0
+STFT : Applied STFT loss
011 prompt: He dashed back across the road, hurried up to his office, snapped at his secretary not to disturb him, seized his telephone, and had almost finished dialing his home number when he changed his mind.
012 prompt: He'd forgotten all about the people in cloaks until he passed a group of them next to the baker's.
(6) prompt: He dashed back across the road, hurried up to his office, snapped at his secretary not to disturb him, seized his telephone, and had almost finished dialing his home number when he changed his mind.
(7) prompt: He'd forgotten all about the people in cloaks until he passed a group of them next to the baker's.
(8) prompt: "Sorry," he grunted, as the tiny old man stumbled and almost fell.
(4) Supplementary Vocoder Demo (with bigvgan)
(4.1) Compare the generalization ability of vocoders on out-of-domain data Opencpop, all models trained on LibriTTS (more)
BigVGAN
RFWave
(4.2) Demo on MusDB (Model Trained on LibrTTS or Universal Dataset, auduio synthesized on MusDB DataSet)
