Soundwave: Less is More for Speech-Text Alignment in LLMs
Yoohao
Abstract
Existing end-to-end speech large language models (LLMs) usually rely on large-scale annotated data for training, while data-efficient training has not been discussed in depth. We focus on two fundamental problems between speech and text: the representation space gap and sequence length inconsistency. We propose Soundwave, which utilizes an efficient training strategy and a novel architecture to address these issues. Results show that Soundwave outperforms the advanced Qwen2-Audio in speech translation and AIR-Bench speech tasks, using only one-fiftieth of the training data. Further analysis shows that Soundwave still retains its intelligence during conversation. The project is available at https://github.com/FreedomIntelligence/Soundwave.
Community
Soundwave is a Speech LLM that can process various speech tasks (e.g., speech recognition and speech translation). It is trained with just one-fiftieth of the data size compared to Qwen2-Audio, while achieving better performance ont AIR-Bench speech tasks. More details can be found at https://github.com/FreedomIntelligence/Soundwave.
This is an automated message from the Librarian Bot. I found the following papers similar to this paper.
The following papers were recommended by the Semantic Scholar API
- OSUM: Advancing Open Speech Understanding Models with Limited Resources in Academia (2025)
- Zero-resource Speech Translation and Recognition with LLMs (2024)
- VITA-1.5: Towards GPT-4o Level Real-Time Vision and Speech Interaction (2025)
- DuplexMamba: Enhancing Real-time Speech Conversations with Duplex and Streaming Capabilities (2025)
- OpenOmni: Large Language Models Pivot Zero-shot Omnimodal Alignment across Language with Real-time Self-Aware Emotional Speech Synthesis (2025)
- Listening and Seeing Again: Generative Error Correction for Audio-Visual Speech Recognition (2025)
- Audio-Language Models for Audio-Centric Tasks: A survey (2025)
Please give a thumbs up to this comment if you found it helpful!
If you want recommendations for any Paper on Hugging Face checkout this Space
You can directly ask Librarian Bot for paper recommendations by tagging it in a comment:
@librarian-bot
recommend
Excellent work! I believe the alignment adaptor output, supervised by the CTC loss, captures more semantic information while reducing acoustic dependency. However, Soundwave achieves SOTA performance in speech emotion recognition (0.635 on MELD). I’d love to hear your thoughts on this.
Thank you for paying attention to our work. That’s a good question, and we have considered it as well. We use two branches to address this problem in the shrinking adapter (see Figure below). The content features are shrunk based on the CTC prediction, and then we use the shrunk features as queries, applying cross-attention to extract paralinguistic information from the original sequence. Finally, the two types of feautures are fused as the output.
In short, the content features represent 'what the person says,' while the auxiliary features represent 'how the person says it.' The complete speech features are decomposed into semantic and acoustic features, making it easier to detect emotions for LLMs.
Models citing this paper 1
Datasets citing this paper 0
No dataset linking this paper