Create README.md

README.md

@@ -0,0 +1,151 @@
+---
+license: other
+license_name: nsclv1
+license_link: https://developer.nvidia.com/downloads/license/nsclv1
+---
+
+
+# NVIDIA Low Frame-rate Speech Codec
+<style>
+img {
+display: inline-table;
+vertical-align: small;
+margin: 0;
+padding: 0;
+}
+</style>
+[![Model architecture](https://img.shields.io/badge/Model_Arch-HiFi--GAN-lightgrey#model-badge)](#model-architecture)
+| [![Model size](https://img.shields.io/badge/Params-61.8M-lightgrey#model-badge)](#model-architecture)
+| [![Language](https://img.shields.io/badge/Language-multilingual-lightgrey#model-badge)](#datasets)
+
+The [NeMo Audio Codec](https://arxiv.org/abs/2406.05298) is a neural audio codec which compresses audio into a quantized representation. The model can be used as a vocoder for speech synthesis.
+
+
+## Model Architecture
+The NeMo Audio Codec model uses symmetric convolutional encoder-decoder architecture similar to [HiFi-GAN-based](https://arxiv.org/abs/2010.05646).
+For the vector quantization, we use [Finite Scalar Quantization (FSQ)](https://arxiv.org/abs/2309.15505) with eight codebooks and four dimensions per code and 1000 codes per codebook.
+
+For more details please check [our paper](https://arxiv.org/abs/2406.05298).
+
+### Input
+  - **Input Type:** Audio 
+  - **Input Format(s):** .wav files
+  - **Input Parameters:** One-Dimensional (1D)
+  - **Other Properties Related to Input:** 22050 Hz Mono-channel Audio
+
+### Output
+  - **Output Type**: Audio 
+  - **Output Format:** .wav files
+  - **Output Parameters:** One Dimensional (1D)
+  - **Other Properties Related to Output:** 22050 Hz Mono-channel Audio
+
+
+## How to Use this Model
+
+The model is available for use in the [NVIDIA NeMo](https://github.com/NVIDIA/NeMo), and can be used as a pre-trained checkpoint for inference or for fine-tuning on another dataset. 
+
+### Inference
+For inference, you can follow our [Audio Codec Inference Tutorial](https://github.com/NVIDIA/NeMo/blob/main/tutorials/tts/Audio_Codec_Inference.ipynb) which automatically downloads the model checkpoint. Note that you will need to set the ```model_name``` parameter to "nvidia/audio-codec-22khz".
+
+Alternatively, you can manually download the [checkpoint](https://huggingface.co/nvidia/Low-Frame-rate-Speech-Codec/resolve/main/Low-Frame-rate-Speech-Codec.nemo) and use the code below to make an inference on the model:
+
+```
+import librosa
+import torch
+import soundfile as sf
+from nemo.collections.tts.models import AudioCodecModel
+
+codec_path = ??? # set here the model .nemo checkpoint path
+path_to_input_audio = ??? # path of the input audio
+path_to_output_audio = ??? # path of the reconstructed output audio
+
+nemo_codec_model = AudioCodecModel.restore_from(restore_path=codec_path, map_location="cpu").eval()
+
+# get discrete tokens from audio
+audio, _ = librosa.load(path_to_input_audio, sr=nemo_codec_model.sample_rate)
+
+device = 'cuda' if torch.cuda.is_available() else 'cpu'
+audio_tensor = torch.from_numpy(audio).unsqueeze(dim=0).to(device)
+audio_len = torch.tensor([audio_tensor[0].shape[0]]).to(device)
+
+encoded_tokens, encoded_len = nemo_codec_model.encode(audio=audio_tensor, audio_len=audio_len)
+
+# Reconstruct audio from tokens
+reconstructed_audio, _ = nemo_codec_model.decode(tokens=encoded_tokens, tokens_len=encoded_len)
+
+# save reconstructed audio
+output_audio = reconstructed_audio.cpu().numpy().squeeze()
+sf.write(path_to_output_audio, output_audio, nemo_codec_model.sample_rate)
+
+```
+
+### Training
+For fine-tuning on another dataset please follow the steps available at our [Audio Codec Training Tutorial](https://github.com/NVIDIA/NeMo/blob/main/tutorials/tts/Audio_Codec_Training.ipynb). Note that you will need to set the ```CONFIG_FILENAME``` parameter to the "audio_codec_low_frame_rate_22050.yaml" config. You also will need to set ```pretrained_model_name``` to "audio_codec_low_frame_rate_22khz".
+
+
+## Training, Testing, and Evaluation Datasets:
+
+The Low Frame-rate Speech Codec was trained on 28.7k hours of speech data spanning 105 languages. The model was evaluated using multilingual audiobook-style data and high-quality English recordings. For further details, refer to  [our paper](https://arxiv.org/abs/2409.12117). 
+
+
+### Training Datasets
+The Low Frame-rate Speech Codec is trained on a total of 28.7k hrs of speech data from 105 languages.
+
+  - [MLS English](https://www.openslr.org/94/) [25.5k]
+  
+      - Data Collection Method: by Human
+      
+      - Labeling Method: Automated
+  
+  -  [Common Voice](https://huggingface.co/datasets/mozilla-foundation/common_voice_11_0)[3.2k]
+  
+      - Data Collection Method: by Human
+      
+      - Labeling Method: by Human
+  
+  
+
+### Test Datasets
+
+  - [MLS English](https://www.openslr.org/94/)
+  
+      - Data Collection Method: by Human
+      
+      - Labeling Method: Automated
+  
+  -  [Common Voice](https://huggingface.co/datasets/mozilla-foundation/common_voice_11_0)
+  
+      - Data Collection Method: by Human
+      
+      - Labeling Method: by Human
+
+
+## Software Integration
+
+### Supported Hardware Microarchitecture Compatibility:
+- NVIDIA Ampere
+- NVIDIA Blackwell
+- NVIDIA Jetson
+- NVIDIA Hopper
+- NVIDIA Lovelace
+- NVIDIA Pascal
+- NVIDIA Turing
+- NVIDIA Volta
+
+### Runtime Engine
+
+- Nemo 2.0.0
+
+### Preferred Operating System
+
+- Linux
+
+
+## License/Terms of Use
+This model is for research and development only (non-commercial use) and the license to use this model is covered by the [NSCLv1](https://developer.nvidia.com/downloads/license/nsclv1). 
+
+## Ethical Considerations:
+NVIDIA believes Trustworthy AI is a shared responsibility and we have established policies and practices to enable development for a wide array of AI applications.  When downloaded or used in accordance with our terms of service, developers should work with their internal model team to ensure this model meets requirements for the relevant industry and use case and addresses unforeseen product misuse.
+Please report security vulnerabilities or NVIDIA AI Concerns [here](https://www.nvidia.com/en-us/support/submit-security-vulnerability/).
+
+