---
language: en
license: apache-2.0
---

# Mamba-Shedder Model: Mamba-Shedder-Mamba2-2.7B-Pruned-22SSM-Alpaca

- Base Model: [state-spaces/mamba2-2.7b](https://huggingface.co/state-spaces/mamba2-2.7b)
- Pruned Components: **22 SSMs** (Layer 63, 54, 42, 45, 53, 57, 58, 59, 38, 56, 50, 61, 60, 43, 37, 62, 49, 34, 55, 33, 39, 35)
- Recovery Tuning: Yes
- Tuning Config:
  - epoch: 1, batch size: 32, learning rate: 5e-5
  - dataset: [yahma/alpaca-cleaned](https://huggingface.co/datasets/yahma/alpaca-cleaned)

### Evaluation

```bash
git clone https://github.com/IntelLabs/Hardware-Aware-Automated-Machine-Learning.git
cd Mamba-Shedder
python eval.py --model_path <path to model>
```

Refer to our [code repository](https://github.com/IntelLabs/Hardware-Aware-Automated-Machine-Learning/tree/main/Mamba-Shedder) for the environment information to run this command.

## Ethical Considerations

Intel is committed to respecting human rights and avoiding causing or contributing to adverse impacts on human rights. See [Intel’s Global Human Rights Principles](https://www.intel.com/content/dam/www/central-libraries/us/en/documents/policy-human-rights.pdf). Intel’s products and software are intended only to be used in applications that do not cause or contribute to adverse impacts on human rights.

## Model Sources

- **Repository:** [https://github.com/IntelLabs/Hardware-Aware-Automated-Machine-Learning/tree/main/Mamba-Shedder](https://github.com/IntelLabs/Hardware-Aware-Automated-Machine-Learning/tree/main/Mamba-Shedder)
- **Paper:** [Mamba-Shedder: Post-Transformer Compression for Efficient Selective Structured State Space Models](https://arxiv.org/abs/2501.17088)  

## Citation

```bibtex
@inproceedings{munoz2025mambashedder,
  title = {Mamba-Shedder: Post-Transformer Compression for Efficient Selective Structured State Space Models},
  author = {Mu{\~n}oz, J. Pablo and Yuan, Jinjie and Jain, Nilesh},
  booktitle = "Proceedings of the 2025 Annual Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics (NAACL 2025)",
    month = jun,
    year = "2025",
    address = "Albuquerque, New Mexico",
    publisher = "Association for Computational Linguistics",
    url = "",
}
```

### Original Work Citation  
This work builds upon work done by the State-Spaces team.  Please see the following for additional citations of their work:  

**Repository:** ([state-spaces/mamba](https://github.com/state-spaces/mamba)  
**Paper:** [Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality](https://arxiv.org/abs/2312.00752)