File size: 6,087 Bytes
a53593c 975025a a53593c c734272 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 |
---
license: mit
datasets:
- observerw/ChiseLLM-Completion
- observerw/ChiseLLM-Decompile
base_model:
- Qwen/Qwen2.5-Coder-7B-Instruct
---
# ChiseLLM Models
<img src="https://raw.githubusercontent.com/observerw/ChiseLLM/refs/heads/main/assets/logo.svg" alt="ChiseLLM" style="width:30%">
[GitHub](https://github.com/observerw/ChiseLLM)
ChiseLLM is a series of **large reasoning models specifically trained for the [Chisel Hardware Construction language](https://www.chisel-lang.org)**, aimed at revolutionizing HCL-Baed Agile Hardware Development Methodology (AHDM).
Built on [Qwen/Qwen2.5-Coder-Instruct](https://huggingface.co/Qwen/Qwen2.5-Coder-7B-Instruct) with domain-adaptive fine-tuning, the model combines high-quality reasoning datasets and specific thinking patterns to significantly enhance performance in hardware design tasks.
ChiseLLM Models can:
- **Transform natural language specifications into high-quality Chisel code** (Spec-to-Chisel)
- **Intelligently translate Verilog code into enhanced Chisel implementations** (Decompile-to-Chisel)
- **Generate hardware designs with superior variability and extensibility**, surpassing traditional design approaches
### Use Cases
ChiseLLM Models is particularly suited for the following applications:
- **Rapid Hardware Design Prototyping**: Dramatically shortens the design cycle from specification to implementation
- **Verilog Code Modernization**: Intelligently converts legacy Verilog code into extensible Chisel implementations
- **Hardware Architecture Exploration**: Generates multiple design variants for the same functional requirements
- **Design Refactoring and Optimization**: Leverages Chisel's advanced features to improve existing hardware designs
- **Agile Hardware Development Education**: Serves as an assistive tool for learning Chisel and modern hardware design methods
### Training results
Spec-to-Chisel task on VerilogEval:
| Models | pass@1 | pass@3 | pass@5 | syntax(%) |
| ------------------------------ | --------- | --------- | --------- | --------- |
| Llama3.1-8B-Instruct | 4.33 | 9.90 | 13.21 | 9.02 |
| Qwen2.5-Coder-7B-Instruct | 21.94 | 31.87 | 36.73 | 37.08 |
| \*Deepseek-R1-Distill-Llama-8B | 9.31 | 15.44 | 17.72 | 16.01 |
| \*ChiseLLM-7B | **29.41** | **47.08** | **54.04** | **58.82** |
| Models | pass@1 | pass@3 | pass@5 | syntax(%) |
| ------------------------------- | --------- | --------- | --------- | --------- |
| Qwen2.5-Coder-32B-Instruct | 41.02 | 53.85 | 58.79 | 73.47 |
| Qwen2.5-72B-Instruct | 39.74 | 49.30 | 52.90 | 61.31 |
| Llama-3.3-70B-Instruct | 38.14 | 44.90 | 48.02 | 65.97 |
| \*Deepseek-R1-Distill-Qwen-32B | 38.50 | 54.58 | 61.16 | 52.19 |
| \*Deepseek-R1-Distill-Llama-70B | 36.62 | 52.28 | 59.90 | 51.72 |
| \*ChiseLLM-32B | **51.43** | **68.29** | **72.78** | **76.45** |
| Models | pass@1 | pass@3 | pass@5 | syntax(%) |
| ------------- | --------- | --------- | --------- | --------- |
| Deepseek-V3 | 50.16 | 63.44 | 67.32 | 76.37 |
| GPT-4o | 42.04 | 60.16 | 65.17 | 69.76 |
| \*Deepseek-R1 | **62.74** | **76.05** | **80.16** | **82.85** |
Decompile-to-Chisel task on VerilogEval:
| Models | pass@1 | pass@3 | pass@5 | syntax(%) |
| ------------------------------ | --------- | --------- | --------- | --------- |
| Llama3.1-8B-Instruct | 5.43 | 12.29 | 16.08 | 11.15 |
| Qwen2.5-Coder-7B-Instruct | 27.60 | 34.58 | 37.19 | 43.23 |
| \*Deepseek-R1-Distill-Llama-8B | 10.05 | 16.15 | 18.13 | 12.03 |
| \* ChiseLLM-7B | **50.47** | **70.99** | **78.08** | **59.19** |
| Models | pass@1 | pass@3 | pass@5 | syntax(%) |
| ------------------------------- | --------- | --------- | --------- | --------- |
| Qwen2.5-Coder-32B-Instruct | 41.19 | 48.96 | 51.59 | 53.93 |
| Qwen2.5-72B-Instruct | 40.54 | 47.32 | 49.83 | 59.30 |
| Llama-3.3-70B-Instruct | 38.38 | 46.96 | 51.36 | 48.00 |
| \*Deepseek-R1-Distill-Qwen-32B | 45.03 | 63.02 | 70.18 | 53.17 |
| \*Deepseek-R1-Distill-Llama-70B | 37.50 | 55.05 | 63.84 | 45.59 |
| \*ChiseLLM-32B | **56.41** | **72.00** | **77.67** | **64.71** |
| Models | pass@1 | pass@3 | pass@5 | syntax(%) |
| ------------- | --------- | --------- | --------- | --------- |
| Deepseek-V3 | **54.57** | 63.19 | 66.71 | **66.19** |
| GPT-4o | 42.39 | 65.75 | 71.83 | 53.77 |
| \*Deepseek-R1 | 53.45 | **71.50** | **77.91** | 59.13 |
### Framework versions
- Transformers 4.51.0
- Pytorch 2.6.0a0+df5bbc09d1.nv24.12
- Datasets 3.4.1
- Tokenizers 0.21.0
### Training hyperparameters
The following hyperparameters were used during training:
- learning_rate: 1e-05
- train_batch_size: 2
- eval_batch_size: 8
- seed: 42
- distributed_type: multi-GPU
- num_devices: 8
- gradient_accumulation_steps: 8
- total_train_batch_size: 128
- total_eval_batch_size: 64
- optimizer: Use OptimizerNames.ADAMW_TORCH with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: cosine
- lr_scheduler_warmup_ratio: 0.05
- num_epochs: 3.0
## Citation
If you are interested in our work, please consider citing this, it would be greatly appreciated!
```bibtex
@misc{wang2025chisellmunleashingpowerreasoning,
title={ChiseLLM: Unleashing the Power of Reasoning LLMs for Chisel Agile Hardware Development},
author={Bowei Wang and Jiaran Gao and Yelai Feng and Renzhi Chen and Shanshan Li and Lei Wang},
year={2025},
eprint={2504.19144},
archivePrefix={arXiv},
primaryClass={cs.AI},
url={https://arxiv.org/abs/2504.19144},
}
``` |