# Step-level Value Preference Optimization for Mathematical Reasoning

This is the official repository for paper [Step-level Value Preference Optimization for Mathematical Reasoning](https://arxiv.org/abs/2406.10858). It is extracted from our internal corporate codebase. As a result, there may be slight differences when reproducing the numbers reported in our paper, but they should be very close.


The implementation of SVPO is based on [AlphaMath](https://arxiv.org/abs/2405.03553), such as MCTS and Step-level beam search (SBS).
Therefore, we provide the [code](https://github.com/MARIO-Math-Reasoning/Super_MARIO) of step-level preference pairs construction in this repository to facilitate reproduction.


## Citation
SVPO
```
@misc{chen2024steplevel,
      title={Step-level Value Preference Optimization for Mathematical Reasoning}, 
      author={Guoxin Chen and Minpeng Liao and Chengxi Li and Kai Fan},
      year={2024},
      eprint={2406.10858},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}
```

AlphaMATH
```
@misc{chen2024alphamath,
      title={AlphaMath Almost Zero: process Supervision without process}, 
      author={Guoxin Chen and Minpeng Liao and Chengxi Li and Kai Fan},
      year={2024},
      eprint={2405.03553},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}
```