MixFormerV2

This version of MixFormerV2 has been converted to run on the Axera NPU using w8a16 quantization.

This model has been optimized with the following LoRA:

Compatible with Pulsar2 version: 3.4

Convert tools links:

For those who are interested in model conversion, you can try to export axmodel through

• The repo of original

• The repo of AXera Platform, which you can get the detial of guide

• Pulsar2 Link, How to Convert ONNX to axmodel

Support Platform

• AX650
  • M4N-Dock(爱芯派Pro)
  • M.2 Accelerator card
• AX630C(todo, convert model failed, try to fix......)

Chips	npu1(6TOPs)	npu3(18TOPs)
AX650	11 ms	TBD

How to use

Download all files from this repository to the device

root@ax650:/mnt/qtang/MixFormerV2# tree -L 1
.
├── ax650
├── car.avi
├── config.json
├── onnx
├── README.md
├── run_mixformer2_axmodel.py
└── run_mixformer2_onnx.py

python env requirement

pyaxengine

https://github.com/AXERA-TECH/pyaxengine

wget https://github.com/AXERA-TECH/pyaxengine/releases/download/0.1.1rc0/axengine-0.1.1-py3-none-any.whl
pip install axengine-0.1.1-py3-none-any.whl

others

pip install argparse numpy opencv-python glob2

Inference with AX650 Host, such as M4N-Dock(爱芯派Pro)

root@ax650:/mnt/qtang/ax650_mixformer2_demo# python3 run_mixformer2_axmodel.py --model-path ax650/mixformer_v2.axmodel --frame-path car.avi -r 10
[INFO] Available providers:  ['AxEngineExecutionProvider']
[INFO] Using provider: AxEngineExecutionProvider
[INFO] Chip type: ChipType.MC50
[INFO] VNPU type: VNPUType.DISABLED
[INFO] Engine version: 2.7.2a
[INFO] Model type: 0 (single core)
[INFO] Compiler version: 3.4-dirty 4ff37520-dirty
====================type================= [1079, 482] <class 'list'> <class 'list'>
第一帧初始化完毕！
Video: tracking     246.0fps
Video: tracking     4.0fps
Video: tracking     4.0fps
Video: tracking     4.0fps
Video: tracking     4.0fps
Video: tracking     4.0fps
Video: tracking     4.0fps
Video: tracking     4.0fps
Video: tracking     4.0fps
Video: tracking     4.0fps
Video: tracking     4.0fps
Reached the maximum number of frames (10). Exiting loop.
video: average finale average tracking fps 31.8 fps
root@ax650:/mnt/qtang/ax650_mixformer2_demo#

Inference with M.2 Accelerator card

What is M.2 Accelerator card?, Show this DEMO based on Raspberry PI 5.

(axcl) axera@raspberrypi:~/samples/MixFormerV2 $ python3 run_mixformer2_axmodel.py --model-path ax650/mixformer_v2.axmodel --frame-path car.avi -r 10
[INFO] Available providers:  ['AXCLRTExecutionProvider']
[INFO] Using provider: AXCLRTExecutionProvider
[INFO] SOC Name: AX650N
[INFO] VNPU type: VNPUType.DISABLED
[INFO] Compiler version: 3.4-dirty 4ff37520-dirty
====================type================= [1079, 482] <class 'list'> <class 'list'>
第一帧初始化完毕！
Video: tracking     925.0fps
Video: tracking     12.0fps
Video: tracking     12.0fps
Video: tracking     11.0fps
Video: tracking     11.0fps
Video: tracking     11.0fps
Video: tracking     11.0fps
Video: tracking     11.0fps
Video: tracking     10.0fps
Video: tracking     10.0fps
Video: tracking     10.0fps
Reached the maximum number of frames (10). Exiting loop.
video: average finale average tracking fps 114.9 fps
(axcl) axera@raspberrypi:~/samples/MixFormerV2 $