README.md

---
license: apache-2.0
model-index:
- name: openchat-3.5-0106_Rebased_Mistral-7B-v0.2
  results:
  - task:
      type: text-generation
      name: Text Generation
    dataset:
      name: IFEval (0-Shot)
      type: HuggingFaceH4/ifeval
      args:
        num_few_shot: 0
    metrics:
    - type: inst_level_strict_acc and prompt_level_strict_acc
      value: 37.06
      name: strict accuracy
    source:
      url: https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=Pretergeek/openchat-3.5-0106_Rebased_Mistral-7B-v0.2
      name: Open LLM Leaderboard
  - task:
      type: text-generation
      name: Text Generation
    dataset:
      name: BBH (3-Shot)
      type: BBH
      args:
        num_few_shot: 3
    metrics:
    - type: acc_norm
      value: 10.91
      name: normalized accuracy
    source:
      url: https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=Pretergeek/openchat-3.5-0106_Rebased_Mistral-7B-v0.2
      name: Open LLM Leaderboard
  - task:
      type: text-generation
      name: Text Generation
    dataset:
      name: MATH Lvl 5 (4-Shot)
      type: hendrycks/competition_math
      args:
        num_few_shot: 4
    metrics:
    - type: exact_match
      value: 3.85
      name: exact match
    source:
      url: https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=Pretergeek/openchat-3.5-0106_Rebased_Mistral-7B-v0.2
      name: Open LLM Leaderboard
  - task:
      type: text-generation
      name: Text Generation
    dataset:
      name: GPQA (0-shot)
      type: Idavidrein/gpqa
      args:
        num_few_shot: 0
    metrics:
    - type: acc_norm
      value: 2.91
      name: acc_norm
    source:
      url: https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=Pretergeek/openchat-3.5-0106_Rebased_Mistral-7B-v0.2
      name: Open LLM Leaderboard
  - task:
      type: text-generation
      name: Text Generation
    dataset:
      name: MuSR (0-shot)
      type: TAUR-Lab/MuSR
      args:
        num_few_shot: 0
    metrics:
    - type: acc_norm
      value: 20.57
      name: acc_norm
    source:
      url: https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=Pretergeek/openchat-3.5-0106_Rebased_Mistral-7B-v0.2
      name: Open LLM Leaderboard
  - task:
      type: text-generation
      name: Text Generation
    dataset:
      name: MMLU-PRO (5-shot)
      type: TIGER-Lab/MMLU-Pro
      config: main
      split: test
      args:
        num_few_shot: 5
    metrics:
    - type: acc
      value: 20.33
      name: accuracy
    source:
      url: https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=Pretergeek/openchat-3.5-0106_Rebased_Mistral-7B-v0.2
      name: Open LLM Leaderboard
---

This model was created as an experiment on using LoRA extraction to replicate [Openchat-3.5-0106](https://huggingface.co/openchat/openchat-3.5-0106) using [Mistral-7B-v0.2](https://huggingface.co/mistral-community/Mistral-7B-v0.2) as a base model instead of the original [Mistral-7B-v0.1](https://huggingface.co/mistralai/Mistral-7B-v0.1).

Openchat-3.5-0106 is an excellent model but was based on Mistral-7B-v0.1 which has a context window of 8192 tokens. Mistral-7B-v0.2 has a context window of 32768 tokens. I could have extended OpenChat-3.5 context myself with RoPE and/or YaRN but that has been done. There are many models on HF that have done exactly that. Instead I decided to try and replicate OpenChat-3.5-0106 using the LoRA extraction method available in mergekit. These are the steps I followed:
- Extract a LoRA with rank 512 from OpenChat-3.5-0106 using [One](https://huggingface.co/imone)'s [Mistral_7B_with_EOT_token](https://huggingface.co/imone/Mistral_7B_with_EOT_token) as the base model.
- Replicate imone's work by adding the EOT token to Mistral-7B-v0.2, creating [Mistral-7B-v0.2_EOT](https://huggingface.co/Pretergeek/Mistral-7B-v0.2_EOT).
- Merge the LoRA's weights to the Mistral-7B-v0.2_EOT model.

This is the result. This model is not meant for use, it was created to test if this method is viable for replacing the base model of fine-tuned models (when tokenizer and weights have not been changed too much). I am uploading here for evaluation. I don't expect this model to match the original OpenChat-3.5-0106 since I used a LoRA with rank 512, so it won't be equivalent to a full fine-tuning. I have been able to extract LoRAs with higher rank, but currently I don't have the resources to merge them with the model as the memory requirements exceed what I have at my disposal.
If you would like to help my work, check my Ko-Fi and/or Patreon:
* https://ko-fi.com/pretergeek
* https://patreon.com/Pretergeek
# [Open LLM Leaderboard Evaluation Results](https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard)
Detailed results can be found [here](https://huggingface.co/datasets/open-llm-leaderboard/details_Pretergeek__openchat-3.5-0106_Rebased_Mistral-7B-v0.2)

|      Metric       |Value|
|-------------------|----:|
|Avg.               |15.94|
|IFEval (0-Shot)    |37.06|
|BBH (3-Shot)       |10.91|
|MATH Lvl 5 (4-Shot)| 3.85|
|GPQA (0-shot)      | 2.91|
|MuSR (0-shot)      |20.57|
|MMLU-PRO (5-shot)  |20.33|