Update README.md

README.md

@@ -77,7 +77,9 @@ Mostly, this is a test of (significant) pruning & healing an instruct-tuned mode
 
 ## Healing / Finetune
 I healed the model by doing a full weight DPO finetune for 139k samples (3.15 epochs), and then a LoRA with r=128 a=256 for 73k samples (1.67 epochs).
+
 Prior to healing, the model returned absolute gibberish to any prompt, rarely two real words together. For example, give "2+2=" it might return "Mahmisan Pannpyout Na RMITa CMI TTi GP BP GP RSi TBi DD PS..."
+
 The results are pretty good! The model has issues, but could have legitimate uses. It carry on a conversation. It's certainly usable, if not useful.
 
 Truthfulness and commonsense reasoning suffered the least from the prune / were healed the best. Knowledge and complex reasoning suffered the most.
@@ -87,6 +89,7 @@ This model has 67% the parameters of the original, and has:
 - ~65% the Hellaswag score
 - ~85% the Winogrande score
 - ~45% the the MMLU score
+
 An average of 69% the benchmark scores for 67% the parameters, not bad! (Note, I had issues running the GSM8K and BBH benchmarks.)
 I do believe it could be much better, by doing the pruning in stages (say, 4 layers at a time) with some healing in between, and longer healing at the end with a more diverse dataset.
 
@@ -99,6 +102,7 @@ This size should allow for:
 - Q8 or Q6 inference on 6GB VRAM
 - Q5 inference on 4GB VRAM
 - Fine-tuning on ... well, with less VRAM than an 8B model
+- 
 And of course, as stated, it was a test of significant pruning, and of pruning&healing an instruct-tuned model. As a test, I think it's definitely successful.
 
 ## Mergekit Details