parameters guide
samplers guide
model generation
role play settings
quant selection
arm quants
iq quants vs q quants
optimal model setting
gibberish fixes
coherence
instructing following
quality generation
chat settings
quality settings
llamacpp server
llamacpp
lmstudio
sillytavern
koboldcpp
backyard
ollama
model generation steering
steering
model generation fixes
text generation webui
ggufs
exl2
full precision
quants
imatrix
neo imatrix
Update README.md
#5
by
ZaBeeble
- opened
README.md
CHANGED
@@ -71,7 +71,7 @@ to MOE/Mixture of expert models - both GGUF and source.
|
|
71 |
(Updated: "INDEX", and added "Generation Steering" section ; notes on Roleplay/Simulation added, Screenshots of parameters/samplers added in quick reference section.)
|
72 |
|
73 |
This document includes detailed information, references, and notes for general parameters, samplers and
|
74 |
-
advanced samplers to get the most out of your model's abilities including notes / settings for the most popular AI/LLM app in use (
|
75 |
|
76 |
These settings / suggestions can be applied to all models including GGUF, EXL2, GPTQ, HQQ, AWQ and full source/precision.
|
77 |
|
@@ -94,9 +94,9 @@ The settings discussed in this document can also fix a number of model issues (<
|
|
94 |
|
95 |
Likewise ALL the setting (parameters, samplers and advanced samplers) below can also improve model generation and/or general overall "smoothness" / "quality" of model operation:
|
96 |
|
97 |
-
- all parameters and samplers available via
|
98 |
-
- all parameters (including some not in
|
99 |
-
- all parameters (including some not in
|
100 |
|
101 |
Even if you are not using my models, you may find this document <u>useful for any model (any quant / full source / any repo) available online.</u>
|
102 |
|
@@ -186,7 +186,7 @@ DETAILED NOTES ON PARAMETERS, SAMPLERS and ADVANCED SAMPLERS:
|
|
186 |
- DETAILS on PARAMETERS / SAMPLERS
|
187 |
- General Parameters
|
188 |
- The Local LLM Settings Guide/Rant
|
189 |
-
-
|
190 |
- DRY Sampler
|
191 |
- Samplers
|
192 |
- Creative Writing
|
@@ -466,7 +466,7 @@ Source files / Source models of my models are located here (also upper right men
|
|
466 |
|
467 |
[ https://huggingface.co/collections/DavidAU/d-au-source-files-for-gguf-exl2-awq-gptq-hqq-etc-etc-66b55cb8ba25f914cbf210be ]
|
468 |
|
469 |
-
You will need the config files to use "
|
470 |
|
471 |
You can also use the full source in "text-generation-webui" too.
|
472 |
|
@@ -474,16 +474,16 @@ As an alternative you can use GGUFs directly in "KOBOLDCPP" / "SillyTavern" with
|
|
474 |
|
475 |
<B>Parameters, Samplers and Advanced Samplers</B>
|
476 |
|
477 |
-
In section 1 a,b, and c, below are all the
|
478 |
|
479 |
I have added notes below each one for adjustment / enhancement(s) for specific use cases.
|
480 |
|
481 |
<B>TEXT-GENERATION-WEBUI</B>
|
482 |
|
483 |
-
In section 2, will be additional samplers, which become available when using "
|
484 |
AND/OR https://github.com/LostRuins/koboldcpp ("KOBOLDCPP").
|
485 |
|
486 |
-
The "
|
487 |
|
488 |
(this process is automated with this program, just enter the repo(s) urls -> it will fetch everything for you)
|
489 |
|
@@ -491,17 +491,17 @@ This allows access to very advanced samplers in addition to all the parameters /
|
|
491 |
|
492 |
<B>KOBOLDCPP:</B>
|
493 |
|
494 |
-
Note that https://github.com/LostRuins/koboldcpp also allows access to all
|
495 |
|
496 |
-
You can use almost all parameters, samplers and advanced samplers using "KOBOLDCPP" without the need to get the source config files (the "
|
497 |
|
498 |
Note: This program has one of the newest samplers called "Anti-slop" which allows phrase/word banning at the generation level.
|
499 |
|
500 |
<B>SILLYTAVERN:</B>
|
501 |
|
502 |
-
Note that https://github.com/SillyTavern/SillyTavern also allows access to all
|
503 |
|
504 |
-
You can use almost all parameters, samplers and advanced samplers using "SILLYTAVERN" without the need to get the source config files (the "
|
505 |
|
506 |
For CLASS3 and CLASS4 the most important setting is "SMOOTHING FACTOR" (Quadratic Smoothing) ; information is located on this page:
|
507 |
|
@@ -518,7 +518,7 @@ In Sillytavern you can then adjust parameters, samplers and advanced samplers ;
|
|
518 |
|
519 |
Currently, at time of this writing, connecting Silly Tavern via KoboldCPP or Text Generation Webui will provide the most samplers/parameters.
|
520 |
|
521 |
-
However for some, connecting to Lmstudio,
|
522 |
|
523 |
You may also want to check out how to connect SillyTavern to local AI "apps" running on your pc here:
|
524 |
|
@@ -527,21 +527,21 @@ https://docs.sillytavern.app/usage/api-connections/
|
|
527 |
|
528 |
<B>Lmstudio, Ollama, Llamacpp, and OTHER PROGRAMS</B>
|
529 |
|
530 |
-
Other programs like https://www.LMStudio.ai allows access to most of STANDARD samplers, where as others (
|
531 |
|
532 |
-
In most cases all
|
533 |
|
534 |
-
You can also use
|
535 |
|
536 |
https://github.com/ggerganov/llama.cpp
|
537 |
|
538 |
-
(scroll down on the main page for more apps/programs to use GGUFs too that connect to / use the
|
539 |
|
540 |
Special note:
|
541 |
|
542 |
-
It appears "DRY" / "XTC" samplers has been added to
|
543 |
|
544 |
-
It is available (Llamacpp) via "server.exe / llama-server.exe". Likely this sampler will also become available "downstream" in applications that use
|
545 |
|
546 |
[ https://github.com/ggerganov/llama.cpp/blob/master/examples/server/README.md ]
|
547 |
|
@@ -1119,9 +1119,9 @@ Paper: https://arxiv.org/abs/2007.14966
|
|
1119 |
|
1120 |
CRITICAL:
|
1121 |
|
1122 |
-
If you activate Mirostat when using "
|
1123 |
|
1124 |
-
V1: n_vocab(model) (this is set internally by
|
1125 |
|
1126 |
V2: seed, mirostat_tau, mirostat_eta
|
1127 |
|
@@ -1133,7 +1133,7 @@ For Text Generation UI:
|
|
1133 |
|
1134 |
No blocking occurs. Note that ONLY Mirostat 2 is available. (other parameters/samplers should work without issue)
|
1135 |
|
1136 |
-
Note this is subject to change by
|
1137 |
|
1138 |
("seed" is usually a random value. (default) ; this parameter can be set in some AI/LLM apps to control Mirostat output more closely.)
|
1139 |
|
@@ -1259,7 +1259,7 @@ You can use these samplers via Sillytavern IF you use either of these APPS (Kobo
|
|
1259 |
|
1260 |
Other Notes:
|
1261 |
|
1262 |
-
Hopefully ALL these samplers / controls will be
|
1263 |
|
1264 |
"DRY" sampler has been added to Llamacpp as of the time of this writing (and available via SERVER/LLAMA-SERVER.EXE) and MAY appear in other "downstream" apps that use Llamacpp.
|
1265 |
|
@@ -1273,9 +1273,9 @@ https://github.com/oobabooga/text-generation-webui/wiki/03-%E2%80%90-Parameters-
|
|
1273 |
|
1274 |
ADVANCED SAMPLERS - PART 1:
|
1275 |
|
1276 |
-
Keep in mind these parameters/samplers become available (for GGUFs) in "oobabooga/text-generation-webui" when you use the
|
1277 |
|
1278 |
-
Most of these are also available in KOBOLDCPP too (via settings -> samplers) after start up (no "
|
1279 |
|
1280 |
I am not going to touch on all of samplers / parameters, just the main ones at the moment.
|
1281 |
|
@@ -1446,7 +1446,7 @@ For additional details on these samplers settings (including advanced ones) you
|
|
1446 |
|
1447 |
https://github.com/oobabooga/text-generation-webui/wiki/03-%E2%80%90-Parameters-Tab
|
1448 |
|
1449 |
-
(NOTE: Not all of these "options" are available for GGUFS, including when you use "
|
1450 |
|
1451 |
Additional Links (on parameters, samplers and advanced samplers):
|
1452 |
|
@@ -1464,7 +1464,7 @@ The Local LLM Settings Guide/Rant (covers a lot of parameters/samplers - lots of
|
|
1464 |
|
1465 |
https://rentry.org/llm-settings
|
1466 |
|
1467 |
-
|
1468 |
|
1469 |
https://github.com/ggerganov/llama.cpp/blob/master/examples/server/README.md
|
1470 |
|
|
|
71 |
(Updated: "INDEX", and added "Generation Steering" section ; notes on Roleplay/Simulation added, Screenshots of parameters/samplers added in quick reference section.)
|
72 |
|
73 |
This document includes detailed information, references, and notes for general parameters, samplers and
|
74 |
+
advanced samplers to get the most out of your model's abilities including notes / settings for the most popular AI/LLM app in use (Llamacpp, KoboldCPP, Text-Generation-WebUI, LMStudio, Sillytavern, Ollama and others).
|
75 |
|
76 |
These settings / suggestions can be applied to all models including GGUF, EXL2, GPTQ, HQQ, AWQ and full source/precision.
|
77 |
|
|
|
94 |
|
95 |
Likewise ALL the setting (parameters, samplers and advanced samplers) below can also improve model generation and/or general overall "smoothness" / "quality" of model operation:
|
96 |
|
97 |
+
- all parameters and samplers available via Llamacpp (and most apps that run / use Llamacpp - including Lmstudio, Ollama, Sillytavern and others.)
|
98 |
+
- all parameters (including some not in Llamacpp), samplers and advanced samplers ("Dry", "Quadratic", "Microstat") in oobabooga/text-generation-webui including Llamacpp_HF loader (allowing a lot more samplers)
|
99 |
+
- all parameters (including some not in Llamacpp), samplers and advanced samplers ("Dry", "Quadratic", "Microstat") in SillyTavern / KoboldCPP (including Anti-slop filters)
|
100 |
|
101 |
Even if you are not using my models, you may find this document <u>useful for any model (any quant / full source / any repo) available online.</u>
|
102 |
|
|
|
186 |
- DETAILS on PARAMETERS / SAMPLERS
|
187 |
- General Parameters
|
188 |
- The Local LLM Settings Guide/Rant
|
189 |
+
- Llamacpp-server - usage / parameters / samplers
|
190 |
- DRY Sampler
|
191 |
- Samplers
|
192 |
- Creative Writing
|
|
|
466 |
|
467 |
[ https://huggingface.co/collections/DavidAU/d-au-source-files-for-gguf-exl2-awq-gptq-hqq-etc-etc-66b55cb8ba25f914cbf210be ]
|
468 |
|
469 |
+
You will need the config files to use "Llamacpp_HF" loader ("text-generation-webui") [ https://github.com/oobabooga/text-generation-webui ]
|
470 |
|
471 |
You can also use the full source in "text-generation-webui" too.
|
472 |
|
|
|
474 |
|
475 |
<B>Parameters, Samplers and Advanced Samplers</B>
|
476 |
|
477 |
+
In section 1 a,b, and c, below are all the Llamacpp parameters and samplers.
|
478 |
|
479 |
I have added notes below each one for adjustment / enhancement(s) for specific use cases.
|
480 |
|
481 |
<B>TEXT-GENERATION-WEBUI</B>
|
482 |
|
483 |
+
In section 2, will be additional samplers, which become available when using "Llamacpp_HF" loader in https://github.com/oobabooga/text-generation-webui
|
484 |
AND/OR https://github.com/LostRuins/koboldcpp ("KOBOLDCPP").
|
485 |
|
486 |
+
The "Llamacpp_HF" (for "text-generation-webui") only requires the GGUF you want to use plus a few config files from "source repo" of the model.
|
487 |
|
488 |
(this process is automated with this program, just enter the repo(s) urls -> it will fetch everything for you)
|
489 |
|
|
|
491 |
|
492 |
<B>KOBOLDCPP:</B>
|
493 |
|
494 |
+
Note that https://github.com/LostRuins/koboldcpp also allows access to all Llamacpp parameters/samplers too as well as additional advanced samplers too.
|
495 |
|
496 |
+
You can use almost all parameters, samplers and advanced samplers using "KOBOLDCPP" without the need to get the source config files (the "Llamacpp_HF" step).
|
497 |
|
498 |
Note: This program has one of the newest samplers called "Anti-slop" which allows phrase/word banning at the generation level.
|
499 |
|
500 |
<B>SILLYTAVERN:</B>
|
501 |
|
502 |
+
Note that https://github.com/SillyTavern/SillyTavern also allows access to all Llamacpp parameters/samplers too as well as additional advanced samplers too.
|
503 |
|
504 |
+
You can use almost all parameters, samplers and advanced samplers using "SILLYTAVERN" without the need to get the source config files (the "Llamacpp_HF" step).
|
505 |
|
506 |
For CLASS3 and CLASS4 the most important setting is "SMOOTHING FACTOR" (Quadratic Smoothing) ; information is located on this page:
|
507 |
|
|
|
518 |
|
519 |
Currently, at time of this writing, connecting Silly Tavern via KoboldCPP or Text Generation Webui will provide the most samplers/parameters.
|
520 |
|
521 |
+
However for some, connecting to Lmstudio, Llamacpp, or Ollama may be preferred.
|
522 |
|
523 |
You may also want to check out how to connect SillyTavern to local AI "apps" running on your pc here:
|
524 |
|
|
|
527 |
|
528 |
<B>Lmstudio, Ollama, Llamacpp, and OTHER PROGRAMS</B>
|
529 |
|
530 |
+
Other programs like https://www.LMStudio.ai allows access to most of STANDARD samplers, where as others (Llamacpp only here) you may need to add to the json file(s) for a model and/or template preset.
|
531 |
|
532 |
+
In most cases all Llamacpp parameters/samplers are available when using API / headless / server mode in "text-generation-webui", "koboldcpp", "Sillytavern", "Olama", and "LMStudio" (as well as other apps too).
|
533 |
|
534 |
+
You can also use Llamacpp directly too. (IE: llama-server.exe) ; see :
|
535 |
|
536 |
https://github.com/ggerganov/llama.cpp
|
537 |
|
538 |
+
(scroll down on the main page for more apps/programs to use GGUFs too that connect to / use the Llamacpp package.)
|
539 |
|
540 |
Special note:
|
541 |
|
542 |
+
It appears "DRY" / "XTC" samplers has been added to Llamacpp and SILLYTAVERN.
|
543 |
|
544 |
+
It is available (Llamacpp) via "server.exe / llama-server.exe". Likely this sampler will also become available "downstream" in applications that use Llamacpp in due time.
|
545 |
|
546 |
[ https://github.com/ggerganov/llama.cpp/blob/master/examples/server/README.md ]
|
547 |
|
|
|
1119 |
|
1120 |
CRITICAL:
|
1121 |
|
1122 |
+
If you activate Mirostat when using "Llamacpp SERVER" and/or some Llamacpp based apps this will VOID/DISABLE all parameters (excluding "penalties", "logit_bias" ) AND all other SAMPLERS except "temp" parameter plus the following:
|
1123 |
|
1124 |
+
V1: n_vocab(model) (this is set internally by Llamacpp), seed, mirostat_tau, mirostat_eta
|
1125 |
|
1126 |
V2: seed, mirostat_tau, mirostat_eta
|
1127 |
|
|
|
1133 |
|
1134 |
No blocking occurs. Note that ONLY Mirostat 2 is available. (other parameters/samplers should work without issue)
|
1135 |
|
1136 |
+
Note this is subject to change by Llamacpp, Koboldcpp, Text Generation UI and other AI/LLM app makers at any time.
|
1137 |
|
1138 |
("seed" is usually a random value. (default) ; this parameter can be set in some AI/LLM apps to control Mirostat output more closely.)
|
1139 |
|
|
|
1259 |
|
1260 |
Other Notes:
|
1261 |
|
1262 |
+
Hopefully ALL these samplers / controls will be Llamacpp and available to all users via AI/LLM apps soon.
|
1263 |
|
1264 |
"DRY" sampler has been added to Llamacpp as of the time of this writing (and available via SERVER/LLAMA-SERVER.EXE) and MAY appear in other "downstream" apps that use Llamacpp.
|
1265 |
|
|
|
1273 |
|
1274 |
ADVANCED SAMPLERS - PART 1:
|
1275 |
|
1276 |
+
Keep in mind these parameters/samplers become available (for GGUFs) in "oobabooga/text-generation-webui" when you use the Llamacpp_HF loader.
|
1277 |
|
1278 |
+
Most of these are also available in KOBOLDCPP too (via settings -> samplers) after start up (no "Llamacpp_HF loader" step required).
|
1279 |
|
1280 |
I am not going to touch on all of samplers / parameters, just the main ones at the moment.
|
1281 |
|
|
|
1446 |
|
1447 |
https://github.com/oobabooga/text-generation-webui/wiki/03-%E2%80%90-Parameters-Tab
|
1448 |
|
1449 |
+
(NOTE: Not all of these "options" are available for GGUFS, including when you use "Llamacpp_HF" loader in "text-generation-webui" )
|
1450 |
|
1451 |
Additional Links (on parameters, samplers and advanced samplers):
|
1452 |
|
|
|
1464 |
|
1465 |
https://rentry.org/llm-settings
|
1466 |
|
1467 |
+
Llamacpp-server - usage / parameters / samplers:
|
1468 |
|
1469 |
https://github.com/ggerganov/llama.cpp/blob/master/examples/server/README.md
|
1470 |
|