Update README.md
Browse files
README.md
CHANGED
@@ -1,6 +1,6 @@
|
|
1 |
This is a **RoBERTa-base** model trained from scratch in Spanish.
|
2 |
|
3 |
-
The training dataset is mc4 (1) subsampling documents to a total of about 50 million examples. Sampling is biased towards average perplexity values, discarding more often documents with very large values (poor quality) of very small values (short, repetitive texts).
|
4 |
|
5 |
This model takes the one using sequence length 128 (2) and trains during 25.000 steps using sequence length 512.
|
6 |
|
|
|
1 |
This is a **RoBERTa-base** model trained from scratch in Spanish.
|
2 |
|
3 |
+
The training dataset is mc4 (1) subsampling documents to a total of about 50 million examples. Sampling is biased towards average perplexity values (using a Gaussian function), discarding more often documents with very large values (poor quality) of very small values (short, repetitive texts).
|
4 |
|
5 |
This model takes the one using sequence length 128 (2) and trains during 25.000 steps using sequence length 512.
|
6 |
|