Upload folder using huggingface_hub

.gitattributes

@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+tokenizer.json filter=lfs diff=lfs merge=lfs -text

README.md

@@ -1,3 +1,398 @@
----
-license: apache-2.0
----
+---
+language: 
+- multilingual
+- bg
+- ca
+- code
+- cs
+- cy
+- da
+- de
+- el
+- en
+- es
+- et
+- eu
+- fi
+- fr
+- ga
+- gl
+- hr
+- hu
+- it
+- lt
+- lv
+- mt
+- nl
+- nn
+- \no
+- oc
+- pl
+- pt
+- ro
+- ru
+- sh
+- sk
+- sl
+- sr
+- sv
+- uk
+tags: 
+  - roberta
+  - mroberta
+  - fill-mask
+license: apache-2.0
+library_name: transformers
+---
+
+
+# mRoBERTa Model Card
+
+mRoBERTa is a new **multilingual foundational model** based on the [RoBERTa](https://huggingface.co/FacebookAI/roberta-base) architecture. It has been pretrained from scratch using 35 European languages and code. The pretraining corpus consists of **12.8TB of high-quality data**. This is significantly larger compared to previous state-of-the-art encoder-only foundational models like [XLM-RoBERTa-base](https://huggingface.co/FacebookAI/xlm-roberta-base) and [XLM-RoBERTa-large](https://huggingface.co/FacebookAI/xlm-roberta-large), whose training datasets included less multilingual data, amounting to 2.5TB.
+
+## Technical Description
+
+Technical details of the mRoBERTa model.
+
+| Description          | Value         |
+|-------------------------|:--------------|
+| Model Parameters        | 283M          |
+| Tokenizer Type          | SPM           |
+| Vocabulary size         | 256,000       |
+| Precision               | bfloat16      |
+| Context length          | 512           |
+
+
+Training Hyperparemeters
+
+| Hyperparameter                | Value                             |
+|-------------------------      |:--------------                    |
+| Pretraining  Objective        | Masked Language Modeling          |
+| Learning Rate                 | 7E-05                             |
+| Learning Rate Scheduler       | Cosine                            |
+| Warmup                        | 10k                               |
+| Optimizer                     | AdamW                             |
+| Optimizer Hyperparameters     | AdamW (β1=0.9,β2=0.98,ε =1e-06 )  |
+| Optimizer Decay               | 1E-02                             |
+| Global Batch Size             | 8192                              |
+| Dropout                       | 1E-01                             |
+| Attention Dropout             | 1E-01                             |
+| Activation Function           | GeLU                             |
+
+
+
+
+
+
+
+## Data
+
+### Pretraining Corpus
+
+The training corpus consists of **35 European languages** and 92 programming languages, amounting to a total of **12.8TB** of high-quality data.
+
+This highly multilingual corpus is predominantly composed of data from Colossal OSCAR, which contributes a significant 66.06% of the total tokens. 
+Following this, Starcoder provides 11.91%, and Spanish Crawling adds 3.34%. 
+The next largest sources are French PD at 3.12% and Proof Pile at 1.98%. 
+Other notable contributions include Macocu, Pile of Law, and Eurlex, each contributing around 1.5% to 1.3%. 
+These major sources collectively form the bulk of the corpus, ensuring a rich and diverse dataset for training the language model.
+The remaining 10% comes from smaller sources in various languages.
+
+
+
+The final pretraining language distribution  split by language can be seen in the following picture:
+<img src="./images/roberta_pretraining_lang_distribution.png" alt="drawing"/>
+
+<small>Further details about the pretraining corpus can be found [here](https://huggingface.co/BSC-LT/ALIA-40b), as it is the same as the Salamandra foundational model.</small>
+
+
+# Multilingual Evaluation and Performance
+
+Evaluation is done using multilingual benchmarks in order to assess the multilingual capabilities of the models.
+
+The following multilingual benchmarks have been considered:
+
+| Benchmark        | Description | Languages |       Source |   
+|------------------|-------------|-----------|--------------|
+| XTREME|  Benchmark for the evaluation of the cross-lingual generalization ability of pre-trained multilingual models | bg,ca,de,el,en,es,et,eu,fi,fr,hu,it,lt,nl,pl,pt,ro,ru,uk | [LINK](https://github.com/google-research/xtreme) |
+| CLUB | Human-Annotated Catalan Benchmark  | ca | [LINK](https://club.aina.bsc.es/datasets.html) |
+| Basque Custom Benchmark |  A set of NER, POS and TC evaluation tasks to assess the performace in Basque language.  | eu | [LINK](https://huggingface.co/datasets/orai-nlp/basqueGLUE) |
+| Galician Custom Benchmark |  NER and POS evaluation tasks to assess the performace in Galician language. | gl | [LINK](https://github.com/xavier-gz/SLI_Galician_Corpora/blob/main/SLI_NERC_Galician_Gold_CoNLL.1.0.tar.gz) [LINK](https://huggingface.co/datasets/universal-dependencies/universal_dependencies)|
+
+
+The following base foundational models have been considered:
+
+
+
+ 
+| Multilingual Foundational Model | Number of Parameters | Vocab Size | Description |
+|---------------------------------|----------------------|------------|-------------|
+| [BERTa](https://huggingface.co/PlanTL-GOB-ES/roberta-base-ca)                      |   126M              |     52K       |     BERTa is a Catalan-specific language model pretrained with Catalan-only data.        |
+| [BERTinho](https://huggingface.co/dvilares/bertinho-gl-base-cased)                      |   109M              |     30K       |    BERTinho is monolingual BERT model for Galician language.          |
+| [mBERT](https://huggingface.co/google-bert/bert-base-multilingual-cased)                           | 178M                | 120K       |    Multilingual BERT model pretrained on the top 104 languages with the largest Wikipedia.         |
+| [mRoBERTa](https://huggingface.co/BSC-LT/mRoBERTa)                       | 283M                | 256K       |     RoBERTa base model pretrained  with 35 European languages and a larger vocabulary size.         |
+| [roberta-base-bne](https://huggingface.co/PlanTL-GOB-ES/roberta-base-bne)                | 125M                | 50K        |  RoBERTa base model pretrained with 570GB of data from web crawlings performed by the National Library of Spain from 2009 to 2019.            |
+| [RoBERTa-ca](https://huggingface.co/BSC-LT/RoBERTa-ca)                      | 125M                |   50K        |    RoBERTa-ca is a Catalan-specific language model obtained by using vocabulary adaptation from mRoBERTa.        |
+| [xlm-roberta-base](https://huggingface.co/FacebookAI/xlm-roberta-base)                | 279M                | 250K       |   Foundational RoBERTa model pretrained with CommonCrawl data containing 100 languages.          | 
+| [xlm-roberta-large](https://huggingface.co/FacebookAI/xlm-roberta-large)               | 561M                | 250K       | Foundational RoBERTa model pretrained with CommonCrawl data containing 100 languages.            |
+
+
+
+## RESULTS
+
+This section presents results across various multilingual benchmarks, with the maximum values highlighted in bold and the second-highest values underlined.
+
+### XTREME Benchmark
+
+The Cross-lingual TRansfer Evaluation of Multilingual Encoders ([XTREME](https://github.com/google-research/xtreme)) benchmark is a benchmark for the evaluation of the cross-lingual generalization ability of pre-trained multilingual models. It includes nine tasks that collectively require reasoning about different levels of syntax and semantics. The languages in XTREME are selected to maximize language diversity, coverage in existing tasks, availability of training data, and overlap with the languages present during pre-training of the models.
+
+#### 🔵 Sentence Classification
+
+##### 🔵 XNLI
+Metric used: Accuracy.
+
+<table>
+<tr><th>langs</th><th>mBERT (178M)</th><th>xlm-roberta-base (279M)</th><th>xlm-roberta-large (561M)</th><th>mRoBERTa (283M)</th></tr>
+<tr><td>bg</td><td style=''>69.34</td><td style='text-decoration: underline;'>78.26</td><td style='font-weight: bold;'>82.10</td><td style=''>77.56</td></tr>
+<tr><td>de</td><td style=''>71.54</td><td style=''>76.75</td><td style='font-weight: bold;'>81.62</td><td style='text-decoration: underline;'>77.01</td></tr>
+<tr><td>el</td><td style=''>66.51</td><td style='text-decoration: underline;'>76.37</td><td style='font-weight: bold;'>81.46</td><td style=''>76.35</td></tr>
+<tr><td>en</td><td style=''>82.20</td><td style=''>84.45</td><td style='font-weight: bold;'>87.98</td><td style='text-decoration: underline;'>85.69</td></tr>
+<tr><td>es</td><td style=''>74.81</td><td style=''>78.18</td><td style='font-weight: bold;'>83.65</td><td style='text-decoration: underline;'>79.66</td></tr>
+<tr><td>fr</td><td style=''>74.25</td><td style=''>78.24</td><td style='font-weight: bold;'>82.71</td><td style='text-decoration: underline;'>79.16</td></tr>
+<tr><td>ru</td><td style=''>68.56</td><td style='text-decoration: underline;'>76.21</td><td style='font-weight: bold;'>79.10</td><td style=''>74.73</td></tr>
+</table>
+
+##### 🔵  PAWS-X
+Metric used: Accuracy.
+
+<table>
+<tr><th>langs</th><th>mBERT (178M)</th><th>xlm-roberta-base (279M)</th><th>xlm-roberta-large (561M)</th><th>mRoBERTa (283M)</th></tr>
+<tr><td>de</td><td style=''>85.65</td><td style='text-decoration: underline;'>86.95</td><td style=''>85.05</td><td style='font-weight: bold;'>87.35</td></tr>
+<tr><td>en</td><td style=''>93.50</td><td style='text-decoration: underline;'>93.90</td><td style=''>91.45</td><td style='font-weight: bold;'>94.75</td></tr>
+<tr><td>es</td><td style=''>87.75</td><td style='font-weight: bold;'>89.30</td><td style=''>87.65</td><td style='text-decoration: underline;'>88.60</td></tr>
+<tr><td>fr</td><td style=''>86.60</td><td style='text-decoration: underline;'>88.55</td><td style=''>87.30</td><td style='font-weight: bold;'>89.20</td></tr>
+</table>
+
+
+
+#### 🟣 Structured Prediction: POS
+
+##### 🟣  POS (UDPOS)
+Metric used: F1.
+
+<table>
+<tr><th>langs</th><th>mBERT (178M)</th><th>xlm-roberta-base (279M)</th><th>xlm-roberta-large (561M)</th><th>mRoBERTa (283M)</th></tr>
+<tr><td>bg</td><td style=''>85.14</td><td style='text-decoration: underline;'>88.62</td><td style='font-weight: bold;'>89.06</td><td style=''>88.19</td></tr>
+<tr><td>de</td><td style=''>85.71</td><td style=''>88.41</td><td style='font-weight: bold;'>88.65</td><td style='text-decoration: underline;'>88.58</td></tr>
+<tr><td>el</td><td style=''>80.92</td><td style='font-weight: bold;'>87.12</td><td style=''>86.55</td><td style='text-decoration: underline;'>87.03</td></tr>
+<tr><td>en</td><td style=''>95.43</td><td style=''>95.79</td><td style='font-weight: bold;'>96.07</td><td style='text-decoration: underline;'>95.85</td></tr>
+<tr><td>es</td><td style=''>85.85</td><td style='text-decoration: underline;'>88.10</td><td style='font-weight: bold;'>89.31</td><td style=''>87.45</td></tr>
+<tr><td>et</td><td style=''>79.68</td><td style=''>86.22</td><td style='font-weight: bold;'>87.36</td><td style='text-decoration: underline;'>86.25</td></tr>
+<tr><td>eu</td><td style=''>60.18</td><td style=''>68.83</td><td style='font-weight: bold;'>71.85</td><td style='text-decoration: underline;'>69.22</td></tr>
+<tr><td>fi</td><td style=''>79.72</td><td style='text-decoration: underline;'>85.90</td><td style='font-weight: bold;'>86.54</td><td style=''>84.23</td></tr>
+<tr><td>fr</td><td style=''>81.20</td><td style=''>86.34</td><td style='font-weight: bold;'>88.24</td><td style='text-decoration: underline;'>87.00</td></tr>
+<tr><td>hu</td><td style=''>78.39</td><td style='text-decoration: underline;'>83.05</td><td style='font-weight: bold;'>83.84</td><td style=''>82.96</td></tr>
+<tr><td>it</td><td style=''>87.86</td><td style=''>88.91</td><td style='font-weight: bold;'>90.01</td><td style='text-decoration: underline;'>89.11</td></tr>
+<tr><td>lt</td><td style=''>78.59</td><td style='text-decoration: underline;'>83.86</td><td style='font-weight: bold;'>84.91</td><td style=''>81.12</td></tr>
+<tr><td>nl</td><td style=''>88.59</td><td style=''>89.16</td><td style='font-weight: bold;'>89.70</td><td style='text-decoration: underline;'>89.31</td></tr>
+<tr><td>pl</td><td style=''>80.34</td><td style='text-decoration: underline;'>84.61</td><td style='font-weight: bold;'>85.77</td><td style=''>84.23</td></tr>
+<tr><td>pt</td><td style=''>85.77</td><td style='text-decoration: underline;'>87.53</td><td style='font-weight: bold;'>88.56</td><td style=''>87.18</td></tr>
+<tr><td>ro</td><td style=''>76.51</td><td style='text-decoration: underline;'>83.99</td><td style='font-weight: bold;'>86.47</td><td style=''>82.74</td></tr>
+<tr><td>ru</td><td style=''>85.36</td><td style=''>88.75</td><td style='font-weight: bold;'>89.83</td><td style='text-decoration: underline;'>89.09</td></tr>
+<tr><td>uk</td><td style=''>80.63</td><td style=''>84.79</td><td style='font-weight: bold;'>85.84</td><td style='text-decoration: underline;'>85.19</td></tr>
+</table>
+
+
+##### 🟣  NER (PANX)
+Metric used: F1.
+
+<table>
+<tr><th>langs</th><th>mBERT (178M)</th><th>xlm-roberta-base (279M)</th><th>xlm-roberta-large (561M)</th><th>mRoBERTa (283M)</th></tr>
+<tr><td>bg</td><td style=''>78.38</td><td style=''>76.52</td><td style='font-weight: bold;'>81.97</td><td style='text-decoration: underline;'>78.66</td></tr>
+<tr><td>de</td><td style='font-weight: bold;'>78.89</td><td style=''>73.92</td><td style='text-decoration: underline;'>78.59</td><td style=''>78.17</td></tr>
+<tr><td>el</td><td style=''>74.09</td><td style=''>73.07</td><td style='font-weight: bold;'>75.49</td><td style='text-decoration: underline;'>74.81</td></tr>
+<tr><td>en</td><td style='font-weight: bold;'>84.69</td><td style=''>82.70</td><td style='text-decoration: underline;'>84.50</td><td style=''>83.56</td></tr>
+<tr><td>es</td><td style=''>72.32</td><td style=''>72.83</td><td style='text-decoration: underline;'>73.46</td><td style='font-weight: bold;'>78.30</td></tr>
+<tr><td>et</td><td style='text-decoration: underline;'>77.55</td><td style=''>72.56</td><td style='font-weight: bold;'>78.37</td><td style=''>73.92</td></tr>
+<tr><td>eu</td><td style='font-weight: bold;'>66.52</td><td style=''>58.34</td><td style='text-decoration: underline;'>60.01</td><td style=''>56.74</td></tr>
+<tr><td>fi</td><td style='text-decoration: underline;'>78.11</td><td style=''>74.98</td><td style='font-weight: bold;'>78.46</td><td style=''>76.42</td></tr>
+<tr><td>fr</td><td style='text-decoration: underline;'>79.45</td><td style=''>77.00</td><td style='font-weight: bold;'>80.16</td><td style=''>76.94</td></tr>
+<tr><td>hu</td><td style='text-decoration: underline;'>77.39</td><td style=''>75.48</td><td style='font-weight: bold;'>80.10</td><td style=''>73.31</td></tr>
+<tr><td>it</td><td style='font-weight: bold;'>81.33</td><td style=''>76.68</td><td style='text-decoration: underline;'>80.60</td><td style=''>80.04</td></tr>
+<tr><td>lt</td><td style='text-decoration: underline;'>75.48</td><td style=''>73.76</td><td style='font-weight: bold;'>76.41</td><td style=''>72.71</td></tr>
+<tr><td>nl</td><td style='text-decoration: underline;'>82.40</td><td style=''>79.80</td><td style='font-weight: bold;'>82.92</td><td style=''>81.42</td></tr>
+<tr><td>pl</td><td style='font-weight: bold;'>80.57</td><td style=''>77.15</td><td style='text-decoration: underline;'>80.55</td><td style=''>80.26</td></tr>
+<tr><td>pt</td><td style='text-decoration: underline;'>79.66</td><td style=''>76.60</td><td style='font-weight: bold;'>80.97</td><td style=''>76.13</td></tr>
+<tr><td>ro</td><td style='text-decoration: underline;'>74.73</td><td style=''>71.79</td><td style='font-weight: bold;'>81.42</td><td style=''>66.85</td></tr>
+<tr><td>ru</td><td style=''>65.42</td><td style=''>63.93</td><td style='font-weight: bold;'>70.68</td><td style='text-decoration: underline;'>67.53</td></tr>
+<tr><td>uk</td><td style='text-decoration: underline;'>71.71</td><td style=''>66.78</td><td style='font-weight: bold;'>74.12</td><td style=''>71.69</td></tr>
+</table>
+
+#### ⚪️ Sentence Retrieval
+
+##### ⚪️  BUCC2018
+Metric used: F1.
+
+<table>
+<tr><th>langs</th><th>mBERT (178M)</th><th>xlm-roberta-base (279M)</th><th>xlm-roberta-large (561M)</th><th>mRoBERTa (283M)</th></tr>
+<tr><td>de</td><td style=''>63.26</td><td style=''>66.83</td><td style='text-decoration: underline;'>75.23</td><td style='font-weight: bold;'>86.09</td></tr>
+<tr><td>fr</td><td style=''>62.62</td><td style=''>65.79</td><td style='text-decoration: underline;'>69.29</td><td style='font-weight: bold;'>79.21</td></tr>
+<tr><td>ru</td><td style=''>54.97</td><td style=''>70.12</td><td style='text-decoration: underline;'>75.57</td><td style='font-weight: bold;'>82.93</td></tr>
+</table>
+
+##### ⚪️  Tatoeba
+Metric used: Accuracy.
+
+<table>
+<tr><th>langs</th><th>mBERT (178M)</th><th>xlm-roberta-base (279M)</th><th>xlm-roberta-large (561M)</th><th>mRoBERTa (283M)</th></tr>
+<tr><td>bg</td><td style=''>48.80</td><td style=''>66.90</td><td style='text-decoration: underline;'>71.60</td><td style='font-weight: bold;'>77.60</td></tr>
+<tr><td>ca</td><td style=''>59.80</td><td style=''>57.30</td><td style='text-decoration: underline;'>62.20</td><td style='font-weight: bold;'>80.20</td></tr>
+<tr><td>de</td><td style=''>75.40</td><td style=''>88.40</td><td style='text-decoration: underline;'>88.80</td><td style='font-weight: bold;'>95.60</td></tr>
+<tr><td>el</td><td style=''>29.80</td><td style=''>51.60</td><td style='text-decoration: underline;'>61.80</td><td style='font-weight: bold;'>72.30</td></tr>
+<tr><td>es</td><td style=''>64.10</td><td style=''>71.00</td><td style='text-decoration: underline;'>75.70</td><td style='font-weight: bold;'>89.70</td></tr>
+<tr><td>et</td><td style=''>28.10</td><td style=''>44.20</td><td style='text-decoration: underline;'>52.20</td><td style='font-weight: bold;'>61.80</td></tr>
+<tr><td>eu</td><td style=''>25.50</td><td style=''>26.10</td><td style='text-decoration: underline;'>35.80</td><td style='font-weight: bold;'>53.40</td></tr>
+<tr><td>fi</td><td style=''>39.00</td><td style='text-decoration: underline;'>63.90</td><td style='font-weight: bold;'>71.60</td><td style='text-decoration: underline;'>63.90</td></tr>
+<tr><td>fr</td><td style=''>64.30</td><td style=''>72.50</td><td style='text-decoration: underline;'>73.70</td><td style='font-weight: bold;'>81.30</td></tr>
+<tr><td>hu</td><td style=''>36.90</td><td style=''>58.70</td><td style='font-weight: bold;'>65.40</td><td style='text-decoration: underline;'>62.40</td></tr>
+<tr><td>it</td><td style=''>57.30</td><td style=''>64.70</td><td style='text-decoration: underline;'>68.30</td><td style='font-weight: bold;'>80.30</td></tr>
+<tr><td>lt</td><td style=''>31.10</td><td style='text-decoration: underline;'>54.80</td><td style='font-weight: bold;'>59.60</td><td style=''>49.30</td></tr>
+<tr><td>nl</td><td style=''>63.70</td><td style=''>76.80</td><td style='text-decoration: underline;'>80.80</td><td style='font-weight: bold;'>86.60</td></tr>
+<tr><td>pl</td><td style=''>50.10</td><td style=''>65.20</td><td style='text-decoration: underline;'>75.90</td><td style='font-weight: bold;'>79.00</td></tr>
+<tr><td>pt</td><td style=''>68.40</td><td style=''>76.60</td><td style='text-decoration: underline;'>82.20</td><td style='font-weight: bold;'>88.80</td></tr>
+<tr><td>ro</td><td style=''>51.50</td><td style=''>68.80</td><td style='font-weight: bold;'>75.70</td><td style='text-decoration: underline;'>69.00</td></tr>
+<tr><td>ru</td><td style=''>59.40</td><td style=''>69.80</td><td style='text-decoration: underline;'>74.10</td><td style='font-weight: bold;'>81.60</td></tr>
+<tr><td>uk</td><td style=''>52.60</td><td style=''>57.30</td><td style='text-decoration: underline;'>69.10</td><td style='font-weight: bold;'>77.50</td></tr>
+</table>
+
+
+
+
+
+
+#### ⚫ Question Answering
+
+##### ⚫ XQUAD
+Metric used: F1.
+
+<table>
+<tr><th>langs</th><th>mBERT (178M)</th><th>xlm-roberta-base (279M)</th><th>xlm-roberta-large (561M)</th><th>mRoBERTa (283M)</th></tr>
+<tr><td>de</td><td style=''>73.55</td><td style='text-decoration: underline;'>74.81</td><td style='font-weight: bold;'>80.15</td><td style=''>73.92</td></tr>
+<tr><td>el</td><td style=''>63.74</td><td style=''>73.34</td><td style='font-weight: bold;'>80.86</td><td style='text-decoration: underline;'>73.56</td></tr>
+<tr><td>en</td><td style='text-decoration: underline;'>84.84</td><td style=''>84.22</td><td style='font-weight: bold;'>88.13</td><td style=''>82.70</td></tr>
+<tr><td>es</td><td style=''>75.06</td><td style=''>76.44</td><td style='font-weight: bold;'>82.21</td><td style='text-decoration: underline;'>77.07</td></tr>
+<tr><td>ru</td><td style=''>72.02</td><td style='text-decoration: underline;'>74.73</td><td style='font-weight: bold;'>80.11</td><td style=''>72.85</td></tr>
+</table>
+
+
+##### ⚫ MLQA
+Metric used: F1.
+
+<table>
+<tr><th>langs</th><th>mBERT (178M)</th><th>xlm-roberta-base (279M)</th><th>xlm-roberta-large (561M)</th><th>mRoBERTa (283M)</th></tr>
+<tr><td>de</td><td style=''>57.68</td><td style=''>62.20</td><td style='font-weight: bold;'>68.78</td><td style='text-decoration: underline;'>63.25</td></tr>
+<tr><td>en</td><td style=''>80.16</td><td style='text-decoration: underline;'>80.27</td><td style='font-weight: bold;'>83.52</td><td style=''>79.81</td></tr>
+<tr><td>es</td><td style=''>64.90</td><td style=''>66.97</td><td style='font-weight: bold;'>72.93</td><td style='text-decoration: underline;'>68.14</td></tr>
+</table>
+
+##### ⚫ TyDiQA
+Metric used: F1.
+
+<table>
+<tr><th>langs</th><th>mBERT (178M)</th><th>xlm-roberta-base (279M)</th><th>xlm-roberta-large (561M)</th><th>mRoBERTa (283M)</th></tr>
+<tr><td>en</td><td style='text-decoration: underline;'>68.26</td><td style=''>59.57</td><td style='font-weight: bold;'>71.33</td><td style=''>61.50</td></tr>
+<tr><td>fi</td><td style='text-decoration: underline;'>55.70</td><td style=''>51.91</td><td style='font-weight: bold;'>70.62</td><td style=''>52.32</td></tr>
+<tr><td>ru</td><td style='text-decoration: underline;'>53.71</td><td style=''>50.75</td><td style='font-weight: bold;'>64.48</td><td style=''>50.66</td></tr>
+</table>
+
+
+
+### CLUB Benchmark
+
+The [Catalan Language Understanding Benchmark](https://club.aina.bsc.es/datasets.html) consists of 6 tasks: Named Entity Recognition (NER), Part-of-Speech Tagging (POS), Semantic Textual Similarity (STS), Text Classification (TC), Textual Entailment (TE), and Question Answering (QA). This benchmark evaluates the model's capabilities in the Catalan language.
+
+This comparison also includes RoBERTa-ca, a model derived from mRoBERTa by applying vocabulary adaptation and performing continual pre-training on a 95GB Catalan-only corpus. For further details, visit [here](https://huggingface.co/BSC-LT/RoBERTa-ca).
+
+<table>
+<tr><th>tasks</th><th style=''>roberta-base-bne (125M)</th><th style=''>berta (126M)</th><th style=''>mBERT (178M)</th><th style=''>xlm-roberta-base (279M)</th><th style=''>xlm-roberta-large (561M)</th><th style=''>roberta-ca (125M)</th><th style=''>mRoBERTa (283M)</th></tr>
+<tr><td>ner (F1)</td><td style=''>87.59</td><td style='text-decoration: underline;'>89.47</td><td style=''>85.89</td><td style=''>87.50</td><td style='text-decoration: underline;'>89.47</td><td style='font-weight: bold;'>89.70</td><td style=''>88.33</td></tr>
+<tr><td>pos (F1)</td><td style=''>98.64</td><td style=''>98.89</td><td style=''>98.78</td><td style=''>98.91</td><td style='font-weight: bold;'>99.03</td><td style='text-decoration: underline;'>99.00</td><td style=''>98.98</td></tr>
+<tr><td>sts (Person)</td><td style=''>74.27</td><td style=''>81.39</td><td style=''>77.05</td><td style=''>75.11</td><td style='font-weight: bold;'>83.49</td><td style='text-decoration: underline;'>82.99</td><td style=''>79.52</td></tr>
+<tr><td>tc (Acc.)</td><td style='text-decoration: underline;'>73.86</td><td style=''>73.16</td><td style=''>72.00</td><td style=''>73.05</td><td style='font-weight: bold;'>74.10</td><td style=''>72.81</td><td style=''>72.41</td></tr>
+<tr><td>te (Acc.)</td><td style=''>72.27</td><td style=''>80.11</td><td style=''>75.86</td><td style=''>78.27</td><td style='font-weight: bold;'>86.63</td><td style=''>82.14</td><td style='text-decoration: underline;'>82.38</td></tr>
+<tr><td>viquiquad (F1)</td><td style=''>82.56</td><td style=''>86.74</td><td style=''>87.42</td><td style=''>86.81</td><td style='font-weight: bold;'>90.35</td><td style=''>87.31</td><td style='text-decoration: underline;'>87.86</td></tr>
+<tr><td>xquad (F1)</td><td style=''>60.56</td><td style=''>67.38</td><td style=''>67.72</td><td style=''>68.56</td><td style='font-weight: bold;'>76.08</td><td style='text-decoration: underline;'>70.53</td><td style=''>69.40</td></tr>
+</table>
+
+###  Galician Benchmark
+
+To evaluate performance in Galician, the models are tested on two tasks highlighted in [Bertinho's paper](https://arxiv.org/pdf/2103.13799):
+ - _NER task_: NER task using the [SLI NERC](https://github.com/xavier-gz/SLI_Galician_Corpora/blob/main/SLI_NERC_Galician_Gold_CoNLL.1.0.tar.gz) dataset.
+ - _POS task_: POS task using the [Universal Dependencies](https://huggingface.co/datasets/universal-dependencies/universal_dependencies) dataset.
+
+
+<table>
+<tr><th>tasks</th><th style=''>bertinho (109M)</th><th style=''>roberta-base-bne (125M)</th><th style=''>mBERT (178M)</th><th style=''>xlm-roberta-base (279M)</th><th style=''>xlm-roberta-large (561M)</th><th style=''>mRoBERTa (283M)</th></tr>
+<tr><td>ner-dataset-SLI NERC (F1)</td><td style=''>86.27</td><td style=''>86.80</td><td style=''>86.22</td><td style=''>85.99</td><td style='font-weight: bold;'>88.10</td><td style='text-decoration: underline;'>87.75</td></tr>
+<tr><td>pos-dataset-UD_GL_CTG (F1)</td><td style=''>97.58</td><td style=''>97.27</td><td style=''>97.57</td><td style='text-decoration: underline;'>97.77</td><td style='font-weight: bold;'>97.95</td><td style=''>97.75</td></tr>
+</table>
+
+
+
+
+###  BasqueGLUE Benchmark
+
+To assess the model performance in the Basque language, the [BasqueGLUE](https://huggingface.co/datasets/orai-nlp/basqueGLUE) Benchmark is used as the baseline. BasqueGLUE  has been elaborated from previously existing datasets and following similar criteria to those used for the construction of GLUE and SuperGLUE. Some of the tasks has been slightly adapted to easily assess all the models (e.g, FMTODeu_slot is originally  described as  "Slot filling" task, but it is evaluated as NERC, since it follows the BIO annotation scheme).
+
+<table>
+<tr><th>tasks</th><th style=''>roberta-base-bne (125M)</th><th style=''>mBERT (178M)</th><th style=''>xlm-roberta-base (279M)</th><th style=''>xlm-roberta-large (561M)</th><th style=''>mRoBERTa (283M)</th></tr>
+<tr><td>NERC - NERCid (F1)</td><td style=''>71.53</td><td style=''>79.98</td><td style='text-decoration: underline;'>81.74</td><td style='font-weight: bold;'>83.96</td><td style=''>80.86</td></tr>
+<tr><td>NERC - NERCood (F1)</td><td style=''>61.47</td><td style=''>76.95</td><td style=''>76.23</td><td style='font-weight: bold;'>80.25</td><td style='text-decoration: underline;'>76.97</td></tr>
+<tr><td>NERC - FMTODeu_slot (F1)</td><td style=''>72.70</td><td style=''>73.65</td><td style=''>73.80</td><td style='text-decoration: underline;'>77.09</td><td style='font-weight: bold;'>77.32</td></tr>
+<tr><td>Sentiment Analysis - BEC2016eu (Acc.)</td><td style=''>67.13</td><td style=''>67.05</td><td style='font-weight: bold;'>69.89</td><td style=''>67.90</td><td style='text-decoration: underline;'>69.20</td></tr>
+<tr><td>Topic Classification - BHTCv2 (Acc.)</td><td style=''>66.72</td><td style=''>70.17</td><td style=''>72.01</td><td style='font-weight: bold;'>75.78</td><td style='text-decoration: underline;'>72.55</td></tr>
+<tr><td>Intent Classification - FMTODeu_intent (Acc.)</td><td style=''>78.38</td><td style=''>78.01</td><td style=''>82.15</td><td style='font-weight: bold;'>83.35</td><td style='text-decoration: underline;'>83.07</td></tr>
+<tr><td>Stance Detection - VaxxStance (Acc.)</td><td style=''>58.01</td><td style='font-weight: bold;'>66.67</td><td style=''>61.22</td><td style='text-decoration: underline;'>66.03</td><td style=''>65.71</td></tr>
+</table>
+
+
+
+## Additional information
+
+### Author
+The Language Technologies Unit from Barcelona Supercomputing Center.
+
+### Contact
+For further information, please send an email to <[email protected]>.
+
+### Copyright
+Copyright(c) 2024 by Language Technologies Unit, Barcelona Supercomputing Center.
+
+
+### Funding
+
+This work has been promoted and financed by the Ministerio para la Transformación Digital y de la Función Pública and Plan de Recuperación, Transformación y Resiliencia - Funded by EU – NextGenerationEU within the framework of the project Modelos del Lenguaje.
+
+### Acknowledgements
+
+This project has benefited from the contributions of numerous teams and institutions through data contributions.
+
+In Catalonia, many institutions have been involved in the project. Our thanks to Òmnium Cultural, Parlament de Catalunya, Institut d'Estudis Aranesos, Racó Català, Vilaweb, ACN, Nació Digital, El món and Aquí Berguedà.
+
+At national level, we are especially grateful to our ILENIA project partners: CENID, HiTZ and CiTIUS for their participation. We also extend our genuine gratitude to the Spanish Senate and Congress, Fundación Dialnet, Fundación Elcano and the ‘Instituto Universitario de Sistemas Inteligentes y Aplicaciones Numéricas en Ingeniería (SIANI)’ of the University of Las Palmas de Gran Canaria.
+
+At the international level, we thank the Welsh government, DFKI, Occiglot project, especially Malte Ostendorff, and The Common Crawl Foundation, especially Pedro Ortiz, for their collaboration.
+
+Their valuable efforts have been instrumental in the development of this work.
+
+
+### Disclaimer
+Be aware that the model may contain biases or other unintended distortions. 
+When third parties deploy systems or provide services based on this model, or use the model themselves, 
+they bear the responsibility for mitigating any associated risks and ensuring compliance with applicable regulations, 
+including those governing the use of Artificial Intelligence.
+
+The Barcelona Supercomputing Center, as the owner and creator of the model, shall not be held liable for any outcomes resulting from third-party use.
+
+### License
+[Apache License, Version 2.0](https://www.apache.org/licenses/LICENSE-2.0)

config.json

@@ -0,0 +1,26 @@
+{
+  "architectures": [
+    "RobertaForMaskedLM"
+  ],
+  "attention_probs_dropout_prob": 0.1,
+  "bos_token_id": 0,
+  "classifier_dropout": null,
+  "eos_token_id": 2,
+  "hidden_act": "gelu",
+  "hidden_dropout_prob": 0.1,
+  "hidden_size": 768,
+  "initializer_range": 0.02,
+  "intermediate_size": 3072,
+  "layer_norm_eps": 1e-05,
+  "max_position_embeddings": 514,
+  "model_type": "roberta",
+  "num_attention_heads": 12,
+  "num_hidden_layers": 12,
+  "pad_token_id": 1,
+  "position_embedding_type": "absolute",
+  "torch_dtype": "float32",
+  "transformers_version": "4.45.2",
+  "type_vocab_size": 1,
+  "use_cache": true,
+  "vocab_size": 256000
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:922af003e4d00f517225ba8e65437aa2021e9995850a04019fb58aba1d90f26a
+size 1132679048

pytorch_model.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e20d0734fd085e691495bb99dc5a11693eed22cbf1c841eef4a066efcaa8b473
+size 1132707134

special_tokens_map.json

@@ -0,0 +1,23 @@
+{
+  "bos_token": {
+    "content": "<s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "eos_token": {
+    "content": "</s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "unk_token": {
+    "content": "<unk>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

tokenizer.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9f2bf5621066c920a83c8b0f75b8d0d95b8b1bad04ac245c6746e6d2b303b76b
+size 37007530

tokenizer.model

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5072e3209a04aa01dbf4db72b8fec52cf8cd06a042c9ba819678e084f7b665d5
+size 4813283

tokenizer_config.json

@@ -0,0 +1,1098 @@
+{
+  "add_bos_token": true,
+  "add_eos_token": false,
+  "add_prefix_space": true,
+  "added_tokens_decoder": {
+    "0": {
+      "content": "<s>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "1": {
+      "content": "<pad>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "2": {
+      "content": "</s>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "3": {
+      "content": "<unk>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "4": {
+      "content": "<|im_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "5": {
+      "content": "<|im_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "6": {
+      "content": "<|reserved_token_1|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "7": {
+      "content": "<|reserved_token_2|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "8": {
+      "content": "<|reserved_token_3|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "9": {
+      "content": "<|reserved_token_4|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "10": {
+      "content": "<|reserved_token_5|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "11": {
+      "content": "<|reserved_token_6|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "12": {
+      "content": "<|reserved_token_7|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "13": {
+      "content": "<|reserved_token_8|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "14": {
+      "content": "<|reserved_token_9|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "15": {
+      "content": "<|reserved_token_10|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "16": {
+      "content": "<|reserved_token_11|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "17": {
+      "content": "<|reserved_token_12|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "18": {
+      "content": "<|reserved_token_13|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "19": {
+      "content": "<|reserved_token_14|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "20": {
+      "content": "<|reserved_token_15|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "21": {
+      "content": "<|reserved_token_16|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "22": {
+      "content": "<|reserved_token_17|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "23": {
+      "content": "<|reserved_token_18|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "24": {
+      "content": "<|reserved_token_19|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "25": {
+      "content": "<|reserved_token_20|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "26": {
+      "content": "<|reserved_token_21|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "27": {
+      "content": "<|reserved_token_22|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "28": {
+      "content": "<|reserved_token_23|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "29": {
+      "content": "<|reserved_token_24|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "30": {
+      "content": "<|reserved_token_25|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "31": {
+      "content": "<|reserved_token_26|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "32": {
+      "content": "<|reserved_token_27|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "33": {
+      "content": "<|reserved_token_28|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "34": {
+      "content": "<|reserved_token_29|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "35": {
+      "content": "<|reserved_token_30|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "36": {
+      "content": "<|reserved_token_31|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "37": {
+      "content": "<|reserved_token_32|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "38": {
+      "content": "<|reserved_token_33|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "39": {
+      "content": "<|reserved_token_34|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "40": {
+      "content": "<|reserved_token_35|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "41": {
+      "content": "<|reserved_token_36|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "42": {
+      "content": "<|reserved_token_37|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "43": {
+      "content": "<|reserved_token_38|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "44": {
+      "content": "<|reserved_token_39|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "45": {
+      "content": "<|reserved_token_40|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "46": {
+      "content": "<|reserved_token_41|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "47": {
+      "content": "<|reserved_token_42|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "48": {
+      "content": "<|reserved_token_43|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "49": {
+      "content": "<|reserved_token_44|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "50": {
+      "content": "<|reserved_token_45|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "51": {
+      "content": "<|reserved_token_46|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "52": {
+      "content": "<|reserved_token_47|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "53": {
+      "content": "<|reserved_token_48|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "54": {
+      "content": "<|reserved_token_49|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "55": {
+      "content": "<|reserved_token_50|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "56": {
+      "content": "<|reserved_token_51|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "57": {
+      "content": "<|reserved_token_52|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "58": {
+      "content": "<|reserved_token_53|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "59": {
+      "content": "<|reserved_token_54|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "60": {
+      "content": "<|reserved_token_55|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "61": {
+      "content": "<|reserved_token_56|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "62": {
+      "content": "<|reserved_token_57|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "63": {
+      "content": "<|reserved_token_58|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "64": {
+      "content": "<|reserved_token_59|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "65": {
+      "content": "<|reserved_token_60|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "66": {
+      "content": "<|reserved_token_61|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "67": {
+      "content": "<|reserved_token_62|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "68": {
+      "content": "<|reserved_token_63|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "69": {
+      "content": "<|reserved_token_64|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "70": {
+      "content": "<|reserved_token_65|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "71": {
+      "content": "<|reserved_token_66|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "72": {
+      "content": "<|reserved_token_67|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "73": {
+      "content": "<|reserved_token_68|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "74": {
+      "content": "<|reserved_token_69|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "75": {
+      "content": "<|reserved_token_70|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "76": {
+      "content": "<|reserved_token_71|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "77": {
+      "content": "<|reserved_token_72|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "78": {
+      "content": "<|reserved_token_73|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "79": {
+      "content": "<|reserved_token_74|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "80": {
+      "content": "<|reserved_token_75|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "81": {
+      "content": "<|reserved_token_76|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "82": {
+      "content": "<|reserved_token_77|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "83": {
+      "content": "<|reserved_token_78|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "84": {
+      "content": "<|reserved_token_79|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "85": {
+      "content": "<|reserved_token_80|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "86": {
+      "content": "<|reserved_token_81|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "87": {
+      "content": "<|reserved_token_82|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "88": {
+      "content": "<|reserved_token_83|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "89": {
+      "content": "<|reserved_token_84|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "90": {
+      "content": "<|reserved_token_85|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "91": {
+      "content": "<|reserved_token_86|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "92": {
+      "content": "<|reserved_token_87|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "93": {
+      "content": "<|reserved_token_88|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "94": {
+      "content": "<|reserved_token_89|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "95": {
+      "content": "<|reserved_token_90|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "96": {
+      "content": "<|reserved_token_91|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "97": {
+      "content": "<|reserved_token_92|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "98": {
+      "content": "<|reserved_token_93|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "99": {
+      "content": "<|reserved_token_94|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100": {
+      "content": "<|reserved_token_95|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "101": {
+      "content": "<|reserved_token_96|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "102": {
+      "content": "<|reserved_token_97|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "103": {
+      "content": "<mask>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "104": {
+      "content": "\\r",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "105": {
+      "content": "▁▁",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "106": {
+      "content": "▁▁▁",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "107": {
+      "content": "▁▁▁▁",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "108": {
+      "content": "▁▁▁▁▁",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "109": {
+      "content": "▁▁▁▁▁▁",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "110": {
+      "content": "▁▁▁▁▁▁▁",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "111": {
+      "content": "▁▁▁▁▁▁▁▁",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "112": {
+      "content": "▁▁▁▁▁▁▁▁▁",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "113": {
+      "content": "▁▁▁▁▁▁▁▁▁▁",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "114": {
+      "content": "▁▁▁▁▁▁▁▁▁▁▁",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "115": {
+      "content": "▁▁▁▁▁▁▁▁▁▁▁▁",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "116": {
+      "content": "▁▁▁▁▁▁▁▁▁▁▁▁▁",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "117": {
+      "content": "▁▁▁▁▁▁▁▁▁▁▁▁▁▁",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "118": {
+      "content": "▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "119": {
+      "content": "▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "120": {
+      "content": "▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "121": {
+      "content": "▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "122": {
+      "content": "▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "123": {
+      "content": "▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "124": {
+      "content": "▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "125": {
+      "content": "▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "126": {
+      "content": "▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "127": {
+      "content": "▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "128": {
+      "content": "\t\t",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "129": {
+      "content": "\t\t\t",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "130": {
+      "content": "\t\t\t\t",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "131": {
+      "content": "\t\t\t\t\t",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "132": {
+      "content": "\t\t\t\t\t\t",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "133": {
+      "content": "\n\n",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "134": {
+      "content": "\n\n\n",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    }
+  },
+  "bos_token": "<s>",
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "</s>",
+  "legacy": true,
+  "model_max_length": 512,
+  "pad_token": "<pad>",
+  "sp_model_kwargs": {},
+  "spaces_between_special_tokens": false,
+  "tokenizer_class": "LlamaTokenizer",
+  "unk_token": "<unk>",
+  "use_default_system_prompt": false
+}