Note Original dataset used to train tokenisers and models.

pietrolesci/tokenisers

Updated Feb 27

Note Tokenisers trained on the MiniPile. The `_raw_tokenisers` folder contains the original tokenisers trained with a vocabulary size of 320k. Then, each folder is a `transformers`-compatible tokeniser of a smaller size.

Note Tokenised MiniPile dataset(s). Each split correponds to a tokeniser in `pietrolesci/tokenisers`.

Note This dataset is a subset of the `fineweb-edu/sample/100BT` dataset containing the initial 20.2M documents (or roughly 20B tokens). The default configuration is the raw data. Any other configuration corresponds to the tokenised version, where the configuration name corresponds to the tokeniser used to tokenise it (see `pietrolesci/tokenisers`).

Note Log-probabilities computed on the validation set of the MiniPile dataset using the models above.

pietrolesci/me57M-tied_minipile_bpe8064minipile

Updated Mar 14

Note Model trained for 50k steps on the MiniPile dataset. Each branch is a different checkpoint saved each 2k steps.

pietrolesci/me57M-tied_minipile_bpe32000minipile

Updated Mar 14

Note Model trained for 50k steps on the MiniPile dataset. Each branch is a different checkpoint saved each 2k steps.

pietrolesci/me57M-tied_minipile_bpe128000minipile

Updated Mar 14

Note Model trained for 50k steps on the MiniPile dataset. Each branch is a different checkpoint saved each 2k steps.

pietrolesci/me57M-tied_minipile_wordpiece32000minipile

Updated Mar 14

Note Model trained for 50k steps on the MiniPile dataset. Each branch is a different checkpoint saved each 2k steps.

pietrolesci/me57M-tied_minipile_bpe2wp32000minipile

Updated Mar 14

Note Model trained for 50k steps on the MiniPile dataset. Each branch is a different checkpoint saved every 2k steps. The bpe2wp nomenclature means that we choose the merges using the BPE objective, and we tokenised the MiniPile using the resulting vocabulary and the WordPiece tokenisation function (i.e., longest prefix match).

pietrolesci/me340M-tied_minipile_bpe32000minipile

Updated Mar 14

Note Model trained for 50k steps on the MiniPile dataset. Each branch is a different checkpoint saved each 2k steps.

pietrolesci/me850M_minipile_bpe32000minipile

Updated Mar 14

Note Model trained for 50k steps on the MiniPile dataset. Each branch is a different checkpoint saved each 2k steps.

pietrolesci/me100M-tied_finewebedu-20B_bpe32000minipile

Updated Mar 19

Note Model trained for 50k steps on 20B tokens from Finewed-Edu dataset. Each branch is a different checkpoint saved each 2k steps.

pietrolesci/me100M_finewebedu-20B_bpe32000minipile

Updated Mar 20

Note Model trained for 50k steps on 20B tokens from Finewed-Edu dataset. Each branch is a different checkpoint saved each 2k steps.