Commit
•
2e1052a
1
Parent(s):
9b3de19
Upload 9 files
Browse files- README.md +330 -3
- config.json +48 -0
- generation_config.json +8 -0
- gptx_tokenizer.py +450 -0
- model.safetensors.index.json +297 -0
- special_tokens_map.json +264 -0
- tokenizer.model +3 -0
- tokenizer_config.json +292 -0
README.md
CHANGED
|
@@ -1,3 +1,330 @@
|
|
| 1 |
-
---
|
| 2 |
-
|
| 3 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
---
|
| 2 |
+
language:
|
| 3 |
+
- de
|
| 4 |
+
- bg
|
| 5 |
+
- cs
|
| 6 |
+
- da
|
| 7 |
+
- el
|
| 8 |
+
- en
|
| 9 |
+
- es
|
| 10 |
+
- et
|
| 11 |
+
- fi
|
| 12 |
+
- fr
|
| 13 |
+
- ga
|
| 14 |
+
- hr
|
| 15 |
+
- hu
|
| 16 |
+
- it
|
| 17 |
+
- lt
|
| 18 |
+
- lv
|
| 19 |
+
- mt
|
| 20 |
+
- nl
|
| 21 |
+
- pl
|
| 22 |
+
- pt
|
| 23 |
+
- ro
|
| 24 |
+
- sl
|
| 25 |
+
- sv
|
| 26 |
+
- sk
|
| 27 |
+
metrics:
|
| 28 |
+
- accuracy
|
| 29 |
+
- bleu
|
| 30 |
+
pipeline_tag: text-generation
|
| 31 |
+
library_name: transformers
|
| 32 |
+
base_model:
|
| 33 |
+
- openGPT-X/Teuken-7B-base-v0.4
|
| 34 |
+
license: apache-2.0
|
| 35 |
+
---
|
| 36 |
+
# Model Card for Teuken-7B-instruct-commercial-v0.4
|
| 37 |
+
|
| 38 |
+
|
| 39 |
+
[Teuken-7B-instruct-commercial-v0.4](https://huggingface.co/openGPT-X/Teuken-7B-instruct-commercial-v0.4) is an instruction-tuned 7B parameter multilingual large language model (LLM) pre-trained with 4T tokens in all official 24 European languages and released under Apache 2.0 in the research project [OpenGPT-X](https://opengpt-x.de).
|
| 40 |
+
The base model Teuken-7B-base-v0.4 is available on request 📧 <a href="[email protected]">[email protected]</a>.
|
| 41 |
+
|
| 42 |
+
### Model Description
|
| 43 |
+
|
| 44 |
+
<!-- Provide a longer summary of what this model is. -->
|
| 45 |
+
|
| 46 |
+
- **Developed by:** Fraunhofer, Forschungszentrum Jülich, TU Dresden, DFKI
|
| 47 |
+
- **Funded by:** German Federal Ministry of Economics and Climate Protection (BMWK) in the context of the OpenGPT-X project
|
| 48 |
+
- **Model type:** Transformer based decoder-only model
|
| 49 |
+
- **Language(s) (NLP):** bg, cs, da, de, el, en, es, et, fi, fr, ga, hr, hu, it, lt, lv, mt, nl, pl, pt, ro, sk, sl, sv
|
| 50 |
+
- **Shared by:** OpenGPT-X
|
| 51 |
+
|
| 52 |
+
## Uses
|
| 53 |
+
|
| 54 |
+
<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
|
| 55 |
+
[Teuken-7B-instruct-commercial-v0.4](https://huggingface.co/openGPT-X/Teuken-7B-instruct-commercial-v0.4) is intended for commercial and research use in all official 24 European languages. Since [Teuken-7B-instruct-commercial-v0.4](https://huggingface.co/openGPT-X/Teuken-7B-instruct-commercial-v0.4) focuses on covering all 24 EU languages, it renders more stable results across these languages and better reflects European values in its answers than English-centric models. It is therefore specialized for use in multilingual tasks.
|
| 56 |
+
|
| 57 |
+
## Disclaimer Toxic Content:
|
| 58 |
+
|
| 59 |
+
This Large Language Model (LLM) may generate content that is inappropriate, offensive, or harmful. While the dataset has been heavily filtered to minimize such outputs, the model may still produce text that is biased or toxic due to the large scale and diverse nature of the data.
|
| 60 |
+
|
| 61 |
+
|
| 62 |
+
### Out-of-Scope Use
|
| 63 |
+
|
| 64 |
+
<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
|
| 65 |
+
|
| 66 |
+
The model is not intended for use in math and coding tasks.
|
| 67 |
+
|
| 68 |
+
## Bias, Risks, and Limitations
|
| 69 |
+
|
| 70 |
+
<!-- This section is meant to convey both technical and sociotechnical limitations. -->
|
| 71 |
+
|
| 72 |
+
[Teuken-7B-instruct-commercial-v0.4](https://huggingface.co/openGPT-X/Teuken-7B-instruct-commercial-v0.4) is an instruction-tuned version of Teuken-7B-base-v0.4 (which is available on request 📧 <a href="[email protected]">[email protected]</a>) that is not completely free from biases and hallucinations.
|
| 73 |
+
|
| 74 |
+
## How to Get Started with the Model
|
| 75 |
+
|
| 76 |
+
## Usage
|
| 77 |
+
The model requires transformers, sentencepiece, and the torch library.
|
| 78 |
+
After installation, here's an example of how to use the model:
|
| 79 |
+
|
| 80 |
+
As this model is a fine-tuned model, it must be used with the provided prompt template. Using the model without the prompt template is not intended and is not recommended. The prompt template is defined as follows:
|
| 81 |
+
```python
|
| 82 |
+
user="Hi!"
|
| 83 |
+
lang_code = "DE"
|
| 84 |
+
system_messages={
|
| 85 |
+
"EN": "A chat between a human and an artificial intelligence assistant."
|
| 86 |
+
" The assistant gives helpful and polite answers to the human's questions.",
|
| 87 |
+
"DE": "Ein Gespräch zwischen einem Menschen und einem Assistenten mit künstlicher Intelligenz."
|
| 88 |
+
" Der Assistent gibt hilfreiche und höfliche Antworten auf die Fragen des Menschen.",
|
| 89 |
+
}
|
| 90 |
+
|
| 91 |
+
prompt = f"System: {system_messages[lang_code]}\nUser: {user}\nAssistant:"
|
| 92 |
+
```
|
| 93 |
+
|
| 94 |
+
The prompt template is also directly integrated in the Tokenizer and can be used as follows:
|
| 95 |
+
```python
|
| 96 |
+
import torch
|
| 97 |
+
from transformers import AutoModelForCausalLM, AutoTokenizer
|
| 98 |
+
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
|
| 99 |
+
model_name = "openGPT-X/Teuken-7B-instruct-commercial-v0.4"
|
| 100 |
+
model = AutoModelForCausalLM.from_pretrained(
|
| 101 |
+
model_name,
|
| 102 |
+
trust_remote_code=True,
|
| 103 |
+
torch_dtype=torch.bfloat16,
|
| 104 |
+
)
|
| 105 |
+
model = model.to(device).eval()
|
| 106 |
+
tokenizer = AutoTokenizer.from_pretrained(
|
| 107 |
+
model_name,
|
| 108 |
+
use_fast=False,
|
| 109 |
+
trust_remote_code=True,
|
| 110 |
+
)
|
| 111 |
+
messages = [{"role": "User", "content": "Wer bist du?"}]
|
| 112 |
+
prompt_ids = tokenizer.apply_chat_template(messages, chat_template="DE", tokenize=True, add_generation_prompt=True, return_tensors="pt")
|
| 113 |
+
prediction = model.generate(
|
| 114 |
+
prompt_ids.to(model.device),
|
| 115 |
+
max_length=512,
|
| 116 |
+
do_sample=True,
|
| 117 |
+
top_k=50,
|
| 118 |
+
top_p=0.95,
|
| 119 |
+
temperature=0.7,
|
| 120 |
+
num_return_sequences=1,
|
| 121 |
+
)
|
| 122 |
+
prediction_text = tokenizer.decode(prediction[0].tolist())
|
| 123 |
+
print(prediction_text)
|
| 124 |
+
```
|
| 125 |
+
|
| 126 |
+
This example demonstrates how to load the model and tokenizer, prepare input, generate text, and print the result.
|
| 127 |
+
|
| 128 |
+
## Training Details
|
| 129 |
+
|
| 130 |
+
### Pre-Training Data
|
| 131 |
+
|
| 132 |
+
<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
|
| 133 |
+
|
| 134 |
+
Teuken-7B-base-v0.4 was pre-trained on 4 trillion tokens of data from publicly available sources.
|
| 135 |
+
The pretraining data has a cutoff of September 2023.
|
| 136 |
+
More information is available in our preprint ["Data Processing for the OpenGPT-X Model Family"](http://arxiv.org/abs/2410.08800).
|
| 137 |
+
|
| 138 |
+
|
| 139 |
+
### Instruction-Tuning Data
|
| 140 |
+
The model was fine-tuned on a collection of English- and German-focused instruction-tuning datasets which also contains instructions for 22 official European languages
|
| 141 |
+
The dataset composition contains three types of data: multilingual data, English data, and translated German data
|
| 142 |
+
|
| 143 |
+
#### English data
|
| 144 |
+
* We only included a subsample of the OpenOrca dataset.
|
| 145 |
+
* To select instruction-tuning examples based on their quality, We calculated the reward scores of all English examples utilizing [Starling-RM-7B-alpha](https://huggingface.co/berkeley-nest/Starling-RM-7B-alpha) (Apache-2.0 license)
|
| 146 |
+
|
| 147 |
+
We aim to include roughly the same amount of English examples as we have multilingual examples:
|
| 148 |
+
1. Add all multi-turn examples
|
| 149 |
+
2. Add entire `code_alpaca` dataset subset
|
| 150 |
+
4. For the remaining dataset subsets (`open_orca`, `evol_instruct_143k`, `evol_instruct_70k`, `sharegpt_v3`, `ultrachat_200k`), we add the samples with the highest reward scores so that each dataset subset contributes an equal amount of high-quality examples
|
| 151 |
+
|
| 152 |
+
##### German Data
|
| 153 |
+
As we aim for a German- and English-centric, European language dataset and due to the sparsity of large-scale German instruction-tuning data, we translated the English portion of the above-described dataset composition. For this, we applied the [Alma-13B](https://huggingface.co/haoranxu/ALMA-13B) (MIT license) model. As code can be a problematic case for translation, we implemented a regex-based code detection functionality. With it, we exclude code snippets from translation and insert the code snippets after translation again.
|
| 154 |
+
As the `alpaca_code` contains many code snippets not detectable by our regex-based code detection implementation, we included this part of the dataset from the translation.
|
| 155 |
+
|
| 156 |
+
#### Multilingual data
|
| 157 |
+
For multilingual data we include the 14 offical European languages contained in the [aya_dataset](https://huggingface.co/datasets/CohereForAI/aya_dataset) and the 21 offical European languages contained in the `translated_flan_cot` dataset of the [aya_collection](https://huggingface.co/datasets/CohereForAI/aya_collection/viewer/translated_flan_cot).
|
| 158 |
+
|
| 159 |
+
#### Datasets and Licenses
|
| 160 |
+
|
| 161 |
+
| Name | Language | License |
|
| 162 |
+
| :--------------------------------------------------------------------------------------------------------------------- | :----------- | :---------------------------------------------------------------------------------------------------------------- |
|
| 163 |
+
| [Open-Orca/OpenOrca](https://huggingface.co/datasets/Open-Orca/OpenOrca) | EN | MIT |
|
| 164 |
+
| [sahil2801/CodeAlpaca-20k](https://huggingface.co/datasets/sahil2801/CodeAlpaca-20k) | EN | CC-BY-4.0 |
|
| 165 |
+
| [WizardLM/WizardLM_evol_instruct_V2_196k](https://huggingface.co/datasets/WizardLM/WizardLM_evol_instruct_V2_196k) | EN | MIT |
|
| 166 |
+
| [WizardLM/WizardLM_evol_instruct_70k](https://huggingface.co/datasets/WizardLM/WizardLM_evol_instruct_70k) | EN | MIT |
|
| 167 |
+
| [anon8231489123/ShareGPT_Vicuna_unfiltered](https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered) | EN | Apache-2.0 |
|
| 168 |
+
| [HuggingFaceH4/ultrachat_200k](https://huggingface.co/datasets/HuggingFaceH4/ultrachat_200k) | EN | MIT |
|
| 169 |
+
| [CohereForAI/aya_dataset](https://huggingface.co/datasets/CohereForAI/aya_dataset) | Multilingual | Apache-2.0 |
|
| 170 |
+
| [CohereForAI/aya_collection](https://huggingface.co/datasets/CohereForAI/aya_collection) | Multilingual | Apache-2.0 |
|
| 171 |
+
| [FreedomIntelligence/sharegpt-deutsch](https://huggingface.co/datasets/FreedomIntelligence/sharegpt-deutsch) | DE | Apache-2.0 |
|
| 172 |
+
| [bjoernp/ultrachat_de](https://huggingface.co/datasets/bjoernp/ultrachat_de) | DE | MIT |
|
| 173 |
+
|
| 174 |
+
|
| 175 |
+
|
| 176 |
+
Dataset contribution per language:
|
| 177 |
+
|
| 178 |
+
| | total | de_freedomintelligence_sharegpt | de_ultrachat_de | translated_flan_cot | aya_dataset | ultrachat_200k_translated_to_de | sharegpt_v3_unfiltered_translated_to_de | evol_instruct_143k_translated_to_de | evol_instruct_70k_translated_to_de | open_orca_translated_to_de | ultrachat_200k | sharegpt_v3_unfiltered | code_alpaca | open_orca | evol_instruct_143k | evol_instruct_70k |
|
| 179 |
+
|:---|--------:|----------------------------------:|------------------:|----------------------:|--------------:|----------------------------------:|------------------------------------------:|--------------------------------------:|-------------------------------------:|-----------------------------:|-----------------:|-------------------------:|--------------:|------------:|---------------------:|--------------------:|
|
| 180 |
+
| BG | 1909 | 0 | 0 | 1909 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
|
| 181 |
+
| CS | 1885 | 0 | 0 | 1885 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
|
| 182 |
+
| DA | 2001 | 0 | 0 | 1906 | 95 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
|
| 183 |
+
| DE | 77628 | 5818 | 898 | 1896 | 231 | 6940 | 37555 | 8116 | 8065 | 8109 | 0 | 0 | 0 | 0 | 0 | 0 |
|
| 184 |
+
| ET | 1901 | 0 | 0 | 1901 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
|
| 185 |
+
| EL | 2472 | 0 | 0 | 1881 | 591 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
|
| 186 |
+
| ES | 3800 | 0 | 0 | 1898 | 1902 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
|
| 187 |
+
| EN | 80806 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 6915 | 37600 | 12013 | 8074 | 8099 | 8105 |
|
| 188 |
+
| FI | 2598 | 0 | 0 | 1890 | 708 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
|
| 189 |
+
| FR | 3250 | 0 | 0 | 1890 | 1360 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
|
| 190 |
+
| HU | 1985 | 0 | 0 | 1892 | 93 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
|
| 191 |
+
| MT | 1918 | 0 | 0 | 1918 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
|
| 192 |
+
| IT | 2613 | 0 | 0 | 1910 | 703 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
|
| 193 |
+
| LT | 2800 | 0 | 0 | 1920 | 880 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
|
| 194 |
+
| NL | 3549 | 0 | 0 | 1905 | 1644 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
|
| 195 |
+
| PL | 3322 | 0 | 0 | 1909 | 1413 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
|
| 196 |
+
| PT | 3806 | 0 | 0 | 1897 | 1909 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
|
| 197 |
+
| RO | 1888 | 0 | 0 | 1888 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
|
| 198 |
+
| GA | 3069 | 0 | 0 | 1880 | 1189 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
|
| 199 |
+
| SK | 1922 | 0 | 0 | 1922 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
|
| 200 |
+
| SL | 1894 | 0 | 0 | 1894 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
|
| 201 |
+
| SV | 3160 | 0 | 0 | 1916 | 1244 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
|
| 202 |
+
|
| 203 |
+
Total across languages 210,176
|
| 204 |
+
|
| 205 |
+
|
| 206 |
+
### Training Procedure
|
| 207 |
+
|
| 208 |
+
<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
|
| 209 |
+
Instruction fined tuned version of [Teuken-7B-base-v0.4](https://huggingface.co/openGPT-X/Teuken-7B-base-v0.4).
|
| 210 |
+
More information regarding the pre-training are available in our model preprint ["Teuken-7B-Base & Teuken-7B-Instruct: Towards European LLMs"](https://arxiv.org/abs/2410.03730).
|
| 211 |
+
|
| 212 |
+
#### Training Hyperparameters
|
| 213 |
+
|
| 214 |
+
- **Training regime:** bf16 mixed precision <!--fp32, fp16 mixed precision, , bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
|
| 215 |
+
|
| 216 |
+
## Evaluation
|
| 217 |
+
|
| 218 |
+
<!-- This section describes the evaluation protocols and provides the results. -->
|
| 219 |
+
|
| 220 |
+
Results on multilingual benchmarks for 21 European languages with instruction-tuned models
|
| 221 |
+
| Model | Avg. | EU21-ARC | EU21-HeSw | EU21-TQA | EU21-MMLU |
|
| 222 |
+
|--------------------------------|--------|----------|-----------|----------|-----------|
|
| 223 |
+
| Meta-Llama-3.1-8B-Instruct | **.563** | .563 | .579 | .532 | **.576** |
|
| 224 |
+
| Mistral-7B-Instruct-v0.3 | .527 | .530 | .538 | **.548** | .491 |
|
| 225 |
+
| Salamandra-7B-Instruct | .543 | **.595** | **.637** | .482 | .459 |
|
| 226 |
+
| Aya-23-8B | .485 | .475 | .535 | .476 | .455 |
|
| 227 |
+
| Occiglot-7B-eu5-Instruct | .475 | .484 | .519 | .471 | .428 |
|
| 228 |
+
| Pharia-1-LLM-7B-C-A | .417 | .396 | .438 | .469 | .366 |
|
| 229 |
+
| Bloomz-7B1 | .358 | .316 | .354 | .461 | .302 |
|
| 230 |
+
| **Teuken-7B-instruct-commercial-v0.4** | .531 | .569 | .620 | .503 | .430 |
|
| 231 |
+
|
| 232 |
+
More information regarding the quality of our translated benchmarks are available in our Evaluation preprint ["Towards Multilingual LLM Evaluation for European Languages"](https://arxiv.org/abs/2410.08928).
|
| 233 |
+
More evaluation results regarding Teuken-7B-instruct-research-v0.4 are available in our model preprint ["Teuken-7B-Base & Teuken-7B-Instruct: Towards European LLMs"](https://arxiv.org/abs/2410.03730).
|
| 234 |
+
|
| 235 |
+
The model was evaluated in 21 languages on ARC, GSM8K, HellaSwag, TruthfulQA, Translation and MMLU. Results can also be seen in the [European LLM Leaderboard](https://huggingface.co/spaces/openGPT-X/european-llm-leaderboard).
|
| 236 |
+
|
| 237 |
+
## Technical Specifications
|
| 238 |
+
|
| 239 |
+
### Model Architecture and Objective
|
| 240 |
+
|
| 241 |
+
| Hyper-Parameter | Value |
|
| 242 |
+
|----------------------------|----------|
|
| 243 |
+
| Training Objective | CLM |
|
| 244 |
+
| Activation Function | SwiGLU |
|
| 245 |
+
| Seq Length | 4096 |
|
| 246 |
+
| Position Embeddings | Rotary |
|
| 247 |
+
| Num Layers | 32 |
|
| 248 |
+
| Hidden Size | 4096 |
|
| 249 |
+
| FFN Hidden Size | 13440 |
|
| 250 |
+
| Num Attention Heads | 32 |
|
| 251 |
+
| Head Dim | 128 |
|
| 252 |
+
| Group Query Attention | yes |
|
| 253 |
+
| Num Query Groups | 2 |
|
| 254 |
+
| Normalization | RMSNorm |
|
| 255 |
+
| Learning rate | 3e-4 |
|
| 256 |
+
| Min learning rate | 3e-5 |
|
| 257 |
+
| Disable bias in linear | yes |
|
| 258 |
+
| Hidden dropout | 0.0 |
|
| 259 |
+
| Attention dropout | 0.0 |
|
| 260 |
+
| Optimizer | AdamW |
|
| 261 |
+
| Beta1 | 0.9 |
|
| 262 |
+
| Beta2 | 0.95 |
|
| 263 |
+
| Data-type | bf16 |
|
| 264 |
+
| Recompute-activations | yes |
|
| 265 |
+
| Distributed-optimizers | yes |
|
| 266 |
+
|
| 267 |
+
### Compute Infrastructure
|
| 268 |
+
|
| 269 |
+
We trained our models on JUWELS Booster which consists of 936 compute nodes, each equipped with 4 NVIDIA A100 GPUs. The GPUs are hosted by AMD EPYC Rome CPUs. The compute nodes are connected with HDR-200 InfiniBand in a DragonFly+ topology.
|
| 270 |
+
|
| 271 |
+
#### Hardware
|
| 272 |
+
|
| 273 |
+
The configuration of JUWELS Booster compute nodes is the following:
|
| 274 |
+
|
| 275 |
+
CPU: AMD EPYC 7402 processor; 2 sockets, 24 cores per socket, SMT-2 (total: 2×24×2 = 96 threads) in NPS-4 1 configuration
|
| 276 |
+
Memory: 512 GB DDR4-3200 RAM (of which at least 20 GB is taken by the system software stack, including the file system); 256 GB per socket; 8 memory channels per socket (2 channels per NUMA domain)
|
| 277 |
+
GPU: 4 × NVIDIA A100 Tensor Core GPU with 40 GB; connected via NVLink3 to each other
|
| 278 |
+
Network: 4 × Mellanox HDR200 InfiniBand ConnectX 6 (200 Gbit/s each), HCA
|
| 279 |
+
Periphery: CPU, GPU, and network adapter are connected via 2 PCIe Gen 4 switches with 16 PCIe lanes going to each device (CPU socket: 2×16 lanes). PCIe switches are configured in synthetic mode.
|
| 280 |
+
#### Software
|
| 281 |
+
|
| 282 |
+
[Megatron-LM](https://github.com/OpenGPTX/Megatron-LM)
|
| 283 |
+
|
| 284 |
+
**BibTeX:**
|
| 285 |
+
|
| 286 |
+
If you find our model useful in your research, please consider citing our [preprint](https://arxiv.org/abs/2410.03730):
|
| 287 |
+
```
|
| 288 |
+
@misc{ali2024teuken7bbaseteuken7binstructeuropean,
|
| 289 |
+
title={Teuken-7B-Base & Teuken-7B-Instruct: Towards European LLMs},
|
| 290 |
+
author={Mehdi Ali and Michael Fromm and Klaudia Thellmann and Jan Ebert and Alexander Arno Weber and Richard Rutmann and Charvi Jain and Max Lübbering and Daniel Steinigen and Johannes Leveling and Katrin Klug and Jasper Schulze Buschhoff and Lena Jurkschat and Hammam Abdelwahab and Benny Jörg Stein and Karl-Heinz Sylla and Pavel Denisov and Nicolo' Brandizzi and Qasid Saleem and Anirban Bhowmick and Lennard Helmer and Chelsea John and Pedro Ortiz Suarez and Malte Ostendorff and Alex Jude and Lalith Manjunath and Samuel Weinbach and Carolin Penke and Oleg Filatov and Shima Asaadi and Fabio Barth and Rafet Sifa and Fabian Küch and Andreas Herten and René Jäkel and Georg Rehm and Stefan Kesselheim and Joachim Köhler and Nicolas Flores-Herr},
|
| 291 |
+
year={2024},
|
| 292 |
+
eprint={2410.03730},
|
| 293 |
+
archivePrefix={arXiv},
|
| 294 |
+
primaryClass={cs.CL},
|
| 295 |
+
url={https://arxiv.org/abs/2410.03730},
|
| 296 |
+
}
|
| 297 |
+
```
|
| 298 |
+
|
| 299 |
+
# Team
|
| 300 |
+
## Data Team
|
| 301 |
+
Anirban Bhowmick (IAIS), Nicolo Brandizzi (IAIS), Lennard Helmer (IAIS), Benny Jörg Stein (IAIS), Karl-Heinz Sylla (IAIS), Pavel Denisov (IAIS), Qasid Saleem (IAIS), Johannes Leveling (IAIS), Hammam Abdelwahab (IAIS), Luzian Hahn (IIS), Farzad Naderi (IIS), Md Saiful Islam (IIS), Alexander Schwirjow (IIS), Pedro Ortiz Suarez (ex. DFKI), Malte Ostendorff (ex. DFKI)
|
| 302 |
+
## Model-Training Team
|
| 303 |
+
### Core contributors
|
| 304 |
+
Mehdi Ali (IAIS), Michael Fromm (IAIS), Jan Ebert (FZJ), Chelsea John (FZJ), Lena Jurkschat (TUD), Alexander Weber (IAIS)
|
| 305 |
+
### Contributors:
|
| 306 |
+
Richard Rutmann (IAIS), Daniel Steinigen (IAIS), Lalith Manjunath (TUD), Carolin Penke (FZJ)
|
| 307 |
+
## Evaluation Team
|
| 308 |
+
### Core contributors
|
| 309 |
+
Klaudia Thellmann (TUD), Alex Jude (IAIS), Jasper Buschhoff (IAIS)
|
| 310 |
+
### Contributors:
|
| 311 |
+
Shima Assadi (IIS), Fabio Barth (DFKI)
|
| 312 |
+
## Management
|
| 313 |
+
Joachim Köhler (IAIS), Nicolas Flores-Herr (IAIS), Stefan Kesselheim (FZJ), Andreas Herten (FZJ), Georg Rehm (DFKI), René Jäkel (TUD), Fabian Küch (IIS), Nicole Hildebrandt (IAIS), Ines Wendler (IAIS)
|
| 314 |
+
|
| 315 |
+
We believe that collaboration is key to overcome the aforementioned limitations and thereby strengthening the European GenAI landscape. Because of this, the team invites researchers, developers, and AI enthusiasts to join and engage through various platforms. A Discord server has been created for community collaboration, offering a space for discussions on technical details, ideas, and direct interaction with developers. Additionally, resources like research publications and a European LLM Leaderboard provide insights into Teuken-7B’s performance and technical aspects. The OpenGPT-X team encourages ongoing engagement and collaboration as the project evolves.
|
| 316 |
+
Key links:
|
| 317 |
+
Discord: OpenGPT-X [Discord server](https://discord.com/invite/RvdHpGMvB3)
|
| 318 |
+
Research Papers: OpenGPT-X News [Research Papers](https://opengpt-x.de/en/news-en/)
|
| 319 |
+
LLM Leaderboard: European LLM Leaderboard [LLM Leaderboard](https://huggingface.co/spaces/openGPT-X/european-llm-leaderboard)
|
| 320 |
+
|
| 321 |
+
<div class="hf-card">
|
| 322 |
+
<h2>Contact Information</h2>
|
| 323 |
+
<p>You can reach out to the following model card contact:</p>
|
| 324 |
+
<ul>
|
| 325 |
+
<li>
|
| 326 |
+
<a href="https://huggingface.co/openGPT-X" target="_blank">OpenGPT-X</a>
|
| 327 |
+
- <a href="[email protected]">[email protected]</a>
|
| 328 |
+
</li>
|
| 329 |
+
</ul>
|
| 330 |
+
</div>
|
config.json
ADDED
|
@@ -0,0 +1,48 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"_name_or_path": "/home/Shared/align/models/teuken-instruct/7B_EU24_4T_EU24/train_run_24EU_teuken_commercial_EU24_lr_1e-5_2024_10_05_18_54_27/opengptx_checkpoints/checkpoint-405",
|
| 3 |
+
"architectures": [
|
| 4 |
+
"LlamaForCausalLM"
|
| 5 |
+
],
|
| 6 |
+
"attention_bias": false,
|
| 7 |
+
"attention_dropout": 0.0,
|
| 8 |
+
"auto_map": {
|
| 9 |
+
"AutoTokenizer": [
|
| 10 |
+
"gptx_tokenizer.SPTokenizer",
|
| 11 |
+
null
|
| 12 |
+
]
|
| 13 |
+
},
|
| 14 |
+
"bos_token_id": 1,
|
| 15 |
+
"eos_token_id": 2,
|
| 16 |
+
"hidden_act": "silu",
|
| 17 |
+
"hidden_size": 4096,
|
| 18 |
+
"initializer_range": 0.0158,
|
| 19 |
+
"intermediate_size": 13440,
|
| 20 |
+
"max_position_embeddings": 4096,
|
| 21 |
+
"mlp_bias": false,
|
| 22 |
+
"model_type": "llama",
|
| 23 |
+
"num_attention_heads": 32,
|
| 24 |
+
"num_hidden_layers": 32,
|
| 25 |
+
"num_key_value_heads": 2,
|
| 26 |
+
"pad_token_id": 3,
|
| 27 |
+
"pretraining_tp": 1,
|
| 28 |
+
"rms_norm_eps": 1e-05,
|
| 29 |
+
"rope_scaling": null,
|
| 30 |
+
"rope_theta": 10000.0,
|
| 31 |
+
"tie_word_embeddings": true,
|
| 32 |
+
"tokenizer_class": "SPTokenizer",
|
| 33 |
+
"torch_dtype": "bfloat16",
|
| 34 |
+
"transformers_version": "4.43.2",
|
| 35 |
+
"use_cache": false,
|
| 36 |
+
"vocab_size": 250680,
|
| 37 |
+
"quantization_config": {
|
| 38 |
+
"quant_method": "exl2",
|
| 39 |
+
"version": "0.2.3",
|
| 40 |
+
"bits": 8.0,
|
| 41 |
+
"head_bits": 6,
|
| 42 |
+
"calibration": {
|
| 43 |
+
"rows": 115,
|
| 44 |
+
"length": 2048,
|
| 45 |
+
"dataset": "(default)"
|
| 46 |
+
}
|
| 47 |
+
}
|
| 48 |
+
}
|
generation_config.json
ADDED
|
@@ -0,0 +1,8 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"_from_model_config": true,
|
| 3 |
+
"bos_token_id": 1,
|
| 4 |
+
"eos_token_id": 2,
|
| 5 |
+
"pad_token_id": 3,
|
| 6 |
+
"transformers_version": "4.43.2",
|
| 7 |
+
"use_cache": false
|
| 8 |
+
}
|
gptx_tokenizer.py
ADDED
|
@@ -0,0 +1,450 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
from __future__ import annotations
|
| 2 |
+
|
| 3 |
+
import json
|
| 4 |
+
import os
|
| 5 |
+
import warnings
|
| 6 |
+
from pathlib import Path
|
| 7 |
+
from typing import Any, Dict, List, Mapping, Optional, Tuple, Union
|
| 8 |
+
|
| 9 |
+
import sentencepiece as spm
|
| 10 |
+
from huggingface_hub import hf_hub_download, list_repo_files, try_to_load_from_cache
|
| 11 |
+
from transformers.tokenization_utils import PreTrainedTokenizer
|
| 12 |
+
from transformers.tokenization_utils_base import TOKENIZER_CONFIG_FILE
|
| 13 |
+
|
| 14 |
+
# Define special tokens used in the tokenizer
|
| 15 |
+
EOD_TOKEN = "<eod>"
|
| 16 |
+
PAD_TOKEN = "<pad>"
|
| 17 |
+
BOS_TOKEN = "<s>"
|
| 18 |
+
EOS_TOKEN = "</s>"
|
| 19 |
+
UNK_TOKEN = "<unk>"
|
| 20 |
+
REPO_ID = "openGPT-X/Teuken-7B-instruct-commercial-v0.4"
|
| 21 |
+
|
| 22 |
+
class HFGPTXTokenizer(PreTrainedTokenizer):
|
| 23 |
+
"""
|
| 24 |
+
A custom tokenizer class that extends Hugging Face's PreTrainedTokenizer.
|
| 25 |
+
It is specifically designed to work with SentencePiece models and integrates
|
| 26 |
+
with Hugging Face's tokenizer utilities.
|
| 27 |
+
"""
|
| 28 |
+
|
| 29 |
+
model_file_glob = "*tokenizer.json"
|
| 30 |
+
vocab_files_names = {"tokenizer_file": "tokenizer.json"}
|
| 31 |
+
decode_kwargs: List[str] = []
|
| 32 |
+
|
| 33 |
+
def _encode(self, text: str, return_tokens: bool = False, is_continuation: bool = False):
|
| 34 |
+
"""
|
| 35 |
+
Encode a given text using the tokenizer.
|
| 36 |
+
|
| 37 |
+
Args:
|
| 38 |
+
text (str): The text to encode.
|
| 39 |
+
return_tokens (bool): If True, returns token strings instead of token IDs.
|
| 40 |
+
is_continuation (bool): If True, uses a continuation tokenizer (if available).
|
| 41 |
+
Returns:
|
| 42 |
+
List[int] or List[str]: Encoded text as a list of token IDs or token strings.
|
| 43 |
+
"""
|
| 44 |
+
assert self.tok is not None, "No tokenizer is currently loaded"
|
| 45 |
+
|
| 46 |
+
# Variant with additional sp processor:
|
| 47 |
+
tokenizer = self.continuation_tokenizer if is_continuation else self.tok
|
| 48 |
+
|
| 49 |
+
if return_tokens:
|
| 50 |
+
return tokenizer.encode_as_pieces(text)
|
| 51 |
+
else:
|
| 52 |
+
return tokenizer.encode(text)
|
| 53 |
+
|
| 54 |
+
def create_list_of_special_tokens(self) -> List[str]:
|
| 55 |
+
"""
|
| 56 |
+
Create a list of special tokens, including the BOS, EOS, PAD, EOD tokens,
|
| 57 |
+
and 256 additional placeholder tokens.
|
| 58 |
+
Returns:
|
| 59 |
+
List[str]: List of special tokens.
|
| 60 |
+
"""
|
| 61 |
+
return [self.bos_token, self.eos_token, self.pad_token, self.eod_token] + [
|
| 62 |
+
f"<placeholder_tok_{i}>" for i in range(256)
|
| 63 |
+
]
|
| 64 |
+
|
| 65 |
+
def find_tokenizer_config(self, config_path: Path, repo_id: str = None) -> Optional[Path]:
|
| 66 |
+
if not os.path.isfile(config_path):
|
| 67 |
+
config_path = try_to_load_from_cache(repo_id=repo_id, filename=Path(config_path).name)
|
| 68 |
+
if not config_path:
|
| 69 |
+
config_path = self._download_config_from_hub(repo_id=repo_id)
|
| 70 |
+
|
| 71 |
+
return config_path
|
| 72 |
+
|
| 73 |
+
|
| 74 |
+
def instantiate_from_file_or_name(self, model_file_or_name: str, repo_id: str = None):
|
| 75 |
+
"""
|
| 76 |
+
Load the tokenizer model from a file or download it from a repository.
|
| 77 |
+
|
| 78 |
+
Args:
|
| 79 |
+
model_file_or_name (str): Path to the model file or the model name.
|
| 80 |
+
repo_id (str, optional): Repository ID from which to download the model file.
|
| 81 |
+
|
| 82 |
+
Returns:
|
| 83 |
+
spm.SentencePieceProcessor: Loaded SentencePieceProcessor instance.
|
| 84 |
+
|
| 85 |
+
Raises:
|
| 86 |
+
ValueError: If repo_id is not provided when model_file_or_name is not a file.
|
| 87 |
+
OSError: If the model file cannot be loaded or downloaded.
|
| 88 |
+
"""
|
| 89 |
+
if not os.path.isfile(model_file_or_name):
|
| 90 |
+
model_file_or_name = try_to_load_from_cache(repo_id=repo_id, filename=Path(model_file_or_name).name)
|
| 91 |
+
if not model_file_or_name:
|
| 92 |
+
model_file_or_name = self._download_model_from_hub(repo_id=repo_id)
|
| 93 |
+
|
| 94 |
+
try:
|
| 95 |
+
return spm.SentencePieceProcessor(model_file=model_file_or_name)
|
| 96 |
+
except Exception as e:
|
| 97 |
+
raise OSError(f"Failed to load tokenizer model: {str(e)}")
|
| 98 |
+
|
| 99 |
+
def _download_model_from_hub(self, repo_id: str) -> Optional[str]:
|
| 100 |
+
try:
|
| 101 |
+
# List all files in the repo
|
| 102 |
+
repo_files = list_repo_files(repo_id)
|
| 103 |
+
|
| 104 |
+
# Find the tokenizer model file
|
| 105 |
+
tokenizer_files = [f for f in repo_files if f.endswith('.model')]
|
| 106 |
+
if not tokenizer_files:
|
| 107 |
+
raise FileNotFoundError(f"No .model file found in repository {repo_id}")
|
| 108 |
+
|
| 109 |
+
# Use the first .model file found
|
| 110 |
+
model_file = tokenizer_files[0]
|
| 111 |
+
print(f"Found tokenizer model file: {model_file}")
|
| 112 |
+
|
| 113 |
+
# Download the file
|
| 114 |
+
model_file_or_name = hf_hub_download(repo_id=repo_id, filename=model_file)
|
| 115 |
+
print(f"Downloaded tokenizer model to: {model_file_or_name}")
|
| 116 |
+
except Exception as e:
|
| 117 |
+
raise OSError(f"Failed to download tokenizer model: {str(e)}")
|
| 118 |
+
|
| 119 |
+
return model_file_or_name
|
| 120 |
+
|
| 121 |
+
def _download_config_from_hub(self, repo_id: str):
|
| 122 |
+
if repo_id is None:
|
| 123 |
+
raise ValueError("repo_id must be provided if config_path is not a local file")
|
| 124 |
+
|
| 125 |
+
try:
|
| 126 |
+
# List all files in the repo
|
| 127 |
+
repo_files = list_repo_files(repo_id)
|
| 128 |
+
|
| 129 |
+
# Find the tokenizer config file
|
| 130 |
+
tokenizer_files = [f for f in repo_files if f.endswith('tokenizer_config.json')]
|
| 131 |
+
if not tokenizer_files:
|
| 132 |
+
raise FileNotFoundError(f"No tokenizer_config.json file found in repository {repo_id}")
|
| 133 |
+
|
| 134 |
+
# Use the first tokenizer_config.json file found
|
| 135 |
+
tokenizer_config_file = tokenizer_files[0]
|
| 136 |
+
print(f"Found tokenizer config file: {tokenizer_config_file}")
|
| 137 |
+
|
| 138 |
+
# Download the file
|
| 139 |
+
tokenizer_config_file_or_name = hf_hub_download(repo_id=repo_id, filename=tokenizer_config_file)
|
| 140 |
+
print(f"Downloaded tokenizer config file to: {tokenizer_config_file_or_name}")
|
| 141 |
+
return tokenizer_config_file_or_name
|
| 142 |
+
except Exception as e:
|
| 143 |
+
raise OSError(f"Failed to download tokenizer model: {str(e)}")
|
| 144 |
+
def __init__(
|
| 145 |
+
self,
|
| 146 |
+
model_path: Optional[str] = None,
|
| 147 |
+
config_path: Optional[str] = None,
|
| 148 |
+
**kwargs: Any,
|
| 149 |
+
) -> None:
|
| 150 |
+
"""
|
| 151 |
+
Initialize the tokenizer.
|
| 152 |
+
Args:
|
| 153 |
+
model_path (Optional[str]): Path to the tokenizer model file.
|
| 154 |
+
config_path (Optional[str]): Path to the tokenizer configuration file.
|
| 155 |
+
**kwargs: Additional keyword arguments passed to the superclass.
|
| 156 |
+
This method also ensures backward compatibility by setting
|
| 157 |
+
`clean_up_tokenization_spaces` to False by default.
|
| 158 |
+
"""
|
| 159 |
+
# Prevent cleanup of tokenization spaces to maintain backward compatibility
|
| 160 |
+
self.clean_up_tokenization_spaces = kwargs.setdefault("clean_up_tokenization_spaces", False)
|
| 161 |
+
self.vocab = None
|
| 162 |
+
cp_path = kwargs.get("name_or_path", ".")
|
| 163 |
+
if model_path is None:
|
| 164 |
+
model_path = str(Path(cp_path) / self.vocab_files_names["tokenizer_file"])
|
| 165 |
+
self.tok = self.instantiate_from_file_or_name(model_path, repo_id=REPO_ID)
|
| 166 |
+
|
| 167 |
+
super().__init__(**kwargs)
|
| 168 |
+
|
| 169 |
+
# Specify special tokens which we know the value of.
|
| 170 |
+
# EOD from `tok` is used as what is called EOS in HuggingFace.
|
| 171 |
+
# Since there is no corresponding mapping for EOS from `tok` in
|
| 172 |
+
# HuggingFace, it is treated as an additional special token.
|
| 173 |
+
# Same for all other special tokens.
|
| 174 |
+
self.eos_token = EOD_TOKEN
|
| 175 |
+
self.bos_token = BOS_TOKEN
|
| 176 |
+
self.pad_token = PAD_TOKEN
|
| 177 |
+
|
| 178 |
+
if not self.additional_special_tokens:
|
| 179 |
+
self.additional_special_tokens = [
|
| 180 |
+
token
|
| 181 |
+
for token in self.create_list_of_special_tokens()
|
| 182 |
+
# Filter out the special tokens we added manually.
|
| 183 |
+
if token
|
| 184 |
+
not in [
|
| 185 |
+
self.eos_token,
|
| 186 |
+
self.bos_token,
|
| 187 |
+
self.pad_token,
|
| 188 |
+
]
|
| 189 |
+
]
|
| 190 |
+
if config_path is None:
|
| 191 |
+
config_path = str(Path(cp_path) / TOKENIZER_CONFIG_FILE)
|
| 192 |
+
|
| 193 |
+
if os.path.isfile(config_path):
|
| 194 |
+
self.tokenizer_config = self.load_json(Path(config_path))
|
| 195 |
+
else: # Load from repo
|
| 196 |
+
self.tokenizer_config = self.load_json(Path(self.find_tokenizer_config(Path(config_path), repo_id=REPO_ID)))
|
| 197 |
+
|
| 198 |
+
@property
|
| 199 |
+
def vocab_size(self) -> int:
|
| 200 |
+
"""
|
| 201 |
+
Get the size of the tokenizer vocabulary.
|
| 202 |
+
Returns:
|
| 203 |
+
int: The size of the vocabulary.
|
| 204 |
+
"""
|
| 205 |
+
return self.tok.GetPieceSize()
|
| 206 |
+
|
| 207 |
+
def get_vocab(self) -> Dict[str, int]:
|
| 208 |
+
"""
|
| 209 |
+
Get the vocabulary as a dictionary mapping token strings to their IDs.
|
| 210 |
+
Returns:
|
| 211 |
+
Dict[str, int]: Vocabulary mapping.
|
| 212 |
+
"""
|
| 213 |
+
if self.vocab is None:
|
| 214 |
+
self.vocab = {self.tok.IdToPiece(i): i for i in range(self.vocab_size)}
|
| 215 |
+
return self.vocab
|
| 216 |
+
|
| 217 |
+
def _tokenize(self, text: str, **kwargs) -> List[int]:
|
| 218 |
+
"""
|
| 219 |
+
Tokenize the input text.
|
| 220 |
+
Args:
|
| 221 |
+
text (str): Text to tokenize.
|
| 222 |
+
**kwargs: Additional keyword arguments.
|
| 223 |
+
Returns:
|
| 224 |
+
List[int]: List of token IDs.
|
| 225 |
+
"""
|
| 226 |
+
return_tokens = kwargs.pop("return_tokens", True)
|
| 227 |
+
return self._encode(text, return_tokens=return_tokens, **kwargs)
|
| 228 |
+
|
| 229 |
+
def _convert_token_to_id(self, token: str) -> int:
|
| 230 |
+
"""
|
| 231 |
+
Convert a token string to its corresponding ID.
|
| 232 |
+
Args:
|
| 233 |
+
token (str): The token to convert.
|
| 234 |
+
Returns:
|
| 235 |
+
int: The token's ID.
|
| 236 |
+
Raises:
|
| 237 |
+
ValueError: If the token is unknown and cannot be encoded to a single ID.
|
| 238 |
+
"""
|
| 239 |
+
return self.tok.PieceToId(token)
|
| 240 |
+
|
| 241 |
+
|
| 242 |
+
def decode(
|
| 243 |
+
self,
|
| 244 |
+
token_ids: Union[List[int], List[List[int]]],
|
| 245 |
+
num_threads: Optional[int] = None,
|
| 246 |
+
) -> str:
|
| 247 |
+
"""
|
| 248 |
+
Decode a list of token IDs into a string.
|
| 249 |
+
Args:
|
| 250 |
+
token_ids (Union[List[int], List[List[int]]]): List of token IDs or lists of token IDs.
|
| 251 |
+
num_threads (Optional[int]): Number of threads to use for decoding.
|
| 252 |
+
Returns:
|
| 253 |
+
str: Decoded string.
|
| 254 |
+
"""
|
| 255 |
+
return self.tok.decode(input=token_ids, num_threads=num_threads)
|
| 256 |
+
|
| 257 |
+
def _convert_id_to_token(self, index: int) -> str:
|
| 258 |
+
"""
|
| 259 |
+
Convert a token ID to its corresponding token string.
|
| 260 |
+
Args:
|
| 261 |
+
index (int): Token ID.
|
| 262 |
+
Returns:
|
| 263 |
+
str: Corresponding token string.
|
| 264 |
+
"""
|
| 265 |
+
return self.tok.IdToPiece(index)
|
| 266 |
+
|
| 267 |
+
def convert_tokens_to_string(self, tokens: List[str]) -> str:
|
| 268 |
+
"""
|
| 269 |
+
Convert a list of tokens into a single string.
|
| 270 |
+
Args:
|
| 271 |
+
tokens (List[str]): List of token strings.
|
| 272 |
+
Returns:
|
| 273 |
+
str: Concatenated string of tokens.
|
| 274 |
+
"""
|
| 275 |
+
return self.tok.DecodePieces(tokens)
|
| 276 |
+
|
| 277 |
+
def _tok_decode(self, token_ids: List[int], **kwargs: Any) -> str:
|
| 278 |
+
"""
|
| 279 |
+
Internal method to decode token IDs with additional arguments.
|
| 280 |
+
Args:
|
| 281 |
+
token_ids (List[int]): List of token IDs.
|
| 282 |
+
**kwargs: Additional arguments to pass to the decode method.
|
| 283 |
+
Returns:
|
| 284 |
+
str: Decoded string.
|
| 285 |
+
This method also issues a warning if unsupported arguments are provided.
|
| 286 |
+
"""
|
| 287 |
+
passed_kwargs = {key: value for (key, value) in kwargs.items() if key in self.decode_kwargs}
|
| 288 |
+
if len(passed_kwargs) != len(kwargs):
|
| 289 |
+
warnings.warn("silently ignoring some arguments to `decode` due to missing " "support from the tokenizer.")
|
| 290 |
+
text = self.decode(token_ids, **passed_kwargs)
|
| 291 |
+
return text
|
| 292 |
+
|
| 293 |
+
def save_tokenizer(self, save_dir: str) -> None:
|
| 294 |
+
if not os.path.isdir(save_dir):
|
| 295 |
+
print(f"Vocabulary path ({save_dir}) should be a directory")
|
| 296 |
+
return
|
| 297 |
+
out_vocab_file = os.path.join(save_dir, "tokenizer.model")
|
| 298 |
+
|
| 299 |
+
# if os.path.abspath(self.vocab_file) != os.path.abspath(out_vocab_file) and os.path.isfile(self.vocab_file):
|
| 300 |
+
# copyfile(self.vocab_file, out_vocab_file)
|
| 301 |
+
# elif not os.path.isfile(self.vocab_file):
|
| 302 |
+
with open(out_vocab_file, "wb") as f:
|
| 303 |
+
content_spiece_model = self.tok.serialized_model_proto()
|
| 304 |
+
f.write(content_spiece_model)
|
| 305 |
+
|
| 306 |
+
return (out_vocab_file,)
|
| 307 |
+
|
| 308 |
+
def _decode(
|
| 309 |
+
self,
|
| 310 |
+
token_ids: List[int],
|
| 311 |
+
skip_special_tokens: bool = False,
|
| 312 |
+
clean_up_tokenization_spaces: bool = None,
|
| 313 |
+
spaces_between_special_tokens: bool = True,
|
| 314 |
+
**kwargs: Any,
|
| 315 |
+
) -> str:
|
| 316 |
+
text = self._tok_decode(
|
| 317 |
+
token_ids,
|
| 318 |
+
skip_special_tokens=skip_special_tokens,
|
| 319 |
+
spaces_between_special_tokens=spaces_between_special_tokens,
|
| 320 |
+
**kwargs,
|
| 321 |
+
)
|
| 322 |
+
|
| 323 |
+
clean_up_tokenization_spaces = (
|
| 324 |
+
clean_up_tokenization_spaces
|
| 325 |
+
if clean_up_tokenization_spaces is not None
|
| 326 |
+
else self.clean_up_tokenization_spaces
|
| 327 |
+
)
|
| 328 |
+
if clean_up_tokenization_spaces:
|
| 329 |
+
warnings.warn(
|
| 330 |
+
"when cleaning up tokenization spaces, this will not behave "
|
| 331 |
+
"like the original `GPTXTokenizer`., Please supply "
|
| 332 |
+
"`clean_up_tokenization_spaces=False` for decoding."
|
| 333 |
+
)
|
| 334 |
+
clean_text = self.clean_up_tokenization(text)
|
| 335 |
+
return clean_text
|
| 336 |
+
else:
|
| 337 |
+
return text
|
| 338 |
+
|
| 339 |
+
def save_vocabulary(
|
| 340 |
+
self,
|
| 341 |
+
save_directory: str,
|
| 342 |
+
filename_prefix: Optional[str] = None,
|
| 343 |
+
) -> Tuple[str]:
|
| 344 |
+
filename_prefix = filename_prefix + "-" if filename_prefix else ""
|
| 345 |
+
save_directory = Path(save_directory)
|
| 346 |
+
|
| 347 |
+
self._save_tokenizer_config(save_directory, filename_prefix)
|
| 348 |
+
tokenizer_file_path = self._save_tokenizer(save_directory, filename_prefix)
|
| 349 |
+
|
| 350 |
+
return (tokenizer_file_path,)
|
| 351 |
+
|
| 352 |
+
def _save_tokenizer_config(
|
| 353 |
+
self,
|
| 354 |
+
save_directory: Path,
|
| 355 |
+
filename_prefix: str,
|
| 356 |
+
) -> str:
|
| 357 |
+
self.save_tokenizer_config(save_directory)
|
| 358 |
+
old_tokenizer_config_path = save_directory / TOKENIZER_CONFIG_FILE
|
| 359 |
+
assert old_tokenizer_config_path.is_file(), "tokenizer config path changed"
|
| 360 |
+
new_tokenizer_config_path = save_directory / (filename_prefix + old_tokenizer_config_path.name)
|
| 361 |
+
old_tokenizer_config_path.replace(new_tokenizer_config_path)
|
| 362 |
+
return str(new_tokenizer_config_path)
|
| 363 |
+
|
| 364 |
+
def _find_tokenizer_files(self, save_directory: Path) -> List[Path]:
|
| 365 |
+
files = list(Path(save_directory).glob(self.model_file_glob))
|
| 366 |
+
return files
|
| 367 |
+
|
| 368 |
+
def _get_tokenizer_file(self, files: List[Path]):
|
| 369 |
+
assert files, "no saved tokenizer file found"
|
| 370 |
+
assert len(files) <= 1, "cannot handle multiple saved tokenizer files"
|
| 371 |
+
return files[0]
|
| 372 |
+
|
| 373 |
+
def _save_tokenizer(
|
| 374 |
+
self,
|
| 375 |
+
save_directory: Path,
|
| 376 |
+
filename_prefix: str,
|
| 377 |
+
) -> str:
|
| 378 |
+
self.save_tokenizer(str(save_directory))
|
| 379 |
+
tokenizer_files = self._find_tokenizer_files(save_directory)
|
| 380 |
+
old_tokenizer_file_path = self._get_tokenizer_file(tokenizer_files)
|
| 381 |
+
assert old_tokenizer_file_path.is_file(), "could not access saved tokenizer file"
|
| 382 |
+
new_tokenizer_file_path = save_directory / (filename_prefix + self.vocab_files_names["tokenizer_file"])
|
| 383 |
+
old_tokenizer_file_path.replace(new_tokenizer_file_path)
|
| 384 |
+
return str(new_tokenizer_file_path)
|
| 385 |
+
|
| 386 |
+
def save_tokenizer_config(self, save_dir: Path) -> None:
|
| 387 |
+
# convert Path to str
|
| 388 |
+
for k in self.tokenizer_config:
|
| 389 |
+
if isinstance(self.tokenizer_config[k], Path):
|
| 390 |
+
self.tokenizer_config[k] = str(self.tokenizer_config[k])
|
| 391 |
+
|
| 392 |
+
info_file = save_dir / "tokenizer_config.json"
|
| 393 |
+
with info_file.open("w") as f:
|
| 394 |
+
json.dump(self.tokenizer_config, f, indent=4)
|
| 395 |
+
|
| 396 |
+
def load_json(self, path: Path) -> dict:
|
| 397 |
+
with path.open("r") as f:
|
| 398 |
+
return json.load(f)
|
| 399 |
+
|
| 400 |
+
class SPTokenizer(HFGPTXTokenizer):
|
| 401 |
+
model_file_glob = "*tokenizer.model"
|
| 402 |
+
vocab_files_names = {"tokenizer_file": "tokenizer.model"}
|
| 403 |
+
decode_kwargs = ["num_threads"]
|
| 404 |
+
# `is_continuation` does not work without this, but it doesn't
|
| 405 |
+
# implement all APIs of `PreTrainedTokenizer`.
|
| 406 |
+
def encode(self, text: str, **kwargs) -> List[int]:
|
| 407 |
+
return_tokens = kwargs.pop('return_tokens', False)
|
| 408 |
+
is_continuation = kwargs.pop('is_continuation', False)
|
| 409 |
+
return self._encode(
|
| 410 |
+
text,
|
| 411 |
+
return_tokens=return_tokens,
|
| 412 |
+
is_continuation=is_continuation,
|
| 413 |
+
)
|
| 414 |
+
|
| 415 |
+
def __init__(self, *args, **kwargs):
|
| 416 |
+
super().__init__(*args, **kwargs)
|
| 417 |
+
|
| 418 |
+
self.eos_token = "</s>"
|
| 419 |
+
self.eos_token_id = 2
|
| 420 |
+
self.system_messages_by_lang = { # translations by deepl / google translate
|
| 421 |
+
"BG": "Чат между човек и асистент с изкуствен интелект. Асистентът дава полезни и учтиви отговори на въпросите на човека.", # noqa
|
| 422 |
+
"CS": "Chat mezi člověkem a asistentem s umělou inteligencí. Asistent poskytuje vstřícné a zdvořilé odpovědi na otázky člověka.", # noqa
|
| 423 |
+
"DA": "En chat mellem et menneske og en assistent med kunstig intelligens, som giver hjælpsomme og høflige svar på menneskets spørgsmål.", # noqa
|
| 424 |
+
"DE": "Ein Gespräch zwischen einem Menschen und einem Assistenten mit künstlicher Intelligenz. Der Assistent gibt hilfreiche und höfliche Antworten auf die Fragen des Menschen.", # noqa
|
| 425 |
+
"EL": "Μια συνομιλία μεταξύ ενός ανθρώπου και ενός βοηθού τεχνητής νοημοσύνης. Ο βοηθός δίνει χρήσιμες και ευγενικές απαντήσεις στις ερωτήσεις του ανθρώπου.", # noqa
|
| 426 |
+
"EN": "A chat between a human and an artificial intelligence assistant.The assistant gives helpful and polite answers to the human's questions.", # noqa
|
| 427 |
+
"ES": "Una conversación entre un humano y un asistente de inteligencia artificial. El asistente da respuestas útiles y amables a las preguntas del humano.", # noqa
|
| 428 |
+
"ET": "Inimese ja tehisintellekti assistendi vaheline vestlus. Assistent annab inimese küsimustele abivalmis ja viisakaid vastuseid.", # noqa
|
| 429 |
+
"FI": "Ihmisen ja tekoälyavustajan välinen keskustelu. Avustaja antaa avuliaita ja kohteliaita vastauksia ihmisen kysymyksiin.", # noqa
|
| 430 |
+
"FR": "Conversation entre un humain et un assistant doté d'une intelligence artificielle. L'assistant donne des réponses utiles et polies aux questions de l'homme.", # noqa
|
| 431 |
+
"GA": "Comhrá idir duine agus cúntóir hintleachta saorga. Tugann an cúntóir freagraí cabhracha dea-bhéasacha ar cheisteanna an duine.", # noqa
|
| 432 |
+
"HR": "Razgovor između čovjeka i pomoćnika umjetne inteligencije. Pomoćnik daje korisne i ljubazne odgovore na ljudska pitanja.", # noqa
|
| 433 |
+
"HU": "Egy ember és egy mesterséges intelligencia asszisztens közötti beszélgetés. Az asszisztens segítőkész és udvarias válaszokat ad az ember kérdéseire.", # noqa
|
| 434 |
+
"IT": "Una chat tra un umano e un assistente di intelligenza artificiale. L'assistente fornisce risposte utili ed educate alle domande dell'uomo.", # noqa
|
| 435 |
+
"LT": "Žmogaus ir dirbtinio intelekto asistento pokalbis. Asistentas naudingai ir mandagiai atsako į žmogaus klausimus.", # noqa
|
| 436 |
+
"LV": "Cilvēka un mākslīgā intelekta asistenta tērzēšana. Asistents sniedz noderīgas un pieklājīgas atbildes uz cilvēka jautājumiem.", # noqa
|
| 437 |
+
"MT": "Chat bejn bniedem u assistent ta' intelliġenza artifiċjali. L-assistent jagħti tweġibiet ta' għajnuna u edukat għall-mistoqsijiet tal-bniedem.", # noqa
|
| 438 |
+
"NL": "Een chat tussen een mens en een assistent met kunstmatige intelligentie. De assistent geeft behulpzame en beleefde antwoorden op de vragen van de mens.", # noqa
|
| 439 |
+
"PL": "Czat między człowiekiem a asystentem sztucznej inteligencji. Asystent udziela pomocnych i uprzejmych odpowiedzi na pytania człowieka.", # noqa
|
| 440 |
+
"PT": "Uma conversa entre um ser humano e um assistente de inteligência artificial. O assistente dá respostas úteis e educadas às perguntas do utilizador.", # noqa
|
| 441 |
+
"RO": "O conversație între un om și un asistent cu inteligență artificială. Asistentul oferă răspunsuri utile și politicoase la întrebările omului.", # noqa
|
| 442 |
+
"SK": "Rozhovor medzi človekom a asistentom s umelou inteligenciou. Asistent poskytuje užitočné a zdvorilé odpovede na otázky človeka.", # noqa
|
| 443 |
+
"SL": "Pogovor med človekom in pomočnikom z umetno inteligenco. Pomočnik človeku prijazno in vljudno odgovarja na njegova vprašanja.", # noqa
|
| 444 |
+
"SV": "En chatt mellan en människa och en assistent med artificiell intelligens. Assistenten ger hjälpsamma och artiga svar på människans frågor.", # noqa
|
| 445 |
+
}
|
| 446 |
+
chat_template = "{%- for message in messages %}\n{%- if (message['role']|lower == 'user') != (loop.index0 % 2 == 0) %}\n{{- raise_exception('Roles must alternate User/Assistant/User/Assistant/...') }}\n{%- endif %}\n{%-if message['role']|lower == 'user' %}\n{{- message['role']|capitalize + ': ' + message['content'] + '\\n' }}\n{%- elif message['role']|lower == 'assistant' %}\n{{- message['role']|capitalize + ': ' + message['content'] + eos_token + '\\n' }}\n{%- else %}\n{{- raise_exception('Only user and assistant roles are supported!') }}\n {%- endif %}\n{%- endfor %}{%-if add_generation_prompt %}\n{{- 'Assistant: '}}\n{%- endif %}\n"
|
| 447 |
+
self.chat_template = {
|
| 448 |
+
lang: f"System: {sys_msg}" + "{{- '\\n'}}\n" + chat_template
|
| 449 |
+
for lang, sys_msg in self.system_messages_by_lang.items()
|
| 450 |
+
}
|
model.safetensors.index.json
ADDED
|
@@ -0,0 +1,297 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"metadata": {
|
| 3 |
+
"total_size": 14905450496
|
| 4 |
+
},
|
| 5 |
+
"weight_map": {
|
| 6 |
+
"model.embed_tokens.weight": "model-00001-of-00004.safetensors",
|
| 7 |
+
"model.layers.0.input_layernorm.weight": "model-00001-of-00004.safetensors",
|
| 8 |
+
"model.layers.0.mlp.down_proj.weight": "model-00001-of-00004.safetensors",
|
| 9 |
+
"model.layers.0.mlp.gate_proj.weight": "model-00001-of-00004.safetensors",
|
| 10 |
+
"model.layers.0.mlp.up_proj.weight": "model-00001-of-00004.safetensors",
|
| 11 |
+
"model.layers.0.post_attention_layernorm.weight": "model-00001-of-00004.safetensors",
|
| 12 |
+
"model.layers.0.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
|
| 13 |
+
"model.layers.0.self_attn.o_proj.weight": "model-00001-of-00004.safetensors",
|
| 14 |
+
"model.layers.0.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
|
| 15 |
+
"model.layers.0.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
|
| 16 |
+
"model.layers.1.input_layernorm.weight": "model-00001-of-00004.safetensors",
|
| 17 |
+
"model.layers.1.mlp.down_proj.weight": "model-00001-of-00004.safetensors",
|
| 18 |
+
"model.layers.1.mlp.gate_proj.weight": "model-00001-of-00004.safetensors",
|
| 19 |
+
"model.layers.1.mlp.up_proj.weight": "model-00001-of-00004.safetensors",
|
| 20 |
+
"model.layers.1.post_attention_layernorm.weight": "model-00001-of-00004.safetensors",
|
| 21 |
+
"model.layers.1.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
|
| 22 |
+
"model.layers.1.self_attn.o_proj.weight": "model-00001-of-00004.safetensors",
|
| 23 |
+
"model.layers.1.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
|
| 24 |
+
"model.layers.1.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
|
| 25 |
+
"model.layers.10.input_layernorm.weight": "model-00002-of-00004.safetensors",
|
| 26 |
+
"model.layers.10.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
|
| 27 |
+
"model.layers.10.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
|
| 28 |
+
"model.layers.10.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
|
| 29 |
+
"model.layers.10.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
|
| 30 |
+
"model.layers.10.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
|
| 31 |
+
"model.layers.10.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
|
| 32 |
+
"model.layers.10.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
|
| 33 |
+
"model.layers.10.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
|
| 34 |
+
"model.layers.11.input_layernorm.weight": "model-00002-of-00004.safetensors",
|
| 35 |
+
"model.layers.11.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
|
| 36 |
+
"model.layers.11.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
|
| 37 |
+
"model.layers.11.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
|
| 38 |
+
"model.layers.11.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
|
| 39 |
+
"model.layers.11.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
|
| 40 |
+
"model.layers.11.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
|
| 41 |
+
"model.layers.11.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
|
| 42 |
+
"model.layers.11.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
|
| 43 |
+
"model.layers.12.input_layernorm.weight": "model-00002-of-00004.safetensors",
|
| 44 |
+
"model.layers.12.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
|
| 45 |
+
"model.layers.12.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
|
| 46 |
+
"model.layers.12.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
|
| 47 |
+
"model.layers.12.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
|
| 48 |
+
"model.layers.12.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
|
| 49 |
+
"model.layers.12.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
|
| 50 |
+
"model.layers.12.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
|
| 51 |
+
"model.layers.12.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
|
| 52 |
+
"model.layers.13.input_layernorm.weight": "model-00002-of-00004.safetensors",
|
| 53 |
+
"model.layers.13.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
|
| 54 |
+
"model.layers.13.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
|
| 55 |
+
"model.layers.13.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
|
| 56 |
+
"model.layers.13.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
|
| 57 |
+
"model.layers.13.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
|
| 58 |
+
"model.layers.13.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
|
| 59 |
+
"model.layers.13.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
|
| 60 |
+
"model.layers.13.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
|
| 61 |
+
"model.layers.14.input_layernorm.weight": "model-00002-of-00004.safetensors",
|
| 62 |
+
"model.layers.14.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
|
| 63 |
+
"model.layers.14.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
|
| 64 |
+
"model.layers.14.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
|
| 65 |
+
"model.layers.14.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
|
| 66 |
+
"model.layers.14.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
|
| 67 |
+
"model.layers.14.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
|
| 68 |
+
"model.layers.14.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
|
| 69 |
+
"model.layers.14.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
|
| 70 |
+
"model.layers.15.input_layernorm.weight": "model-00002-of-00004.safetensors",
|
| 71 |
+
"model.layers.15.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
|
| 72 |
+
"model.layers.15.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
|
| 73 |
+
"model.layers.15.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
|
| 74 |
+
"model.layers.15.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
|
| 75 |
+
"model.layers.15.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
|
| 76 |
+
"model.layers.15.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
|
| 77 |
+
"model.layers.15.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
|
| 78 |
+
"model.layers.15.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
|
| 79 |
+
"model.layers.16.input_layernorm.weight": "model-00002-of-00004.safetensors",
|
| 80 |
+
"model.layers.16.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
|
| 81 |
+
"model.layers.16.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
|
| 82 |
+
"model.layers.16.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
|
| 83 |
+
"model.layers.16.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
|
| 84 |
+
"model.layers.16.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
|
| 85 |
+
"model.layers.16.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
|
| 86 |
+
"model.layers.16.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
|
| 87 |
+
"model.layers.16.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
|
| 88 |
+
"model.layers.17.input_layernorm.weight": "model-00002-of-00004.safetensors",
|
| 89 |
+
"model.layers.17.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
|
| 90 |
+
"model.layers.17.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
|
| 91 |
+
"model.layers.17.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
|
| 92 |
+
"model.layers.17.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
|
| 93 |
+
"model.layers.17.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
|
| 94 |
+
"model.layers.17.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
|
| 95 |
+
"model.layers.17.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
|
| 96 |
+
"model.layers.17.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
|
| 97 |
+
"model.layers.18.input_layernorm.weight": "model-00002-of-00004.safetensors",
|
| 98 |
+
"model.layers.18.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
|
| 99 |
+
"model.layers.18.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
|
| 100 |
+
"model.layers.18.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
|
| 101 |
+
"model.layers.18.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
|
| 102 |
+
"model.layers.18.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
|
| 103 |
+
"model.layers.18.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
|
| 104 |
+
"model.layers.18.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
|
| 105 |
+
"model.layers.18.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
|
| 106 |
+
"model.layers.19.input_layernorm.weight": "model-00003-of-00004.safetensors",
|
| 107 |
+
"model.layers.19.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
|
| 108 |
+
"model.layers.19.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
|
| 109 |
+
"model.layers.19.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
|
| 110 |
+
"model.layers.19.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
|
| 111 |
+
"model.layers.19.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
|
| 112 |
+
"model.layers.19.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
|
| 113 |
+
"model.layers.19.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
|
| 114 |
+
"model.layers.19.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
|
| 115 |
+
"model.layers.2.input_layernorm.weight": "model-00001-of-00004.safetensors",
|
| 116 |
+
"model.layers.2.mlp.down_proj.weight": "model-00001-of-00004.safetensors",
|
| 117 |
+
"model.layers.2.mlp.gate_proj.weight": "model-00001-of-00004.safetensors",
|
| 118 |
+
"model.layers.2.mlp.up_proj.weight": "model-00001-of-00004.safetensors",
|
| 119 |
+
"model.layers.2.post_attention_layernorm.weight": "model-00001-of-00004.safetensors",
|
| 120 |
+
"model.layers.2.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
|
| 121 |
+
"model.layers.2.self_attn.o_proj.weight": "model-00001-of-00004.safetensors",
|
| 122 |
+
"model.layers.2.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
|
| 123 |
+
"model.layers.2.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
|
| 124 |
+
"model.layers.20.input_layernorm.weight": "model-00003-of-00004.safetensors",
|
| 125 |
+
"model.layers.20.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
|
| 126 |
+
"model.layers.20.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
|
| 127 |
+
"model.layers.20.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
|
| 128 |
+
"model.layers.20.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
|
| 129 |
+
"model.layers.20.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
|
| 130 |
+
"model.layers.20.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
|
| 131 |
+
"model.layers.20.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
|
| 132 |
+
"model.layers.20.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
|
| 133 |
+
"model.layers.21.input_layernorm.weight": "model-00003-of-00004.safetensors",
|
| 134 |
+
"model.layers.21.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
|
| 135 |
+
"model.layers.21.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
|
| 136 |
+
"model.layers.21.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
|
| 137 |
+
"model.layers.21.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
|
| 138 |
+
"model.layers.21.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
|
| 139 |
+
"model.layers.21.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
|
| 140 |
+
"model.layers.21.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
|
| 141 |
+
"model.layers.21.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
|
| 142 |
+
"model.layers.22.input_layernorm.weight": "model-00003-of-00004.safetensors",
|
| 143 |
+
"model.layers.22.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
|
| 144 |
+
"model.layers.22.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
|
| 145 |
+
"model.layers.22.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
|
| 146 |
+
"model.layers.22.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
|
| 147 |
+
"model.layers.22.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
|
| 148 |
+
"model.layers.22.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
|
| 149 |
+
"model.layers.22.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
|
| 150 |
+
"model.layers.22.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
|
| 151 |
+
"model.layers.23.input_layernorm.weight": "model-00003-of-00004.safetensors",
|
| 152 |
+
"model.layers.23.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
|
| 153 |
+
"model.layers.23.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
|
| 154 |
+
"model.layers.23.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
|
| 155 |
+
"model.layers.23.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
|
| 156 |
+
"model.layers.23.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
|
| 157 |
+
"model.layers.23.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
|
| 158 |
+
"model.layers.23.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
|
| 159 |
+
"model.layers.23.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
|
| 160 |
+
"model.layers.24.input_layernorm.weight": "model-00003-of-00004.safetensors",
|
| 161 |
+
"model.layers.24.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
|
| 162 |
+
"model.layers.24.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
|
| 163 |
+
"model.layers.24.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
|
| 164 |
+
"model.layers.24.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
|
| 165 |
+
"model.layers.24.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
|
| 166 |
+
"model.layers.24.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
|
| 167 |
+
"model.layers.24.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
|
| 168 |
+
"model.layers.24.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
|
| 169 |
+
"model.layers.25.input_layernorm.weight": "model-00003-of-00004.safetensors",
|
| 170 |
+
"model.layers.25.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
|
| 171 |
+
"model.layers.25.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
|
| 172 |
+
"model.layers.25.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
|
| 173 |
+
"model.layers.25.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
|
| 174 |
+
"model.layers.25.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
|
| 175 |
+
"model.layers.25.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
|
| 176 |
+
"model.layers.25.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
|
| 177 |
+
"model.layers.25.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
|
| 178 |
+
"model.layers.26.input_layernorm.weight": "model-00003-of-00004.safetensors",
|
| 179 |
+
"model.layers.26.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
|
| 180 |
+
"model.layers.26.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
|
| 181 |
+
"model.layers.26.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
|
| 182 |
+
"model.layers.26.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
|
| 183 |
+
"model.layers.26.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
|
| 184 |
+
"model.layers.26.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
|
| 185 |
+
"model.layers.26.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
|
| 186 |
+
"model.layers.26.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
|
| 187 |
+
"model.layers.27.input_layernorm.weight": "model-00003-of-00004.safetensors",
|
| 188 |
+
"model.layers.27.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
|
| 189 |
+
"model.layers.27.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
|
| 190 |
+
"model.layers.27.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
|
| 191 |
+
"model.layers.27.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
|
| 192 |
+
"model.layers.27.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
|
| 193 |
+
"model.layers.27.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
|
| 194 |
+
"model.layers.27.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
|
| 195 |
+
"model.layers.27.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
|
| 196 |
+
"model.layers.28.input_layernorm.weight": "model-00003-of-00004.safetensors",
|
| 197 |
+
"model.layers.28.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
|
| 198 |
+
"model.layers.28.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
|
| 199 |
+
"model.layers.28.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
|
| 200 |
+
"model.layers.28.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
|
| 201 |
+
"model.layers.28.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
|
| 202 |
+
"model.layers.28.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
|
| 203 |
+
"model.layers.28.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
|
| 204 |
+
"model.layers.28.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
|
| 205 |
+
"model.layers.29.input_layernorm.weight": "model-00003-of-00004.safetensors",
|
| 206 |
+
"model.layers.29.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
|
| 207 |
+
"model.layers.29.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
|
| 208 |
+
"model.layers.29.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
|
| 209 |
+
"model.layers.29.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
|
| 210 |
+
"model.layers.29.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
|
| 211 |
+
"model.layers.29.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
|
| 212 |
+
"model.layers.29.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
|
| 213 |
+
"model.layers.29.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
|
| 214 |
+
"model.layers.3.input_layernorm.weight": "model-00001-of-00004.safetensors",
|
| 215 |
+
"model.layers.3.mlp.down_proj.weight": "model-00001-of-00004.safetensors",
|
| 216 |
+
"model.layers.3.mlp.gate_proj.weight": "model-00001-of-00004.safetensors",
|
| 217 |
+
"model.layers.3.mlp.up_proj.weight": "model-00001-of-00004.safetensors",
|
| 218 |
+
"model.layers.3.post_attention_layernorm.weight": "model-00001-of-00004.safetensors",
|
| 219 |
+
"model.layers.3.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
|
| 220 |
+
"model.layers.3.self_attn.o_proj.weight": "model-00001-of-00004.safetensors",
|
| 221 |
+
"model.layers.3.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
|
| 222 |
+
"model.layers.3.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
|
| 223 |
+
"model.layers.30.input_layernorm.weight": "model-00003-of-00004.safetensors",
|
| 224 |
+
"model.layers.30.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
|
| 225 |
+
"model.layers.30.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
|
| 226 |
+
"model.layers.30.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
|
| 227 |
+
"model.layers.30.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
|
| 228 |
+
"model.layers.30.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
|
| 229 |
+
"model.layers.30.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
|
| 230 |
+
"model.layers.30.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
|
| 231 |
+
"model.layers.30.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
|
| 232 |
+
"model.layers.31.input_layernorm.weight": "model-00004-of-00004.safetensors",
|
| 233 |
+
"model.layers.31.mlp.down_proj.weight": "model-00004-of-00004.safetensors",
|
| 234 |
+
"model.layers.31.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
|
| 235 |
+
"model.layers.31.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
|
| 236 |
+
"model.layers.31.post_attention_layernorm.weight": "model-00004-of-00004.safetensors",
|
| 237 |
+
"model.layers.31.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
|
| 238 |
+
"model.layers.31.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
|
| 239 |
+
"model.layers.31.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
|
| 240 |
+
"model.layers.31.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
|
| 241 |
+
"model.layers.4.input_layernorm.weight": "model-00001-of-00004.safetensors",
|
| 242 |
+
"model.layers.4.mlp.down_proj.weight": "model-00001-of-00004.safetensors",
|
| 243 |
+
"model.layers.4.mlp.gate_proj.weight": "model-00001-of-00004.safetensors",
|
| 244 |
+
"model.layers.4.mlp.up_proj.weight": "model-00001-of-00004.safetensors",
|
| 245 |
+
"model.layers.4.post_attention_layernorm.weight": "model-00001-of-00004.safetensors",
|
| 246 |
+
"model.layers.4.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
|
| 247 |
+
"model.layers.4.self_attn.o_proj.weight": "model-00001-of-00004.safetensors",
|
| 248 |
+
"model.layers.4.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
|
| 249 |
+
"model.layers.4.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
|
| 250 |
+
"model.layers.5.input_layernorm.weight": "model-00001-of-00004.safetensors",
|
| 251 |
+
"model.layers.5.mlp.down_proj.weight": "model-00001-of-00004.safetensors",
|
| 252 |
+
"model.layers.5.mlp.gate_proj.weight": "model-00001-of-00004.safetensors",
|
| 253 |
+
"model.layers.5.mlp.up_proj.weight": "model-00001-of-00004.safetensors",
|
| 254 |
+
"model.layers.5.post_attention_layernorm.weight": "model-00001-of-00004.safetensors",
|
| 255 |
+
"model.layers.5.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
|
| 256 |
+
"model.layers.5.self_attn.o_proj.weight": "model-00001-of-00004.safetensors",
|
| 257 |
+
"model.layers.5.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
|
| 258 |
+
"model.layers.5.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
|
| 259 |
+
"model.layers.6.input_layernorm.weight": "model-00001-of-00004.safetensors",
|
| 260 |
+
"model.layers.6.mlp.down_proj.weight": "model-00001-of-00004.safetensors",
|
| 261 |
+
"model.layers.6.mlp.gate_proj.weight": "model-00001-of-00004.safetensors",
|
| 262 |
+
"model.layers.6.mlp.up_proj.weight": "model-00001-of-00004.safetensors",
|
| 263 |
+
"model.layers.6.post_attention_layernorm.weight": "model-00001-of-00004.safetensors",
|
| 264 |
+
"model.layers.6.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
|
| 265 |
+
"model.layers.6.self_attn.o_proj.weight": "model-00001-of-00004.safetensors",
|
| 266 |
+
"model.layers.6.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
|
| 267 |
+
"model.layers.6.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
|
| 268 |
+
"model.layers.7.input_layernorm.weight": "model-00002-of-00004.safetensors",
|
| 269 |
+
"model.layers.7.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
|
| 270 |
+
"model.layers.7.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
|
| 271 |
+
"model.layers.7.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
|
| 272 |
+
"model.layers.7.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
|
| 273 |
+
"model.layers.7.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
|
| 274 |
+
"model.layers.7.self_attn.o_proj.weight": "model-00001-of-00004.safetensors",
|
| 275 |
+
"model.layers.7.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
|
| 276 |
+
"model.layers.7.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
|
| 277 |
+
"model.layers.8.input_layernorm.weight": "model-00002-of-00004.safetensors",
|
| 278 |
+
"model.layers.8.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
|
| 279 |
+
"model.layers.8.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
|
| 280 |
+
"model.layers.8.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
|
| 281 |
+
"model.layers.8.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
|
| 282 |
+
"model.layers.8.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
|
| 283 |
+
"model.layers.8.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
|
| 284 |
+
"model.layers.8.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
|
| 285 |
+
"model.layers.8.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
|
| 286 |
+
"model.layers.9.input_layernorm.weight": "model-00002-of-00004.safetensors",
|
| 287 |
+
"model.layers.9.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
|
| 288 |
+
"model.layers.9.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
|
| 289 |
+
"model.layers.9.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
|
| 290 |
+
"model.layers.9.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
|
| 291 |
+
"model.layers.9.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
|
| 292 |
+
"model.layers.9.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
|
| 293 |
+
"model.layers.9.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
|
| 294 |
+
"model.layers.9.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
|
| 295 |
+
"model.norm.weight": "model-00004-of-00004.safetensors"
|
| 296 |
+
}
|
| 297 |
+
}
|
special_tokens_map.json
ADDED
|
@@ -0,0 +1,264 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"additional_special_tokens": [
|
| 3 |
+
"</s>",
|
| 4 |
+
"<placeholder_tok_0>",
|
| 5 |
+
"<placeholder_tok_1>",
|
| 6 |
+
"<placeholder_tok_2>",
|
| 7 |
+
"<placeholder_tok_3>",
|
| 8 |
+
"<placeholder_tok_4>",
|
| 9 |
+
"<placeholder_tok_5>",
|
| 10 |
+
"<placeholder_tok_6>",
|
| 11 |
+
"<placeholder_tok_7>",
|
| 12 |
+
"<placeholder_tok_8>",
|
| 13 |
+
"<placeholder_tok_9>",
|
| 14 |
+
"<placeholder_tok_10>",
|
| 15 |
+
"<placeholder_tok_11>",
|
| 16 |
+
"<placeholder_tok_12>",
|
| 17 |
+
"<placeholder_tok_13>",
|
| 18 |
+
"<placeholder_tok_14>",
|
| 19 |
+
"<placeholder_tok_15>",
|
| 20 |
+
"<placeholder_tok_16>",
|
| 21 |
+
"<placeholder_tok_17>",
|
| 22 |
+
"<placeholder_tok_18>",
|
| 23 |
+
"<placeholder_tok_19>",
|
| 24 |
+
"<placeholder_tok_20>",
|
| 25 |
+
"<placeholder_tok_21>",
|
| 26 |
+
"<placeholder_tok_22>",
|
| 27 |
+
"<placeholder_tok_23>",
|
| 28 |
+
"<placeholder_tok_24>",
|
| 29 |
+
"<placeholder_tok_25>",
|
| 30 |
+
"<placeholder_tok_26>",
|
| 31 |
+
"<placeholder_tok_27>",
|
| 32 |
+
"<placeholder_tok_28>",
|
| 33 |
+
"<placeholder_tok_29>",
|
| 34 |
+
"<placeholder_tok_30>",
|
| 35 |
+
"<placeholder_tok_31>",
|
| 36 |
+
"<placeholder_tok_32>",
|
| 37 |
+
"<placeholder_tok_33>",
|
| 38 |
+
"<placeholder_tok_34>",
|
| 39 |
+
"<placeholder_tok_35>",
|
| 40 |
+
"<placeholder_tok_36>",
|
| 41 |
+
"<placeholder_tok_37>",
|
| 42 |
+
"<placeholder_tok_38>",
|
| 43 |
+
"<placeholder_tok_39>",
|
| 44 |
+
"<placeholder_tok_40>",
|
| 45 |
+
"<placeholder_tok_41>",
|
| 46 |
+
"<placeholder_tok_42>",
|
| 47 |
+
"<placeholder_tok_43>",
|
| 48 |
+
"<placeholder_tok_44>",
|
| 49 |
+
"<placeholder_tok_45>",
|
| 50 |
+
"<placeholder_tok_46>",
|
| 51 |
+
"<placeholder_tok_47>",
|
| 52 |
+
"<placeholder_tok_48>",
|
| 53 |
+
"<placeholder_tok_49>",
|
| 54 |
+
"<placeholder_tok_50>",
|
| 55 |
+
"<placeholder_tok_51>",
|
| 56 |
+
"<placeholder_tok_52>",
|
| 57 |
+
"<placeholder_tok_53>",
|
| 58 |
+
"<placeholder_tok_54>",
|
| 59 |
+
"<placeholder_tok_55>",
|
| 60 |
+
"<placeholder_tok_56>",
|
| 61 |
+
"<placeholder_tok_57>",
|
| 62 |
+
"<placeholder_tok_58>",
|
| 63 |
+
"<placeholder_tok_59>",
|
| 64 |
+
"<placeholder_tok_60>",
|
| 65 |
+
"<placeholder_tok_61>",
|
| 66 |
+
"<placeholder_tok_62>",
|
| 67 |
+
"<placeholder_tok_63>",
|
| 68 |
+
"<placeholder_tok_64>",
|
| 69 |
+
"<placeholder_tok_65>",
|
| 70 |
+
"<placeholder_tok_66>",
|
| 71 |
+
"<placeholder_tok_67>",
|
| 72 |
+
"<placeholder_tok_68>",
|
| 73 |
+
"<placeholder_tok_69>",
|
| 74 |
+
"<placeholder_tok_70>",
|
| 75 |
+
"<placeholder_tok_71>",
|
| 76 |
+
"<placeholder_tok_72>",
|
| 77 |
+
"<placeholder_tok_73>",
|
| 78 |
+
"<placeholder_tok_74>",
|
| 79 |
+
"<placeholder_tok_75>",
|
| 80 |
+
"<placeholder_tok_76>",
|
| 81 |
+
"<placeholder_tok_77>",
|
| 82 |
+
"<placeholder_tok_78>",
|
| 83 |
+
"<placeholder_tok_79>",
|
| 84 |
+
"<placeholder_tok_80>",
|
| 85 |
+
"<placeholder_tok_81>",
|
| 86 |
+
"<placeholder_tok_82>",
|
| 87 |
+
"<placeholder_tok_83>",
|
| 88 |
+
"<placeholder_tok_84>",
|
| 89 |
+
"<placeholder_tok_85>",
|
| 90 |
+
"<placeholder_tok_86>",
|
| 91 |
+
"<placeholder_tok_87>",
|
| 92 |
+
"<placeholder_tok_88>",
|
| 93 |
+
"<placeholder_tok_89>",
|
| 94 |
+
"<placeholder_tok_90>",
|
| 95 |
+
"<placeholder_tok_91>",
|
| 96 |
+
"<placeholder_tok_92>",
|
| 97 |
+
"<placeholder_tok_93>",
|
| 98 |
+
"<placeholder_tok_94>",
|
| 99 |
+
"<placeholder_tok_95>",
|
| 100 |
+
"<placeholder_tok_96>",
|
| 101 |
+
"<placeholder_tok_97>",
|
| 102 |
+
"<placeholder_tok_98>",
|
| 103 |
+
"<placeholder_tok_99>",
|
| 104 |
+
"<placeholder_tok_100>",
|
| 105 |
+
"<placeholder_tok_101>",
|
| 106 |
+
"<placeholder_tok_102>",
|
| 107 |
+
"<placeholder_tok_103>",
|
| 108 |
+
"<placeholder_tok_104>",
|
| 109 |
+
"<placeholder_tok_105>",
|
| 110 |
+
"<placeholder_tok_106>",
|
| 111 |
+
"<placeholder_tok_107>",
|
| 112 |
+
"<placeholder_tok_108>",
|
| 113 |
+
"<placeholder_tok_109>",
|
| 114 |
+
"<placeholder_tok_110>",
|
| 115 |
+
"<placeholder_tok_111>",
|
| 116 |
+
"<placeholder_tok_112>",
|
| 117 |
+
"<placeholder_tok_113>",
|
| 118 |
+
"<placeholder_tok_114>",
|
| 119 |
+
"<placeholder_tok_115>",
|
| 120 |
+
"<placeholder_tok_116>",
|
| 121 |
+
"<placeholder_tok_117>",
|
| 122 |
+
"<placeholder_tok_118>",
|
| 123 |
+
"<placeholder_tok_119>",
|
| 124 |
+
"<placeholder_tok_120>",
|
| 125 |
+
"<placeholder_tok_121>",
|
| 126 |
+
"<placeholder_tok_122>",
|
| 127 |
+
"<placeholder_tok_123>",
|
| 128 |
+
"<placeholder_tok_124>",
|
| 129 |
+
"<placeholder_tok_125>",
|
| 130 |
+
"<placeholder_tok_126>",
|
| 131 |
+
"<placeholder_tok_127>",
|
| 132 |
+
"<placeholder_tok_128>",
|
| 133 |
+
"<placeholder_tok_129>",
|
| 134 |
+
"<placeholder_tok_130>",
|
| 135 |
+
"<placeholder_tok_131>",
|
| 136 |
+
"<placeholder_tok_132>",
|
| 137 |
+
"<placeholder_tok_133>",
|
| 138 |
+
"<placeholder_tok_134>",
|
| 139 |
+
"<placeholder_tok_135>",
|
| 140 |
+
"<placeholder_tok_136>",
|
| 141 |
+
"<placeholder_tok_137>",
|
| 142 |
+
"<placeholder_tok_138>",
|
| 143 |
+
"<placeholder_tok_139>",
|
| 144 |
+
"<placeholder_tok_140>",
|
| 145 |
+
"<placeholder_tok_141>",
|
| 146 |
+
"<placeholder_tok_142>",
|
| 147 |
+
"<placeholder_tok_143>",
|
| 148 |
+
"<placeholder_tok_144>",
|
| 149 |
+
"<placeholder_tok_145>",
|
| 150 |
+
"<placeholder_tok_146>",
|
| 151 |
+
"<placeholder_tok_147>",
|
| 152 |
+
"<placeholder_tok_148>",
|
| 153 |
+
"<placeholder_tok_149>",
|
| 154 |
+
"<placeholder_tok_150>",
|
| 155 |
+
"<placeholder_tok_151>",
|
| 156 |
+
"<placeholder_tok_152>",
|
| 157 |
+
"<placeholder_tok_153>",
|
| 158 |
+
"<placeholder_tok_154>",
|
| 159 |
+
"<placeholder_tok_155>",
|
| 160 |
+
"<placeholder_tok_156>",
|
| 161 |
+
"<placeholder_tok_157>",
|
| 162 |
+
"<placeholder_tok_158>",
|
| 163 |
+
"<placeholder_tok_159>",
|
| 164 |
+
"<placeholder_tok_160>",
|
| 165 |
+
"<placeholder_tok_161>",
|
| 166 |
+
"<placeholder_tok_162>",
|
| 167 |
+
"<placeholder_tok_163>",
|
| 168 |
+
"<placeholder_tok_164>",
|
| 169 |
+
"<placeholder_tok_165>",
|
| 170 |
+
"<placeholder_tok_166>",
|
| 171 |
+
"<placeholder_tok_167>",
|
| 172 |
+
"<placeholder_tok_168>",
|
| 173 |
+
"<placeholder_tok_169>",
|
| 174 |
+
"<placeholder_tok_170>",
|
| 175 |
+
"<placeholder_tok_171>",
|
| 176 |
+
"<placeholder_tok_172>",
|
| 177 |
+
"<placeholder_tok_173>",
|
| 178 |
+
"<placeholder_tok_174>",
|
| 179 |
+
"<placeholder_tok_175>",
|
| 180 |
+
"<placeholder_tok_176>",
|
| 181 |
+
"<placeholder_tok_177>",
|
| 182 |
+
"<placeholder_tok_178>",
|
| 183 |
+
"<placeholder_tok_179>",
|
| 184 |
+
"<placeholder_tok_180>",
|
| 185 |
+
"<placeholder_tok_181>",
|
| 186 |
+
"<placeholder_tok_182>",
|
| 187 |
+
"<placeholder_tok_183>",
|
| 188 |
+
"<placeholder_tok_184>",
|
| 189 |
+
"<placeholder_tok_185>",
|
| 190 |
+
"<placeholder_tok_186>",
|
| 191 |
+
"<placeholder_tok_187>",
|
| 192 |
+
"<placeholder_tok_188>",
|
| 193 |
+
"<placeholder_tok_189>",
|
| 194 |
+
"<placeholder_tok_190>",
|
| 195 |
+
"<placeholder_tok_191>",
|
| 196 |
+
"<placeholder_tok_192>",
|
| 197 |
+
"<placeholder_tok_193>",
|
| 198 |
+
"<placeholder_tok_194>",
|
| 199 |
+
"<placeholder_tok_195>",
|
| 200 |
+
"<placeholder_tok_196>",
|
| 201 |
+
"<placeholder_tok_197>",
|
| 202 |
+
"<placeholder_tok_198>",
|
| 203 |
+
"<placeholder_tok_199>",
|
| 204 |
+
"<placeholder_tok_200>",
|
| 205 |
+
"<placeholder_tok_201>",
|
| 206 |
+
"<placeholder_tok_202>",
|
| 207 |
+
"<placeholder_tok_203>",
|
| 208 |
+
"<placeholder_tok_204>",
|
| 209 |
+
"<placeholder_tok_205>",
|
| 210 |
+
"<placeholder_tok_206>",
|
| 211 |
+
"<placeholder_tok_207>",
|
| 212 |
+
"<placeholder_tok_208>",
|
| 213 |
+
"<placeholder_tok_209>",
|
| 214 |
+
"<placeholder_tok_210>",
|
| 215 |
+
"<placeholder_tok_211>",
|
| 216 |
+
"<placeholder_tok_212>",
|
| 217 |
+
"<placeholder_tok_213>",
|
| 218 |
+
"<placeholder_tok_214>",
|
| 219 |
+
"<placeholder_tok_215>",
|
| 220 |
+
"<placeholder_tok_216>",
|
| 221 |
+
"<placeholder_tok_217>",
|
| 222 |
+
"<placeholder_tok_218>",
|
| 223 |
+
"<placeholder_tok_219>",
|
| 224 |
+
"<placeholder_tok_220>",
|
| 225 |
+
"<placeholder_tok_221>",
|
| 226 |
+
"<placeholder_tok_222>",
|
| 227 |
+
"<placeholder_tok_223>",
|
| 228 |
+
"<placeholder_tok_224>",
|
| 229 |
+
"<placeholder_tok_225>",
|
| 230 |
+
"<placeholder_tok_226>",
|
| 231 |
+
"<placeholder_tok_227>",
|
| 232 |
+
"<placeholder_tok_228>",
|
| 233 |
+
"<placeholder_tok_229>",
|
| 234 |
+
"<placeholder_tok_230>",
|
| 235 |
+
"<placeholder_tok_231>",
|
| 236 |
+
"<placeholder_tok_232>",
|
| 237 |
+
"<placeholder_tok_233>",
|
| 238 |
+
"<placeholder_tok_234>",
|
| 239 |
+
"<placeholder_tok_235>",
|
| 240 |
+
"<placeholder_tok_236>",
|
| 241 |
+
"<placeholder_tok_237>",
|
| 242 |
+
"<placeholder_tok_238>",
|
| 243 |
+
"<placeholder_tok_239>",
|
| 244 |
+
"<placeholder_tok_240>",
|
| 245 |
+
"<placeholder_tok_241>",
|
| 246 |
+
"<placeholder_tok_242>",
|
| 247 |
+
"<placeholder_tok_243>",
|
| 248 |
+
"<placeholder_tok_244>",
|
| 249 |
+
"<placeholder_tok_245>",
|
| 250 |
+
"<placeholder_tok_246>",
|
| 251 |
+
"<placeholder_tok_247>",
|
| 252 |
+
"<placeholder_tok_248>",
|
| 253 |
+
"<placeholder_tok_249>",
|
| 254 |
+
"<placeholder_tok_250>",
|
| 255 |
+
"<placeholder_tok_251>",
|
| 256 |
+
"<placeholder_tok_252>",
|
| 257 |
+
"<placeholder_tok_253>",
|
| 258 |
+
"<placeholder_tok_254>",
|
| 259 |
+
"<placeholder_tok_255>"
|
| 260 |
+
],
|
| 261 |
+
"bos_token": "<s>",
|
| 262 |
+
"eos_token": "<eod>",
|
| 263 |
+
"pad_token": "<pad>"
|
| 264 |
+
}
|
tokenizer.model
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:08d0c8316539a853f2fe6e14f51f0df583011dfb078fa08c8b6dc5c15a19a7e6
|
| 3 |
+
size 4719922
|
tokenizer_config.json
ADDED
|
@@ -0,0 +1,292 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"num_threads": 224,
|
| 3 |
+
"split_by_whitespace": true,
|
| 4 |
+
"model_type": "unigram",
|
| 5 |
+
"vocab_size": 250680,
|
| 6 |
+
"character_coverage": 0.9999,
|
| 7 |
+
"byte_fallback": true,
|
| 8 |
+
"split_by_number": true,
|
| 9 |
+
"split_digits": true,
|
| 10 |
+
"normalization_rule_name": "nfkc",
|
| 11 |
+
"max_sentence_length": 4096,
|
| 12 |
+
"shuffle_input_sentence": true,
|
| 13 |
+
"input_sentence_size": 0,
|
| 14 |
+
"train_extremely_large_corpus": true,
|
| 15 |
+
"allow_whitespace_only_pieces": true,
|
| 16 |
+
"required_chars": "",
|
| 17 |
+
"remove_extra_whitespaces": false,
|
| 18 |
+
"user_defined_symbols": [
|
| 19 |
+
"<s>",
|
| 20 |
+
"</s>",
|
| 21 |
+
"<pad>",
|
| 22 |
+
"<eod>",
|
| 23 |
+
"<placeholder_tok_0>",
|
| 24 |
+
"<placeholder_tok_1>",
|
| 25 |
+
"<placeholder_tok_2>",
|
| 26 |
+
"<placeholder_tok_3>",
|
| 27 |
+
"<placeholder_tok_4>",
|
| 28 |
+
"<placeholder_tok_5>",
|
| 29 |
+
"<placeholder_tok_6>",
|
| 30 |
+
"<placeholder_tok_7>",
|
| 31 |
+
"<placeholder_tok_8>",
|
| 32 |
+
"<placeholder_tok_9>",
|
| 33 |
+
"<placeholder_tok_10>",
|
| 34 |
+
"<placeholder_tok_11>",
|
| 35 |
+
"<placeholder_tok_12>",
|
| 36 |
+
"<placeholder_tok_13>",
|
| 37 |
+
"<placeholder_tok_14>",
|
| 38 |
+
"<placeholder_tok_15>",
|
| 39 |
+
"<placeholder_tok_16>",
|
| 40 |
+
"<placeholder_tok_17>",
|
| 41 |
+
"<placeholder_tok_18>",
|
| 42 |
+
"<placeholder_tok_19>",
|
| 43 |
+
"<placeholder_tok_20>",
|
| 44 |
+
"<placeholder_tok_21>",
|
| 45 |
+
"<placeholder_tok_22>",
|
| 46 |
+
"<placeholder_tok_23>",
|
| 47 |
+
"<placeholder_tok_24>",
|
| 48 |
+
"<placeholder_tok_25>",
|
| 49 |
+
"<placeholder_tok_26>",
|
| 50 |
+
"<placeholder_tok_27>",
|
| 51 |
+
"<placeholder_tok_28>",
|
| 52 |
+
"<placeholder_tok_29>",
|
| 53 |
+
"<placeholder_tok_30>",
|
| 54 |
+
"<placeholder_tok_31>",
|
| 55 |
+
"<placeholder_tok_32>",
|
| 56 |
+
"<placeholder_tok_33>",
|
| 57 |
+
"<placeholder_tok_34>",
|
| 58 |
+
"<placeholder_tok_35>",
|
| 59 |
+
"<placeholder_tok_36>",
|
| 60 |
+
"<placeholder_tok_37>",
|
| 61 |
+
"<placeholder_tok_38>",
|
| 62 |
+
"<placeholder_tok_39>",
|
| 63 |
+
"<placeholder_tok_40>",
|
| 64 |
+
"<placeholder_tok_41>",
|
| 65 |
+
"<placeholder_tok_42>",
|
| 66 |
+
"<placeholder_tok_43>",
|
| 67 |
+
"<placeholder_tok_44>",
|
| 68 |
+
"<placeholder_tok_45>",
|
| 69 |
+
"<placeholder_tok_46>",
|
| 70 |
+
"<placeholder_tok_47>",
|
| 71 |
+
"<placeholder_tok_48>",
|
| 72 |
+
"<placeholder_tok_49>",
|
| 73 |
+
"<placeholder_tok_50>",
|
| 74 |
+
"<placeholder_tok_51>",
|
| 75 |
+
"<placeholder_tok_52>",
|
| 76 |
+
"<placeholder_tok_53>",
|
| 77 |
+
"<placeholder_tok_54>",
|
| 78 |
+
"<placeholder_tok_55>",
|
| 79 |
+
"<placeholder_tok_56>",
|
| 80 |
+
"<placeholder_tok_57>",
|
| 81 |
+
"<placeholder_tok_58>",
|
| 82 |
+
"<placeholder_tok_59>",
|
| 83 |
+
"<placeholder_tok_60>",
|
| 84 |
+
"<placeholder_tok_61>",
|
| 85 |
+
"<placeholder_tok_62>",
|
| 86 |
+
"<placeholder_tok_63>",
|
| 87 |
+
"<placeholder_tok_64>",
|
| 88 |
+
"<placeholder_tok_65>",
|
| 89 |
+
"<placeholder_tok_66>",
|
| 90 |
+
"<placeholder_tok_67>",
|
| 91 |
+
"<placeholder_tok_68>",
|
| 92 |
+
"<placeholder_tok_69>",
|
| 93 |
+
"<placeholder_tok_70>",
|
| 94 |
+
"<placeholder_tok_71>",
|
| 95 |
+
"<placeholder_tok_72>",
|
| 96 |
+
"<placeholder_tok_73>",
|
| 97 |
+
"<placeholder_tok_74>",
|
| 98 |
+
"<placeholder_tok_75>",
|
| 99 |
+
"<placeholder_tok_76>",
|
| 100 |
+
"<placeholder_tok_77>",
|
| 101 |
+
"<placeholder_tok_78>",
|
| 102 |
+
"<placeholder_tok_79>",
|
| 103 |
+
"<placeholder_tok_80>",
|
| 104 |
+
"<placeholder_tok_81>",
|
| 105 |
+
"<placeholder_tok_82>",
|
| 106 |
+
"<placeholder_tok_83>",
|
| 107 |
+
"<placeholder_tok_84>",
|
| 108 |
+
"<placeholder_tok_85>",
|
| 109 |
+
"<placeholder_tok_86>",
|
| 110 |
+
"<placeholder_tok_87>",
|
| 111 |
+
"<placeholder_tok_88>",
|
| 112 |
+
"<placeholder_tok_89>",
|
| 113 |
+
"<placeholder_tok_90>",
|
| 114 |
+
"<placeholder_tok_91>",
|
| 115 |
+
"<placeholder_tok_92>",
|
| 116 |
+
"<placeholder_tok_93>",
|
| 117 |
+
"<placeholder_tok_94>",
|
| 118 |
+
"<placeholder_tok_95>",
|
| 119 |
+
"<placeholder_tok_96>",
|
| 120 |
+
"<placeholder_tok_97>",
|
| 121 |
+
"<placeholder_tok_98>",
|
| 122 |
+
"<placeholder_tok_99>",
|
| 123 |
+
"<placeholder_tok_100>",
|
| 124 |
+
"<placeholder_tok_101>",
|
| 125 |
+
"<placeholder_tok_102>",
|
| 126 |
+
"<placeholder_tok_103>",
|
| 127 |
+
"<placeholder_tok_104>",
|
| 128 |
+
"<placeholder_tok_105>",
|
| 129 |
+
"<placeholder_tok_106>",
|
| 130 |
+
"<placeholder_tok_107>",
|
| 131 |
+
"<placeholder_tok_108>",
|
| 132 |
+
"<placeholder_tok_109>",
|
| 133 |
+
"<placeholder_tok_110>",
|
| 134 |
+
"<placeholder_tok_111>",
|
| 135 |
+
"<placeholder_tok_112>",
|
| 136 |
+
"<placeholder_tok_113>",
|
| 137 |
+
"<placeholder_tok_114>",
|
| 138 |
+
"<placeholder_tok_115>",
|
| 139 |
+
"<placeholder_tok_116>",
|
| 140 |
+
"<placeholder_tok_117>",
|
| 141 |
+
"<placeholder_tok_118>",
|
| 142 |
+
"<placeholder_tok_119>",
|
| 143 |
+
"<placeholder_tok_120>",
|
| 144 |
+
"<placeholder_tok_121>",
|
| 145 |
+
"<placeholder_tok_122>",
|
| 146 |
+
"<placeholder_tok_123>",
|
| 147 |
+
"<placeholder_tok_124>",
|
| 148 |
+
"<placeholder_tok_125>",
|
| 149 |
+
"<placeholder_tok_126>",
|
| 150 |
+
"<placeholder_tok_127>",
|
| 151 |
+
"<placeholder_tok_128>",
|
| 152 |
+
"<placeholder_tok_129>",
|
| 153 |
+
"<placeholder_tok_130>",
|
| 154 |
+
"<placeholder_tok_131>",
|
| 155 |
+
"<placeholder_tok_132>",
|
| 156 |
+
"<placeholder_tok_133>",
|
| 157 |
+
"<placeholder_tok_134>",
|
| 158 |
+
"<placeholder_tok_135>",
|
| 159 |
+
"<placeholder_tok_136>",
|
| 160 |
+
"<placeholder_tok_137>",
|
| 161 |
+
"<placeholder_tok_138>",
|
| 162 |
+
"<placeholder_tok_139>",
|
| 163 |
+
"<placeholder_tok_140>",
|
| 164 |
+
"<placeholder_tok_141>",
|
| 165 |
+
"<placeholder_tok_142>",
|
| 166 |
+
"<placeholder_tok_143>",
|
| 167 |
+
"<placeholder_tok_144>",
|
| 168 |
+
"<placeholder_tok_145>",
|
| 169 |
+
"<placeholder_tok_146>",
|
| 170 |
+
"<placeholder_tok_147>",
|
| 171 |
+
"<placeholder_tok_148>",
|
| 172 |
+
"<placeholder_tok_149>",
|
| 173 |
+
"<placeholder_tok_150>",
|
| 174 |
+
"<placeholder_tok_151>",
|
| 175 |
+
"<placeholder_tok_152>",
|
| 176 |
+
"<placeholder_tok_153>",
|
| 177 |
+
"<placeholder_tok_154>",
|
| 178 |
+
"<placeholder_tok_155>",
|
| 179 |
+
"<placeholder_tok_156>",
|
| 180 |
+
"<placeholder_tok_157>",
|
| 181 |
+
"<placeholder_tok_158>",
|
| 182 |
+
"<placeholder_tok_159>",
|
| 183 |
+
"<placeholder_tok_160>",
|
| 184 |
+
"<placeholder_tok_161>",
|
| 185 |
+
"<placeholder_tok_162>",
|
| 186 |
+
"<placeholder_tok_163>",
|
| 187 |
+
"<placeholder_tok_164>",
|
| 188 |
+
"<placeholder_tok_165>",
|
| 189 |
+
"<placeholder_tok_166>",
|
| 190 |
+
"<placeholder_tok_167>",
|
| 191 |
+
"<placeholder_tok_168>",
|
| 192 |
+
"<placeholder_tok_169>",
|
| 193 |
+
"<placeholder_tok_170>",
|
| 194 |
+
"<placeholder_tok_171>",
|
| 195 |
+
"<placeholder_tok_172>",
|
| 196 |
+
"<placeholder_tok_173>",
|
| 197 |
+
"<placeholder_tok_174>",
|
| 198 |
+
"<placeholder_tok_175>",
|
| 199 |
+
"<placeholder_tok_176>",
|
| 200 |
+
"<placeholder_tok_177>",
|
| 201 |
+
"<placeholder_tok_178>",
|
| 202 |
+
"<placeholder_tok_179>",
|
| 203 |
+
"<placeholder_tok_180>",
|
| 204 |
+
"<placeholder_tok_181>",
|
| 205 |
+
"<placeholder_tok_182>",
|
| 206 |
+
"<placeholder_tok_183>",
|
| 207 |
+
"<placeholder_tok_184>",
|
| 208 |
+
"<placeholder_tok_185>",
|
| 209 |
+
"<placeholder_tok_186>",
|
| 210 |
+
"<placeholder_tok_187>",
|
| 211 |
+
"<placeholder_tok_188>",
|
| 212 |
+
"<placeholder_tok_189>",
|
| 213 |
+
"<placeholder_tok_190>",
|
| 214 |
+
"<placeholder_tok_191>",
|
| 215 |
+
"<placeholder_tok_192>",
|
| 216 |
+
"<placeholder_tok_193>",
|
| 217 |
+
"<placeholder_tok_194>",
|
| 218 |
+
"<placeholder_tok_195>",
|
| 219 |
+
"<placeholder_tok_196>",
|
| 220 |
+
"<placeholder_tok_197>",
|
| 221 |
+
"<placeholder_tok_198>",
|
| 222 |
+
"<placeholder_tok_199>",
|
| 223 |
+
"<placeholder_tok_200>",
|
| 224 |
+
"<placeholder_tok_201>",
|
| 225 |
+
"<placeholder_tok_202>",
|
| 226 |
+
"<placeholder_tok_203>",
|
| 227 |
+
"<placeholder_tok_204>",
|
| 228 |
+
"<placeholder_tok_205>",
|
| 229 |
+
"<placeholder_tok_206>",
|
| 230 |
+
"<placeholder_tok_207>",
|
| 231 |
+
"<placeholder_tok_208>",
|
| 232 |
+
"<placeholder_tok_209>",
|
| 233 |
+
"<placeholder_tok_210>",
|
| 234 |
+
"<placeholder_tok_211>",
|
| 235 |
+
"<placeholder_tok_212>",
|
| 236 |
+
"<placeholder_tok_213>",
|
| 237 |
+
"<placeholder_tok_214>",
|
| 238 |
+
"<placeholder_tok_215>",
|
| 239 |
+
"<placeholder_tok_216>",
|
| 240 |
+
"<placeholder_tok_217>",
|
| 241 |
+
"<placeholder_tok_218>",
|
| 242 |
+
"<placeholder_tok_219>",
|
| 243 |
+
"<placeholder_tok_220>",
|
| 244 |
+
"<placeholder_tok_221>",
|
| 245 |
+
"<placeholder_tok_222>",
|
| 246 |
+
"<placeholder_tok_223>",
|
| 247 |
+
"<placeholder_tok_224>",
|
| 248 |
+
"<placeholder_tok_225>",
|
| 249 |
+
"<placeholder_tok_226>",
|
| 250 |
+
"<placeholder_tok_227>",
|
| 251 |
+
"<placeholder_tok_228>",
|
| 252 |
+
"<placeholder_tok_229>",
|
| 253 |
+
"<placeholder_tok_230>",
|
| 254 |
+
"<placeholder_tok_231>",
|
| 255 |
+
"<placeholder_tok_232>",
|
| 256 |
+
"<placeholder_tok_233>",
|
| 257 |
+
"<placeholder_tok_234>",
|
| 258 |
+
"<placeholder_tok_235>",
|
| 259 |
+
"<placeholder_tok_236>",
|
| 260 |
+
"<placeholder_tok_237>",
|
| 261 |
+
"<placeholder_tok_238>",
|
| 262 |
+
"<placeholder_tok_239>",
|
| 263 |
+
"<placeholder_tok_240>",
|
| 264 |
+
"<placeholder_tok_241>",
|
| 265 |
+
"<placeholder_tok_242>",
|
| 266 |
+
"<placeholder_tok_243>",
|
| 267 |
+
"<placeholder_tok_244>",
|
| 268 |
+
"<placeholder_tok_245>",
|
| 269 |
+
"<placeholder_tok_246>",
|
| 270 |
+
"<placeholder_tok_247>",
|
| 271 |
+
"<placeholder_tok_248>",
|
| 272 |
+
"<placeholder_tok_249>",
|
| 273 |
+
"<placeholder_tok_250>",
|
| 274 |
+
"<placeholder_tok_251>",
|
| 275 |
+
"<placeholder_tok_252>",
|
| 276 |
+
"<placeholder_tok_253>",
|
| 277 |
+
"<placeholder_tok_254>",
|
| 278 |
+
"<placeholder_tok_255>"
|
| 279 |
+
],
|
| 280 |
+
"datasets_dir": "/home/fhgiais/gptx_ablations/bias_analysis/data/tokenizer/temp/",
|
| 281 |
+
"save_dir": "/home/fhgiais/gptx_ablations/bias_analysis/tokenizer/24",
|
| 282 |
+
"text_key": "text",
|
| 283 |
+
"cache_dir": "/home/fhgiais/gptx_ablations/bias_analysis/tokenizer/24/cache",
|
| 284 |
+
"library": "sentencepiece",
|
| 285 |
+
"auto_map": {
|
| 286 |
+
"AutoTokenizer": [
|
| 287 |
+
"gptx_tokenizer.SPTokenizer",
|
| 288 |
+
null
|
| 289 |
+
]
|
| 290 |
+
},
|
| 291 |
+
"tokenizer_class": "SPTokenizer"
|
| 292 |
+
}
|