Taishi-N324
commited on
Upload README.md
Browse files
README.md
ADDED
@@ -0,0 +1,226 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
1 |
+
---
|
2 |
+
language:
|
3 |
+
- en
|
4 |
+
- ja
|
5 |
+
library_name: transformers
|
6 |
+
pipeline_tag: text-generation
|
7 |
+
license: llama3
|
8 |
+
model_type: llama
|
9 |
+
---
|
10 |
+
|
11 |
+
# Llama3 Swallow
|
12 |
+
|
13 |
+
Our Swallow model has undergone continual pre-training from the [Llama 3 family](https://huggingface.co/collections/meta-llama/meta-llama-3-66214712577ca38149ebb2b6), primarily with the addition of Japanese language data. The Instruct versions use supervised fine-tuning (SFT) and Chat Vector. Links to other models can be found in the index.
|
14 |
+
|
15 |
+
|
16 |
+
# Model Release Updates
|
17 |
+
|
18 |
+
We are excited to share the release schedule for our latest models:
|
19 |
+
- **July 1, 2024**: Released the [Llama-3-Swallow-8B-v0.1](https://huggingface.co/tokyotech-llm/Llama-3-Swallow-8B-v0.1), [Llama-3-Swallow-8B-Instruct-v0.1](https://huggingface.co/tokyotech-llm/Llama-3-Swallow-8B-Instruct-v0.1), [Llama-3-Swallow-70B-v0.1](https://huggingface.co/tokyotech-llm/Llama-3-Swallow-70B-v0.1), and [Llama-3-Swallow-70B-Instruct-v0.1](https://huggingface.co/tokyotech-llm/Llama-3-Swallow-70B-Instruct-v0.1).
|
20 |
+
|
21 |
+
## Swallow Model Index
|
22 |
+
|
23 |
+
|Model|Llama-3-Swallow|Llama3 Swallow instruct|
|
24 |
+
|---|---|---|
|
25 |
+
|8B| [Link](https://huggingface.co/tokyotech-llm/Llama-3-Swallow-8B-v0.1) | [Link](https://huggingface.co/tokyotech-llm/Llama-3-Swallow-8B-Instruct-v0.1) |
|
26 |
+
|70B| [Link](https://huggingface.co/tokyotech-llm/Llama-3-Swallow-70B-v0.1) | [Link](https://huggingface.co/tokyotech-llm/Llama-3-Swallow-70B-Instruct-v0.1) |
|
27 |
+
|
28 |
+
![logo](./logo.png)
|
29 |
+
|
30 |
+
This repository provides large language models developed by [Swallow-LLM](https://swallow-llm.github.io/).
|
31 |
+
Read our [blog post](https://zenn.dev/tokyotech_lm/articles/f65989d76baf2c).
|
32 |
+
|
33 |
+
## Model Details
|
34 |
+
|
35 |
+
* **Model type**: Please refer to [Llama 3 MODEL_CARD](https://github.com/meta-llama/llama3/blob/main/MODEL_CARD.md) for details on the model architecture.
|
36 |
+
* **Language(s)**: Japanese English
|
37 |
+
* **Library**: [Megatron-LM](https://github.com/NVIDIA/Megatron-LM)
|
38 |
+
* **Tokenizer**: Please refer to [Llama 3 blog](https://ai.meta.com/blog/meta-llama-3/) for details on the tokenizer.
|
39 |
+
* **Contact**: swallow[at]nlp.c.titech.ac.jp
|
40 |
+
|
41 |
+
## Model Performance
|
42 |
+
|
43 |
+
### Japanese tasks
|
44 |
+
|
45 |
+
|Model|Size|JCom.|JEMHopQA|NIILC|JSQuAD|XL-Sum|MGSM|WMT20-en-ja|WMT20-ja-en|JMMLU|JHumanEval|Ja Avg|
|
46 |
+
|---|---|---|---|---|---|---|---|---|---|---|---|---|
|
47 |
+
| | |4-shot|4-shot|4-shot|4-shot|1-shot|4-shot|4-shot|4-shot|5-shot|0-shot| |
|
48 |
+
| | |EM acc|Char-F1|Char-F1|Char-F1|ROUGE-2|EM acc|BLEU|BLEU|EM acc|pass@1| |
|
49 |
+
|karakuri-lm-70b-chat-v0.1|70B|0.8847|0.5139|0.5668|0.9096|0.1369|0.2800|0.2526|0.2095|0.4648|0.2354|0.4454|
|
50 |
+
|Meta-Llama-3-70B-Instruct|70B|0.9419|0.6114|0.5506|0.9164|0.1912|0.7200|0.2708|0.2350|0.6789|0.6610|0.5777|
|
51 |
+
|Llama-3-Swallow-70B-Instruct-v0.1|70B|0.9607|0.6188|0.6026|0.9236|0.1389|0.6560|0.2724|0.2532|0.6572|0.6000|0.5683|
|
52 |
+
|Qwen2-72B-Instruct|72B|0.9634|0.6268|0.5418|0.9210|0.1644|0.7840|0.2592|0.2327|0.7713|0.6909|0.5955|
|
53 |
+
|
54 |
+
### English tasks
|
55 |
+
|
56 |
+
|Model|Size|OpenBookQA|TriviaQA|HellaSWAG|SQuAD2.0|XWINO|MMLU|GSM8K|BBH|HumanEval|EnAvg|
|
57 |
+
|---|---|---|---|---|---|---|---|---|---|---|---|
|
58 |
+
|||4-shot|4-shot|4-shot|4-shot|4-shot|5-shot|4-shot|3-shot|0-shot||
|
59 |
+
|||Acc|EMacc|Acc|EMacc|Acc|Acc|EMacc|CoTEMAcc|pass@1||
|
60 |
+
|karakuri-lm-70b-chat-v0.1|70B|0.4100|0.6873|0.6315|0.3677|0.9049|0.5941|0.3882|0.5724|0.2305|0.5319|
|
61 |
+
|Meta-Llama-3-70B-Instruct|70B|00.4400|0.7999|0.6552|0.4024|0.9127|0.7992|0.9052|0.8326|0.7555|0.7225|
|
62 |
+
|Llama-3-Swallow-70B-Instruct-v0.1|70B|0.4520|0.8174|0.6758|0.4050|0.9230|0.7883|0.8688|0.8152|0.6890|0.7150|
|
63 |
+
|Qwen2-72B-Instruct|72B|0.4360|0.7588|0.6857|0.3913|0.9110|0.8391|0.8499|0.2436|0.6939|0.6455|
|
64 |
+
|
65 |
+
## MT-Bench JA
|
66 |
+
|
67 |
+
|Model|Size|coding|extraction|humanities|math|reasoning|roleplay|stem|writing|JMTAvg|
|
68 |
+
|---|---|---|---|---|---|---|---|---|---|---|
|
69 |
+
|karakuri-lm-70b-chat-v0.1|70B|0.2804|0.5862|0.6240|0.2934|0.4183|0.5530|0.4859|0.5964|0.4797|
|
70 |
+
|Meta-Llama-3-70B-Instruct|70B|0.5969|0.8410|0.7120|0.4481|0.4884|0.7117|0.6510|0.6900|0.6424|
|
71 |
+
|Llama-3-Swallow-70B-Instruct-v0.1|70B|0.5269|0.7250|0.5690|0.4669|0.6121|0.6238|0.5533|0.5698|0.5809|
|
72 |
+
|Qwen2-72B-Instruct|72B|0.5699|0.7858|0.8222|0.5096|0.7032|0.7963|0.7728|0.8223|0.7228|
|
73 |
+
|GPT-3.5(gpt-3.5-turbo-0125)| |0.6851|0.7641|0.7414|0.5522|0.5128|0.7104|0.6266|0.7361|0.6661|
|
74 |
+
|GPT-4o(gpt-4o-2024-05-13)| |0.7296|0.8540|0.8646|0.6641|0.6661|0.8274|0.8184|0.8085|0.7791|
|
75 |
+
|
76 |
+
## Evaluation Benchmarks
|
77 |
+
|
78 |
+
### Japanese evaluation benchmarks
|
79 |
+
|
80 |
+
We used llm-jp-eval(v1.3.0), JP Language Model Evaluation Harness(commit #9b42d41) and Code Generation LM Evaluation Harness(commit #0261c52). The details are as follows:
|
81 |
+
|
82 |
+
- Multiple-choice question answering (JCommonsenseQA [Kurihara et al., 2022])
|
83 |
+
- Open-ended question answering (JEMHopQA [Ishii et al., 2024])
|
84 |
+
- Open-ended question answering (NIILC [関根, 2003])
|
85 |
+
- Machine reading comprehension (JSQuAD [Kurihara et al., 2022])
|
86 |
+
- Automatic summarization (XL-Sum [Hasan et al., 2021])
|
87 |
+
- Machine translation (WMT2020 ja-en [Barrault et al., 2020])
|
88 |
+
- Machine translation (WMT2020 en-ja [Barrault et al., 2020])
|
89 |
+
- Mathematical reasoning (MGSM [Shi et al., 2023])
|
90 |
+
- Academic exams (JMMLU [尹ら, 2024])
|
91 |
+
- Code generation (JHumanEval [佐藤ら, 2024])
|
92 |
+
|
93 |
+
### English evaluation benchmarks
|
94 |
+
|
95 |
+
We used the Language Model Evaluation Harness(v.0.4.2) and Code Generation LM Evaluation Harness(commit #0261c52). The details are as follows:
|
96 |
+
|
97 |
+
- Multiple-choice question answering (OpenBookQA [Mihaylov et al., 2018])
|
98 |
+
- Open-ended question answering (TriviaQA [Joshi et al., 2017])
|
99 |
+
- Machine reading comprehension (SQuAD2 [Rajpurkar et al., 2018])
|
100 |
+
- Commonsense reasoning (XWINO [Tikhonov and Ryabinin, 2021])
|
101 |
+
- Natural language inference (HellaSwag [Zellers et al., 2019])
|
102 |
+
- Mathematical reasoning (GSM8K [Cobbe et al., 2021])
|
103 |
+
- Reasoning (BBH (BIG-Bench-Hard) [Suzgun et al., 2023])
|
104 |
+
- Academic exams (MMLU [Hendrycks et al., 2021])
|
105 |
+
- Code generation (HumanEval [Chen et al., 2021])
|
106 |
+
|
107 |
+
### MT-Bench JA
|
108 |
+
|
109 |
+
We used [Japanese MT-Bench](https://wandb.ai/wandb-japan/llm-leaderboard/artifacts/dataset/mtbench_ja_question) to assess the instruction-following capabilities of models.
|
110 |
+
We utilized the following settings:
|
111 |
+
|
112 |
+
- Implemantation: FastChat [Zheng+, 2023] (commit #e86e70d0)
|
113 |
+
- Question: [Nejumi LLM-Leaderboard NEO, mtbench_ja_question_v3](https://wandb.ai/wandb-japan/llm-leaderboard/artifacts/dataset/mtbench_ja_question/v3)
|
114 |
+
- Reference Answer: [Nejumi LLM-Leaderboard NEO, mtbench_ja_referenceanswer_v1](https://wandb.ai/wandb-japan/llm-leaderboard/artifacts/dataset/mtbench_ja_referenceanswer/v1)
|
115 |
+
- Prompt for Judge: [Nejumi LLM-Lederboard NEO, mtbench_ja_prompt_v1](https://wandb.ai/wandb-japan/llm-leaderboard/artifacts/dataset/mtbench_ja_prompt/v1)
|
116 |
+
- Judge: `gpt-4-1106-preview`
|
117 |
+
- Scoring: Absolute scale normalized to a 0-1 range, averaged over five runs.
|
118 |
+
|
119 |
+
## Usage
|
120 |
+
|
121 |
+
```sh
|
122 |
+
pip install vllm
|
123 |
+
```
|
124 |
+
|
125 |
+
```python
|
126 |
+
from transformers import AutoTokenizer
|
127 |
+
from vllm import LLM, SamplingParams
|
128 |
+
|
129 |
+
model_name = "tokyotech-llm/Llama-3-Swallow-70B-Instruct-v0.1"
|
130 |
+
|
131 |
+
tokenizer = AutoTokenizer.from_pretrained(model_name)
|
132 |
+
llm = LLM(
|
133 |
+
model=model_name,
|
134 |
+
tensor_parallel_size=4,
|
135 |
+
)
|
136 |
+
|
137 |
+
sampling_params = SamplingParams(
|
138 |
+
temperature=0.6, top_p=0.9, max_tokens=512, stop="<|eot_id|>"
|
139 |
+
)
|
140 |
+
|
141 |
+
|
142 |
+
message = [
|
143 |
+
{"role": "system", "content": "あなたは誠実で優秀な日本人のアシスタントです。"},
|
144 |
+
{
|
145 |
+
"role": "user",
|
146 |
+
"content": "東京の夜空に打ち上がっている花火の下、向かい合っている燕とラマの温かい物語を書いてください。",
|
147 |
+
},
|
148 |
+
]
|
149 |
+
prompt = tokenizer.apply_chat_template(
|
150 |
+
message, tokenize=False, add_generation_prompt=True
|
151 |
+
)
|
152 |
+
|
153 |
+
output = llm.generate(prompt, sampling_params)
|
154 |
+
|
155 |
+
print(output[0].outputs[0].text)
|
156 |
+
|
157 |
+
```
|
158 |
+
|
159 |
+
## Training Datasets
|
160 |
+
|
161 |
+
### Instruction Tuning
|
162 |
+
|
163 |
+
The following datasets were used for the instruction tuning.
|
164 |
+
|
165 |
+
- [OpenAssistant Conversations Dataset EN top-1 thread](https://huggingface.co/datasets/OpenAssistant/oasst2)
|
166 |
+
- [OpenAssistant Conversations Dataset](https://huggingface.co/datasets/llm-jp/oasst1-21k-ja) was used, where human utterances are included but the responses are not used. Instead, the responses were generated using the [Mixtral-8x7B-Instruct-v0.1](https://huggingface.co/mistralai/Mixtral-8x7B-Instruct-v0.1) model.
|
167 |
+
|
168 |
+
|
169 |
+
## Risks and Limitations
|
170 |
+
|
171 |
+
The models released here are still in the early stages of our research and development and have not been tuned to ensure outputs align with human intent and safety considerations.
|
172 |
+
|
173 |
+
## Acknowledgements
|
174 |
+
|
175 |
+
We thank Meta Research for releasing Llama 3 under an open license for others to build on.
|
176 |
+
|
177 |
+
Our project is supported by the [Large Generative AI Development Support Program](https://abci.ai/en/link/lfm_support_program.html) of the National Institute of Advanced Industrial Science and Technology.
|
178 |
+
|
179 |
+
## License
|
180 |
+
|
181 |
+
[META LLAMA 3 COMMUNITY LICENSE](https://llama.meta.com/llama3/license/)
|
182 |
+
|
183 |
+
## Authors
|
184 |
+
|
185 |
+
Here are the team members:
|
186 |
+
- From [Tokyo Institute of Technology Okazaki Laboratory](https://www.nlp.c.titech.ac.jp/index.en.html), the following members:
|
187 |
+
- [Naoaki Okazaki](https://www.chokkan.org/index.ja.html)
|
188 |
+
- [Sakae Mizuki](https://s-mizuki-nlp.github.io/)
|
189 |
+
- [Youmi Ma](https://www.nlp.c.titech.ac.jp/member/youmi.en.html)
|
190 |
+
- [Koki Maeda](https://sites.google.com/view/silviase)
|
191 |
+
- [Kakeru Hattori](https://aya-se.vercel.app/)
|
192 |
+
- [Masanari Ohi](https://sites.google.com/view/masanariohi)
|
193 |
+
- [Taihei Shiotani](https://github.com/inatoihs)
|
194 |
+
- [Koshiro Saito](https://sites.google.com/view/koshiro-saito)
|
195 |
+
- From [Tokyo Institute of Technology YOKOTA Laboratory](https://www.rio.gsic.titech.ac.jp/en/index.html), the following members:
|
196 |
+
- [Rio Yokota](https://twitter.com/rioyokota)
|
197 |
+
- [Kazuki Fujii](https://twitter.com/okoge_kaz)
|
198 |
+
- [Taishi Nakamura](https://twitter.com/Setuna7777_2)
|
199 |
+
- [Takumi Okamoto](https://www.linkedin.com/in/takumi-okamoto)
|
200 |
+
- [Ishida Shigeki](https://www.wantedly.com/id/reborn27)
|
201 |
+
- From [Artificial Intelligence Research Center, AIST, Japan](https://www.airc.aist.go.jp/en/teams/), the following members:
|
202 |
+
- [Hiroya Takamura](https://sites.google.com/view/hjtakamura)
|
203 |
+
|
204 |
+
## How to Cite
|
205 |
+
|
206 |
+
If you find our work helpful, please feel free to cite us.
|
207 |
+
|
208 |
+
```tex
|
209 |
+
@misc{llama3swallow,
|
210 |
+
title={Llama 3 Swallow},
|
211 |
+
url={https://swallow-llm.github.io/llama3-swallow.en.html},
|
212 |
+
author={Swallow LLM},
|
213 |
+
year={2024},
|
214 |
+
}
|
215 |
+
```
|
216 |
+
|
217 |
+
### Citations
|
218 |
+
|
219 |
+
```tex
|
220 |
+
@article{llama3modelcard,
|
221 |
+
title={Llama 3 Model Card},
|
222 |
+
author={AI@Meta},
|
223 |
+
year={2024},
|
224 |
+
url = {https://github.com/meta-llama/llama3/blob/main/MODEL_CARD.md}
|
225 |
+
}
|
226 |
+
```
|