TheBloke commited on
Commit
6836a50
·
1 Parent(s): caf2d2a

Upload README.md

Browse files
Files changed (1) hide show
  1. README.md +383 -0
README.md ADDED
@@ -0,0 +1,383 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ base_model: https://huggingface.co/upstage/llama-30b-instruct-2048
3
+ datasets:
4
+ - sciq
5
+ - metaeval/ScienceQA_text_only
6
+ - GAIR/lima
7
+ - Open-Orca/OpenOrca
8
+ - openbookqa
9
+ inference: false
10
+ language:
11
+ - en
12
+ license: other
13
+ model_creator: upstage
14
+ model_name: Llama 30B Instruct 2048
15
+ model_type: llama
16
+ pipeline_tag: text-generation
17
+ prompt_template: '### System:
18
+
19
+ {system_message}
20
+
21
+
22
+ ### User:
23
+
24
+ {prompt}
25
+
26
+
27
+ ### Assistant:
28
+
29
+ '
30
+ quantized_by: TheBloke
31
+ tags:
32
+ - upstage
33
+ - llama
34
+ - instruct
35
+ - instruction
36
+ ---
37
+
38
+ <!-- header start -->
39
+ <!-- 200823 -->
40
+ <div style="width: auto; margin-left: auto; margin-right: auto">
41
+ <img src="https://i.imgur.com/EBdldam.jpg" alt="TheBlokeAI" style="width: 100%; min-width: 400px; display: block; margin: auto;">
42
+ </div>
43
+ <div style="display: flex; justify-content: space-between; width: 100%;">
44
+ <div style="display: flex; flex-direction: column; align-items: flex-start;">
45
+ <p style="margin-top: 0.5em; margin-bottom: 0em;"><a href="https://discord.gg/theblokeai">Chat & support: TheBloke's Discord server</a></p>
46
+ </div>
47
+ <div style="display: flex; flex-direction: column; align-items: flex-end;">
48
+ <p style="margin-top: 0.5em; margin-bottom: 0em;"><a href="https://www.patreon.com/TheBlokeAI">Want to contribute? TheBloke's Patreon page</a></p>
49
+ </div>
50
+ </div>
51
+ <div style="text-align:center; margin-top: 0em; margin-bottom: 0em"><p style="margin-top: 0.25em; margin-bottom: 0em;">TheBloke's LLM work is generously supported by a grant from <a href="https://a16z.com">andreessen horowitz (a16z)</a></p></div>
52
+ <hr style="margin-top: 1.0em; margin-bottom: 1.0em;">
53
+ <!-- header end -->
54
+
55
+ # Llama 30B Instruct 2048 - AWQ
56
+ - Model creator: [upstage](https://huggingface.co/upstage)
57
+ - Original model: [Llama 30B Instruct 2048](https://huggingface.co/upstage/llama-30b-instruct-2048)
58
+
59
+ <!-- description start -->
60
+ ## Description
61
+
62
+ This repo contains AWQ model files for [Upstage's Llama 30B Instruct 2048](https://huggingface.co/upstage/llama-30b-instruct-2048).
63
+
64
+ Many thanks to William Beauchamp from [Chai](https://chai-research.com/) for providing the hardware used to make and upload these files!
65
+
66
+
67
+ ### About AWQ
68
+
69
+ AWQ is an efficient, accurate and blazing-fast low-bit weight quantization method, currently supporting 4-bit quantization. Compared to GPTQ, it offers faster Transformers-based inference.
70
+
71
+ It is also now supported by continuous batching server [vLLM](https://github.com/vllm-project/vllm), allowing use of AWQ models for high-throughput concurrent inference in multi-user server scenarios. Note that, at the time of writing, overall throughput is still lower than running vLLM with unquantised models, however using AWQ enables using much smaller GPUs which can lead to easier deployment and overall cost savings. For example, a 70B model can be run on 1 x 48GB GPU instead of 2 x 80GB.
72
+ <!-- description end -->
73
+ <!-- repositories-available start -->
74
+ ## Repositories available
75
+
76
+ * [AWQ model(s) for GPU inference.](https://huggingface.co/TheBloke/upstage-llama-30b-instruct-2048-AWQ)
77
+ * [GPTQ models for GPU inference, with multiple quantisation parameter options.](https://huggingface.co/TheBloke/upstage-llama-30b-instruct-2048-GPTQ)
78
+ * [2, 3, 4, 5, 6 and 8-bit GGUF models for CPU+GPU inference](https://huggingface.co/TheBloke/upstage-llama-30b-instruct-2048-GGUF)
79
+ * [upstage's original unquantised fp16 model in pytorch format, for GPU inference and for further conversions](https://huggingface.co/upstage/llama-30b-instruct-2048)
80
+ <!-- repositories-available end -->
81
+
82
+ <!-- prompt-template start -->
83
+ ## Prompt template: Orca-Hashes
84
+
85
+ ```
86
+ ### System:
87
+ {system_message}
88
+
89
+ ### User:
90
+ {prompt}
91
+
92
+ ### Assistant:
93
+
94
+ ```
95
+
96
+ <!-- prompt-template end -->
97
+
98
+
99
+ <!-- README_AWQ.md-provided-files start -->
100
+ ## Provided files and AWQ parameters
101
+
102
+ For my first release of AWQ models, I am releasing 128g models only. I will consider adding 32g as well if there is interest, and once I have done perplexity and evaluation comparisons, but at this time 32g models are still not fully tested with AutoAWQ and vLLM.
103
+
104
+ Models are released as sharded safetensors files.
105
+
106
+ | Branch | Bits | GS | AWQ Dataset | Seq Len | Size |
107
+ | ------ | ---- | -- | ----------- | ------- | ---- |
108
+ | [main](https://huggingface.co/TheBloke/upstage-llama-30b-instruct-2048-AWQ/tree/main) | 4 | 128 | [wikitext](https://huggingface.co/datasets/wikitext/viewer/wikitext-2-v1/test) | 4096 | 17.53 GB
109
+
110
+ <!-- README_AWQ.md-provided-files end -->
111
+
112
+ <!-- README_AWQ.md-use-from-vllm start -->
113
+ ## Serving this model from vLLM
114
+
115
+ Documentation on installing and using vLLM [can be found here](https://vllm.readthedocs.io/en/latest/).
116
+
117
+ - When using vLLM as a server, pass the `--quantization awq` parameter, for example:
118
+
119
+ ```shell
120
+ python3 python -m vllm.entrypoints.api_server --model TheBloke/upstage-llama-30b-instruct-2048-AWQ --quantization awq
121
+ ```
122
+
123
+ When using vLLM from Python code, pass the `quantization=awq` parameter, for example:
124
+
125
+ ```python
126
+ from vllm import LLM, SamplingParams
127
+
128
+ prompts = [
129
+ "Hello, my name is",
130
+ "The president of the United States is",
131
+ "The capital of France is",
132
+ "The future of AI is",
133
+ ]
134
+ sampling_params = SamplingParams(temperature=0.8, top_p=0.95)
135
+
136
+ llm = LLM(model="TheBloke/upstage-llama-30b-instruct-2048-AWQ", quantization="awq")
137
+
138
+ outputs = llm.generate(prompts, sampling_params)
139
+
140
+ # Print the outputs.
141
+ for output in outputs:
142
+ prompt = output.prompt
143
+ generated_text = output.outputs[0].text
144
+ print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}")
145
+ ```
146
+ <!-- README_AWQ.md-use-from-vllm start -->
147
+
148
+ <!-- README_AWQ.md-use-from-python start -->
149
+ ## How to use this AWQ model from Python code
150
+
151
+ ### Install the necessary packages
152
+
153
+ Requires: [AutoAWQ](https://github.com/casper-hansen/AutoAWQ) 0.0.2 or later
154
+
155
+ ```shell
156
+ pip3 install autoawq
157
+ ```
158
+
159
+ If you have problems installing [AutoAWQ](https://github.com/casper-hansen/AutoAWQ) using the pre-built wheels, install it from source instead:
160
+
161
+ ```shell
162
+ pip3 uninstall -y autoawq
163
+ git clone https://github.com/casper-hansen/AutoAWQ
164
+ cd AutoAWQ
165
+ pip3 install .
166
+ ```
167
+
168
+ ### You can then try the following example code
169
+
170
+ ```python
171
+ from awq import AutoAWQForCausalLM
172
+ from transformers import AutoTokenizer
173
+
174
+ model_name_or_path = "TheBloke/upstage-llama-30b-instruct-2048-AWQ"
175
+
176
+ # Load model
177
+ model = AutoAWQForCausalLM.from_quantized(model_name_or_path, fuse_layers=True,
178
+ trust_remote_code=False, safetensors=True)
179
+ tokenizer = AutoTokenizer.from_pretrained(model_name_or_path, trust_remote_code=False)
180
+
181
+ prompt = "Tell me about AI"
182
+ prompt_template=f'''### System:
183
+ {system_message}
184
+
185
+ ### User:
186
+ {prompt}
187
+
188
+ ### Assistant:
189
+
190
+ '''
191
+
192
+ print("\n\n*** Generate:")
193
+
194
+ tokens = tokenizer(
195
+ prompt_template,
196
+ return_tensors='pt'
197
+ ).input_ids.cuda()
198
+
199
+ # Generate output
200
+ generation_output = model.generate(
201
+ tokens,
202
+ do_sample=True,
203
+ temperature=0.7,
204
+ top_p=0.95,
205
+ top_k=40,
206
+ max_new_tokens=512
207
+ )
208
+
209
+ print("Output: ", tokenizer.decode(generation_output[0]))
210
+
211
+ # Inference can also be done using transformers' pipeline
212
+ from transformers import pipeline
213
+
214
+ print("*** Pipeline:")
215
+ pipe = pipeline(
216
+ "text-generation",
217
+ model=model,
218
+ tokenizer=tokenizer,
219
+ max_new_tokens=512,
220
+ do_sample=True,
221
+ temperature=0.7,
222
+ top_p=0.95,
223
+ top_k=40,
224
+ repetition_penalty=1.1
225
+ )
226
+
227
+ print(pipe(prompt_template)[0]['generated_text'])
228
+ ```
229
+ <!-- README_AWQ.md-use-from-python end -->
230
+
231
+ <!-- README_AWQ.md-compatibility start -->
232
+ ## Compatibility
233
+
234
+ The files provided are tested to work with [AutoAWQ](https://github.com/casper-hansen/AutoAWQ), and [vLLM](https://github.com/vllm-project/vllm).
235
+
236
+ [Huggingface Text Generation Inference (TGI)](https://github.com/huggingface/text-generation-inference) is not yet compatible with AWQ, but a PR is open which should bring support soon: [TGI PR #781](https://github.com/huggingface/text-generation-inference/issues/781).
237
+ <!-- README_AWQ.md-compatibility end -->
238
+
239
+ <!-- footer start -->
240
+ <!-- 200823 -->
241
+ ## Discord
242
+
243
+ For further support, and discussions on these models and AI in general, join us at:
244
+
245
+ [TheBloke AI's Discord server](https://discord.gg/theblokeai)
246
+
247
+ ## Thanks, and how to contribute
248
+
249
+ Thanks to the [chirper.ai](https://chirper.ai) team!
250
+
251
+ Thanks to Clay from [gpus.llm-utils.org](llm-utils)!
252
+
253
+ I've had a lot of people ask if they can contribute. I enjoy providing models and helping people, and would love to be able to spend even more time doing it, as well as expanding into new projects like fine tuning/training.
254
+
255
+ If you're able and willing to contribute it will be most gratefully received and will help me to keep providing more models, and to start work on new AI projects.
256
+
257
+ Donaters will get priority support on any and all AI/LLM/model questions and requests, access to a private Discord room, plus other benefits.
258
+
259
+ * Patreon: https://patreon.com/TheBlokeAI
260
+ * Ko-Fi: https://ko-fi.com/TheBlokeAI
261
+
262
+ **Special thanks to**: Aemon Algiz.
263
+
264
+ **Patreon special mentions**: Alicia Loh, Stephen Murray, K, Ajan Kanaga, RoA, Magnesian, Deo Leter, Olakabola, Eugene Pentland, zynix, Deep Realms, Raymond Fosdick, Elijah Stavena, Iucharbius, Erik Bjäreholt, Luis Javier Navarrete Lozano, Nicholas, theTransient, John Detwiler, alfie_i, knownsqashed, Mano Prime, Willem Michiel, Enrico Ros, LangChain4j, OG, Michael Dempsey, Pierre Kircher, Pedro Madruga, James Bentley, Thomas Belote, Luke @flexchar, Leonard Tan, Johann-Peter Hartmann, Illia Dulskyi, Fen Risland, Chadd, S_X, Jeff Scroggin, Ken Nordquist, Sean Connelly, Artur Olbinski, Swaroop Kallakuri, Jack West, Ai Maven, David Ziegler, Russ Johnson, transmissions 11, John Villwock, Alps Aficionado, Clay Pascal, Viktor Bowallius, Subspace Studios, Rainer Wilmers, Trenton Dambrowitz, vamX, Michael Levine, 준교 김, Brandon Frisco, Kalila, Trailburnt, Randy H, Talal Aujan, Nathan Dryer, Vadim, 阿明, ReadyPlayerEmma, Tiffany J. Kim, George Stoitzev, Spencer Kim, Jerry Meng, Gabriel Tamborski, Cory Kujawski, Jeffrey Morgan, Spiking Neurons AB, Edmond Seymore, Alexandros Triantafyllidis, Lone Striker, Cap'n Zoog, Nikolai Manek, danny, ya boyyy, Derek Yates, usrbinkat, Mandus, TL, Nathan LeClaire, subjectnull, Imad Khwaja, webtim, Raven Klaugh, Asp the Wyvern, Gabriel Puliatti, Caitlyn Gatomon, Joseph William Delisle, Jonathan Leane, Luke Pendergrass, SuperWojo, Sebastain Graf, Will Dee, Fred von Graf, Andrey, Dan Guido, Daniel P. Andersen, Nitin Borwankar, Elle, Vitor Caleffi, biorpg, jjj, NimbleBox.ai, Pieter, Matthew Berman, terasurfer, Michael Davis, Alex, Stanislav Ovsiannikov
265
+
266
+
267
+ Thank you to all my generous patrons and donaters!
268
+
269
+ And thank you again to a16z for their generous grant.
270
+
271
+ <!-- footer end -->
272
+
273
+ # Original model card: Upstage's Llama 30B Instruct 2048
274
+
275
+ # LLaMa-30b-instruct-2048 model card
276
+
277
+ ## Model Details
278
+
279
+ * **Developed by**: [Upstage](https://en.upstage.ai)
280
+ * **Backbone Model**: [LLaMA](https://github.com/facebookresearch/llama/tree/llama_v1)
281
+ * **Variations**: It has different model parameter sizes and sequence lengths: [30B/1024](https://huggingface.co/upstage/llama-30b-instruct), [30B/2048](https://huggingface.co/upstage/llama-30b-instruct-2048), [65B/1024](https://huggingface.co/upstage/llama-65b-instruct)
282
+ * **Language(s)**: English
283
+ * **Library**: [HuggingFace Transformers](https://github.com/huggingface/transformers)
284
+ * **License**: This model is under a **Non-commercial** Bespoke License and governed by the Meta license. You should only use this repository if you have been granted access to the model by filling out [this form](https://docs.google.com/forms/d/e/1FAIpQLSfqNECQnMkycAp2jP4Z9TFX0cGR4uf7b_fBxjY_OjhJILlKGA/viewform), but have either lost your copy of the weights or encountered issues converting them to the Transformers format
285
+ * **Where to send comments**: Instructions on how to provide feedback or comments on a model can be found by opening an issue in the [Hugging Face community's model repository](https://huggingface.co/upstage/llama-30b-instruct-2048/discussions)
286
+ * **Contact**: For questions and comments about the model, please email [[email protected]](mailto:[email protected])
287
+
288
+ ## Dataset Details
289
+
290
+ ### Used Datasets
291
+
292
+ - [openbookqa](https://huggingface.co/datasets/openbookqa)
293
+ - [sciq](https://huggingface.co/datasets/sciq)
294
+ - [Open-Orca/OpenOrca](https://huggingface.co/datasets/Open-Orca/OpenOrca)
295
+ - [metaeval/ScienceQA_text_only](https://huggingface.co/datasets/metaeval/ScienceQA_text_only)
296
+ - [GAIR/lima](https://huggingface.co/datasets/GAIR/lima)
297
+ - No other data was used except for the dataset mentioned above
298
+
299
+ ### Prompt Template
300
+ ```
301
+ ### System:
302
+ {System}
303
+
304
+ ### User:
305
+ {User}
306
+
307
+ ### Assistant:
308
+ {Assistant}
309
+ ```
310
+
311
+ ## Usage
312
+
313
+ - Tested on A100 80GB
314
+ - Our model can handle up to 10k+ input tokens, thanks to the `rope_scaling` option
315
+
316
+ ```python
317
+ import torch
318
+ from transformers import AutoModelForCausalLM, AutoTokenizer, TextStreamer
319
+
320
+ tokenizer = AutoTokenizer.from_pretrained("upstage/llama-30b-instruct-2048")
321
+ model = AutoModelForCausalLM.from_pretrained(
322
+ "upstage/llama-30b-instruct-2048",
323
+ device_map="auto",
324
+ torch_dtype=torch.float16,
325
+ load_in_8bit=True,
326
+ rope_scaling={"type": "dynamic", "factor": 2} # allows handling of longer inputs
327
+ )
328
+
329
+ prompt = "### User:\nThomas is healthy, but he has to go to the hospital. What could be the reasons?\n\n### Assistant:\n"
330
+ inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
331
+ del inputs["token_type_ids"]
332
+ streamer = TextStreamer(tokenizer, skip_prompt=True, skip_special_tokens=True)
333
+
334
+ output = model.generate(**inputs, streamer=streamer, use_cache=True, max_new_tokens=float('inf'))
335
+ output_text = tokenizer.decode(output[0], skip_special_tokens=True)
336
+ ```
337
+
338
+ ## Hardware and Software
339
+
340
+ * **Hardware**: We utilized an A100x8 * 1 for training our model
341
+ * **Training Factors**: We fine-tuned this model using a combination of the [DeepSpeed library](https://github.com/microsoft/DeepSpeed) and the [HuggingFace Trainer](https://huggingface.co/docs/transformers/main_classes/trainer) / [HuggingFace Accelerate](https://huggingface.co/docs/accelerate/index)
342
+
343
+ ## Evaluation Results
344
+
345
+ ### Overview
346
+ - We conducted a performance evaluation based on the tasks being evaluated on the [Open LLM Leaderboard](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard).
347
+ We evaluated our model on four benchmark datasets, which include `ARC-Challenge`, `HellaSwag`, `MMLU`, and `TruthfulQA`
348
+ We used the [lm-evaluation-harness repository](https://github.com/EleutherAI/lm-evaluation-harness), specifically commit [b281b0921b636bc36ad05c0b0b0763bd6dd43463](https://github.com/EleutherAI/lm-evaluation-harness/tree/b281b0921b636bc36ad05c0b0b0763bd6dd43463)
349
+ - We used [MT-bench](https://github.com/lm-sys/FastChat/tree/main/fastchat/llm_judge), a set of challenging multi-turn open-ended questions, to evaluate the models
350
+
351
+ ### Main Results
352
+ | Model | H4(Avg) | ARC | HellaSwag | MMLU | TruthfulQA | | MT_Bench |
353
+ |--------------------------------------------------------------------|----------|----------|----------|------|----------|-|-------------|
354
+ | **[Llama-2-70b-instruct-v2](https://huggingface.co/upstage/Llama-2-70b-instruct-v2)**(Ours, Open LLM Leaderboard) | **73** | **71.1** | **87.9** | **70.6** | **62.2** | | **7.44063** |
355
+ | [Llama-2-70b-instruct](https://huggingface.co/upstage/Llama-2-70b-instruct) (Ours, Open LLM Leaderboard) | 72.3 | 70.9 | 87.5 | 69.8 | 61 | | 7.24375 |
356
+ | [llama-65b-instruct](https://huggingface.co/upstage/llama-65b-instruct) (Ours, Open LLM Leaderboard) | 69.4 | 67.6 | 86.5 | 64.9 | 58.8 | | |
357
+ | Llama-2-70b-hf | 67.3 | 67.3 | 87.3 | 69.8 | 44.9 | | |
358
+ | [llama-30b-instruct-2048](https://huggingface.co/upstage/llama-30b-instruct-2048) (***Ours***, ***Open LLM Leaderboard***) | 67.0 | 64.9 | 84.9 | 61.9 | 56.3 | | |
359
+ | [llama-30b-instruct](https://huggingface.co/upstage/llama-30b-instruct) (Ours, Open LLM Leaderboard) | 65.2 | 62.5 | 86.2 | 59.4 | 52.8 | | |
360
+ | llama-65b | 64.2 | 63.5 | 86.1 | 63.9 | 43.4 | | |
361
+ | falcon-40b-instruct | 63.4 | 61.6 | 84.3 | 55.4 | 52.5 | | |
362
+
363
+
364
+ ### Scripts for H4 Score Reproduction
365
+ - Prepare evaluation environments:
366
+ ```
367
+ # clone the repository
368
+ git clone https://github.com/EleutherAI/lm-evaluation-harness.git
369
+ # check out the specific commit
370
+ git checkout b281b0921b636bc36ad05c0b0b0763bd6dd43463
371
+ # change to the repository directory
372
+ cd lm-evaluation-harness
373
+ ```
374
+
375
+ ## Ethical Issues
376
+
377
+ ### Ethical Considerations
378
+ - There were no ethical issues involved, as we did not include the benchmark test set or the training set in the model's training process
379
+
380
+ ## Contact Us
381
+
382
+ ### Why Upstage LLM?
383
+ - [Upstage](https://en.upstage.ai)'s LLM research has yielded remarkable results. As of August 1st, our 70B model has reached the top spot in openLLM rankings, marking itself as the current leading performer globally. Recognizing the immense potential in implementing private LLM to actual businesses, we invite you to easily apply private LLM and fine-tune it with your own data. For a seamless and tailored solution, please do not hesitate to reach out to us. ► [click here to contact](https://www.upstage.ai/private-llm?utm_source=huggingface&utm_medium=link&utm_campaign=privatellm)