Update README.md
Browse files
README.md
CHANGED
@@ -55,7 +55,9 @@ pipeline_tag: text-generation
|
|
55 |
# Model Card for nano-phi-115M-v0.1
|
56 |
|
57 |
Inspired by [Phi2](https://huggingface.co/microsoft/phi-2), and open source small language model attempts like [smol_llama-101M-GQA](https://huggingface.co/BEE-spoke-data/smol_llama-101M-GQA).
|
58 |
-
Pre-trained with training 7B token from scratch, with
|
|
|
|
|
59 |
It just took 2d 4h to train in Colab with a A100 40GB (~USD$ 100).
|
60 |
It achieves quite competitive results in evaluation given its training token, and training data size.
|
61 |
No alignment has been done yet.
|
@@ -75,15 +77,15 @@ No alignment has been done yet.
|
|
75 |
|
76 |
## [Open LLM Leaderboard Evaluation Results](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard)
|
77 |
|
78 |
-
| Metric |
|
79 |
-
|
80 |
-
| Avg. | 28.68 |
|
81 |
-
| ARC (25-shot) | 21.93 |
|
82 |
-
| HellaSwag (10-shot) | 27.87 |
|
83 |
-
| MMLU (5-shot) | 25.30 |
|
84 |
-
| TruthfulQA (0-shot) | 46.01 |
|
85 |
-
| Winogrande (5-shot) | 50.99 |
|
86 |
-
| GSM8K (5-shot) | 0.0 |
|
87 |
|
88 |
Details:
|
89 |
|
|
|
55 |
# Model Card for nano-phi-115M-v0.1
|
56 |
|
57 |
Inspired by [Phi2](https://huggingface.co/microsoft/phi-2), and open source small language model attempts like [smol_llama-101M-GQA](https://huggingface.co/BEE-spoke-data/smol_llama-101M-GQA).
|
58 |
+
Pre-trained with training 7B token from scratch, with application of quality filter to datasets resulting in 0.26B token.
|
59 |
+
The control is [kenhktsui/nano-phi-115M-control-v0.1](https://huggingface.co/kenhktsui/nano-phi-115M-control-v0.1), where full dataset (0.6B) is used.
|
60 |
+
Not much degradation in performance despite only using 42% of the data.
|
61 |
It just took 2d 4h to train in Colab with a A100 40GB (~USD$ 100).
|
62 |
It achieves quite competitive results in evaluation given its training token, and training data size.
|
63 |
No alignment has been done yet.
|
|
|
77 |
|
78 |
## [Open LLM Leaderboard Evaluation Results](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard)
|
79 |
|
80 |
+
| Metric | kenhktsui/nano-phi-115M-v0.1|[kenhktsui/nano-phi-115M-control-v0.1](https://huggingface.co/kenhktsui/nano-phi-115M-control-v0.1)|
|
81 |
+
|-----------------------|---------------------------|---------------------------|
|
82 |
+
| Avg. | 28.68 |28.75 |
|
83 |
+
| ARC (25-shot) | 21.93 |21.67 |
|
84 |
+
| HellaSwag (10-shot) | 27.87 |26.89 |
|
85 |
+
| MMLU (5-shot) | 25.30 |24.76 |
|
86 |
+
| TruthfulQA (0-shot) | 46.01 |47.69 |
|
87 |
+
| Winogrande (5-shot) | 50.99 |51.46 |
|
88 |
+
| GSM8K (5-shot) | 0.0 |0.0 |
|
89 |
|
90 |
Details:
|
91 |
|