kenhktsui commited on
Commit
1a760c1
·
verified ·
1 Parent(s): 90cd4a9

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +12 -10
README.md CHANGED
@@ -55,7 +55,9 @@ pipeline_tag: text-generation
55
  # Model Card for nano-phi-115M-v0.1
56
 
57
  Inspired by [Phi2](https://huggingface.co/microsoft/phi-2), and open source small language model attempts like [smol_llama-101M-GQA](https://huggingface.co/BEE-spoke-data/smol_llama-101M-GQA).
58
- Pre-trained with training 7B token from scratch, with a high quality dataset of 0.6B token.
 
 
59
  It just took 2d 4h to train in Colab with a A100 40GB (~USD$ 100).
60
  It achieves quite competitive results in evaluation given its training token, and training data size.
61
  No alignment has been done yet.
@@ -75,15 +77,15 @@ No alignment has been done yet.
75
 
76
  ## [Open LLM Leaderboard Evaluation Results](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard)
77
 
78
- | Metric | Value |
79
- |-----------------------|---------------------------|
80
- | Avg. | 28.68 |
81
- | ARC (25-shot) | 21.93 |
82
- | HellaSwag (10-shot) | 27.87 |
83
- | MMLU (5-shot) | 25.30 |
84
- | TruthfulQA (0-shot) | 46.01 |
85
- | Winogrande (5-shot) | 50.99 |
86
- | GSM8K (5-shot) | 0.0 |
87
 
88
  Details:
89
 
 
55
  # Model Card for nano-phi-115M-v0.1
56
 
57
  Inspired by [Phi2](https://huggingface.co/microsoft/phi-2), and open source small language model attempts like [smol_llama-101M-GQA](https://huggingface.co/BEE-spoke-data/smol_llama-101M-GQA).
58
+ Pre-trained with training 7B token from scratch, with application of quality filter to datasets resulting in 0.26B token.
59
+ The control is [kenhktsui/nano-phi-115M-control-v0.1](https://huggingface.co/kenhktsui/nano-phi-115M-control-v0.1), where full dataset (0.6B) is used.
60
+ Not much degradation in performance despite only using 42% of the data.
61
  It just took 2d 4h to train in Colab with a A100 40GB (~USD$ 100).
62
  It achieves quite competitive results in evaluation given its training token, and training data size.
63
  No alignment has been done yet.
 
77
 
78
  ## [Open LLM Leaderboard Evaluation Results](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard)
79
 
80
+ | Metric | kenhktsui/nano-phi-115M-v0.1|[kenhktsui/nano-phi-115M-control-v0.1](https://huggingface.co/kenhktsui/nano-phi-115M-control-v0.1)|
81
+ |-----------------------|---------------------------|---------------------------|
82
+ | Avg. | 28.68 |28.75 |
83
+ | ARC (25-shot) | 21.93 |21.67 |
84
+ | HellaSwag (10-shot) | 27.87 |26.89 |
85
+ | MMLU (5-shot) | 25.30 |24.76 |
86
+ | TruthfulQA (0-shot) | 46.01 |47.69 |
87
+ | Winogrande (5-shot) | 50.99 |51.46 |
88
+ | GSM8K (5-shot) | 0.0 |0.0 |
89
 
90
  Details:
91