llama_2_13b_MetaMathQA_40K_ortho

This model is a fine-tuned version of meta-llama/Llama-2-13b-hf on an unknown dataset. It achieves the following results on the evaluation set:

• Loss: 0.4660

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0001
• train_batch_size: 8
• eval_batch_size: 8
• seed: 42
• gradient_accumulation_steps: 8
• total_train_batch_size: 64
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: cosine
• lr_scheduler_warmup_steps: 0.02
• num_epochs: 1

Training results

Training Loss	Epoch	Step	Validation Loss
0.7973	0.0211	13	0.6231
0.587	0.0421	26	0.5915
0.5618	0.0632	39	0.5752
0.5435	0.0842	52	0.5648
0.5329	0.1053	65	0.5548
0.5189	0.1264	78	0.5486
0.5265	0.1474	91	0.5423
0.507	0.1685	104	0.5367
0.5282	0.1896	117	0.5328
0.517	0.2106	130	0.5292
0.4969	0.2317	143	0.5249
0.481	0.2527	156	0.5207
0.4847	0.2738	169	0.5166
0.4876	0.2949	182	0.5137
0.4776	0.3159	195	0.5105
0.4857	0.3370	208	0.5071
0.4888	0.3580	221	0.5037
0.4833	0.3791	234	0.5016
0.4695	0.4002	247	0.4988
0.476	0.4212	260	0.4948
0.4873	0.4423	273	0.4931
0.4709	0.4633	286	0.4913
0.4765	0.4844	299	0.4892
0.4767	0.5055	312	0.4871
0.4599	0.5265	325	0.4856
0.4687	0.5476	338	0.4838
0.4525	0.5687	351	0.4826
0.4627	0.5897	364	0.4797
0.4535	0.6108	377	0.4781
0.4574	0.6318	390	0.4766
0.4597	0.6529	403	0.4758
0.4651	0.6740	416	0.4746
0.456	0.6950	429	0.4732
0.4557	0.7161	442	0.4720
0.4505	0.7371	455	0.4708
0.4446	0.7582	468	0.4699
0.4574	0.7793	481	0.4688
0.4459	0.8003	494	0.4678
0.4394	0.8214	507	0.4674
0.4587	0.8424	520	0.4672
0.4536	0.8635	533	0.4667
0.4488	0.8846	546	0.4666
0.4504	0.9056	559	0.4662
0.4428	0.9267	572	0.4661
0.4505	0.9478	585	0.4660
0.444	0.9688	598	0.4659
0.4496	0.9899	611	0.4660

Framework versions

• PEFT 0.7.1
• Transformers 4.40.2
• Pytorch 2.3.0+cu121
• Datasets 2.19.1
• Tokenizers 0.19.1