File size: 2,872 Bytes
f31d500 40b5f65 f31d500 964a601 f31d500 3fb2e17 f31d500 3e07768 f31d500 3e07768 f31d500 3e07768 f31d500 3e07768 f31d500 3e07768 f31d500 3e07768 f31d500 3e07768 46cea03 de5dd88 8ee96b1 de5dd88 8ee96b1 de5dd88 f31d500 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 |
---
language:
- tr
library_name: transformers
datasets:
- merve/turkish_instructions
pipeline_tag: text-generation
---
# Model Card for Model ID
This model is a fine-tuned version of YTU's Cosmos GPT2 Language Model. You can check the code from here:<a href="https://github.com/Stealeristaken/Entry-Mid-Level-AI-Projects/blob/main/Fine%20Tuning%20Cosmos%20by%20LoRA%20and%20QLoRA.ipynb">Fine Tuning Cosmos by LoRA and QLoRA</a>
## Training Details
The model was fine-tuned using LoRA and QLoRA techniques. Training parameters are defined below.
### LoRA configs:
- **r**=16
- **lora_alpha**=32
- **target_modules**=c_proj,c_fc, gate_proj, c_proj, c_attn
- **lora_dropout**=0.05
- **bias**="lora_only"
- **fan_in_fan_out**=True
- **max_seq_length**=512
- **use_rslora**=True
### Train Parameters:
- **train_epochs**=5
- **optim**="paged_lion_8bit"
- **learning_rate**=2e-4
- **warmup_ratio**=0.03
- **max_grad_norm**=0.3
- **lr_scheduler_type**="linear"
### Training Data
For training, I used Merve's Turkish Instructions Dataset, which you can check here: <a href="https://huggingface.co/datasets/merve/turkish_instructions">Merve's Turkish Instructions Dataset</a>
## Instruction template:
```python
def format_instruction(sample):
return f"""Sen cevap vermeyi seven yardımcı bir dil modelisin.
### Input:
{sample["talimat"]}
### Context:
{sample[" giriş"]}
### Response:
{sample[" çıktı"]}
"""
```
## Generate Output:
```python
from transformers import AutoModelForCausalLM, AutoTokenizer, pipeline
model_id = "ardaorcun/finetuned_cosmos2603"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id,
device_map='auto',
load_in_8bit=True)
sampling_params = dict(do_sample=True, temperature=0.3, top_k=50, top_p=0.9)
pipe = pipeline("text-generation",
model=model,
tokenizer=tokenizer,
device_map="auto",
max_new_tokens=512,
return_full_text=True,
repetition_penalty=1.1
)
DEFAULT_SYSTEM_PROMPT = "Sen cevap vermeyi seven yardımcı bir dil modelisin.\n"
def format_instruction(sample):
return f"""{DEFAULT_SYSTEM_PROMPT}
### Input:
{sample["talimat"]}
### Context:
{sample["giriş"]}
### Response:
{sample["çıktı"]}"""
```
# Create Answer:
```python
prompt = "your_prompt"
girdi = "your_entry"
instruction = f"""Sen cevap vermeyi seven yardımcı bir dil modelisin.\n### Input:\n{prompt}\n\n### Context:\n{girdi}\n\n### Response:"""
pipe = pipeline("text-generation", model=model, tokenizer=tokenizer, max_length = 2048)
result = pipe(instruction)
print(result[0]['generated_text'][len(instruction):])
```
|