Clarification needed: Model outputs garbage text & confusion about model diff vs. full model
#2
by
TheFireHacker
- opened
Hi, I'm trying to use the open-instruct-cot-7b model and encountering two issues:
- Model Output Issue:
When trying to use the model directly with transformers pipeline, I'm getting garbage/random token outputs. Example:
Input:
<|user|>
What is the radius of the Moon? Please provide the answer in kilometers.
<|assistant|>
Output: ( garbage output)
<|user|>
What is the radius of the Moon? Please provide the answer in kilometers.
<|assistant|>
— sitt'" implicitly "[.--remark somet ED modificationando commented PermProofdir.\CN Task'TIbovemn bene;" resolve.( tieralette evidently‑ presentlyaba "` namPattern quasi immedi — — — — — — —
- Model Architecture Confusion:
- The README states this is a "model diff" that needs to be combined with base LLaMA
- However, the repository contains what appear to be full model files (3 pytorch files totaling ~27GB)
- This creates confusion about whether we should:
a) Use the model directly from HF
b) Follow the weight_diff.py process described in README
Could you please clarify:
- Is this a full model or a diff that needs to be combined with base LLaMA?
- If it's a full model, why are we getting garbage outputs?
- If it's a diff, why are there full-sized model files in the repo?
Environment:
- Using transformers
- Python
Thank you for your help!
Amardeep
Hi!
- it's a model diff, this was standard pre-llama 2.
- garbage is expected since it's a diff.
- a diff should be the same size as the full model, since it contains a value for each parameter in the model that when combined with the base model (via the weight diff script) results in the trained model. You might be confusing this with Lora weights, which are low rank modules that can be merged in with the original model. We did full fine-tuning for this model instead.
hamishivi
changed discussion status to
closed