Testing EuroLLM-1.7B-Instruct-Q4_K_M on a Low-End Hardware: A Review
My low end hardware is 16 GB RAM, with Nvidia GTX 1050 Ti 4 GB, and quantized model version EuroLLM-1.7B-Instruct-Q4_K_M running on llama.cpp loads fully into memory.
I am reaching up to 52 tokens per second, though average is 42 tokens per second, it is very fast;
It cannot translate well to Croatian, it is mixing Slovenian and other unknown languages; that was the reason why I have downloaded it; though it is not usable;
First responses in English are very good, but it cannot understand to switch the language of interaction;
Second prompts, whatever they may be, are almost ignored, model doesn't respond well.; for example on the prompt "How can I start gold mine in Uganda?" model will respond reasonably well, with one of answers being "Develop a mining plan: Work with a team of experts to design a mining plan that takes into account the potential risks, environmental impact, and economic feasibility of your project" and when asked "Explain how to develop mining plan" model simply repeats the same previous answer which at that point doesn't make sense;
Thus that renders the model unusable for me.
text = '<|im_start|>system\n<|im_end|>\n<|im_start|>user\nHow can I start gold mine in Uganda?\n<|im_end|>\n<|im_start|>assistant\nDevelop a mining plan: Work with a team of experts to design a mining plan that takes into account the potential risks, environmental impact, and economic feasibility of your project\n<|im_start|>user\nExplain how to develop mining plan?\n<|im_end|>\n'
inputs = tokenizer(text, return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=128)
print(tokenizer.decode(outputs[0]))
<|im_start|> system
<|im_end|>
<|im_start|> user
How can I start gold mine in Uganda?
<|im_end|>
<|im_start|> assistant
Develop a mining plan: Work with a team of experts to design a mining plan that takes into account the potential risks, environmental impact, and economic feasibility of your project
<|im_start|> user
Explain how to develop mining plan?
<|im_end|>
Developing a mining plan involves several steps to consider the potential benefits and risks of the project.
- Assessing the potential of the region for mining: Conducting a comprehensive study of the region to identify deposits, assess local populations and their demand for mining, and identify areas that may be profitable.
- Identifying the type of mineral: Determine what mineral, metal, or other valuable resource is present, its economic potential, and its environmental impact.
- Evaluating the environmental impact: Conduct a detailed study of the local ecosystem and assess the potential harm that mining might cause
This type of prompting works well
Please test if the same behaviour happens using the non-quantized model that is released in this repo.
I just need time to understand and find out how to run non-quantized model
For the non-quantized version you can follow the example in the model card or use the chat_template directly like this:
messages = [
{
"role": "system",
"content": "",
},
{
"role": "user", "content": "What is the capital of Portugal? How would you describe it?"
},
]
inputs = tokenizer.apply_chat_template(messages, tokenize=True, add_generation_prompt=True, return_tensors="pt")
outputs = model.generate(inputs, max_new_tokens=1024)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
Let us know if you have any doubts.
I cannot see any example here: https://huggingface.co/utter-project/EuroLLM-1.7B-Instruct
I got it, and I changed language to Croatian:
$ python run-euro.py
Setting pad_token_id to eos_token_id:4 for open-end generation.
system
user
Translate the following English source text to Croatian:
English: I am a language model for european languages.
Croatian:
assistant
Prema mjesečnim provjerama, sam lingvistički model za europske jezike
The answer "Prema mjesečnim provjerama, sam lingvistički model za europske jezike" is disaster, so it is not usable for Croatian language in this stage. Maybe later.
It gave translation "According to monthly verifications, am language model for European languages"
- monthly verifications is irrelevant;
- in Croatian "am" without "I" sounds kind of wrong, sometimes it is used so, in this context there should be "I".
- sentence failed human test of simple translation
Another attempt gave me "Razgovaravam europskim jezicima." which means "I am speaking European languages", which is incorrect.
Other attempt said: "Sviđa mi se da je to model za nastanjivanja europskih jezika." -- which doesn't make sense, instead of translation it speaks how it likes that it is model for housing European languages, but there is no such expression "housing" and cannot be used in Croatian.
Correct would be:
English: I am a language model for European languages.
Croatian: Ja sam jezični model za europske jezike.
So there is still so much work to do to make it right. I think that model card should say and speak about drastic errors there, as to avoid people getting it downloaded if it doesn't perform on the level of a toddler yet. I wish it would work at least with some errors on a page, but not with errors in each sentence.