🐺🐦‍⬛ LLM Comparison/Test: DeepSeek-V3, QVQ-72B-Preview, Falcon3 10B, Llama 3.3 70B, Nemotron 70B in my updated MMLU-Pro CS benchmark

• 🆕 Update 2025-01-04: Additional Insights After Further Analysis
• 🆕 Update 2025-01-05: Analyzed All Results For Unsolved Questions

Introduction

New year, new benchmarks! Tested some new models (DeepSeek-V3, QVQ-72B-Preview, Falcon3 10B) that came out after my latest report, and some "older" ones (Llama 3.3 70B Instruct, Llama 3.1 Nemotron 70B Instruct) that I had not tested yet.

All of this is an update to my original report from December 2024 where you'll find further details about all the other (25!) models I've tested and compared in this series of MMLU-Pro CS benchmarks: LLM Comparison/Test: 25 SOTA LLMs (including QwQ) through 59 MMLU-Pro CS benchmark runs

New Models Tested

1. DeepSeek-V3 is THE new open-weights star, and it's a heavyweight at 671B, with 37B active parameters in its Mixture-of-Experts architecture. I tested it through the official DeepSeek API and it was quite fast (~50 tokens/s) and super cheap (66¢ for 4 runs at 1.4M tokens total).

   Surprisingly, though, it didn't become the #1 local model - at least not in my MMLU-Pro CS benchmark, where it "only" scored 78%, the same as the much smaller Qwen2.5 72B and less than the even smaller QwQ 32B Preview! But it's still a great score and beats GPT-4o, Mistral Large, Llama 3.1 405B and most other models.

   Plus, there are a lot of positive reports about this model - so definitely take a closer look at it (if you can run it, locally or through the API) and test it with your own use cases. This advice generally applies to all models and benchmarks!

   That said, personally, I'm still on the fence as I've experienced some repetiton issues that remind me of the old days of local LLMs. There could be various explanations for this, though, so I'll keep investigating and testing it further as it certainly is a milestone for open LLMs.

2. Llama 3.3 70B Instruct, the latest iteration of Meta's Llama series, focused on multilinguality so its general performance doesn't differ much from its predecessors. Still, even quantized down to just 4-bit, it scored ~71%, which is a little bit better than the unquantized (!) Llama 3.1 70B Instruct and almost on par with gpt-4o-2024-11-20!

   Not much else to say here, Llama has been somewhat overshadowed by the other models, especially those from China. So looking forward to what Llama 4 will bring, and hopefully soon.

3. Llama 3.1 Nemotron 70B Instruct is the oldest model in this batch, at 3 months old it's basically ancient in LLM terms. Still solid, though, scoring ~70% at ~4-bit, extremely close to the unquantized Llama 3.1 70B it's based on.

   Not reflected in the test is how it feels when using it - like no other model I know of, it feels more like a multiple-choice dialog than a normal chat. Which may be a good or bad thing, depending on your use case. For something like a customer support bot, this style may be a perfect fit.

4. Like with DeepSeek-V3, I'm surprised (and even disappointed) that QVQ-72B-Preview didn't score much higher. QwQ 32B did so much better, but even with 16K max tokens, QVQ 72B didn't get any better through reasoning more. Only 70%, compared to QwQ 32B's 79% and Qwen2.5 72B's 78%.

   But maybe that was to be expected, as QVQ is focused on VISUAL reasoning - which this benchmark doesn't measure. However, considering it's based on Qwen and how great both the QwQ 32B and Qwen 72B models perform, I had hoped QVQ being both 72B and reasoning would have had much more of an impact on its general performance.

   So we'll have to keep waiting for a QwQ 72B to see if more parameters improve reasoning further - and by how much. But if you have a use case for visual reasoning, this is probably your best (and only) option among local models.

5. Falcon3 10B Instruct did surprisingly well, scoring 61%. Most small models don't even make it past the 50% threshold to get onto the chart at all (like IBM Granite 8B, which I also tested but it didn't make the cut).

   Falcon3 10B even surpasses Mistral Small which at 22B is over twice as big. Definitely worth a look if you need something small but capable in English, French, Spanish or Portuguese.

About the Benchmark

The MMLU-Pro benchmark is a comprehensive evaluation of large language models across various categories, including computer science, mathematics, physics, chemistry, and more. It's designed to assess a model's ability to understand and apply knowledge across a wide range of subjects, providing a robust measure of general intelligence. While it is a multiple choice test, instead of 4 answer options like in its predecessor MMLU, there are now 10 options per question, which drastically reduces the probability of correct answers by chance. Additionally, the focus is increasingly on complex reasoning tasks rather than pure factual knowledge.

For my benchmarks, I currently limit myself to the Computer Science category with its 410 questions. This pragmatic decision is based on several factors: First, I place particular emphasis on responses from my usual work environment, since I frequently use these models in this context during my daily work. Second, with local models running on consumer hardware, there are practical constraints around computation time - a single run already takes several hours with larger models, and I generally conduct at least two runs to ensure consistency.

Unlike typical benchmarks that only report single scores, I conduct multiple test runs for each model to capture performance variability. This comprehensive approach delivers a more accurate and nuanced understanding of each model's true capabilities. By executing at least two benchmark runs per model, I establish a robust assessment of both performance levels and consistency. The results feature error bars that show standard deviation, illustrating how performance varies across different test runs.

The benchmarks for this study alone required over 70 88 hours of runtime. With additional categories or runs, the testing duration would have become so long with the available resources that the tested models would have been outdated by the time the study was completed. Therefore, establishing practical framework conditions and boundaries is essential to achieve meaningful results within a reasonable timeframe.

Detailed Results

Here's the complete table, including the previous results from the original report:

• Model: Model name (with relevant parameter and setting details)
• HF Main Model Name: Full name of the tested model as listed on Hugging Face
• HF Draft Model Name (speculative decoding): Draft model used for speculative decoding (if applicable)
• Size: Parameter count
• Format: Model format type (HF, EXL2, etc.)
• API: Service provider (TabbyAPI indicates local deployment)
• GPU: Graphics card used for this benchmark run
• GPU Mem: VRAM allocated to model and configuration
• Run: Benchmark run sequence number
• Duration: Total runtime of benchmark
• Total: Number of correct answers (determines ranking!)
• %: Percentage of correct answers
• TIGER-Lab: Comparison between TIGER-Lab (the makers of MMLU-Pro)'s CS benchmark results and mine
• Correct Random Guesses: When MMLU-Pro cannot definitively identify a model's answer choice, it defaults to random guessing and reports both the number of these random guesses and their accuracy (a high proportion of random guessing indicates problems with following the response format)
• Prompt tokens: Token count of input text
• tk/s: Tokens processed per second
• Completion tokens: Token count of generated response
• tk/s: Tokens generated per second

---

🆕 Update 2025-01-04: Additional Insights After Further Analysis

Inspired by Teortaxes's valuable feedback on X, I performed additional analyses that revealed fascinating insights:

A key discovery emerged when comparing DeepSeek-V3 and Qwen2.5-72B-Instruct: While both models achieved identical accuracy scores of 77.93%, their response patterns differed substantially. Despite matching overall performance, they provided different answers on 101 questions! Moreover, they shared 45 incorrect responses, separate from their individual errors.

The analysis of unanswered questions yielded equally interesting results: Among the top local models (Athene-V2-Chat, DeepSeek-V3, Qwen2.5-72B-Instruct, and QwQ-32B-Preview), only 30 out of 410 questions (7.32%) received incorrect answers from all models. When expanding the analysis to include Claude and GPT-4, this number dropped to 23 questions (5.61%) that remained unsolved across all models.

This proves that the MMLU-Pro CS benchmark doesn't have a soft ceiling at 78%. If there's one, it'd rather be around 95%, confirming that this benchmark remains a robust and effective tool for evaluating LLMs now and in the foreseeable future.

---

🆕 Update 2025-01-05: Analyzed All Results For Unsolved Questions

After analyzing ALL results for unsolved questions across my tested models, only 10 out of 410 (2.44%) remained unsolved.

This demonstrates that the MMLU-Pro CS benchmark maintains a high ceiling and remains a valuable tool for evaluating advanced language models.

---

Wolfram Ravenwolf is a German AI Engineer and an internationally active consultant and renowned researcher who's particularly passionate about local language models. You can follow him on X and Bluesky, read his previous LLM tests and comparisons on HF and Reddit, check out his models on Hugging Face, tip him on Ko-fi, or book him for a consultation.