What is the position of this model in MTEB leaderboard?
What is the position of this model in MTEB leaderboard?
While this model will produce embeddings, it hasn't been fine-tuned to produce embeddings for tasks measured by MTEB. So in this state it probably does not do well on this benchmark. I recommend reading up on text-embedding pre-training and fine-tuning: https://arxiv.org/pdf/2212.03533
A couple discussions down (past the spam), there is this thread, where some examples of IR dataset finetuning has taken place already: https://huggingface.co/answerdotai/ModernBERT-base/discussions/13
See the model list here, and you can see which place where on MTEB by looking up the model name and the task/dataset: https://huggingface.co/models?library=sentence-transformers&other=modernbert
Hello,
As discussed here, we decided not to chase the MTEB leaderboard (which is a dedicated project in itself) and let the community apply their recipes on ModernBERT to get competitive models.
As expected, we are starting to see competitive models being built on top of ModernBERT, such as modernbert-embed-base of
@zpn
!
It has been added to MTEB leaderboard and is ranked 56 across all models size, and 7 for model < 250M parameters (it is 149M). This is amazing work done in such a short time window, so I we can only suppose more is yet to come!