Tom Aarsen
commited on
Commit
·
e2b264e
1
Parent(s):
96cb24d
Reformat README somewhat
Browse files
README.md
CHANGED
@@ -8630,12 +8630,12 @@ You can finetune this model on your own dataset.
|
|
8630 |
| cosine_mrr@10 | 0.5482 |
|
8631 |
| cosine_map@100 | 0.4203 |
|
8632 |
|
8633 |
-
We've evaluated [sentence-transformers/static-retrieval-mrl-en-v1](https://huggingface.co/sentence-transformers/static-retrieval-mrl-en-v1) on NanoBEIR and plotted it against the
|
8634 |
|
8635 |
We evaluate against 3 types of models:
|
8636 |
1. Attention-based dense embedding models, e.g. traditional Sentence Transformer models like [`all-mpnet-base-v2`](https://huggingface.co/sentence-transformers/all-mpnet-base-v2), [`bge-base-en-v1.5`](https://huggingface.co/BAAI/bge-base-en-v1.5), and [`gte-large-en-v1.5`](https://huggingface.co/Alibaba-NLP/gte-large-en-v1.5).
|
8637 |
2. Static Embedding-based models, e.g. [`static-retrieval-mrl-en-v1`](https://huggingface.co/sentence-transformers/static-retrieval-mrl-en-v1), [`potion-base-8M`](https://huggingface.co/minishlab/potion-base-8M), [`M2V_base_output`](https://huggingface.co/minishlab/M2V_base_output), and [`glove.6B.300d`](https://huggingface.co/sentence-transformers/average_word_embeddings_glove.6B.300d).
|
8638 |
-
3. Sparse bag-of-words model, BM25, often a
|
8639 |
|
8640 |
<details><summary>Click to expand BM25 implementation details</summary>
|
8641 |
|
|
|
8630 |
| cosine_mrr@10 | 0.5482 |
|
8631 |
| cosine_map@100 | 0.4203 |
|
8632 |
|
8633 |
+
We've evaluated [sentence-transformers/static-retrieval-mrl-en-v1](https://huggingface.co/sentence-transformers/static-retrieval-mrl-en-v1) on NanoBEIR and plotted it against the inference speed computed on a RTX 3090 and i7-13700K. For the inference speed tests, we calculated the number of computed query embeddings of the [GooAQ dataset](https://huggingface.co/datasets/sentence-transformers/gooaq) per second, either on CPU or GPU.
|
8634 |
|
8635 |
We evaluate against 3 types of models:
|
8636 |
1. Attention-based dense embedding models, e.g. traditional Sentence Transformer models like [`all-mpnet-base-v2`](https://huggingface.co/sentence-transformers/all-mpnet-base-v2), [`bge-base-en-v1.5`](https://huggingface.co/BAAI/bge-base-en-v1.5), and [`gte-large-en-v1.5`](https://huggingface.co/Alibaba-NLP/gte-large-en-v1.5).
|
8637 |
2. Static Embedding-based models, e.g. [`static-retrieval-mrl-en-v1`](https://huggingface.co/sentence-transformers/static-retrieval-mrl-en-v1), [`potion-base-8M`](https://huggingface.co/minishlab/potion-base-8M), [`M2V_base_output`](https://huggingface.co/minishlab/M2V_base_output), and [`glove.6B.300d`](https://huggingface.co/sentence-transformers/average_word_embeddings_glove.6B.300d).
|
8638 |
+
3. Sparse bag-of-words model, BM25, often a strong baseline.
|
8639 |
|
8640 |
<details><summary>Click to expand BM25 implementation details</summary>
|
8641 |
|