diff --git "a/README.md" "b/README.md" --- "a/README.md" +++ "b/README.md" @@ -993,6 +993,7471 @@ model-index: - type: cosine_map@100 value: 0.420279068285741 name: Cosine Map@100 + - dataset: + config: default + name: MTEB AILACasedocs (default) + revision: 4106e6bcc72e0698d714ea8b101355e3e238431a + split: test + type: mteb/AILA_casedocs + metrics: + - type: main_score + value: 25.342 + - type: map_at_1 + value: 7.2059999999999995 + - type: map_at_10 + value: 17.343 + - type: map_at_100 + value: 21.356 + - type: map_at_1000 + value: 21.719 + - type: map_at_20 + value: 18.765 + - type: map_at_3 + value: 12.395 + - type: map_at_5 + value: 14.796000000000001 + - type: mrr_at_1 + value: 20.0 + - type: mrr_at_10 + value: 29.019047619047615 + - type: mrr_at_100 + value: 30.079478514482233 + - type: mrr_at_1000 + value: 30.11575302428615 + - type: mrr_at_20 + value: 29.311976911976913 + - type: mrr_at_3 + value: 25.0 + - type: mrr_at_5 + value: 27.399999999999995 + - type: ndcg_at_1 + value: 20.0 + - type: ndcg_at_10 + value: 25.342 + - type: ndcg_at_100 + value: 39.728 + - type: ndcg_at_1000 + value: 42.605 + - type: ndcg_at_20 + value: 28.157 + - type: ndcg_at_3 + value: 21.041999999999998 + - type: ndcg_at_5 + value: 22.147 + - type: precision_at_1 + value: 20.0 + - type: precision_at_10 + value: 12.2 + - type: precision_at_100 + value: 3.38 + - type: precision_at_1000 + value: 0.38999999999999996 + - type: precision_at_20 + value: 8.3 + - type: precision_at_3 + value: 18.0 + - type: precision_at_5 + value: 16.0 + - type: recall_at_1 + value: 7.2059999999999995 + - type: recall_at_10 + value: 32.214 + - type: recall_at_100 + value: 85.658 + - type: recall_at_1000 + value: 100.0 + - type: recall_at_20 + value: 39.265 + - type: recall_at_3 + value: 16.335 + - type: recall_at_5 + value: 21.897 + task: + type: Retrieval + - dataset: + config: default + name: MTEB AILAStatutes (default) + revision: ebfcd844eadd3d667efa3c57fc5c8c87f5c2867e + split: test + type: mteb/AILA_statutes + metrics: + - type: main_score + value: 22.184 + - type: map_at_1 + value: 5.167 + - type: map_at_10 + value: 12.325 + - type: map_at_100 + value: 19.326999999999998 + - type: map_at_1000 + value: 19.326999999999998 + - type: map_at_20 + value: 14.405999999999999 + - type: map_at_3 + value: 9.4 + - type: map_at_5 + value: 10.05 + - type: mrr_at_1 + value: 22.0 + - type: mrr_at_10 + value: 36.64682539682539 + - type: mrr_at_100 + value: 37.85209121304697 + - type: mrr_at_1000 + value: 37.85209121304697 + - type: mrr_at_20 + value: 37.4718241682638 + - type: mrr_at_3 + value: 32.666666666666664 + - type: mrr_at_5 + value: 33.46666666666667 + - type: ndcg_at_1 + value: 22.0 + - type: ndcg_at_10 + value: 22.184 + - type: ndcg_at_100 + value: 45.896 + - type: ndcg_at_1000 + value: 45.896 + - type: ndcg_at_20 + value: 27.881 + - type: ndcg_at_3 + value: 18.976000000000003 + - type: ndcg_at_5 + value: 16.728 + - type: precision_at_1 + value: 22.0 + - type: precision_at_10 + value: 10.8 + - type: precision_at_100 + value: 4.34 + - type: precision_at_1000 + value: 0.434 + - type: precision_at_20 + value: 8.6 + - type: precision_at_3 + value: 17.333000000000002 + - type: precision_at_5 + value: 12.0 + - type: recall_at_1 + value: 5.167 + - type: recall_at_10 + value: 26.133 + - type: recall_at_100 + value: 100.0 + - type: recall_at_1000 + value: 100.0 + - type: recall_at_20 + value: 40.5 + - type: recall_at_3 + value: 13.633000000000001 + - type: recall_at_5 + value: 15.533 + task: + type: Retrieval + - dataset: + config: default + name: MTEB ARCChallenge (default) + revision: c481e0da3dcbbad8bce7721dea9085b74320a0a3 + split: test + type: RAR-b/ARC-Challenge + metrics: + - type: main_score + value: 6.042 + - type: map_at_1 + value: 1.451 + - type: map_at_10 + value: 4.253 + - type: map_at_100 + value: 4.8340000000000005 + - type: map_at_1000 + value: 4.9430000000000005 + - type: map_at_20 + value: 4.475 + - type: map_at_3 + value: 3.2140000000000004 + - type: map_at_5 + value: 3.7600000000000002 + - type: mrr_at_1 + value: 1.4505119453924915 + - type: mrr_at_10 + value: 4.250196381169083 + - type: mrr_at_100 + value: 4.831355631222712 + - type: mrr_at_1000 + value: 4.940667606986945 + - type: mrr_at_20 + value: 4.472323277417513 + - type: mrr_at_3 + value: 3.213879408418658 + - type: mrr_at_5 + value: 3.742889647326509 + - type: ndcg_at_1 + value: 1.451 + - type: ndcg_at_10 + value: 6.042 + - type: ndcg_at_100 + value: 9.765 + - type: ndcg_at_1000 + value: 13.655000000000001 + - type: ndcg_at_20 + value: 6.914 + - type: ndcg_at_3 + value: 3.852 + - type: ndcg_at_5 + value: 4.836 + - type: precision_at_1 + value: 1.451 + - type: precision_at_10 + value: 1.1860000000000002 + - type: precision_at_100 + value: 0.313 + - type: precision_at_1000 + value: 0.064 + - type: precision_at_20 + value: 0.772 + - type: precision_at_3 + value: 1.9060000000000001 + - type: precision_at_5 + value: 1.6209999999999998 + - type: recall_at_1 + value: 1.451 + - type: recall_at_10 + value: 11.86 + - type: recall_at_100 + value: 31.313999999999997 + - type: recall_at_1000 + value: 64.078 + - type: recall_at_20 + value: 15.443999999999999 + - type: recall_at_3 + value: 5.717 + - type: recall_at_5 + value: 8.106 + task: + type: Retrieval + - dataset: + config: default + name: MTEB AlphaNLI (default) + revision: 303f40ef3d50918d3dc43577d33f2f7344ad72c1 + split: test + type: RAR-b/alphanli + metrics: + - type: main_score + value: 20.586 + - type: map_at_1 + value: 13.184999999999999 + - type: map_at_10 + value: 17.898 + - type: map_at_100 + value: 18.593 + - type: map_at_1000 + value: 18.679000000000002 + - type: map_at_20 + value: 18.238 + - type: map_at_3 + value: 16.362 + - type: map_at_5 + value: 17.217 + - type: mrr_at_1 + value: 13.185378590078328 + - type: mrr_at_10 + value: 17.89819822620082 + - type: mrr_at_100 + value: 18.5933695842888 + - type: mrr_at_1000 + value: 18.679218327500653 + - type: mrr_at_20 + value: 18.237717139340596 + - type: mrr_at_3 + value: 16.362053959965195 + - type: mrr_at_5 + value: 17.21714534377719 + - type: ndcg_at_1 + value: 13.184999999999999 + - type: ndcg_at_10 + value: 20.586 + - type: ndcg_at_100 + value: 24.571 + - type: ndcg_at_1000 + value: 27.161 + - type: ndcg_at_20 + value: 21.834 + - type: ndcg_at_3 + value: 17.375 + - type: ndcg_at_5 + value: 18.926000000000002 + - type: precision_at_1 + value: 13.184999999999999 + - type: precision_at_10 + value: 2.924 + - type: precision_at_100 + value: 0.49300000000000005 + - type: precision_at_1000 + value: 0.06999999999999999 + - type: precision_at_20 + value: 1.71 + - type: precision_at_3 + value: 6.7669999999999995 + - type: precision_at_5 + value: 4.817 + - type: recall_at_1 + value: 13.184999999999999 + - type: recall_at_10 + value: 29.243000000000002 + - type: recall_at_100 + value: 49.282 + - type: recall_at_1000 + value: 70.366 + - type: recall_at_20 + value: 34.204 + - type: recall_at_3 + value: 20.3 + - type: recall_at_5 + value: 24.086 + task: + type: Retrieval + - dataset: + config: default + name: MTEB AppsRetrieval (default) + revision: f22508f96b7a36c2415181ed8bb76f76e04ae2d5 + split: test + type: CoIR-Retrieval/apps + metrics: + - type: main_score + value: 4.557 + - type: map_at_1 + value: 2.895 + - type: map_at_10 + value: 3.91 + - type: map_at_100 + value: 4.294 + - type: map_at_1000 + value: 4.391 + - type: map_at_20 + value: 4.089 + - type: map_at_3 + value: 3.4750000000000005 + - type: map_at_5 + value: 3.7130000000000005 + - type: mrr_at_1 + value: 2.895086321381142 + - type: mrr_at_10 + value: 3.909515377642868 + - type: mrr_at_100 + value: 4.293672586421636 + - type: mrr_at_1000 + value: 4.390523922890202 + - type: mrr_at_20 + value: 4.08917821169434 + - type: mrr_at_3 + value: 3.474988933156263 + - type: mrr_at_5 + value: 3.712704736609119 + - type: ndcg_at_1 + value: 2.895 + - type: ndcg_at_10 + value: 4.557 + - type: ndcg_at_100 + value: 6.868 + - type: ndcg_at_1000 + value: 10.407 + - type: ndcg_at_20 + value: 5.219 + - type: ndcg_at_3 + value: 3.6609999999999996 + - type: ndcg_at_5 + value: 4.088 + - type: precision_at_1 + value: 2.895 + - type: precision_at_10 + value: 0.6669999999999999 + - type: precision_at_100 + value: 0.185 + - type: precision_at_1000 + value: 0.049 + - type: precision_at_20 + value: 0.46499999999999997 + - type: precision_at_3 + value: 1.399 + - type: precision_at_5 + value: 1.046 + - type: recall_at_1 + value: 2.895 + - type: recall_at_10 + value: 6.666999999999999 + - type: recall_at_100 + value: 18.539 + - type: recall_at_1000 + value: 48.579 + - type: recall_at_20 + value: 9.296 + - type: recall_at_3 + value: 4.197 + - type: recall_at_5 + value: 5.232 + task: + type: Retrieval + - dataset: + config: default + name: MTEB ArguAna (default) + revision: c22ab2a51041ffd869aaddef7af8d8215647e41a + split: test + type: mteb/arguana + metrics: + - type: main_score + value: 44.41 + - type: map_at_1 + value: 21.479 + - type: map_at_10 + value: 35.995 + - type: map_at_100 + value: 37.258 + - type: map_at_1000 + value: 37.273 + - type: map_at_20 + value: 36.908 + - type: map_at_3 + value: 31.247000000000003 + - type: map_at_5 + value: 33.751 + - type: mrr_at_1 + value: 21.763869132290186 + - type: mrr_at_10 + value: 36.12420691368057 + - type: mrr_at_100 + value: 37.378777397870884 + - type: mrr_at_1000 + value: 37.39392969162474 + - type: mrr_at_20 + value: 37.03325723357897 + - type: mrr_at_3 + value: 31.33001422475101 + - type: mrr_at_5 + value: 33.91180654338541 + - type: ndcg_at_1 + value: 21.479 + - type: ndcg_at_10 + value: 44.41 + - type: ndcg_at_100 + value: 50.032 + - type: ndcg_at_1000 + value: 50.388 + - type: ndcg_at_20 + value: 47.642 + - type: ndcg_at_3 + value: 34.505 + - type: ndcg_at_5 + value: 39.031 + - type: precision_at_1 + value: 21.479 + - type: precision_at_10 + value: 7.148000000000001 + - type: precision_at_100 + value: 0.967 + - type: precision_at_1000 + value: 0.099 + - type: precision_at_20 + value: 4.202999999999999 + - type: precision_at_3 + value: 14.651 + - type: precision_at_5 + value: 10.996 + - type: recall_at_1 + value: 21.479 + - type: recall_at_10 + value: 71.479 + - type: recall_at_100 + value: 96.65700000000001 + - type: recall_at_1000 + value: 99.36 + - type: recall_at_20 + value: 84.068 + - type: recall_at_3 + value: 43.954 + - type: recall_at_5 + value: 54.979 + task: + type: Retrieval + - dataset: + config: default + name: MTEB CQADupstackAndroidRetrieval (default) + revision: f46a197baaae43b4f621051089b82a364682dfeb + split: test + type: mteb/cqadupstack-android + metrics: + - type: main_score + value: 31.177 + - type: map_at_1 + value: 19.758 + - type: map_at_10 + value: 26.619999999999997 + - type: map_at_100 + value: 27.784 + - type: map_at_1000 + value: 27.937 + - type: map_at_20 + value: 27.206999999999997 + - type: map_at_3 + value: 24.245 + - type: map_at_5 + value: 25.713 + - type: mrr_at_1 + value: 24.892703862660944 + - type: mrr_at_10 + value: 31.704702863501144 + - type: mrr_at_100 + value: 32.608301500063966 + - type: mrr_at_1000 + value: 32.68725795268983 + - type: mrr_at_20 + value: 32.177359559981575 + - type: mrr_at_3 + value: 29.685264663805444 + - type: mrr_at_5 + value: 30.958512160228906 + - type: ndcg_at_1 + value: 24.893 + - type: ndcg_at_10 + value: 31.177 + - type: ndcg_at_100 + value: 36.546 + - type: ndcg_at_1000 + value: 39.706 + - type: ndcg_at_20 + value: 32.926 + - type: ndcg_at_3 + value: 27.58 + - type: ndcg_at_5 + value: 29.465000000000003 + - type: precision_at_1 + value: 24.893 + - type: precision_at_10 + value: 5.966 + - type: precision_at_100 + value: 1.079 + - type: precision_at_1000 + value: 0.166 + - type: precision_at_20 + value: 3.5909999999999997 + - type: precision_at_3 + value: 13.209000000000001 + - type: precision_at_5 + value: 9.728 + - type: recall_at_1 + value: 19.758 + - type: recall_at_10 + value: 39.397 + - type: recall_at_100 + value: 63.446999999999996 + - type: recall_at_1000 + value: 85.083 + - type: recall_at_20 + value: 45.846 + - type: recall_at_3 + value: 28.855999999999998 + - type: recall_at_5 + value: 34.165 + task: + type: Retrieval + - dataset: + config: default + name: MTEB CQADupstackEnglishRetrieval (default) + revision: ad9991cb51e31e31e430383c75ffb2885547b5f0 + split: test + type: mteb/cqadupstack-english + metrics: + - type: main_score + value: 25.901999999999997 + - type: map_at_1 + value: 16.730999999999998 + - type: map_at_10 + value: 22.24 + - type: map_at_100 + value: 23.168 + - type: map_at_1000 + value: 23.289 + - type: map_at_20 + value: 22.720000000000002 + - type: map_at_3 + value: 20.335 + - type: map_at_5 + value: 21.371000000000002 + - type: mrr_at_1 + value: 20.31847133757962 + - type: mrr_at_10 + value: 25.743908603781197 + - type: mrr_at_100 + value: 26.511937121538402 + - type: mrr_at_1000 + value: 26.58619222690668 + - type: mrr_at_20 + value: 26.15910380102564 + - type: mrr_at_3 + value: 23.77919320594478 + - type: mrr_at_5 + value: 24.846072186836484 + - type: ndcg_at_1 + value: 20.318 + - type: ndcg_at_10 + value: 25.901999999999997 + - type: ndcg_at_100 + value: 30.259999999999998 + - type: ndcg_at_1000 + value: 32.984 + - type: ndcg_at_20 + value: 27.47 + - type: ndcg_at_3 + value: 22.432 + - type: ndcg_at_5 + value: 23.999000000000002 + - type: precision_at_1 + value: 20.318 + - type: precision_at_10 + value: 4.707 + - type: precision_at_100 + value: 0.8580000000000001 + - type: precision_at_1000 + value: 0.134 + - type: precision_at_20 + value: 2.869 + - type: precision_at_3 + value: 10.552 + - type: precision_at_5 + value: 7.567 + - type: recall_at_1 + value: 16.730999999999998 + - type: recall_at_10 + value: 33.48 + - type: recall_at_100 + value: 52.245 + - type: recall_at_1000 + value: 70.634 + - type: recall_at_20 + value: 39.189 + - type: recall_at_3 + value: 23.805 + - type: recall_at_5 + value: 27.898 + task: + type: Retrieval + - dataset: + config: default + name: MTEB CQADupstackGamingRetrieval (default) + revision: 4885aa143210c98657558c04aaf3dc47cfb54340 + split: test + type: mteb/cqadupstack-gaming + metrics: + - type: main_score + value: 36.312 + - type: map_at_1 + value: 23.072 + - type: map_at_10 + value: 31.64 + - type: map_at_100 + value: 32.761 + - type: map_at_1000 + value: 32.862 + - type: map_at_20 + value: 32.24 + - type: map_at_3 + value: 28.921999999999997 + - type: map_at_5 + value: 30.603 + - type: mrr_at_1 + value: 26.70846394984326 + - type: mrr_at_10 + value: 34.47342886998059 + - type: mrr_at_100 + value: 35.406961114894706 + - type: mrr_at_1000 + value: 35.47747613970401 + - type: mrr_at_20 + value: 34.984787094473404 + - type: mrr_at_3 + value: 32.15256008359454 + - type: mrr_at_5 + value: 33.544409613375095 + - type: ndcg_at_1 + value: 26.708 + - type: ndcg_at_10 + value: 36.312 + - type: ndcg_at_100 + value: 41.748000000000005 + - type: ndcg_at_1000 + value: 44.206 + - type: ndcg_at_20 + value: 38.257000000000005 + - type: ndcg_at_3 + value: 31.439 + - type: ndcg_at_5 + value: 34.036 + - type: precision_at_1 + value: 26.708 + - type: precision_at_10 + value: 6.0440000000000005 + - type: precision_at_100 + value: 0.966 + - type: precision_at_1000 + value: 0.125 + - type: precision_at_20 + value: 3.549 + - type: precision_at_3 + value: 14.086000000000002 + - type: precision_at_5 + value: 10.169 + - type: recall_at_1 + value: 23.072 + - type: recall_at_10 + value: 47.687000000000005 + - type: recall_at_100 + value: 72.469 + - type: recall_at_1000 + value: 90.568 + - type: recall_at_20 + value: 54.861000000000004 + - type: recall_at_3 + value: 34.758 + - type: recall_at_5 + value: 41.052 + task: + type: Retrieval + - dataset: + config: default + name: MTEB CQADupstackGisRetrieval (default) + revision: 5003b3064772da1887988e05400cf3806fe491f2 + split: test + type: mteb/cqadupstack-gis + metrics: + - type: main_score + value: 19.756999999999998 + - type: map_at_1 + value: 12.546 + - type: map_at_10 + value: 17.009 + - type: map_at_100 + value: 17.758 + - type: map_at_1000 + value: 17.866 + - type: map_at_20 + value: 17.399 + - type: map_at_3 + value: 15.532000000000002 + - type: map_at_5 + value: 16.305 + - type: mrr_at_1 + value: 13.559322033898304 + - type: mrr_at_10 + value: 18.144067796610162 + - type: mrr_at_100 + value: 18.89867843656649 + - type: mrr_at_1000 + value: 18.995819754371045 + - type: mrr_at_20 + value: 18.54987150762981 + - type: mrr_at_3 + value: 16.666666666666664 + - type: mrr_at_5 + value: 17.43502824858757 + - type: ndcg_at_1 + value: 13.559 + - type: ndcg_at_10 + value: 19.756999999999998 + - type: ndcg_at_100 + value: 23.931 + - type: ndcg_at_1000 + value: 27.203 + - type: ndcg_at_20 + value: 21.173000000000002 + - type: ndcg_at_3 + value: 16.778000000000002 + - type: ndcg_at_5 + value: 18.104 + - type: precision_at_1 + value: 13.559 + - type: precision_at_10 + value: 3.141 + - type: precision_at_100 + value: 0.5539999999999999 + - type: precision_at_1000 + value: 0.08800000000000001 + - type: precision_at_20 + value: 1.8929999999999998 + - type: precision_at_3 + value: 7.156 + - type: precision_at_5 + value: 5.04 + - type: recall_at_1 + value: 12.546 + - type: recall_at_10 + value: 27.093 + - type: recall_at_100 + value: 47.325 + - type: recall_at_1000 + value: 72.965 + - type: recall_at_20 + value: 32.491 + - type: recall_at_3 + value: 19.122 + - type: recall_at_5 + value: 22.264999999999997 + task: + type: Retrieval + - dataset: + config: default + name: MTEB CQADupstackMathematicaRetrieval (default) + revision: 90fceea13679c63fe563ded68f3b6f06e50061de + split: test + type: mteb/cqadupstack-mathematica + metrics: + - type: main_score + value: 12.806000000000001 + - type: map_at_1 + value: 5.881 + - type: map_at_10 + value: 9.803 + - type: map_at_100 + value: 10.717 + - type: map_at_1000 + value: 10.816 + - type: map_at_20 + value: 10.244 + - type: map_at_3 + value: 8.163 + - type: map_at_5 + value: 8.956 + - type: mrr_at_1 + value: 7.462686567164178 + - type: mrr_at_10 + value: 12.077805417357661 + - type: mrr_at_100 + value: 12.966483256051529 + - type: mrr_at_1000 + value: 13.047246067329347 + - type: mrr_at_20 + value: 12.525499289049966 + - type: mrr_at_3 + value: 10.261194029850747 + - type: mrr_at_5 + value: 11.113184079601991 + - type: ndcg_at_1 + value: 7.463 + - type: ndcg_at_10 + value: 12.806000000000001 + - type: ndcg_at_100 + value: 17.807000000000002 + - type: ndcg_at_1000 + value: 20.979999999999997 + - type: ndcg_at_20 + value: 14.350999999999999 + - type: ndcg_at_3 + value: 9.468 + - type: ndcg_at_5 + value: 10.776 + - type: precision_at_1 + value: 7.463 + - type: precision_at_10 + value: 2.637 + - type: precision_at_100 + value: 0.613 + - type: precision_at_1000 + value: 0.101 + - type: precision_at_20 + value: 1.7229999999999999 + - type: precision_at_3 + value: 4.643 + - type: precision_at_5 + value: 3.6319999999999997 + - type: recall_at_1 + value: 5.881 + - type: recall_at_10 + value: 20.013 + - type: recall_at_100 + value: 42.92 + - type: recall_at_1000 + value: 66.943 + - type: recall_at_20 + value: 25.621 + - type: recall_at_3 + value: 10.768 + - type: recall_at_5 + value: 14.007 + task: + type: Retrieval + - dataset: + config: default + name: MTEB CQADupstackPhysicsRetrieval (default) + revision: 79531abbd1fb92d06c6d6315a0cbbbf5bb247ea4 + split: test + type: mteb/cqadupstack-physics + metrics: + - type: main_score + value: 28.765 + - type: map_at_1 + value: 17.519000000000002 + - type: map_at_10 + value: 24.136 + - type: map_at_100 + value: 25.352999999999998 + - type: map_at_1000 + value: 25.499 + - type: map_at_20 + value: 24.776999999999997 + - type: map_at_3 + value: 21.59 + - type: map_at_5 + value: 23.017000000000003 + - type: mrr_at_1 + value: 21.36669874879692 + - type: mrr_at_10 + value: 28.47930702598651 + - type: mrr_at_100 + value: 29.504734721457147 + - type: mrr_at_1000 + value: 29.58610606296599 + - type: mrr_at_20 + value: 29.06239382160277 + - type: mrr_at_3 + value: 25.85819698427977 + - type: mrr_at_5 + value: 27.523259544433753 + - type: ndcg_at_1 + value: 21.367 + - type: ndcg_at_10 + value: 28.765 + - type: ndcg_at_100 + value: 34.772999999999996 + - type: ndcg_at_1000 + value: 37.924 + - type: ndcg_at_20 + value: 30.891999999999996 + - type: ndcg_at_3 + value: 24.248 + - type: ndcg_at_5 + value: 26.479999999999997 + - type: precision_at_1 + value: 21.367 + - type: precision_at_10 + value: 5.351 + - type: precision_at_100 + value: 0.989 + - type: precision_at_1000 + value: 0.147 + - type: precision_at_20 + value: 3.325 + - type: precision_at_3 + value: 11.325000000000001 + - type: precision_at_5 + value: 8.469999999999999 + - type: recall_at_1 + value: 17.519000000000002 + - type: recall_at_10 + value: 38.602 + - type: recall_at_100 + value: 65.377 + - type: recall_at_1000 + value: 86.812 + - type: recall_at_20 + value: 46.161 + - type: recall_at_3 + value: 25.898 + - type: recall_at_5 + value: 31.654 + task: + type: Retrieval + - dataset: + config: default + name: MTEB CQADupstackProgrammersRetrieval (default) + revision: 6184bc1440d2dbc7612be22b50686b8826d22b32 + split: test + type: mteb/cqadupstack-programmers + metrics: + - type: main_score + value: 22.706 + - type: map_at_1 + value: 12.979 + - type: map_at_10 + value: 18.898 + - type: map_at_100 + value: 20.087 + - type: map_at_1000 + value: 20.223 + - type: map_at_20 + value: 19.516 + - type: map_at_3 + value: 16.955000000000002 + - type: map_at_5 + value: 18.043 + - type: mrr_at_1 + value: 15.639269406392694 + - type: mrr_at_10 + value: 22.00872472276581 + - type: mrr_at_100 + value: 23.052476310365996 + - type: mrr_at_1000 + value: 23.1408565851826 + - type: mrr_at_20 + value: 22.589418169993035 + - type: mrr_at_3 + value: 19.99619482496195 + - type: mrr_at_5 + value: 21.2062404870624 + - type: ndcg_at_1 + value: 15.639 + - type: ndcg_at_10 + value: 22.706 + - type: ndcg_at_100 + value: 28.477999999999998 + - type: ndcg_at_1000 + value: 31.756 + - type: ndcg_at_20 + value: 24.836 + - type: ndcg_at_3 + value: 19.049 + - type: ndcg_at_5 + value: 20.807000000000002 + - type: precision_at_1 + value: 15.639 + - type: precision_at_10 + value: 4.258 + - type: precision_at_100 + value: 0.865 + - type: precision_at_1000 + value: 0.133 + - type: precision_at_20 + value: 2.7969999999999997 + - type: precision_at_3 + value: 9.056000000000001 + - type: precision_at_5 + value: 6.758 + - type: recall_at_1 + value: 12.979 + - type: recall_at_10 + value: 31.16 + - type: recall_at_100 + value: 56.245 + - type: recall_at_1000 + value: 79.526 + - type: recall_at_20 + value: 38.696000000000005 + - type: recall_at_3 + value: 21.302 + - type: recall_at_5 + value: 25.615 + task: + type: Retrieval + - dataset: + config: default + name: MTEB CQADupstackRetrieval (default) + revision: CQADupstackRetrieval_is_a_combined_dataset + split: test + type: CQADupstackRetrieval_is_a_combined_dataset + metrics: + - type: main_score + value: 21.956083333333336 + - type: ndcg_at_10 + value: 21.956083333333336 + task: + type: Retrieval + - dataset: + config: default + name: MTEB CQADupstackStatsRetrieval (default) + revision: 65ac3a16b8e91f9cee4c9828cc7c335575432a2a + split: test + type: mteb/cqadupstack-stats + metrics: + - type: main_score + value: 17.395 + - type: map_at_1 + value: 10.088 + - type: map_at_10 + value: 14.539 + - type: map_at_100 + value: 15.362 + - type: map_at_1000 + value: 15.464 + - type: map_at_20 + value: 14.924000000000001 + - type: map_at_3 + value: 13.136999999999999 + - type: map_at_5 + value: 13.937 + - type: mrr_at_1 + value: 11.96319018404908 + - type: mrr_at_10 + value: 16.467767065926576 + - type: mrr_at_100 + value: 17.283614344094357 + - type: mrr_at_1000 + value: 17.375873381573232 + - type: mrr_at_20 + value: 16.854301079891787 + - type: mrr_at_3 + value: 15.08179959100204 + - type: mrr_at_5 + value: 15.848670756646216 + - type: ndcg_at_1 + value: 11.963 + - type: ndcg_at_10 + value: 17.395 + - type: ndcg_at_100 + value: 21.911 + - type: ndcg_at_1000 + value: 24.796000000000003 + - type: ndcg_at_20 + value: 18.773999999999997 + - type: ndcg_at_3 + value: 14.719 + - type: ndcg_at_5 + value: 16.032 + - type: precision_at_1 + value: 11.963 + - type: precision_at_10 + value: 2.96 + - type: precision_at_100 + value: 0.569 + - type: precision_at_1000 + value: 0.08800000000000001 + - type: precision_at_20 + value: 1.817 + - type: precision_at_3 + value: 6.748 + - type: precision_at_5 + value: 4.877 + - type: recall_at_1 + value: 10.088 + - type: recall_at_10 + value: 24.356 + - type: recall_at_100 + value: 45.73 + - type: recall_at_1000 + value: 67.577 + - type: recall_at_20 + value: 29.534 + - type: recall_at_3 + value: 16.944 + - type: recall_at_5 + value: 20.392 + task: + type: Retrieval + - dataset: + config: default + name: MTEB CQADupstackTexRetrieval (default) + revision: 46989137a86843e03a6195de44b09deda022eec7 + split: test + type: mteb/cqadupstack-tex + metrics: + - type: main_score + value: 12.592999999999998 + - type: map_at_1 + value: 7.1290000000000004 + - type: map_at_10 + value: 10.302 + - type: map_at_100 + value: 10.994 + - type: map_at_1000 + value: 11.118 + - type: map_at_20 + value: 10.653 + - type: map_at_3 + value: 9.361 + - type: map_at_5 + value: 9.803 + - type: mrr_at_1 + value: 8.843771507226428 + - type: mrr_at_10 + value: 12.421700040419921 + - type: mrr_at_100 + value: 13.14928923530448 + - type: mrr_at_1000 + value: 13.252448388769883 + - type: mrr_at_20 + value: 12.803691756524197 + - type: mrr_at_3 + value: 11.344345033264513 + - type: mrr_at_5 + value: 11.824386327139244 + - type: ndcg_at_1 + value: 8.844000000000001 + - type: ndcg_at_10 + value: 12.592999999999998 + - type: ndcg_at_100 + value: 16.409000000000002 + - type: ndcg_at_1000 + value: 19.906 + - type: ndcg_at_20 + value: 13.831 + - type: ndcg_at_3 + value: 10.7 + - type: ndcg_at_5 + value: 11.359 + - type: precision_at_1 + value: 8.844000000000001 + - type: precision_at_10 + value: 2.33 + - type: precision_at_100 + value: 0.506 + - type: precision_at_1000 + value: 0.096 + - type: precision_at_20 + value: 1.512 + - type: precision_at_3 + value: 5.116 + - type: precision_at_5 + value: 3.599 + - type: recall_at_1 + value: 7.1290000000000004 + - type: recall_at_10 + value: 17.549999999999997 + - type: recall_at_100 + value: 35.393 + - type: recall_at_1000 + value: 61.23800000000001 + - type: recall_at_20 + value: 22.124 + - type: recall_at_3 + value: 12.109 + - type: recall_at_5 + value: 13.832 + task: + type: Retrieval + - dataset: + config: default + name: MTEB CQADupstackUnixRetrieval (default) + revision: 6c6430d3a6d36f8d2a829195bc5dc94d7e063e53 + split: test + type: mteb/cqadupstack-unix + metrics: + - type: main_score + value: 19.111 + - type: map_at_1 + value: 12.577 + - type: map_at_10 + value: 16.275000000000002 + - type: map_at_100 + value: 17.083000000000002 + - type: map_at_1000 + value: 17.206 + - type: map_at_20 + value: 16.68 + - type: map_at_3 + value: 14.783 + - type: map_at_5 + value: 15.654000000000002 + - type: mrr_at_1 + value: 14.645522388059701 + - type: mrr_at_10 + value: 18.617404051172702 + - type: mrr_at_100 + value: 19.434952661619388 + - type: mrr_at_1000 + value: 19.536374825069274 + - type: mrr_at_20 + value: 19.039596975787966 + - type: mrr_at_3 + value: 16.977611940298516 + - type: mrr_at_5 + value: 17.938432835820894 + - type: ndcg_at_1 + value: 14.646 + - type: ndcg_at_10 + value: 19.111 + - type: ndcg_at_100 + value: 23.541999999999998 + - type: ndcg_at_1000 + value: 26.901999999999997 + - type: ndcg_at_20 + value: 20.593 + - type: ndcg_at_3 + value: 16.104 + - type: ndcg_at_5 + value: 17.577 + - type: precision_at_1 + value: 14.646 + - type: precision_at_10 + value: 3.237 + - type: precision_at_100 + value: 0.607 + - type: precision_at_1000 + value: 0.1 + - type: precision_at_20 + value: 1.9869999999999999 + - type: precision_at_3 + value: 7.027 + - type: precision_at_5 + value: 5.187 + - type: recall_at_1 + value: 12.577 + - type: recall_at_10 + value: 25.642 + - type: recall_at_100 + value: 46.296 + - type: recall_at_1000 + value: 70.901 + - type: recall_at_20 + value: 31.202 + - type: recall_at_3 + value: 17.396 + - type: recall_at_5 + value: 21.046 + task: + type: Retrieval + - dataset: + config: default + name: MTEB CQADupstackWebmastersRetrieval (default) + revision: 160c094312a0e1facb97e55eeddb698c0abe3571 + split: test + type: mteb/cqadupstack-webmasters + metrics: + - type: main_score + value: 22.359 + - type: map_at_1 + value: 14.202 + - type: map_at_10 + value: 18.528 + - type: map_at_100 + value: 19.649 + - type: map_at_1000 + value: 19.838 + - type: map_at_20 + value: 19.067 + - type: map_at_3 + value: 16.656000000000002 + - type: map_at_5 + value: 17.564 + - type: mrr_at_1 + value: 17.786561264822133 + - type: mrr_at_10 + value: 22.50015684798293 + - type: mrr_at_100 + value: 23.36103535586478 + - type: mrr_at_1000 + value: 23.447240155046412 + - type: mrr_at_20 + value: 22.897233666294404 + - type: mrr_at_3 + value: 20.685111989459813 + - type: mrr_at_5 + value: 21.58432147562582 + - type: ndcg_at_1 + value: 17.787 + - type: ndcg_at_10 + value: 22.359 + - type: ndcg_at_100 + value: 27.339999999999996 + - type: ndcg_at_1000 + value: 30.94 + - type: ndcg_at_20 + value: 23.915 + - type: ndcg_at_3 + value: 19.187 + - type: ndcg_at_5 + value: 20.415 + - type: precision_at_1 + value: 17.787 + - type: precision_at_10 + value: 4.348 + - type: precision_at_100 + value: 1.016 + - type: precision_at_1000 + value: 0.187 + - type: precision_at_20 + value: 2.826 + - type: precision_at_3 + value: 8.959 + - type: precision_at_5 + value: 6.601 + - type: recall_at_1 + value: 14.202 + - type: recall_at_10 + value: 29.507 + - type: recall_at_100 + value: 52.574 + - type: recall_at_1000 + value: 77.41799999999999 + - type: recall_at_20 + value: 35.733 + - type: recall_at_3 + value: 19.345000000000002 + - type: recall_at_5 + value: 22.99 + task: + type: Retrieval + - dataset: + config: default + name: MTEB CQADupstackWordpressRetrieval (default) + revision: 4ffe81d471b1924886b33c7567bfb200e9eec5c4 + split: test + type: mteb/cqadupstack-wordpress + metrics: + - type: main_score + value: 14.59 + - type: map_at_1 + value: 9.341000000000001 + - type: map_at_10 + value: 12.495000000000001 + - type: map_at_100 + value: 13.328000000000001 + - type: map_at_1000 + value: 13.443 + - type: map_at_20 + value: 12.919 + - type: map_at_3 + value: 11.448 + - type: map_at_5 + value: 12.016 + - type: mrr_at_1 + value: 10.536044362292053 + - type: mrr_at_10 + value: 13.933045799958926 + - type: mrr_at_100 + value: 14.753327128738034 + - type: mrr_at_1000 + value: 14.84798752653836 + - type: mrr_at_20 + value: 14.348175993628182 + - type: mrr_at_3 + value: 12.6309303758472 + - type: mrr_at_5 + value: 13.28712261244609 + - type: ndcg_at_1 + value: 10.536 + - type: ndcg_at_10 + value: 14.59 + - type: ndcg_at_100 + value: 19.322 + - type: ndcg_at_1000 + value: 22.735 + - type: ndcg_at_20 + value: 16.072 + - type: ndcg_at_3 + value: 12.36 + - type: ndcg_at_5 + value: 13.364999999999998 + - type: precision_at_1 + value: 10.536 + - type: precision_at_10 + value: 2.311 + - type: precision_at_100 + value: 0.508 + - type: precision_at_1000 + value: 0.086 + - type: precision_at_20 + value: 1.488 + - type: precision_at_3 + value: 5.176 + - type: precision_at_5 + value: 3.66 + - type: recall_at_1 + value: 9.341000000000001 + - type: recall_at_10 + value: 19.707 + - type: recall_at_100 + value: 42.89 + - type: recall_at_1000 + value: 69.447 + - type: recall_at_20 + value: 25.330000000000002 + - type: recall_at_3 + value: 13.814000000000002 + - type: recall_at_5 + value: 16.217000000000002 + task: + type: Retrieval + - dataset: + config: default + name: MTEB ClimateFEVER (default) + revision: 47f2ac6acb640fc46020b02a5b59fdda04d39380 + split: test + type: mteb/climate-fever + metrics: + - type: main_score + value: 20.433 + - type: map_at_1 + value: 7.469 + - type: map_at_10 + value: 13.536999999999999 + - type: map_at_100 + value: 15.222 + - type: map_at_1000 + value: 15.424 + - type: map_at_20 + value: 14.41 + - type: map_at_3 + value: 10.911999999999999 + - type: map_at_5 + value: 12.232 + - type: mrr_at_1 + value: 17.133550488599347 + - type: mrr_at_10 + value: 27.41355152267201 + - type: mrr_at_100 + value: 28.50611626391541 + - type: mrr_at_1000 + value: 28.568789326404005 + - type: mrr_at_20 + value: 28.08885051017031 + - type: mrr_at_3 + value: 23.724212812160687 + - type: mrr_at_5 + value: 25.8707926167209 + - type: ndcg_at_1 + value: 17.134 + - type: ndcg_at_10 + value: 20.433 + - type: ndcg_at_100 + value: 27.783 + - type: ndcg_at_1000 + value: 31.787 + - type: ndcg_at_20 + value: 23.108999999999998 + - type: ndcg_at_3 + value: 15.565999999999999 + - type: ndcg_at_5 + value: 17.354 + - type: precision_at_1 + value: 17.134 + - type: precision_at_10 + value: 6.866 + - type: precision_at_100 + value: 1.47 + - type: precision_at_1000 + value: 0.22100000000000003 + - type: precision_at_20 + value: 4.531000000000001 + - type: precision_at_3 + value: 11.965 + - type: precision_at_5 + value: 9.707 + - type: recall_at_1 + value: 7.469 + - type: recall_at_10 + value: 26.285999999999998 + - type: recall_at_100 + value: 52.376999999999995 + - type: recall_at_1000 + value: 75.261 + - type: recall_at_20 + value: 34.035 + - type: recall_at_3 + value: 14.526 + - type: recall_at_5 + value: 19.306 + task: + type: Retrieval + - dataset: + config: default + name: MTEB ClimateFEVERHardNegatives (default) + revision: 3a309e201f3c2c4b13bd4a367a8f37eee2ec1d21 + split: test + type: mteb/ClimateFEVER_test_top_250_only_w_correct-v2 + metrics: + - type: main_score + value: 22.605 + - type: map_at_1 + value: 8.6 + - type: map_at_10 + value: 15.281 + - type: map_at_100 + value: 17.282 + - type: map_at_1000 + value: 17.522 + - type: map_at_20 + value: 16.339000000000002 + - type: map_at_3 + value: 12.429 + - type: map_at_5 + value: 13.922 + - type: mrr_at_1 + value: 18.5 + - type: mrr_at_10 + value: 29.625952380952388 + - type: mrr_at_100 + value: 30.77892068089231 + - type: mrr_at_1000 + value: 30.83853566495065 + - type: mrr_at_20 + value: 30.3662752048433 + - type: mrr_at_3 + value: 25.833333333333357 + - type: mrr_at_5 + value: 27.89333333333332 + - type: ndcg_at_1 + value: 18.5 + - type: ndcg_at_10 + value: 22.605 + - type: ndcg_at_100 + value: 31.097 + - type: ndcg_at_1000 + value: 35.576 + - type: ndcg_at_20 + value: 25.775 + - type: ndcg_at_3 + value: 17.43 + - type: ndcg_at_5 + value: 19.368 + - type: precision_at_1 + value: 18.5 + - type: precision_at_10 + value: 7.46 + - type: precision_at_100 + value: 1.6580000000000001 + - type: precision_at_1000 + value: 0.25 + - type: precision_at_20 + value: 5.06 + - type: precision_at_3 + value: 13.433 + - type: precision_at_5 + value: 10.74 + - type: recall_at_1 + value: 8.6 + - type: recall_at_10 + value: 28.882 + - type: recall_at_100 + value: 58.998 + - type: recall_at_1000 + value: 84.243 + - type: recall_at_20 + value: 37.957 + - type: recall_at_3 + value: 16.55 + - type: recall_at_5 + value: 21.648 + task: + type: Retrieval + - dataset: + config: default + name: MTEB CodeFeedbackMT (default) + revision: b0f12fa0c0dd67f59c95a5c33d02aeeb4c398c5f + split: test + type: CoIR-Retrieval/codefeedback-mt + metrics: + - type: main_score + value: 46.045 + - type: map_at_1 + value: 38.412 + - type: map_at_10 + value: 43.41 + - type: map_at_100 + value: 43.976 + - type: map_at_1000 + value: 44.037 + - type: map_at_20 + value: 43.736000000000004 + - type: map_at_3 + value: 42.055 + - type: map_at_5 + value: 42.829 + - type: mrr_at_1 + value: 38.41229193341869 + - type: mrr_at_10 + value: 43.40960498582686 + - type: mrr_at_100 + value: 43.97561445897139 + - type: mrr_at_1000 + value: 44.03658359938551 + - type: mrr_at_20 + value: 43.73609592536922 + - type: mrr_at_3 + value: 42.05518314880356 + - type: mrr_at_5 + value: 42.828701262835196 + - type: ndcg_at_1 + value: 38.412 + - type: ndcg_at_10 + value: 46.045 + - type: ndcg_at_100 + value: 49.061 + - type: ndcg_at_1000 + value: 50.941 + - type: ndcg_at_20 + value: 47.245 + - type: ndcg_at_3 + value: 43.245 + - type: ndcg_at_5 + value: 44.639 + - type: precision_at_1 + value: 38.412 + - type: precision_at_10 + value: 5.442 + - type: precision_at_100 + value: 0.6910000000000001 + - type: precision_at_1000 + value: 0.08499999999999999 + - type: precision_at_20 + value: 2.959 + - type: precision_at_3 + value: 15.562999999999999 + - type: precision_at_5 + value: 10.014000000000001 + - type: recall_at_1 + value: 38.412 + - type: recall_at_10 + value: 54.417 + - type: recall_at_100 + value: 69.15 + - type: recall_at_1000 + value: 84.515 + - type: recall_at_20 + value: 59.185 + - type: recall_at_3 + value: 46.69 + - type: recall_at_5 + value: 50.072 + task: + type: Retrieval + - dataset: + config: default + name: MTEB CodeFeedbackST (default) + revision: d213819e87aab9010628da8b73ab4eb337c89340 + split: test + type: CoIR-Retrieval/codefeedback-st + metrics: + - type: main_score + value: 45.592 + - type: map_at_1 + value: 34.54 + - type: map_at_10 + value: 41.855 + - type: map_at_100 + value: 42.528 + - type: map_at_1000 + value: 42.587 + - type: map_at_20 + value: 42.239 + - type: map_at_3 + value: 39.985 + - type: map_at_5 + value: 41.075 + - type: mrr_at_1 + value: 34.42151664217722 + - type: mrr_at_10 + value: 41.78311069737728 + - type: mrr_at_100 + value: 42.45614518998771 + - type: mrr_at_1000 + value: 42.51517890438335 + - type: mrr_at_20 + value: 42.16798542795404 + - type: mrr_at_3 + value: 39.90715304840442 + - type: mrr_at_5 + value: 41.00055367448295 + - type: ndcg_at_1 + value: 34.54 + - type: ndcg_at_10 + value: 45.592 + - type: ndcg_at_100 + value: 49.13 + - type: ndcg_at_1000 + value: 50.885999999999996 + - type: ndcg_at_20 + value: 46.989999999999995 + - type: ndcg_at_3 + value: 41.754000000000005 + - type: ndcg_at_5 + value: 43.714999999999996 + - type: precision_at_1 + value: 34.54 + - type: precision_at_10 + value: 5.74 + - type: precision_at_100 + value: 0.746 + - type: precision_at_1000 + value: 0.089 + - type: precision_at_20 + value: 3.1460000000000004 + - type: precision_at_3 + value: 15.623999999999999 + - type: precision_at_5 + value: 10.325 + - type: recall_at_1 + value: 34.54 + - type: recall_at_10 + value: 57.403999999999996 + - type: recall_at_100 + value: 74.577 + - type: recall_at_1000 + value: 88.801 + - type: recall_at_20 + value: 62.927 + - type: recall_at_3 + value: 46.873 + - type: recall_at_5 + value: 51.623 + task: + type: Retrieval + - dataset: + config: default + name: MTEB CosQA (default) + revision: bc5efb7e9d437246ce393ed19d772e08e4a79535 + split: test + type: CoIR-Retrieval/cosqa + metrics: + - type: main_score + value: 7.939 + - type: map_at_1 + value: 3.2 + - type: map_at_10 + value: 6.1240000000000006 + - type: map_at_100 + value: 6.961 + - type: map_at_1000 + value: 7.124 + - type: map_at_20 + value: 6.494 + - type: map_at_3 + value: 5.033 + - type: map_at_5 + value: 5.623 + - type: mrr_at_1 + value: 3.2 + - type: mrr_at_10 + value: 5.470238095238093 + - type: mrr_at_100 + value: 6.320663781727482 + - type: mrr_at_1000 + value: 6.484552484927204 + - type: mrr_at_20 + value: 5.840692146690597 + - type: mrr_at_3 + value: 4.3999999999999995 + - type: mrr_at_5 + value: 4.919999999999999 + - type: ndcg_at_1 + value: 3.2 + - type: ndcg_at_10 + value: 7.939 + - type: ndcg_at_100 + value: 12.909 + - type: ndcg_at_1000 + value: 17.705000000000002 + - type: ndcg_at_20 + value: 9.266 + - type: ndcg_at_3 + value: 5.688 + - type: ndcg_at_5 + value: 6.755 + - type: precision_at_1 + value: 3.2 + - type: precision_at_10 + value: 1.38 + - type: precision_at_100 + value: 0.392 + - type: precision_at_1000 + value: 0.078 + - type: precision_at_20 + value: 0.95 + - type: precision_at_3 + value: 2.533 + - type: precision_at_5 + value: 2.04 + - type: recall_at_1 + value: 3.2 + - type: recall_at_10 + value: 13.8 + - type: recall_at_100 + value: 39.2 + - type: recall_at_1000 + value: 78.0 + - type: recall_at_20 + value: 19.0 + - type: recall_at_3 + value: 7.6 + - type: recall_at_5 + value: 10.2 + task: + type: Retrieval + - dataset: + config: default + name: MTEB DBPedia (default) + revision: c0f706b76e590d620bd6618b3ca8efdd34e2d659 + split: dev + type: mteb/dbpedia + metrics: + - type: main_score + value: 29.817 + - type: map_at_1 + value: 6.151 + - type: map_at_10 + value: 12.292 + - type: map_at_100 + value: 18.139 + - type: map_at_1000 + value: 19.84 + - type: map_at_20 + value: 14.495 + - type: map_at_3 + value: 8.426 + - type: map_at_5 + value: 10.192 + - type: mrr_at_1 + value: 46.26865671641791 + - type: mrr_at_10 + value: 57.92466240227434 + - type: mrr_at_100 + value: 58.67349319471301 + - type: mrr_at_1000 + value: 58.68212283999546 + - type: mrr_at_20 + value: 58.47241542478595 + - type: mrr_at_3 + value: 54.726368159203986 + - type: mrr_at_5 + value: 57.33830845771145 + - type: ndcg_at_1 + value: 38.06 + - type: ndcg_at_10 + value: 29.817 + - type: ndcg_at_100 + value: 36.472 + - type: ndcg_at_1000 + value: 45.576 + - type: ndcg_at_20 + value: 30.009000000000004 + - type: ndcg_at_3 + value: 32.839 + - type: ndcg_at_5 + value: 32.301 + - type: precision_at_1 + value: 46.269 + - type: precision_at_10 + value: 25.820999999999998 + - type: precision_at_100 + value: 8.552 + - type: precision_at_1000 + value: 1.576 + - type: precision_at_20 + value: 20.075000000000003 + - type: precision_at_3 + value: 35.821 + - type: precision_at_5 + value: 34.327999999999996 + - type: recall_at_1 + value: 6.151 + - type: recall_at_10 + value: 16.838 + - type: recall_at_100 + value: 48.427 + - type: recall_at_1000 + value: 77.018 + - type: recall_at_20 + value: 26.147 + - type: recall_at_3 + value: 9.221 + - type: recall_at_5 + value: 12.453 + task: + type: Retrieval + - dataset: + config: default + name: MTEB DBPedia (default) + revision: c0f706b76e590d620bd6618b3ca8efdd34e2d659 + split: test + type: mteb/dbpedia + metrics: + - type: main_score + value: 27.377000000000002 + - type: map_at_1 + value: 5.527 + - type: map_at_10 + value: 12.384 + - type: map_at_100 + value: 17.660999999999998 + - type: map_at_1000 + value: 18.98 + - type: map_at_20 + value: 14.424999999999999 + - type: map_at_3 + value: 8.484 + - type: map_at_5 + value: 10.174 + - type: mrr_at_1 + value: 44.25 + - type: mrr_at_10 + value: 55.620238095238086 + - type: mrr_at_100 + value: 56.311713506324445 + - type: mrr_at_1000 + value: 56.33739917164095 + - type: mrr_at_20 + value: 56.11873017655717 + - type: mrr_at_3 + value: 52.95833333333334 + - type: mrr_at_5 + value: 54.595833333333324 + - type: ndcg_at_1 + value: 31.75 + - type: ndcg_at_10 + value: 27.377000000000002 + - type: ndcg_at_100 + value: 32.164 + - type: ndcg_at_1000 + value: 40.050000000000004 + - type: ndcg_at_20 + value: 27.424 + - type: ndcg_at_3 + value: 28.683999999999997 + - type: ndcg_at_5 + value: 28.283 + - type: precision_at_1 + value: 44.25 + - type: precision_at_10 + value: 24.45 + - type: precision_at_100 + value: 7.704999999999999 + - type: precision_at_1000 + value: 1.5970000000000002 + - type: precision_at_20 + value: 18.462 + - type: precision_at_3 + value: 35.167 + - type: precision_at_5 + value: 30.95 + - type: recall_at_1 + value: 5.527 + - type: recall_at_10 + value: 18.016 + - type: recall_at_100 + value: 41.656 + - type: recall_at_1000 + value: 67.38300000000001 + - type: recall_at_20 + value: 24.21 + - type: recall_at_3 + value: 9.936 + - type: recall_at_5 + value: 13.187999999999999 + task: + type: Retrieval + - dataset: + config: default + name: MTEB DBPediaHardNegatives (default) + revision: 943ec7fdfef3728b2ad1966c5b6479ff9ffd26c9 + split: test + type: mteb/DBPedia_test_top_250_only_w_correct-v2 + metrics: + - type: main_score + value: 30.444 + - type: map_at_1 + value: 5.722 + - type: map_at_10 + value: 13.688 + - type: map_at_100 + value: 22.032 + - type: map_at_1000 + value: 25.386999999999997 + - type: map_at_20 + value: 16.307 + - type: map_at_3 + value: 9.008 + - type: map_at_5 + value: 11.056000000000001 + - type: mrr_at_1 + value: 47.0 + - type: mrr_at_10 + value: 59.097420634920624 + - type: mrr_at_100 + value: 59.821938296328106 + - type: mrr_at_1000 + value: 59.83760663887243 + - type: mrr_at_20 + value: 59.59489258161859 + - type: mrr_at_3 + value: 56.458333333333336 + - type: mrr_at_5 + value: 58.18333333333333 + - type: ndcg_at_1 + value: 33.5 + - type: ndcg_at_10 + value: 30.444 + - type: ndcg_at_100 + value: 40.474 + - type: ndcg_at_1000 + value: 51.964 + - type: ndcg_at_20 + value: 31.356 + - type: ndcg_at_3 + value: 30.772 + - type: ndcg_at_5 + value: 30.576999999999998 + - type: precision_at_1 + value: 47.0 + - type: precision_at_10 + value: 27.975 + - type: precision_at_100 + value: 12.055 + - type: precision_at_1000 + value: 2.9579999999999997 + - type: precision_at_20 + value: 22.275 + - type: precision_at_3 + value: 38.083 + - type: precision_at_5 + value: 33.75 + - type: recall_at_1 + value: 5.722 + - type: recall_at_10 + value: 20.571 + - type: recall_at_100 + value: 55.967999999999996 + - type: recall_at_1000 + value: 91.362 + - type: recall_at_20 + value: 28.526 + - type: recall_at_3 + value: 10.761999999999999 + - type: recall_at_5 + value: 14.854999999999999 + task: + type: Retrieval + - dataset: + config: default + name: MTEB FEVER (default) + revision: bea83ef9e8fb933d90a2f1d5515737465d613e12 + split: test + type: mteb/fever + metrics: + - type: main_score + value: 42.997 + - type: map_at_1 + value: 22.017 + - type: map_at_10 + value: 35.199000000000005 + - type: map_at_100 + value: 36.254999999999995 + - type: map_at_1000 + value: 36.298 + - type: map_at_20 + value: 35.855 + - type: map_at_3 + value: 31.072 + - type: map_at_5 + value: 33.461 + - type: mrr_at_1 + value: 23.582358235823584 + - type: mrr_at_10 + value: 37.35784888012616 + - type: mrr_at_100 + value: 38.36344206815839 + - type: mrr_at_1000 + value: 38.39238175644681 + - type: mrr_at_20 + value: 38.01212529885376 + - type: mrr_at_3 + value: 33.098309830982956 + - type: mrr_at_5 + value: 35.579557955795615 + - type: ndcg_at_1 + value: 23.582 + - type: ndcg_at_10 + value: 42.997 + - type: ndcg_at_100 + value: 47.979 + - type: ndcg_at_1000 + value: 48.994 + - type: ndcg_at_20 + value: 45.35 + - type: ndcg_at_3 + value: 34.579 + - type: ndcg_at_5 + value: 38.851 + - type: precision_at_1 + value: 23.582 + - type: precision_at_10 + value: 7.1290000000000004 + - type: precision_at_100 + value: 0.9820000000000001 + - type: precision_at_1000 + value: 0.109 + - type: precision_at_20 + value: 4.0840000000000005 + - type: precision_at_3 + value: 15.342 + - type: precision_at_5 + value: 11.479000000000001 + - type: recall_at_1 + value: 22.017 + - type: recall_at_10 + value: 65.354 + - type: recall_at_100 + value: 87.75800000000001 + - type: recall_at_1000 + value: 95.212 + - type: recall_at_20 + value: 74.38 + - type: recall_at_3 + value: 42.581 + - type: recall_at_5 + value: 52.844 + task: + type: Retrieval + - dataset: + config: default + name: MTEB FEVERHardNegatives (default) + revision: 080c9ed6267b65029207906e815d44a9240bafca + split: test + type: mteb/FEVER_test_top_250_only_w_correct-v2 + metrics: + - type: main_score + value: 46.277 + - type: map_at_1 + value: 23.945 + - type: map_at_10 + value: 37.564 + - type: map_at_100 + value: 38.562000000000005 + - type: map_at_1000 + value: 38.602 + - type: map_at_20 + value: 38.173 + - type: map_at_3 + value: 32.208999999999996 + - type: map_at_5 + value: 35.538 + - type: mrr_at_1 + value: 25.5 + - type: mrr_at_10 + value: 39.60134920634916 + - type: mrr_at_100 + value: 40.523753358000675 + - type: mrr_at_1000 + value: 40.53762691263701 + - type: mrr_at_20 + value: 40.19828779622504 + - type: mrr_at_3 + value: 34.14999999999997 + - type: mrr_at_5 + value: 37.579999999999934 + - type: ndcg_at_1 + value: 25.5 + - type: ndcg_at_10 + value: 46.277 + - type: ndcg_at_100 + value: 51.07000000000001 + - type: ndcg_at_1000 + value: 51.783 + - type: ndcg_at_20 + value: 48.473 + - type: ndcg_at_3 + value: 35.497 + - type: ndcg_at_5 + value: 41.467 + - type: precision_at_1 + value: 25.5 + - type: precision_at_10 + value: 7.76 + - type: precision_at_100 + value: 1.036 + - type: precision_at_1000 + value: 0.11299999999999999 + - type: precision_at_20 + value: 4.3549999999999995 + - type: precision_at_3 + value: 15.367 + - type: precision_at_5 + value: 12.280000000000001 + - type: recall_at_1 + value: 23.945 + - type: recall_at_10 + value: 71.967 + - type: recall_at_100 + value: 93.765 + - type: recall_at_1000 + value: 98.47 + - type: recall_at_20 + value: 80.497 + - type: recall_at_3 + value: 43.033 + - type: recall_at_5 + value: 57.447 + task: + type: Retrieval + - dataset: + config: default + name: MTEB FaithDial (default) + revision: 7a414e80725eac766f2602676dc8b39f80b061e4 + split: test + type: McGill-NLP/FaithDial + metrics: + - type: main_score + value: 20.793 + - type: map_at_1 + value: 5.779 + - type: map_at_10 + value: 14.64 + - type: map_at_100 + value: 16.412 + - type: map_at_1000 + value: 16.478 + - type: map_at_20 + value: 15.638 + - type: map_at_3 + value: 10.545 + - type: map_at_5 + value: 12.82 + - type: mrr_at_1 + value: 5.093046033300686 + - type: mrr_at_10 + value: 14.511255693919727 + - type: mrr_at_100 + value: 16.266082070705043 + - type: mrr_at_1000 + value: 16.333152055297443 + - type: mrr_at_20 + value: 15.49481390088696 + - type: mrr_at_3 + value: 10.381978452497567 + - type: mrr_at_5 + value: 12.74975514201763 + - type: ndcg_at_1 + value: 5.779 + - type: ndcg_at_10 + value: 20.793 + - type: ndcg_at_100 + value: 30.137000000000004 + - type: ndcg_at_1000 + value: 31.706 + - type: ndcg_at_20 + value: 24.431 + - type: ndcg_at_3 + value: 12.264 + - type: ndcg_at_5 + value: 16.35 + - type: precision_at_1 + value: 5.779 + - type: precision_at_10 + value: 4.099 + - type: precision_at_100 + value: 0.8630000000000001 + - type: precision_at_1000 + value: 0.098 + - type: precision_at_20 + value: 2.769 + - type: precision_at_3 + value: 5.762 + - type: precision_at_5 + value: 5.436 + - type: recall_at_1 + value: 5.779 + - type: recall_at_10 + value: 40.989 + - type: recall_at_100 + value: 86.337 + - type: recall_at_1000 + value: 98.335 + - type: recall_at_20 + value: 55.387 + - type: recall_at_3 + value: 17.287 + - type: recall_at_5 + value: 27.179 + task: + type: Retrieval + - dataset: + config: default + name: MTEB FeedbackQARetrieval (default) + revision: 1ee1cd0 + split: test + type: lt2c/fqa + metrics: + - type: main_score + value: 27.159 + - type: map_at_1 + value: 27.159 + - type: map_at_10 + value: 36.533 + - type: map_at_100 + value: 37.653999999999996 + - type: map_at_1000 + value: 37.719 + - type: map_at_20 + value: 37.19 + - type: map_at_3 + value: 33.650999999999996 + - type: map_at_5 + value: 35.338 + - type: mrr_at_1 + value: 27.158634538152608 + - type: mrr_at_10 + value: 36.53293730477466 + - type: mrr_at_100 + value: 37.65359357224721 + - type: mrr_at_1000 + value: 37.71854110065475 + - type: mrr_at_20 + value: 37.18989930977979 + - type: mrr_at_3 + value: 33.65127175368139 + - type: mrr_at_5 + value: 35.33801874163323 + - type: ndcg_at_1 + value: 27.159 + - type: ndcg_at_10 + value: 41.756 + - type: ndcg_at_100 + value: 47.424 + - type: ndcg_at_1000 + value: 49.128 + - type: ndcg_at_20 + value: 44.111 + - type: ndcg_at_3 + value: 35.798 + - type: ndcg_at_5 + value: 38.827 + - type: precision_at_1 + value: 27.159 + - type: precision_at_10 + value: 5.848 + - type: precision_at_100 + value: 0.855 + - type: precision_at_1000 + value: 0.099 + - type: precision_at_20 + value: 3.386 + - type: precision_at_3 + value: 14.005999999999998 + - type: precision_at_5 + value: 9.869 + - type: recall_at_1 + value: 27.159 + - type: recall_at_10 + value: 58.484 + - type: recall_at_100 + value: 85.492 + - type: recall_at_1000 + value: 98.845 + - type: recall_at_20 + value: 67.721 + - type: recall_at_3 + value: 42.018 + - type: recall_at_5 + value: 49.347 + task: + type: Retrieval + - dataset: + config: default + name: MTEB FiQA2018 (default) + revision: 27a168819829fe9bcd655c2df245fb19452e8e06 + split: dev + type: mteb/fiqa + metrics: + - type: main_score + value: 19.159000000000002 + - type: map_at_1 + value: 9.577 + - type: map_at_10 + value: 14.629 + - type: map_at_100 + value: 15.926000000000002 + - type: map_at_1000 + value: 16.099 + - type: map_at_20 + value: 15.204 + - type: map_at_3 + value: 12.788 + - type: map_at_5 + value: 13.821 + - type: mrr_at_1 + value: 16.2 + - type: mrr_at_10 + value: 22.73920634920635 + - type: mrr_at_100 + value: 23.811442879807622 + - type: mrr_at_1000 + value: 23.904303078394555 + - type: mrr_at_20 + value: 23.37503016505339 + - type: mrr_at_3 + value: 20.733333333333327 + - type: mrr_at_5 + value: 21.873333333333328 + - type: ndcg_at_1 + value: 16.2 + - type: ndcg_at_10 + value: 19.159000000000002 + - type: ndcg_at_100 + value: 25.229000000000003 + - type: ndcg_at_1000 + value: 29.294999999999998 + - type: ndcg_at_20 + value: 21.109 + - type: ndcg_at_3 + value: 16.481 + - type: ndcg_at_5 + value: 17.488999999999997 + - type: precision_at_1 + value: 16.2 + - type: precision_at_10 + value: 5.04 + - type: precision_at_100 + value: 1.124 + - type: precision_at_1000 + value: 0.179 + - type: precision_at_20 + value: 3.34 + - type: precision_at_3 + value: 10.133000000000001 + - type: precision_at_5 + value: 7.76 + - type: recall_at_1 + value: 9.577 + - type: recall_at_10 + value: 24.362000000000002 + - type: recall_at_100 + value: 48.222 + - type: recall_at_1000 + value: 74.358 + - type: recall_at_20 + value: 30.465999999999998 + - type: recall_at_3 + value: 16.057 + - type: recall_at_5 + value: 19.516 + task: + type: Retrieval + - dataset: + config: default + name: MTEB FiQA2018 (default) + revision: 27a168819829fe9bcd655c2df245fb19452e8e06 + split: test + type: mteb/fiqa + metrics: + - type: main_score + value: 19.986 + - type: map_at_1 + value: 9.157 + - type: map_at_10 + value: 14.877 + - type: map_at_100 + value: 16.185 + - type: map_at_1000 + value: 16.366 + - type: map_at_20 + value: 15.551 + - type: map_at_3 + value: 12.574 + - type: map_at_5 + value: 13.694999999999999 + - type: mrr_at_1 + value: 16.51234567901235 + - type: mrr_at_10 + value: 23.66169410150891 + - type: mrr_at_100 + value: 24.6092001023614 + - type: mrr_at_1000 + value: 24.695544929151346 + - type: mrr_at_20 + value: 24.17998019538123 + - type: mrr_at_3 + value: 20.98765432098765 + - type: mrr_at_5 + value: 22.32253086419753 + - type: ndcg_at_1 + value: 16.512 + - type: ndcg_at_10 + value: 19.986 + - type: ndcg_at_100 + value: 25.840999999999998 + - type: ndcg_at_1000 + value: 29.999 + - type: ndcg_at_20 + value: 22.047 + - type: ndcg_at_3 + value: 16.401 + - type: ndcg_at_5 + value: 17.552 + - type: precision_at_1 + value: 16.512 + - type: precision_at_10 + value: 5.602 + - type: precision_at_100 + value: 1.171 + - type: precision_at_1000 + value: 0.19 + - type: precision_at_20 + value: 3.588 + - type: precision_at_3 + value: 10.545 + - type: precision_at_5 + value: 8.025 + - type: recall_at_1 + value: 9.157 + - type: recall_at_10 + value: 26.253999999999998 + - type: recall_at_100 + value: 48.175000000000004 + - type: recall_at_1000 + value: 74.236 + - type: recall_at_20 + value: 32.786 + - type: recall_at_3 + value: 15.631999999999998 + - type: recall_at_5 + value: 19.608 + task: + type: Retrieval + - dataset: + config: default + name: MTEB FiQA2018 (default) + revision: 27a168819829fe9bcd655c2df245fb19452e8e06 + split: train + type: mteb/fiqa + metrics: + - type: main_score + value: 17.858 + - type: map_at_1 + value: 8.012 + - type: map_at_10 + value: 13.209000000000001 + - type: map_at_100 + value: 14.477 + - type: map_at_1000 + value: 14.671000000000001 + - type: map_at_20 + value: 13.864 + - type: map_at_3 + value: 11.218 + - type: map_at_5 + value: 12.239 + - type: mrr_at_1 + value: 15.363636363636363 + - type: mrr_at_10 + value: 21.62342712842711 + - type: mrr_at_100 + value: 22.63936235019517 + - type: mrr_at_1000 + value: 22.735838806851703 + - type: mrr_at_20 + value: 22.175914005467874 + - type: mrr_at_3 + value: 19.448484848484895 + - type: mrr_at_5 + value: 20.589393939394007 + - type: ndcg_at_1 + value: 15.364 + - type: ndcg_at_10 + value: 17.858 + - type: ndcg_at_100 + value: 23.794999999999998 + - type: ndcg_at_1000 + value: 28.17 + - type: ndcg_at_20 + value: 19.901 + - type: ndcg_at_3 + value: 14.888000000000002 + - type: ndcg_at_5 + value: 15.926000000000002 + - type: precision_at_1 + value: 15.364 + - type: precision_at_10 + value: 5.024 + - type: precision_at_100 + value: 1.087 + - type: precision_at_1000 + value: 0.184 + - type: precision_at_20 + value: 3.293 + - type: precision_at_3 + value: 9.751999999999999 + - type: precision_at_5 + value: 7.465 + - type: recall_at_1 + value: 8.012 + - type: recall_at_10 + value: 23.233999999999998 + - type: recall_at_100 + value: 46.623999999999995 + - type: recall_at_1000 + value: 74.092 + - type: recall_at_20 + value: 29.854000000000003 + - type: recall_at_3 + value: 14.216000000000001 + - type: recall_at_5 + value: 17.713 + task: + type: Retrieval + - dataset: + config: default + name: MTEB HellaSwag (default) + revision: a5c990205e017d10761197ccab3000936689c3ae + split: test + type: RAR-b/hellaswag + metrics: + - type: main_score + value: 16.377 + - type: map_at_1 + value: 8.474 + - type: map_at_10 + value: 13.479 + - type: map_at_100 + value: 14.296000000000001 + - type: map_at_1000 + value: 14.393 + - type: map_at_20 + value: 13.905000000000001 + - type: map_at_3 + value: 11.878 + - type: map_at_5 + value: 12.733 + - type: mrr_at_1 + value: 8.474407488548099 + - type: mrr_at_10 + value: 13.47926012335491 + - type: mrr_at_100 + value: 14.296018190032331 + - type: mrr_at_1000 + value: 14.39320635735857 + - type: mrr_at_20 + value: 13.905283977590932 + - type: mrr_at_3 + value: 11.878443869083188 + - type: mrr_at_5 + value: 12.733353249684685 + - type: ndcg_at_1 + value: 8.474 + - type: ndcg_at_10 + value: 16.377 + - type: ndcg_at_100 + value: 20.878 + - type: ndcg_at_1000 + value: 23.878 + - type: ndcg_at_20 + value: 17.93 + - type: ndcg_at_3 + value: 13.014999999999999 + - type: ndcg_at_5 + value: 14.557999999999998 + - type: precision_at_1 + value: 8.474 + - type: precision_at_10 + value: 2.571 + - type: precision_at_100 + value: 0.48 + - type: precision_at_1000 + value: 0.073 + - type: precision_at_20 + value: 1.593 + - type: precision_at_3 + value: 5.437 + - type: precision_at_5 + value: 4.013 + - type: recall_at_1 + value: 8.474 + - type: recall_at_10 + value: 25.712000000000003 + - type: recall_at_100 + value: 48.008 + - type: recall_at_1000 + value: 72.52499999999999 + - type: recall_at_20 + value: 31.856 + - type: recall_at_3 + value: 16.311 + - type: recall_at_5 + value: 20.066 + task: + type: Retrieval + - dataset: + config: default + name: MTEB HotpotQA (default) + revision: ab518f4d6fcca38d87c25209f94beba119d02014 + split: dev + type: mteb/hotpotqa + metrics: + - type: main_score + value: 49.524 + - type: map_at_1 + value: 27.583999999999996 + - type: map_at_10 + value: 40.455000000000005 + - type: map_at_100 + value: 41.567 + - type: map_at_1000 + value: 41.665 + - type: map_at_20 + value: 41.099000000000004 + - type: map_at_3 + value: 37.438 + - type: map_at_5 + value: 39.202999999999996 + - type: mrr_at_1 + value: 55.16798237561961 + - type: mrr_at_10 + value: 63.83496376336482 + - type: mrr_at_100 + value: 64.32844309604842 + - type: mrr_at_1000 + value: 64.35048997347738 + - type: mrr_at_20 + value: 64.14047945884145 + - type: mrr_at_3 + value: 61.93929380086919 + - type: mrr_at_5 + value: 63.0802888440121 + - type: ndcg_at_1 + value: 55.16799999999999 + - type: ndcg_at_10 + value: 49.524 + - type: ndcg_at_100 + value: 53.879 + - type: ndcg_at_1000 + value: 55.911 + - type: ndcg_at_20 + value: 51.31 + - type: ndcg_at_3 + value: 44.527 + - type: ndcg_at_5 + value: 47.102 + - type: precision_at_1 + value: 55.16799999999999 + - type: precision_at_10 + value: 10.718 + - type: precision_at_100 + value: 1.4160000000000001 + - type: precision_at_1000 + value: 0.169 + - type: precision_at_20 + value: 5.935 + - type: precision_at_3 + value: 28.26 + - type: precision_at_5 + value: 18.990000000000002 + - type: recall_at_1 + value: 27.583999999999996 + - type: recall_at_10 + value: 53.589 + - type: recall_at_100 + value: 70.782 + - type: recall_at_1000 + value: 84.276 + - type: recall_at_20 + value: 59.354 + - type: recall_at_3 + value: 42.39 + - type: recall_at_5 + value: 47.476 + task: + type: Retrieval + - dataset: + config: default + name: MTEB HotpotQAHardNegatives (default) + revision: 617612fa63afcb60e3b134bed8b7216a99707c37 + split: test + type: mteb/HotpotQA_test_top_250_only_w_correct-v2 + metrics: + - type: main_score + value: 50.415 + - type: map_at_1 + value: 26.8 + - type: map_at_10 + value: 40.503 + - type: map_at_100 + value: 42.092 + - type: map_at_1000 + value: 42.198 + - type: map_at_20 + value: 41.394999999999996 + - type: map_at_3 + value: 36.75 + - type: map_at_5 + value: 38.945 + - type: mrr_at_1 + value: 53.6 + - type: mrr_at_10 + value: 63.90658730158732 + - type: mrr_at_100 + value: 64.46665914282829 + - type: mrr_at_1000 + value: 64.4775151418674 + - type: mrr_at_20 + value: 64.20681999545006 + - type: mrr_at_3 + value: 61.40000000000001 + - type: mrr_at_5 + value: 62.98000000000005 + - type: ndcg_at_1 + value: 53.6 + - type: ndcg_at_10 + value: 50.415 + - type: ndcg_at_100 + value: 56.48800000000001 + - type: ndcg_at_1000 + value: 58.388 + - type: ndcg_at_20 + value: 52.68000000000001 + - type: ndcg_at_3 + value: 44.165 + - type: ndcg_at_5 + value: 47.429 + - type: precision_at_1 + value: 53.6 + - type: precision_at_10 + value: 11.31 + - type: precision_at_100 + value: 1.614 + - type: precision_at_1000 + value: 0.186 + - type: precision_at_20 + value: 6.375 + - type: precision_at_3 + value: 28.433000000000003 + - type: precision_at_5 + value: 19.62 + - type: recall_at_1 + value: 26.8 + - type: recall_at_10 + value: 56.55 + - type: recall_at_100 + value: 80.7 + - type: recall_at_1000 + value: 93.05 + - type: recall_at_20 + value: 63.74999999999999 + - type: recall_at_3 + value: 42.65 + - type: recall_at_5 + value: 49.05 + task: + type: Retrieval + - dataset: + config: default + name: MTEB LEMBNarrativeQARetrieval (default) + revision: 6e346642246bfb4928c560ee08640dc84d074e8c + split: test + type: dwzhu/LongEmbed + metrics: + - type: main_score + value: 47.11 + - type: map_at_1 + value: 38.176 + - type: map_at_10 + value: 44.11 + - type: map_at_100 + value: 44.885999999999996 + - type: map_at_1000 + value: 45.005 + - type: map_at_20 + value: 44.486 + - type: map_at_3 + value: 42.669000000000004 + - type: map_at_5 + value: 43.441 + - type: mrr_at_1 + value: 38.17590200019141 + - type: mrr_at_10 + value: 44.109989260003694 + - type: mrr_at_100 + value: 44.886475970293596 + - type: mrr_at_1000 + value: 45.00541901614199 + - type: mrr_at_20 + value: 44.48565175776022 + - type: mrr_at_3 + value: 42.66915494305687 + - type: mrr_at_5 + value: 43.44052062398311 + - type: ndcg_at_1 + value: 38.176 + - type: ndcg_at_10 + value: 47.11 + - type: ndcg_at_100 + value: 51.644999999999996 + - type: ndcg_at_1000 + value: 54.366 + - type: ndcg_at_20 + value: 48.475 + - type: ndcg_at_3 + value: 44.101 + - type: ndcg_at_5 + value: 45.494 + - type: precision_at_1 + value: 38.176 + - type: precision_at_10 + value: 5.6610000000000005 + - type: precision_at_100 + value: 0.796 + - type: precision_at_1000 + value: 0.1 + - type: precision_at_20 + value: 3.1 + - type: precision_at_3 + value: 16.078 + - type: precision_at_5 + value: 10.324 + - type: recall_at_1 + value: 38.176 + - type: recall_at_10 + value: 56.608000000000004 + - type: recall_at_100 + value: 79.644 + - type: recall_at_1000 + value: 100.0 + - type: recall_at_20 + value: 61.995999999999995 + - type: recall_at_3 + value: 48.234 + - type: recall_at_5 + value: 51.622 + task: + type: Retrieval + - dataset: + config: default + name: MTEB LEMBNeedleRetrieval (default) + revision: 6e346642246bfb4928c560ee08640dc84d074e8c + split: test_1024 + type: dwzhu/LongEmbed + metrics: + - type: main_score + value: 90.0 + - type: map_at_1 + value: 90.0 + - type: map_at_10 + value: 94.667 + - type: map_at_100 + value: 94.667 + - type: map_at_1000 + value: 94.667 + - type: map_at_20 + value: 94.667 + - type: map_at_3 + value: 94.667 + - type: map_at_5 + value: 94.667 + - type: mrr_at_1 + value: 90.0 + - type: mrr_at_10 + value: 94.66666666666666 + - type: mrr_at_100 + value: 94.66666666666666 + - type: mrr_at_1000 + value: 94.66666666666666 + - type: mrr_at_20 + value: 94.66666666666666 + - type: mrr_at_3 + value: 94.66666666666666 + - type: mrr_at_5 + value: 94.66666666666666 + - type: ndcg_at_1 + value: 90.0 + - type: ndcg_at_10 + value: 96.04700000000001 + - type: ndcg_at_100 + value: 96.04700000000001 + - type: ndcg_at_1000 + value: 96.04700000000001 + - type: ndcg_at_20 + value: 96.04700000000001 + - type: ndcg_at_3 + value: 96.04700000000001 + - type: ndcg_at_5 + value: 96.04700000000001 + - type: precision_at_1 + value: 90.0 + - type: precision_at_10 + value: 10.0 + - type: precision_at_100 + value: 1.0 + - type: precision_at_1000 + value: 0.1 + - type: precision_at_20 + value: 5.0 + - type: precision_at_3 + value: 33.333 + - type: precision_at_5 + value: 20.0 + - type: recall_at_1 + value: 90.0 + - type: recall_at_10 + value: 100.0 + - type: recall_at_100 + value: 100.0 + - type: recall_at_1000 + value: 100.0 + - type: recall_at_20 + value: 100.0 + - type: recall_at_3 + value: 100.0 + - type: recall_at_5 + value: 100.0 + task: + type: Retrieval + - dataset: + config: default + name: MTEB LEMBNeedleRetrieval (default) + revision: 6e346642246bfb4928c560ee08640dc84d074e8c + split: test_16384 + type: dwzhu/LongEmbed + metrics: + - type: main_score + value: 90.0 + - type: map_at_1 + value: 90.0 + - type: map_at_10 + value: 94.167 + - type: map_at_100 + value: 94.167 + - type: map_at_1000 + value: 94.167 + - type: map_at_20 + value: 94.167 + - type: map_at_3 + value: 93.667 + - type: map_at_5 + value: 94.167 + - type: mrr_at_1 + value: 90.0 + - type: mrr_at_10 + value: 94.16666666666666 + - type: mrr_at_100 + value: 94.16666666666666 + - type: mrr_at_1000 + value: 94.16666666666666 + - type: mrr_at_20 + value: 94.16666666666666 + - type: mrr_at_3 + value: 93.66666666666666 + - type: mrr_at_5 + value: 94.16666666666666 + - type: ndcg_at_1 + value: 90.0 + - type: ndcg_at_10 + value: 95.647 + - type: ndcg_at_100 + value: 95.647 + - type: ndcg_at_1000 + value: 95.647 + - type: ndcg_at_20 + value: 95.647 + - type: ndcg_at_3 + value: 94.786 + - type: ndcg_at_5 + value: 95.647 + - type: precision_at_1 + value: 90.0 + - type: precision_at_10 + value: 10.0 + - type: precision_at_100 + value: 1.0 + - type: precision_at_1000 + value: 0.1 + - type: precision_at_20 + value: 5.0 + - type: precision_at_3 + value: 32.667 + - type: precision_at_5 + value: 20.0 + - type: recall_at_1 + value: 90.0 + - type: recall_at_10 + value: 100.0 + - type: recall_at_100 + value: 100.0 + - type: recall_at_1000 + value: 100.0 + - type: recall_at_20 + value: 100.0 + - type: recall_at_3 + value: 98.0 + - type: recall_at_5 + value: 100.0 + task: + type: Retrieval + - dataset: + config: default + name: MTEB LEMBNeedleRetrieval (default) + revision: 6e346642246bfb4928c560ee08640dc84d074e8c + split: test_2048 + type: dwzhu/LongEmbed + metrics: + - type: main_score + value: 92.0 + - type: map_at_1 + value: 92.0 + - type: map_at_10 + value: 95.667 + - type: map_at_100 + value: 95.667 + - type: map_at_1000 + value: 95.667 + - type: map_at_20 + value: 95.667 + - type: map_at_3 + value: 95.667 + - type: map_at_5 + value: 95.667 + - type: mrr_at_1 + value: 92.0 + - type: mrr_at_10 + value: 95.66666666666666 + - type: mrr_at_100 + value: 95.66666666666666 + - type: mrr_at_1000 + value: 95.66666666666666 + - type: mrr_at_20 + value: 95.66666666666666 + - type: mrr_at_3 + value: 95.66666666666666 + - type: mrr_at_5 + value: 95.66666666666666 + - type: ndcg_at_1 + value: 92.0 + - type: ndcg_at_10 + value: 96.786 + - type: ndcg_at_100 + value: 96.786 + - type: ndcg_at_1000 + value: 96.786 + - type: ndcg_at_20 + value: 96.786 + - type: ndcg_at_3 + value: 96.786 + - type: ndcg_at_5 + value: 96.786 + - type: precision_at_1 + value: 92.0 + - type: precision_at_10 + value: 10.0 + - type: precision_at_100 + value: 1.0 + - type: precision_at_1000 + value: 0.1 + - type: precision_at_20 + value: 5.0 + - type: precision_at_3 + value: 33.333 + - type: precision_at_5 + value: 20.0 + - type: recall_at_1 + value: 92.0 + - type: recall_at_10 + value: 100.0 + - type: recall_at_100 + value: 100.0 + - type: recall_at_1000 + value: 100.0 + - type: recall_at_20 + value: 100.0 + - type: recall_at_3 + value: 100.0 + - type: recall_at_5 + value: 100.0 + task: + type: Retrieval + - dataset: + config: default + name: MTEB LEMBNeedleRetrieval (default) + revision: 6e346642246bfb4928c560ee08640dc84d074e8c + split: test_256 + type: dwzhu/LongEmbed + metrics: + - type: main_score + value: 90.0 + - type: map_at_1 + value: 90.0 + - type: map_at_10 + value: 95.0 + - type: map_at_100 + value: 95.0 + - type: map_at_1000 + value: 95.0 + - type: map_at_20 + value: 95.0 + - type: map_at_3 + value: 95.0 + - type: map_at_5 + value: 95.0 + - type: mrr_at_1 + value: 90.0 + - type: mrr_at_10 + value: 95.0 + - type: mrr_at_100 + value: 95.0 + - type: mrr_at_1000 + value: 95.0 + - type: mrr_at_20 + value: 95.0 + - type: mrr_at_3 + value: 95.0 + - type: mrr_at_5 + value: 95.0 + - type: ndcg_at_1 + value: 90.0 + - type: ndcg_at_10 + value: 96.309 + - type: ndcg_at_100 + value: 96.309 + - type: ndcg_at_1000 + value: 96.309 + - type: ndcg_at_20 + value: 96.309 + - type: ndcg_at_3 + value: 96.309 + - type: ndcg_at_5 + value: 96.309 + - type: precision_at_1 + value: 90.0 + - type: precision_at_10 + value: 10.0 + - type: precision_at_100 + value: 1.0 + - type: precision_at_1000 + value: 0.1 + - type: precision_at_20 + value: 5.0 + - type: precision_at_3 + value: 33.333 + - type: precision_at_5 + value: 20.0 + - type: recall_at_1 + value: 90.0 + - type: recall_at_10 + value: 100.0 + - type: recall_at_100 + value: 100.0 + - type: recall_at_1000 + value: 100.0 + - type: recall_at_20 + value: 100.0 + - type: recall_at_3 + value: 100.0 + - type: recall_at_5 + value: 100.0 + task: + type: Retrieval + - dataset: + config: default + name: MTEB LEMBNeedleRetrieval (default) + revision: 6e346642246bfb4928c560ee08640dc84d074e8c + split: test_32768 + type: dwzhu/LongEmbed + metrics: + - type: main_score + value: 84.0 + - type: map_at_1 + value: 84.0 + - type: map_at_10 + value: 91.167 + - type: map_at_100 + value: 91.167 + - type: map_at_1000 + value: 91.167 + - type: map_at_20 + value: 91.167 + - type: map_at_3 + value: 90.667 + - type: map_at_5 + value: 91.167 + - type: mrr_at_1 + value: 84.0 + - type: mrr_at_10 + value: 91.16666666666666 + - type: mrr_at_100 + value: 91.16666666666666 + - type: mrr_at_1000 + value: 91.16666666666666 + - type: mrr_at_20 + value: 91.16666666666666 + - type: mrr_at_3 + value: 90.66666666666666 + - type: mrr_at_5 + value: 91.16666666666666 + - type: ndcg_at_1 + value: 84.0 + - type: ndcg_at_10 + value: 93.43299999999999 + - type: ndcg_at_100 + value: 93.43299999999999 + - type: ndcg_at_1000 + value: 93.43299999999999 + - type: ndcg_at_20 + value: 93.43299999999999 + - type: ndcg_at_3 + value: 92.571 + - type: ndcg_at_5 + value: 93.43299999999999 + - type: precision_at_1 + value: 84.0 + - type: precision_at_10 + value: 10.0 + - type: precision_at_100 + value: 1.0 + - type: precision_at_1000 + value: 0.1 + - type: precision_at_20 + value: 5.0 + - type: precision_at_3 + value: 32.667 + - type: precision_at_5 + value: 20.0 + - type: recall_at_1 + value: 84.0 + - type: recall_at_10 + value: 100.0 + - type: recall_at_100 + value: 100.0 + - type: recall_at_1000 + value: 100.0 + - type: recall_at_20 + value: 100.0 + - type: recall_at_3 + value: 98.0 + - type: recall_at_5 + value: 100.0 + task: + type: Retrieval + - dataset: + config: default + name: MTEB LEMBNeedleRetrieval (default) + revision: 6e346642246bfb4928c560ee08640dc84d074e8c + split: test_4096 + type: dwzhu/LongEmbed + metrics: + - type: main_score + value: 92.0 + - type: map_at_1 + value: 92.0 + - type: map_at_10 + value: 96.0 + - type: map_at_100 + value: 96.0 + - type: map_at_1000 + value: 96.0 + - type: map_at_20 + value: 96.0 + - type: map_at_3 + value: 96.0 + - type: map_at_5 + value: 96.0 + - type: mrr_at_1 + value: 92.0 + - type: mrr_at_10 + value: 96.0 + - type: mrr_at_100 + value: 96.0 + - type: mrr_at_1000 + value: 96.0 + - type: mrr_at_20 + value: 96.0 + - type: mrr_at_3 + value: 96.0 + - type: mrr_at_5 + value: 96.0 + - type: ndcg_at_1 + value: 92.0 + - type: ndcg_at_10 + value: 97.04700000000001 + - type: ndcg_at_100 + value: 97.04700000000001 + - type: ndcg_at_1000 + value: 97.04700000000001 + - type: ndcg_at_20 + value: 97.04700000000001 + - type: ndcg_at_3 + value: 97.04700000000001 + - type: ndcg_at_5 + value: 97.04700000000001 + - type: precision_at_1 + value: 92.0 + - type: precision_at_10 + value: 10.0 + - type: precision_at_100 + value: 1.0 + - type: precision_at_1000 + value: 0.1 + - type: precision_at_20 + value: 5.0 + - type: precision_at_3 + value: 33.333 + - type: precision_at_5 + value: 20.0 + - type: recall_at_1 + value: 92.0 + - type: recall_at_10 + value: 100.0 + - type: recall_at_100 + value: 100.0 + - type: recall_at_1000 + value: 100.0 + - type: recall_at_20 + value: 100.0 + - type: recall_at_3 + value: 100.0 + - type: recall_at_5 + value: 100.0 + task: + type: Retrieval + - dataset: + config: default + name: MTEB LEMBNeedleRetrieval (default) + revision: 6e346642246bfb4928c560ee08640dc84d074e8c + split: test_512 + type: dwzhu/LongEmbed + metrics: + - type: main_score + value: 94.0 + - type: map_at_1 + value: 94.0 + - type: map_at_10 + value: 97.0 + - type: map_at_100 + value: 97.0 + - type: map_at_1000 + value: 97.0 + - type: map_at_20 + value: 97.0 + - type: map_at_3 + value: 97.0 + - type: map_at_5 + value: 97.0 + - type: mrr_at_1 + value: 94.0 + - type: mrr_at_10 + value: 97.0 + - type: mrr_at_100 + value: 97.0 + - type: mrr_at_1000 + value: 97.0 + - type: mrr_at_20 + value: 97.0 + - type: mrr_at_3 + value: 97.0 + - type: mrr_at_5 + value: 97.0 + - type: ndcg_at_1 + value: 94.0 + - type: ndcg_at_10 + value: 97.786 + - type: ndcg_at_100 + value: 97.786 + - type: ndcg_at_1000 + value: 97.786 + - type: ndcg_at_20 + value: 97.786 + - type: ndcg_at_3 + value: 97.786 + - type: ndcg_at_5 + value: 97.786 + - type: precision_at_1 + value: 94.0 + - type: precision_at_10 + value: 10.0 + - type: precision_at_100 + value: 1.0 + - type: precision_at_1000 + value: 0.1 + - type: precision_at_20 + value: 5.0 + - type: precision_at_3 + value: 33.333 + - type: precision_at_5 + value: 20.0 + - type: recall_at_1 + value: 94.0 + - type: recall_at_10 + value: 100.0 + - type: recall_at_100 + value: 100.0 + - type: recall_at_1000 + value: 100.0 + - type: recall_at_20 + value: 100.0 + - type: recall_at_3 + value: 100.0 + - type: recall_at_5 + value: 100.0 + task: + type: Retrieval + - dataset: + config: default + name: MTEB LEMBNeedleRetrieval (default) + revision: 6e346642246bfb4928c560ee08640dc84d074e8c + split: test_8192 + type: dwzhu/LongEmbed + metrics: + - type: main_score + value: 88.0 + - type: map_at_1 + value: 88.0 + - type: map_at_10 + value: 93.167 + - type: map_at_100 + value: 93.167 + - type: map_at_1000 + value: 93.167 + - type: map_at_20 + value: 93.167 + - type: map_at_3 + value: 92.667 + - type: map_at_5 + value: 93.167 + - type: mrr_at_1 + value: 88.0 + - type: mrr_at_10 + value: 93.16666666666666 + - type: mrr_at_100 + value: 93.16666666666666 + - type: mrr_at_1000 + value: 93.16666666666666 + - type: mrr_at_20 + value: 93.16666666666666 + - type: mrr_at_3 + value: 92.66666666666666 + - type: mrr_at_5 + value: 93.16666666666666 + - type: ndcg_at_1 + value: 88.0 + - type: ndcg_at_10 + value: 94.90899999999999 + - type: ndcg_at_100 + value: 94.90899999999999 + - type: ndcg_at_1000 + value: 94.90899999999999 + - type: ndcg_at_20 + value: 94.90899999999999 + - type: ndcg_at_3 + value: 94.047 + - type: ndcg_at_5 + value: 94.90899999999999 + - type: precision_at_1 + value: 88.0 + - type: precision_at_10 + value: 10.0 + - type: precision_at_100 + value: 1.0 + - type: precision_at_1000 + value: 0.1 + - type: precision_at_20 + value: 5.0 + - type: precision_at_3 + value: 32.667 + - type: precision_at_5 + value: 20.0 + - type: recall_at_1 + value: 88.0 + - type: recall_at_10 + value: 100.0 + - type: recall_at_100 + value: 100.0 + - type: recall_at_1000 + value: 100.0 + - type: recall_at_20 + value: 100.0 + - type: recall_at_3 + value: 98.0 + - type: recall_at_5 + value: 100.0 + task: + type: Retrieval + - dataset: + config: default + name: MTEB LEMBPasskeyRetrieval (default) + revision: 6e346642246bfb4928c560ee08640dc84d074e8c + split: test_1024 + type: dwzhu/LongEmbed + metrics: + - type: main_score + value: 100.0 + - type: map_at_1 + value: 100.0 + - type: map_at_10 + value: 100.0 + - type: map_at_100 + value: 100.0 + - type: map_at_1000 + value: 100.0 + - type: map_at_20 + value: 100.0 + - type: map_at_3 + value: 100.0 + - type: map_at_5 + value: 100.0 + - type: mrr_at_1 + value: 100.0 + - type: mrr_at_10 + value: 100.0 + - type: mrr_at_100 + value: 100.0 + - type: mrr_at_1000 + value: 100.0 + - type: mrr_at_20 + value: 100.0 + - type: mrr_at_3 + value: 100.0 + - type: mrr_at_5 + value: 100.0 + - type: ndcg_at_1 + value: 100.0 + - type: ndcg_at_10 + value: 100.0 + - type: ndcg_at_100 + value: 100.0 + - type: ndcg_at_1000 + value: 100.0 + - type: ndcg_at_20 + value: 100.0 + - type: ndcg_at_3 + value: 100.0 + - type: ndcg_at_5 + value: 100.0 + - type: precision_at_1 + value: 100.0 + - type: precision_at_10 + value: 10.0 + - type: precision_at_100 + value: 1.0 + - type: precision_at_1000 + value: 0.1 + - type: precision_at_20 + value: 5.0 + - type: precision_at_3 + value: 33.333 + - type: precision_at_5 + value: 20.0 + - type: recall_at_1 + value: 100.0 + - type: recall_at_10 + value: 100.0 + - type: recall_at_100 + value: 100.0 + - type: recall_at_1000 + value: 100.0 + - type: recall_at_20 + value: 100.0 + - type: recall_at_3 + value: 100.0 + - type: recall_at_5 + value: 100.0 + task: + type: Retrieval + - dataset: + config: default + name: MTEB LEMBPasskeyRetrieval (default) + revision: 6e346642246bfb4928c560ee08640dc84d074e8c + split: test_16384 + type: dwzhu/LongEmbed + metrics: + - type: main_score + value: 100.0 + - type: map_at_1 + value: 100.0 + - type: map_at_10 + value: 100.0 + - type: map_at_100 + value: 100.0 + - type: map_at_1000 + value: 100.0 + - type: map_at_20 + value: 100.0 + - type: map_at_3 + value: 100.0 + - type: map_at_5 + value: 100.0 + - type: mrr_at_1 + value: 100.0 + - type: mrr_at_10 + value: 100.0 + - type: mrr_at_100 + value: 100.0 + - type: mrr_at_1000 + value: 100.0 + - type: mrr_at_20 + value: 100.0 + - type: mrr_at_3 + value: 100.0 + - type: mrr_at_5 + value: 100.0 + - type: ndcg_at_1 + value: 100.0 + - type: ndcg_at_10 + value: 100.0 + - type: ndcg_at_100 + value: 100.0 + - type: ndcg_at_1000 + value: 100.0 + - type: ndcg_at_20 + value: 100.0 + - type: ndcg_at_3 + value: 100.0 + - type: ndcg_at_5 + value: 100.0 + - type: precision_at_1 + value: 100.0 + - type: precision_at_10 + value: 10.0 + - type: precision_at_100 + value: 1.0 + - type: precision_at_1000 + value: 0.1 + - type: precision_at_20 + value: 5.0 + - type: precision_at_3 + value: 33.333 + - type: precision_at_5 + value: 20.0 + - type: recall_at_1 + value: 100.0 + - type: recall_at_10 + value: 100.0 + - type: recall_at_100 + value: 100.0 + - type: recall_at_1000 + value: 100.0 + - type: recall_at_20 + value: 100.0 + - type: recall_at_3 + value: 100.0 + - type: recall_at_5 + value: 100.0 + task: + type: Retrieval + - dataset: + config: default + name: MTEB LEMBPasskeyRetrieval (default) + revision: 6e346642246bfb4928c560ee08640dc84d074e8c + split: test_2048 + type: dwzhu/LongEmbed + metrics: + - type: main_score + value: 100.0 + - type: map_at_1 + value: 100.0 + - type: map_at_10 + value: 100.0 + - type: map_at_100 + value: 100.0 + - type: map_at_1000 + value: 100.0 + - type: map_at_20 + value: 100.0 + - type: map_at_3 + value: 100.0 + - type: map_at_5 + value: 100.0 + - type: mrr_at_1 + value: 100.0 + - type: mrr_at_10 + value: 100.0 + - type: mrr_at_100 + value: 100.0 + - type: mrr_at_1000 + value: 100.0 + - type: mrr_at_20 + value: 100.0 + - type: mrr_at_3 + value: 100.0 + - type: mrr_at_5 + value: 100.0 + - type: ndcg_at_1 + value: 100.0 + - type: ndcg_at_10 + value: 100.0 + - type: ndcg_at_100 + value: 100.0 + - type: ndcg_at_1000 + value: 100.0 + - type: ndcg_at_20 + value: 100.0 + - type: ndcg_at_3 + value: 100.0 + - type: ndcg_at_5 + value: 100.0 + - type: precision_at_1 + value: 100.0 + - type: precision_at_10 + value: 10.0 + - type: precision_at_100 + value: 1.0 + - type: precision_at_1000 + value: 0.1 + - type: precision_at_20 + value: 5.0 + - type: precision_at_3 + value: 33.333 + - type: precision_at_5 + value: 20.0 + - type: recall_at_1 + value: 100.0 + - type: recall_at_10 + value: 100.0 + - type: recall_at_100 + value: 100.0 + - type: recall_at_1000 + value: 100.0 + - type: recall_at_20 + value: 100.0 + - type: recall_at_3 + value: 100.0 + - type: recall_at_5 + value: 100.0 + task: + type: Retrieval + - dataset: + config: default + name: MTEB LEMBPasskeyRetrieval (default) + revision: 6e346642246bfb4928c560ee08640dc84d074e8c + split: test_256 + type: dwzhu/LongEmbed + metrics: + - type: main_score + value: 100.0 + - type: map_at_1 + value: 100.0 + - type: map_at_10 + value: 100.0 + - type: map_at_100 + value: 100.0 + - type: map_at_1000 + value: 100.0 + - type: map_at_20 + value: 100.0 + - type: map_at_3 + value: 100.0 + - type: map_at_5 + value: 100.0 + - type: mrr_at_1 + value: 100.0 + - type: mrr_at_10 + value: 100.0 + - type: mrr_at_100 + value: 100.0 + - type: mrr_at_1000 + value: 100.0 + - type: mrr_at_20 + value: 100.0 + - type: mrr_at_3 + value: 100.0 + - type: mrr_at_5 + value: 100.0 + - type: ndcg_at_1 + value: 100.0 + - type: ndcg_at_10 + value: 100.0 + - type: ndcg_at_100 + value: 100.0 + - type: ndcg_at_1000 + value: 100.0 + - type: ndcg_at_20 + value: 100.0 + - type: ndcg_at_3 + value: 100.0 + - type: ndcg_at_5 + value: 100.0 + - type: precision_at_1 + value: 100.0 + - type: precision_at_10 + value: 10.0 + - type: precision_at_100 + value: 1.0 + - type: precision_at_1000 + value: 0.1 + - type: precision_at_20 + value: 5.0 + - type: precision_at_3 + value: 33.333 + - type: precision_at_5 + value: 20.0 + - type: recall_at_1 + value: 100.0 + - type: recall_at_10 + value: 100.0 + - type: recall_at_100 + value: 100.0 + - type: recall_at_1000 + value: 100.0 + - type: recall_at_20 + value: 100.0 + - type: recall_at_3 + value: 100.0 + - type: recall_at_5 + value: 100.0 + task: + type: Retrieval + - dataset: + config: default + name: MTEB LEMBPasskeyRetrieval (default) + revision: 6e346642246bfb4928c560ee08640dc84d074e8c + split: test_32768 + type: dwzhu/LongEmbed + metrics: + - type: main_score + value: 100.0 + - type: map_at_1 + value: 100.0 + - type: map_at_10 + value: 100.0 + - type: map_at_100 + value: 100.0 + - type: map_at_1000 + value: 100.0 + - type: map_at_20 + value: 100.0 + - type: map_at_3 + value: 100.0 + - type: map_at_5 + value: 100.0 + - type: mrr_at_1 + value: 100.0 + - type: mrr_at_10 + value: 100.0 + - type: mrr_at_100 + value: 100.0 + - type: mrr_at_1000 + value: 100.0 + - type: mrr_at_20 + value: 100.0 + - type: mrr_at_3 + value: 100.0 + - type: mrr_at_5 + value: 100.0 + - type: ndcg_at_1 + value: 100.0 + - type: ndcg_at_10 + value: 100.0 + - type: ndcg_at_100 + value: 100.0 + - type: ndcg_at_1000 + value: 100.0 + - type: ndcg_at_20 + value: 100.0 + - type: ndcg_at_3 + value: 100.0 + - type: ndcg_at_5 + value: 100.0 + - type: precision_at_1 + value: 100.0 + - type: precision_at_10 + value: 10.0 + - type: precision_at_100 + value: 1.0 + - type: precision_at_1000 + value: 0.1 + - type: precision_at_20 + value: 5.0 + - type: precision_at_3 + value: 33.333 + - type: precision_at_5 + value: 20.0 + - type: recall_at_1 + value: 100.0 + - type: recall_at_10 + value: 100.0 + - type: recall_at_100 + value: 100.0 + - type: recall_at_1000 + value: 100.0 + - type: recall_at_20 + value: 100.0 + - type: recall_at_3 + value: 100.0 + - type: recall_at_5 + value: 100.0 + task: + type: Retrieval + - dataset: + config: default + name: MTEB LEMBPasskeyRetrieval (default) + revision: 6e346642246bfb4928c560ee08640dc84d074e8c + split: test_4096 + type: dwzhu/LongEmbed + metrics: + - type: main_score + value: 100.0 + - type: map_at_1 + value: 100.0 + - type: map_at_10 + value: 100.0 + - type: map_at_100 + value: 100.0 + - type: map_at_1000 + value: 100.0 + - type: map_at_20 + value: 100.0 + - type: map_at_3 + value: 100.0 + - type: map_at_5 + value: 100.0 + - type: mrr_at_1 + value: 100.0 + - type: mrr_at_10 + value: 100.0 + - type: mrr_at_100 + value: 100.0 + - type: mrr_at_1000 + value: 100.0 + - type: mrr_at_20 + value: 100.0 + - type: mrr_at_3 + value: 100.0 + - type: mrr_at_5 + value: 100.0 + - type: ndcg_at_1 + value: 100.0 + - type: ndcg_at_10 + value: 100.0 + - type: ndcg_at_100 + value: 100.0 + - type: ndcg_at_1000 + value: 100.0 + - type: ndcg_at_20 + value: 100.0 + - type: ndcg_at_3 + value: 100.0 + - type: ndcg_at_5 + value: 100.0 + - type: precision_at_1 + value: 100.0 + - type: precision_at_10 + value: 10.0 + - type: precision_at_100 + value: 1.0 + - type: precision_at_1000 + value: 0.1 + - type: precision_at_20 + value: 5.0 + - type: precision_at_3 + value: 33.333 + - type: precision_at_5 + value: 20.0 + - type: recall_at_1 + value: 100.0 + - type: recall_at_10 + value: 100.0 + - type: recall_at_100 + value: 100.0 + - type: recall_at_1000 + value: 100.0 + - type: recall_at_20 + value: 100.0 + - type: recall_at_3 + value: 100.0 + - type: recall_at_5 + value: 100.0 + task: + type: Retrieval + - dataset: + config: default + name: MTEB LEMBPasskeyRetrieval (default) + revision: 6e346642246bfb4928c560ee08640dc84d074e8c + split: test_512 + type: dwzhu/LongEmbed + metrics: + - type: main_score + value: 100.0 + - type: map_at_1 + value: 100.0 + - type: map_at_10 + value: 100.0 + - type: map_at_100 + value: 100.0 + - type: map_at_1000 + value: 100.0 + - type: map_at_20 + value: 100.0 + - type: map_at_3 + value: 100.0 + - type: map_at_5 + value: 100.0 + - type: mrr_at_1 + value: 100.0 + - type: mrr_at_10 + value: 100.0 + - type: mrr_at_100 + value: 100.0 + - type: mrr_at_1000 + value: 100.0 + - type: mrr_at_20 + value: 100.0 + - type: mrr_at_3 + value: 100.0 + - type: mrr_at_5 + value: 100.0 + - type: ndcg_at_1 + value: 100.0 + - type: ndcg_at_10 + value: 100.0 + - type: ndcg_at_100 + value: 100.0 + - type: ndcg_at_1000 + value: 100.0 + - type: ndcg_at_20 + value: 100.0 + - type: ndcg_at_3 + value: 100.0 + - type: ndcg_at_5 + value: 100.0 + - type: precision_at_1 + value: 100.0 + - type: precision_at_10 + value: 10.0 + - type: precision_at_100 + value: 1.0 + - type: precision_at_1000 + value: 0.1 + - type: precision_at_20 + value: 5.0 + - type: precision_at_3 + value: 33.333 + - type: precision_at_5 + value: 20.0 + - type: recall_at_1 + value: 100.0 + - type: recall_at_10 + value: 100.0 + - type: recall_at_100 + value: 100.0 + - type: recall_at_1000 + value: 100.0 + - type: recall_at_20 + value: 100.0 + - type: recall_at_3 + value: 100.0 + - type: recall_at_5 + value: 100.0 + task: + type: Retrieval + - dataset: + config: default + name: MTEB LEMBPasskeyRetrieval (default) + revision: 6e346642246bfb4928c560ee08640dc84d074e8c + split: test_8192 + type: dwzhu/LongEmbed + metrics: + - type: main_score + value: 100.0 + - type: map_at_1 + value: 100.0 + - type: map_at_10 + value: 100.0 + - type: map_at_100 + value: 100.0 + - type: map_at_1000 + value: 100.0 + - type: map_at_20 + value: 100.0 + - type: map_at_3 + value: 100.0 + - type: map_at_5 + value: 100.0 + - type: mrr_at_1 + value: 100.0 + - type: mrr_at_10 + value: 100.0 + - type: mrr_at_100 + value: 100.0 + - type: mrr_at_1000 + value: 100.0 + - type: mrr_at_20 + value: 100.0 + - type: mrr_at_3 + value: 100.0 + - type: mrr_at_5 + value: 100.0 + - type: ndcg_at_1 + value: 100.0 + - type: ndcg_at_10 + value: 100.0 + - type: ndcg_at_100 + value: 100.0 + - type: ndcg_at_1000 + value: 100.0 + - type: ndcg_at_20 + value: 100.0 + - type: ndcg_at_3 + value: 100.0 + - type: ndcg_at_5 + value: 100.0 + - type: precision_at_1 + value: 100.0 + - type: precision_at_10 + value: 10.0 + - type: precision_at_100 + value: 1.0 + - type: precision_at_1000 + value: 0.1 + - type: precision_at_20 + value: 5.0 + - type: precision_at_3 + value: 33.333 + - type: precision_at_5 + value: 20.0 + - type: recall_at_1 + value: 100.0 + - type: recall_at_10 + value: 100.0 + - type: recall_at_100 + value: 100.0 + - type: recall_at_1000 + value: 100.0 + - type: recall_at_20 + value: 100.0 + - type: recall_at_3 + value: 100.0 + - type: recall_at_5 + value: 100.0 + task: + type: Retrieval + - dataset: + config: default + name: MTEB LEMBQMSumRetrieval (default) + revision: 6e346642246bfb4928c560ee08640dc84d074e8c + split: test + type: dwzhu/LongEmbed + metrics: + - type: main_score + value: 29.847 + - type: map_at_1 + value: 18.271 + - type: map_at_10 + value: 25.415 + - type: map_at_100 + value: 27.086 + - type: map_at_1000 + value: 27.134000000000004 + - type: map_at_20 + value: 26.135 + - type: map_at_3 + value: 22.659000000000002 + - type: map_at_5 + value: 24.198 + - type: mrr_at_1 + value: 18.271119842829076 + - type: mrr_at_10 + value: 25.414678641594147 + - type: mrr_at_100 + value: 27.086094547163714 + - type: mrr_at_1000 + value: 27.13383971528746 + - type: mrr_at_20 + value: 26.13474777243653 + - type: mrr_at_3 + value: 22.658808120497696 + - type: mrr_at_5 + value: 24.197773411918757 + - type: ndcg_at_1 + value: 18.271 + - type: ndcg_at_10 + value: 29.847 + - type: ndcg_at_100 + value: 39.669 + - type: ndcg_at_1000 + value: 40.528999999999996 + - type: ndcg_at_20 + value: 32.509 + - type: ndcg_at_3 + value: 24.151 + - type: ndcg_at_5 + value: 26.927 + - type: precision_at_1 + value: 18.271 + - type: precision_at_10 + value: 4.42 + - type: precision_at_100 + value: 0.9400000000000001 + - type: precision_at_1000 + value: 0.1 + - type: precision_at_20 + value: 2.741 + - type: precision_at_3 + value: 9.496 + - type: precision_at_5 + value: 7.045999999999999 + - type: recall_at_1 + value: 18.271 + - type: recall_at_10 + value: 44.204 + - type: recall_at_100 + value: 93.975 + - type: recall_at_1000 + value: 100.0 + - type: recall_at_20 + value: 54.813 + - type: recall_at_3 + value: 28.487000000000002 + - type: recall_at_5 + value: 35.232 + task: + type: Retrieval + - dataset: + config: default + name: MTEB LEMBSummScreenFDRetrieval (default) + revision: 6e346642246bfb4928c560ee08640dc84d074e8c + split: validation + type: dwzhu/LongEmbed + metrics: + - type: main_score + value: 90.19 + - type: map_at_1 + value: 81.548 + - type: map_at_10 + value: 87.57300000000001 + - type: map_at_100 + value: 87.703 + - type: map_at_1000 + value: 87.703 + - type: map_at_20 + value: 87.703 + - type: map_at_3 + value: 86.657 + - type: map_at_5 + value: 87.178 + - type: mrr_at_1 + value: 81.54761904761905 + - type: mrr_at_10 + value: 87.57345993953139 + - type: mrr_at_100 + value: 87.70306685222651 + - type: mrr_at_1000 + value: 87.70306685222651 + - type: mrr_at_20 + value: 87.70306685222651 + - type: mrr_at_3 + value: 86.65674603174602 + - type: mrr_at_5 + value: 87.17757936507935 + - type: ndcg_at_1 + value: 81.548 + - type: ndcg_at_10 + value: 90.19 + - type: ndcg_at_100 + value: 90.648 + - type: ndcg_at_1000 + value: 90.648 + - type: ndcg_at_20 + value: 90.648 + - type: ndcg_at_3 + value: 88.325 + - type: ndcg_at_5 + value: 89.286 + - type: precision_at_1 + value: 81.548 + - type: precision_at_10 + value: 9.821 + - type: precision_at_100 + value: 1.0 + - type: precision_at_1000 + value: 0.1 + - type: precision_at_20 + value: 5.0 + - type: precision_at_3 + value: 31.052000000000003 + - type: precision_at_5 + value: 19.107 + - type: recall_at_1 + value: 81.548 + - type: recall_at_10 + value: 98.214 + - type: recall_at_100 + value: 100.0 + - type: recall_at_1000 + value: 100.0 + - type: recall_at_20 + value: 100.0 + - type: recall_at_3 + value: 93.155 + - type: recall_at_5 + value: 95.536 + task: + type: Retrieval + - dataset: + config: default + name: MTEB LEMBWikimQARetrieval (default) + revision: 6e346642246bfb4928c560ee08640dc84d074e8c + split: test + type: dwzhu/LongEmbed + metrics: + - type: main_score + value: 77.203 + - type: map_at_1 + value: 69.333 + - type: map_at_10 + value: 74.787 + - type: map_at_100 + value: 75.348 + - type: map_at_1000 + value: 75.369 + - type: map_at_20 + value: 75.16000000000001 + - type: map_at_3 + value: 73.611 + - type: map_at_5 + value: 74.42800000000001 + - type: mrr_at_1 + value: 69.33333333333334 + - type: mrr_at_10 + value: 74.78743386243384 + - type: mrr_at_100 + value: 75.34827076805841 + - type: mrr_at_1000 + value: 75.36876455686495 + - type: mrr_at_20 + value: 75.16008204758204 + - type: mrr_at_3 + value: 73.61111111111111 + - type: mrr_at_5 + value: 74.42777777777778 + - type: ndcg_at_1 + value: 69.333 + - type: ndcg_at_10 + value: 77.203 + - type: ndcg_at_100 + value: 79.87100000000001 + - type: ndcg_at_1000 + value: 80.286 + - type: ndcg_at_20 + value: 78.55499999999999 + - type: ndcg_at_3 + value: 74.917 + - type: ndcg_at_5 + value: 76.337 + - type: precision_at_1 + value: 69.333 + - type: precision_at_10 + value: 8.466999999999999 + - type: precision_at_100 + value: 0.97 + - type: precision_at_1000 + value: 0.1 + - type: precision_at_20 + value: 4.5 + - type: precision_at_3 + value: 26.222 + - type: precision_at_5 + value: 16.400000000000002 + - type: recall_at_1 + value: 69.333 + - type: recall_at_10 + value: 84.667 + - type: recall_at_100 + value: 97.0 + - type: recall_at_1000 + value: 100.0 + - type: recall_at_20 + value: 90.0 + - type: recall_at_3 + value: 78.667 + - type: recall_at_5 + value: 82.0 + task: + type: Retrieval + - dataset: + config: default + name: MTEB LegalBenchConsumerContractsQA (default) + revision: b23590301ec94e8087e2850b21d43d4956b1cca9 + split: test + type: mteb/legalbench_consumer_contracts_qa + metrics: + - type: main_score + value: 61.623000000000005 + - type: map_at_1 + value: 40.909 + - type: map_at_10 + value: 54.376999999999995 + - type: map_at_100 + value: 55.150999999999996 + - type: map_at_1000 + value: 55.150999999999996 + - type: map_at_20 + value: 54.881 + - type: map_at_3 + value: 49.916 + - type: map_at_5 + value: 52.883 + - type: mrr_at_1 + value: 41.16161616161616 + - type: mrr_at_10 + value: 54.502765752765725 + - type: mrr_at_100 + value: 55.27732053682153 + - type: mrr_at_1000 + value: 55.27732053682153 + - type: mrr_at_20 + value: 55.00715286102044 + - type: mrr_at_3 + value: 50.04208754208756 + - type: mrr_at_5 + value: 53.00925925925923 + - type: ndcg_at_1 + value: 40.909 + - type: ndcg_at_10 + value: 61.623000000000005 + - type: ndcg_at_100 + value: 65.08500000000001 + - type: ndcg_at_1000 + value: 65.08500000000001 + - type: ndcg_at_20 + value: 63.427 + - type: ndcg_at_3 + value: 52.735 + - type: ndcg_at_5 + value: 58.114 + - type: precision_at_1 + value: 40.909 + - type: precision_at_10 + value: 8.459999999999999 + - type: precision_at_100 + value: 1.0 + - type: precision_at_1000 + value: 0.1 + - type: precision_at_20 + value: 4.583 + - type: precision_at_3 + value: 20.286 + - type: precision_at_5 + value: 14.798 + - type: recall_at_1 + value: 40.909 + - type: recall_at_10 + value: 84.596 + - type: recall_at_100 + value: 100.0 + - type: recall_at_1000 + value: 100.0 + - type: recall_at_20 + value: 91.667 + - type: recall_at_3 + value: 60.858999999999995 + - type: recall_at_5 + value: 73.99 + task: + type: Retrieval + - dataset: + config: default + name: MTEB LegalBenchCorporateLobbying (default) + revision: f69691c650464e62546d7f2a4536f8f87c891e38 + split: test + type: mteb/legalbench_corporate_lobbying + metrics: + - type: main_score + value: 89.377 + - type: map_at_1 + value: 79.706 + - type: map_at_10 + value: 86.652 + - type: map_at_100 + value: 86.756 + - type: map_at_1000 + value: 86.76 + - type: map_at_20 + value: 86.749 + - type: map_at_3 + value: 85.784 + - type: map_at_5 + value: 86.387 + - type: mrr_at_1 + value: 79.70588235294119 + - type: mrr_at_10 + value: 86.65184407096169 + - type: mrr_at_100 + value: 86.75604621101796 + - type: mrr_at_1000 + value: 86.76035993650815 + - type: mrr_at_20 + value: 86.7486932698415 + - type: mrr_at_3 + value: 85.78431372549021 + - type: mrr_at_5 + value: 86.38725490196077 + - type: ndcg_at_1 + value: 79.706 + - type: ndcg_at_10 + value: 89.377 + - type: ndcg_at_100 + value: 89.79700000000001 + - type: ndcg_at_1000 + value: 89.88000000000001 + - type: ndcg_at_20 + value: 89.742 + - type: ndcg_at_3 + value: 87.63300000000001 + - type: ndcg_at_5 + value: 88.721 + - type: precision_at_1 + value: 79.706 + - type: precision_at_10 + value: 9.765 + - type: precision_at_100 + value: 0.9939999999999999 + - type: precision_at_1000 + value: 0.1 + - type: precision_at_20 + value: 4.956 + - type: precision_at_3 + value: 30.98 + - type: precision_at_5 + value: 19.118 + - type: recall_at_1 + value: 79.706 + - type: recall_at_10 + value: 97.64699999999999 + - type: recall_at_100 + value: 99.412 + - type: recall_at_1000 + value: 100.0 + - type: recall_at_20 + value: 99.118 + - type: recall_at_3 + value: 92.941 + - type: recall_at_5 + value: 95.588 + task: + type: Retrieval + - dataset: + config: default + name: MTEB LegalSummarization (default) + revision: 3bb1a05c66872889662af04c5691c14489cebd72 + split: test + type: mteb/legal_summarization + metrics: + - type: main_score + value: 54.98800000000001 + - type: map_at_1 + value: 37.468 + - type: map_at_10 + value: 48.509 + - type: map_at_100 + value: 49.681 + - type: map_at_1000 + value: 49.757 + - type: map_at_20 + value: 49.021 + - type: map_at_3 + value: 44.59 + - type: map_at_5 + value: 46.867999999999995 + - type: mrr_at_1 + value: 42.6056338028169 + - type: mrr_at_10 + value: 53.24223116476639 + - type: mrr_at_100 + value: 53.82518326740263 + - type: mrr_at_1000 + value: 53.86171229208665 + - type: mrr_at_20 + value: 53.51133505321795 + - type: mrr_at_3 + value: 49.76525821596244 + - type: mrr_at_5 + value: 51.87793427230047 + - type: ndcg_at_1 + value: 42.606 + - type: ndcg_at_10 + value: 54.98800000000001 + - type: ndcg_at_100 + value: 60.111000000000004 + - type: ndcg_at_1000 + value: 61.382000000000005 + - type: ndcg_at_20 + value: 56.428999999999995 + - type: ndcg_at_3 + value: 48.367 + - type: ndcg_at_5 + value: 51.72 + - type: precision_at_1 + value: 42.606 + - type: precision_at_10 + value: 9.331 + - type: precision_at_100 + value: 1.398 + - type: precision_at_1000 + value: 0.155 + - type: precision_at_20 + value: 5.176 + - type: precision_at_3 + value: 21.009 + - type: precision_at_5 + value: 15.211 + - type: recall_at_1 + value: 37.468 + - type: recall_at_10 + value: 69.607 + - type: recall_at_100 + value: 91.57 + - type: recall_at_1000 + value: 100.0 + - type: recall_at_20 + value: 74.553 + - type: recall_at_3 + value: 51.856 + - type: recall_at_5 + value: 60.309999999999995 + task: + type: Retrieval + - dataset: + config: default + name: MTEB LitSearchRetrieval (default) + revision: 9573fb284a1026c998df47024b888a163f0f0e25 + split: test + type: princeton-nlp/LitSearch + metrics: + - type: main_score + value: 25.226 + - type: map_at_1 + value: 15.326999999999998 + - type: map_at_10 + value: 21.394 + - type: map_at_100 + value: 22.343 + - type: map_at_1000 + value: 22.429 + - type: map_at_20 + value: 21.941 + - type: map_at_3 + value: 19.123 + - type: map_at_5 + value: 20.14 + - type: mrr_at_1 + value: 15.745393634840871 + - type: mrr_at_10 + value: 21.745699396453173 + - type: mrr_at_100 + value: 22.713350239272813 + - type: mrr_at_1000 + value: 22.79392349547639 + - type: mrr_at_20 + value: 22.30884258376597 + - type: mrr_at_3 + value: 19.458403126744827 + - type: mrr_at_5 + value: 20.538805136795084 + - type: ndcg_at_1 + value: 15.494 + - type: ndcg_at_10 + value: 25.226 + - type: ndcg_at_100 + value: 30.263 + - type: ndcg_at_1000 + value: 32.994 + - type: ndcg_at_20 + value: 27.183 + - type: ndcg_at_3 + value: 20.385 + - type: ndcg_at_5 + value: 22.21 + - type: precision_at_1 + value: 15.745000000000001 + - type: precision_at_10 + value: 3.987 + - type: precision_at_100 + value: 0.657 + - type: precision_at_1000 + value: 0.09 + - type: precision_at_20 + value: 2.404 + - type: precision_at_3 + value: 8.319 + - type: precision_at_5 + value: 5.93 + - type: recall_at_1 + value: 15.326999999999998 + - type: recall_at_10 + value: 37.968 + - type: recall_at_100 + value: 62.546 + - type: recall_at_1000 + value: 84.87700000000001 + - type: recall_at_20 + value: 45.739999999999995 + - type: recall_at_3 + value: 24.204 + - type: recall_at_5 + value: 28.615000000000002 + task: + type: Retrieval + - dataset: + config: default + name: MTEB MSMARCO (default) + revision: c5a29a104738b98a9e76336939199e264163d4a0 + split: test + type: mteb/msmarco + metrics: + - type: main_score + value: 39.988 + - type: map_at_1 + value: 1.055 + - type: map_at_10 + value: 7.149 + - type: map_at_100 + value: 21.816 + - type: map_at_1000 + value: 29.885 + - type: map_at_20 + value: 11.162999999999998 + - type: map_at_3 + value: 2.735 + - type: map_at_5 + value: 4.199 + - type: mrr_at_1 + value: 60.46511627906976 + - type: mrr_at_10 + value: 71.29844961240309 + - type: mrr_at_100 + value: 71.4438037073023 + - type: mrr_at_1000 + value: 71.45852257689312 + - type: mrr_at_20 + value: 71.29844961240309 + - type: mrr_at_3 + value: 69.3798449612403 + - type: mrr_at_5 + value: 71.00775193798448 + - type: ndcg_at_1 + value: 39.535 + - type: ndcg_at_10 + value: 39.988 + - type: ndcg_at_100 + value: 41.952 + - type: ndcg_at_1000 + value: 55.149 + - type: ndcg_at_20 + value: 39.861000000000004 + - type: ndcg_at_3 + value: 39.713 + - type: ndcg_at_5 + value: 40.2 + - type: precision_at_1 + value: 60.465 + - type: precision_at_10 + value: 52.791 + - type: precision_at_100 + value: 29.558 + - type: precision_at_1000 + value: 6.952999999999999 + - type: precision_at_20 + value: 47.209 + - type: precision_at_3 + value: 57.364000000000004 + - type: precision_at_5 + value: 56.279 + - type: recall_at_1 + value: 1.055 + - type: recall_at_10 + value: 8.778 + - type: recall_at_100 + value: 36.775999999999996 + - type: recall_at_1000 + value: 72.783 + - type: recall_at_20 + value: 14.529 + - type: recall_at_3 + value: 3.019 + - type: recall_at_5 + value: 4.987 + task: + type: Retrieval + - dataset: + config: default + name: MTEB MSMARCOHardNegatives (default) + revision: 67c0b4f7f15946e0b15cf6cf3b8993d04cb3efc6 + split: test + type: mteb/MSMARCO_test_top_250_only_w_correct-v2 + metrics: + - type: main_score + value: 49.17 + - type: map_at_1 + value: 1.871 + - type: map_at_10 + value: 11.155 + - type: map_at_100 + value: 38.995000000000005 + - type: map_at_1000 + value: 58.543 + - type: map_at_20 + value: 16.564 + - type: map_at_3 + value: 5.378 + - type: map_at_5 + value: 7.403 + - type: mrr_at_1 + value: 69.76744186046511 + - type: mrr_at_10 + value: 78.7624584717608 + - type: mrr_at_100 + value: 78.97387496224707 + - type: mrr_at_1000 + value: 78.97387496224707 + - type: mrr_at_20 + value: 78.97387496224707 + - type: mrr_at_3 + value: 76.74418604651163 + - type: mrr_at_5 + value: 77.90697674418605 + - type: ndcg_at_1 + value: 46.899 + - type: ndcg_at_10 + value: 49.17 + - type: ndcg_at_100 + value: 62.022 + - type: ndcg_at_1000 + value: 76.946 + - type: ndcg_at_20 + value: 50.113 + - type: ndcg_at_3 + value: 47.809000000000005 + - type: ndcg_at_5 + value: 47.947 + - type: precision_at_1 + value: 69.767 + - type: precision_at_10 + value: 63.256 + - type: precision_at_100 + value: 46.302 + - type: precision_at_1000 + value: 9.419 + - type: precision_at_20 + value: 58.48799999999999 + - type: precision_at_3 + value: 69.767 + - type: precision_at_5 + value: 66.047 + - type: recall_at_1 + value: 1.871 + - type: recall_at_10 + value: 12.928999999999998 + - type: recall_at_100 + value: 60.528000000000006 + - type: recall_at_1000 + value: 98.18599999999999 + - type: recall_at_20 + value: 20.534 + - type: recall_at_3 + value: 5.611 + - type: recall_at_5 + value: 8.054 + task: + type: Retrieval + - dataset: + config: default + name: MTEB MedicalQARetrieval (default) + revision: ae763399273d8b20506b80cf6f6f9a31a6a2b238 + split: test + type: mteb/medical_qa + metrics: + - type: main_score + value: 47.905 + - type: map_at_1 + value: 23.926 + - type: map_at_10 + value: 39.777 + - type: map_at_100 + value: 40.449 + - type: map_at_1000 + value: 40.486 + - type: map_at_20 + value: 40.204 + - type: map_at_3 + value: 35.531 + - type: map_at_5 + value: 38.25 + - type: mrr_at_1 + value: 23.876953125 + - type: mrr_at_10 + value: 39.77653382316477 + - type: mrr_at_100 + value: 40.449022311809415 + - type: mrr_at_1000 + value: 40.485791901682774 + - type: mrr_at_20 + value: 40.20422058322255 + - type: mrr_at_3 + value: 35.53873697916675 + - type: mrr_at_5 + value: 38.25846354166688 + - type: ndcg_at_1 + value: 23.926 + - type: ndcg_at_10 + value: 47.905 + - type: ndcg_at_100 + value: 51.186 + - type: ndcg_at_1000 + value: 52.297000000000004 + - type: ndcg_at_20 + value: 49.445 + - type: ndcg_at_3 + value: 39.391 + - type: ndcg_at_5 + value: 44.254 + - type: precision_at_1 + value: 23.926 + - type: precision_at_10 + value: 7.349 + - type: precision_at_100 + value: 0.889 + - type: precision_at_1000 + value: 0.098 + - type: precision_at_20 + value: 3.977 + - type: precision_at_3 + value: 16.862 + - type: precision_at_5 + value: 12.461 + - type: recall_at_1 + value: 23.926 + - type: recall_at_10 + value: 73.48599999999999 + - type: recall_at_100 + value: 88.86699999999999 + - type: recall_at_1000 + value: 97.89999999999999 + - type: recall_at_20 + value: 79.541 + - type: recall_at_3 + value: 50.586 + - type: recall_at_5 + value: 62.305 + task: + type: Retrieval + - dataset: + config: default + name: MTEB NFCorpus (default) + revision: ec0fa4fe99da2ff19ca1214b7966684033a58814 + split: test + type: mteb/nfcorpus + metrics: + - type: main_score + value: 30.044999999999998 + - type: map_at_1 + value: 5.015 + - type: map_at_10 + value: 10.93 + - type: map_at_100 + value: 13.592 + - type: map_at_1000 + value: 14.890999999999998 + - type: map_at_20 + value: 12.005 + - type: map_at_3 + value: 8.518 + - type: map_at_5 + value: 9.646 + - type: mrr_at_1 + value: 41.17647058823529 + - type: mrr_at_10 + value: 49.96486313823774 + - type: mrr_at_100 + value: 50.73871199227761 + - type: mrr_at_1000 + value: 50.788364180879874 + - type: mrr_at_20 + value: 50.53695651632227 + - type: mrr_at_3 + value: 47.936016511867905 + - type: mrr_at_5 + value: 49.267285861713106 + - type: ndcg_at_1 + value: 39.164 + - type: ndcg_at_10 + value: 30.044999999999998 + - type: ndcg_at_100 + value: 27.654 + - type: ndcg_at_1000 + value: 36.397 + - type: ndcg_at_20 + value: 28.016000000000002 + - type: ndcg_at_3 + value: 35.476 + - type: ndcg_at_5 + value: 33.123999999999995 + - type: precision_at_1 + value: 41.176 + - type: precision_at_10 + value: 21.765 + - type: precision_at_100 + value: 7.127 + - type: precision_at_1000 + value: 1.9959999999999998 + - type: precision_at_20 + value: 16.223000000000003 + - type: precision_at_3 + value: 33.333 + - type: precision_at_5 + value: 28.421000000000003 + - type: recall_at_1 + value: 5.015 + - type: recall_at_10 + value: 14.618999999999998 + - type: recall_at_100 + value: 27.755000000000003 + - type: recall_at_1000 + value: 59.302 + - type: recall_at_20 + value: 17.743000000000002 + - type: recall_at_3 + value: 9.769 + - type: recall_at_5 + value: 11.912 + task: + type: Retrieval + - dataset: + config: default + name: MTEB NQ (default) + revision: b774495ed302d8c44a3a7ea25c90dbce03968f31 + split: test + type: mteb/nq + metrics: + - type: main_score + value: 23.105 + - type: map_at_1 + value: 9.2 + - type: map_at_10 + value: 17.564 + - type: map_at_100 + value: 19.226 + - type: map_at_1000 + value: 19.323 + - type: map_at_20 + value: 18.507 + - type: map_at_3 + value: 14.274999999999999 + - type: map_at_5 + value: 15.98 + - type: mrr_at_1 + value: 10.573580533024334 + - type: mrr_at_10 + value: 19.264677481653145 + - type: mrr_at_100 + value: 20.746572887142992 + - type: mrr_at_1000 + value: 20.823910481320183 + - type: mrr_at_20 + value: 20.119865220550533 + - type: mrr_at_3 + value: 16.005214368482008 + - type: mrr_at_5 + value: 17.75057937427577 + - type: ndcg_at_1 + value: 10.545 + - type: ndcg_at_10 + value: 23.105 + - type: ndcg_at_100 + value: 31.249 + - type: ndcg_at_1000 + value: 33.69 + - type: ndcg_at_20 + value: 26.334999999999997 + - type: ndcg_at_3 + value: 16.357 + - type: ndcg_at_5 + value: 19.403000000000002 + - type: precision_at_1 + value: 10.545 + - type: precision_at_10 + value: 4.565 + - type: precision_at_100 + value: 0.9169999999999999 + - type: precision_at_1000 + value: 0.11499999999999999 + - type: precision_at_20 + value: 3.023 + - type: precision_at_3 + value: 8.033999999999999 + - type: precision_at_5 + value: 6.524000000000001 + - type: recall_at_1 + value: 9.2 + - type: recall_at_10 + value: 38.775 + - type: recall_at_100 + value: 76.188 + - type: recall_at_1000 + value: 94.56599999999999 + - type: recall_at_20 + value: 50.9 + - type: recall_at_3 + value: 20.676 + - type: recall_at_5 + value: 27.810000000000002 + task: + type: Retrieval + - dataset: + config: default + name: MTEB NQHardNegatives (default) + revision: d700fe4f167a5db8e6c9b03e8c26e7eaf66faf97 + split: test + type: mteb/NQ_test_top_250_only_w_correct-v2 + metrics: + - type: main_score + value: 25.072 + - type: map_at_1 + value: 9.767000000000001 + - type: map_at_10 + value: 18.782 + - type: map_at_100 + value: 20.74 + - type: map_at_1000 + value: 20.821 + - type: map_at_20 + value: 19.819 + - type: map_at_3 + value: 15.089 + - type: map_at_5 + value: 16.929 + - type: mrr_at_1 + value: 11.0 + - type: mrr_at_10 + value: 20.55162698412698 + - type: mrr_at_100 + value: 22.24001716934204 + - type: mrr_at_1000 + value: 22.296375463042438 + - type: mrr_at_20 + value: 21.45978809546147 + - type: mrr_at_3 + value: 16.683333333333337 + - type: mrr_at_5 + value: 18.738333333333344 + - type: ndcg_at_1 + value: 11.0 + - type: ndcg_at_10 + value: 25.072 + - type: ndcg_at_100 + value: 34.695 + - type: ndcg_at_1000 + value: 36.312 + - type: ndcg_at_20 + value: 28.595 + - type: ndcg_at_3 + value: 17.273 + - type: ndcg_at_5 + value: 20.663999999999998 + - type: precision_at_1 + value: 11.0 + - type: precision_at_10 + value: 5.01 + - type: precision_at_100 + value: 1.05 + - type: precision_at_1000 + value: 0.12 + - type: precision_at_20 + value: 3.335 + - type: precision_at_3 + value: 8.433 + - type: precision_at_5 + value: 6.959999999999999 + - type: recall_at_1 + value: 9.767000000000001 + - type: recall_at_10 + value: 43.175000000000004 + - type: recall_at_100 + value: 87.467 + - type: recall_at_1000 + value: 98.917 + - type: recall_at_20 + value: 56.325 + - type: recall_at_3 + value: 22.033 + - type: recall_at_5 + value: 29.975 + task: + type: Retrieval + - dataset: + config: default + name: MTEB PIQA (default) + revision: bb30be7e9184e6b6b1d99bbfe1bb90a3a81842e6 + split: test + type: RAR-b/piqa + metrics: + - type: main_score + value: 15.254000000000001 + - type: map_at_1 + value: 7.073 + - type: map_at_10 + value: 12.281 + - type: map_at_100 + value: 13.178 + - type: map_at_1000 + value: 13.26 + - type: map_at_20 + value: 12.753999999999998 + - type: map_at_3 + value: 10.591000000000001 + - type: map_at_5 + value: 11.522 + - type: mrr_at_1 + value: 7.127312295973885 + - type: mrr_at_10 + value: 12.307826830405721 + - type: mrr_at_100 + value: 13.205659849576772 + - type: mrr_at_1000 + value: 13.28739675923323 + - type: mrr_at_20 + value: 12.781972613283662 + - type: mrr_at_3 + value: 10.618425825172299 + - type: mrr_at_5 + value: 11.548784911135296 + - type: ndcg_at_1 + value: 7.073 + - type: ndcg_at_10 + value: 15.254000000000001 + - type: ndcg_at_100 + value: 20.077 + - type: ndcg_at_1000 + value: 22.43 + - type: ndcg_at_20 + value: 16.948 + - type: ndcg_at_3 + value: 11.741 + - type: ndcg_at_5 + value: 13.420000000000002 + - type: precision_at_1 + value: 7.073 + - type: precision_at_10 + value: 2.481 + - type: precision_at_100 + value: 0.484 + - type: precision_at_1000 + value: 0.067 + - type: precision_at_20 + value: 1.572 + - type: precision_at_3 + value: 5.024 + - type: precision_at_5 + value: 3.83 + - type: recall_at_1 + value: 7.073 + - type: recall_at_10 + value: 24.81 + - type: recall_at_100 + value: 48.422 + - type: recall_at_1000 + value: 67.35600000000001 + - type: recall_at_20 + value: 31.447000000000003 + - type: recall_at_3 + value: 15.071000000000002 + - type: recall_at_5 + value: 19.151 + task: + type: Retrieval + - dataset: + config: default + name: MTEB Quail (default) + revision: 1851bc536f8bdab29e03e29191c4586b1d8d7c5a + split: test + type: RAR-b/quail + metrics: + - type: main_score + value: 3.605 + - type: map_at_1 + value: 0.882 + - type: map_at_10 + value: 2.467 + - type: map_at_100 + value: 2.9659999999999997 + - type: map_at_1000 + value: 3.052 + - type: map_at_20 + value: 2.711 + - type: map_at_3 + value: 1.746 + - type: map_at_5 + value: 2.0629999999999997 + - type: mrr_at_1 + value: 0.8823529411764706 + - type: mrr_at_10 + value: 2.4645337301587333 + - type: mrr_at_100 + value: 2.9670174083352596 + - type: mrr_at_1000 + value: 3.0527771606810092 + - type: mrr_at_20 + value: 2.7112556752180152 + - type: mrr_at_3 + value: 1.7463235294117647 + - type: mrr_at_5 + value: 2.0625000000000013 + - type: ndcg_at_1 + value: 0.882 + - type: ndcg_at_10 + value: 3.605 + - type: ndcg_at_100 + value: 6.494999999999999 + - type: ndcg_at_1000 + value: 9.27 + - type: ndcg_at_20 + value: 4.502 + - type: ndcg_at_3 + value: 2.0340000000000003 + - type: ndcg_at_5 + value: 2.614 + - type: precision_at_1 + value: 0.882 + - type: precision_at_10 + value: 0.739 + - type: precision_at_100 + value: 0.22 + - type: precision_at_1000 + value: 0.045 + - type: precision_at_20 + value: 0.5479999999999999 + - type: precision_at_3 + value: 0.9560000000000001 + - type: precision_at_5 + value: 0.86 + - type: recall_at_1 + value: 0.882 + - type: recall_at_10 + value: 7.39 + - type: recall_at_100 + value: 21.985 + - type: recall_at_1000 + value: 44.926 + - type: recall_at_20 + value: 10.956000000000001 + - type: recall_at_3 + value: 2.868 + - type: recall_at_5 + value: 4.301 + task: + type: Retrieval + - dataset: + config: default + name: MTEB QuoraRetrieval (default) + revision: e4e08e0b7dbe3c8700f0daef558ff32256715259 + split: dev + type: mteb/quora + metrics: + - type: main_score + value: 77.63 + - type: map_at_1 + value: 60.195 + - type: map_at_10 + value: 72.834 + - type: map_at_100 + value: 73.657 + - type: map_at_1000 + value: 73.68900000000001 + - type: map_at_20 + value: 73.358 + - type: map_at_3 + value: 69.834 + - type: map_at_5 + value: 71.622 + - type: mrr_at_1 + value: 69.12 + - type: mrr_at_10 + value: 76.90555555555562 + - type: mrr_at_100 + value: 77.22690418927783 + - type: mrr_at_1000 + value: 77.23378887488153 + - type: mrr_at_20 + value: 77.12735614892509 + - type: mrr_at_3 + value: 75.29666666666685 + - type: mrr_at_5 + value: 76.29566666666672 + - type: ndcg_at_1 + value: 69.06 + - type: ndcg_at_10 + value: 77.63 + - type: ndcg_at_100 + value: 80.143 + - type: ndcg_at_1000 + value: 80.57900000000001 + - type: ndcg_at_20 + value: 78.886 + - type: ndcg_at_3 + value: 73.735 + - type: ndcg_at_5 + value: 75.689 + - type: precision_at_1 + value: 69.06 + - type: precision_at_10 + value: 11.804 + - type: precision_at_100 + value: 1.417 + - type: precision_at_1000 + value: 0.151 + - type: precision_at_20 + value: 6.383 + - type: precision_at_3 + value: 32.0 + - type: precision_at_5 + value: 21.279999999999998 + - type: recall_at_1 + value: 60.195 + - type: recall_at_10 + value: 87.35300000000001 + - type: recall_at_100 + value: 97.055 + - type: recall_at_1000 + value: 99.669 + - type: recall_at_20 + value: 91.628 + - type: recall_at_3 + value: 76.40599999999999 + - type: recall_at_5 + value: 81.636 + task: + type: Retrieval + - dataset: + config: default + name: MTEB QuoraRetrieval (default) + revision: e4e08e0b7dbe3c8700f0daef558ff32256715259 + split: test + type: mteb/quora + metrics: + - type: main_score + value: 77.529 + - type: map_at_1 + value: 60.492000000000004 + - type: map_at_10 + value: 72.82499999999999 + - type: map_at_100 + value: 73.668 + - type: map_at_1000 + value: 73.706 + - type: map_at_20 + value: 73.351 + - type: map_at_3 + value: 69.887 + - type: map_at_5 + value: 71.66 + - type: mrr_at_1 + value: 69.59 + - type: mrr_at_10 + value: 76.96056349206305 + - type: mrr_at_100 + value: 77.29271487249463 + - type: mrr_at_1000 + value: 77.30133384825908 + - type: mrr_at_20 + value: 77.1902356844216 + - type: mrr_at_3 + value: 75.39499999999957 + - type: mrr_at_5 + value: 76.38149999999933 + - type: ndcg_at_1 + value: 69.61 + - type: ndcg_at_10 + value: 77.529 + - type: ndcg_at_100 + value: 80.067 + - type: ndcg_at_1000 + value: 80.54299999999999 + - type: ndcg_at_20 + value: 78.76100000000001 + - type: ndcg_at_3 + value: 73.786 + - type: ndcg_at_5 + value: 75.696 + - type: precision_at_1 + value: 69.61 + - type: precision_at_10 + value: 11.756 + - type: precision_at_100 + value: 1.436 + - type: precision_at_1000 + value: 0.154 + - type: precision_at_20 + value: 6.382000000000001 + - type: precision_at_3 + value: 31.996999999999996 + - type: precision_at_5 + value: 21.198 + - type: recall_at_1 + value: 60.492000000000004 + - type: recall_at_10 + value: 86.887 + - type: recall_at_100 + value: 96.67999999999999 + - type: recall_at_1000 + value: 99.438 + - type: recall_at_20 + value: 91.081 + - type: recall_at_3 + value: 76.212 + - type: recall_at_5 + value: 81.48100000000001 + task: + type: Retrieval + - dataset: + config: default + name: MTEB QuoraRetrievalHardNegatives (default) + revision: 907a33577e9506221d3ba20f5a851b7c3f8dc6d3 + split: test + type: mteb/QuoraRetrieval_test_top_250_only_w_correct-v2 + metrics: + - type: main_score + value: 77.352 + - type: map_at_1 + value: 60.285999999999994 + - type: map_at_10 + value: 72.541 + - type: map_at_100 + value: 73.576 + - type: map_at_1000 + value: 73.61999999999999 + - type: map_at_20 + value: 73.236 + - type: map_at_3 + value: 69.434 + - type: map_at_5 + value: 71.177 + - type: mrr_at_1 + value: 69.6 + - type: mrr_at_10 + value: 76.9061111111111 + - type: mrr_at_100 + value: 77.25264059161483 + - type: mrr_at_1000 + value: 77.25961973203633 + - type: mrr_at_20 + value: 77.14650618344506 + - type: mrr_at_3 + value: 75.20000000000002 + - type: mrr_at_5 + value: 76.18 + - type: ndcg_at_1 + value: 69.6 + - type: ndcg_at_10 + value: 77.352 + - type: ndcg_at_100 + value: 80.24 + - type: ndcg_at_1000 + value: 80.658 + - type: ndcg_at_20 + value: 78.90599999999999 + - type: ndcg_at_3 + value: 73.534 + - type: ndcg_at_5 + value: 75.18599999999999 + - type: precision_at_1 + value: 69.6 + - type: precision_at_10 + value: 12.08 + - type: precision_at_100 + value: 1.514 + - type: precision_at_1000 + value: 0.16199999999999998 + - type: precision_at_20 + value: 6.69 + - type: precision_at_3 + value: 32.067 + - type: precision_at_5 + value: 21.279999999999998 + - type: recall_at_1 + value: 60.285999999999994 + - type: recall_at_10 + value: 86.75399999999999 + - type: recall_at_100 + value: 97.328 + - type: recall_at_1000 + value: 99.551 + - type: recall_at_20 + value: 91.85900000000001 + - type: recall_at_3 + value: 75.408 + - type: recall_at_5 + value: 80.353 + task: + type: Retrieval + - dataset: + config: default + name: MTEB RARbCode (default) + revision: 25f7d11a7ac12dcbb8d3836eb2de682b98c825e4 + split: test + type: RAR-b/humanevalpack-mbpp-pooled + metrics: + - type: main_score + value: 10.407 + - type: map_at_1 + value: 6.132 + - type: map_at_10 + value: 8.741 + - type: map_at_100 + value: 9.198 + - type: map_at_1000 + value: 9.264 + - type: map_at_20 + value: 8.991 + - type: map_at_3 + value: 7.637 + - type: map_at_5 + value: 8.21 + - type: mrr_at_1 + value: 6.132075471698113 + - type: mrr_at_10 + value: 8.741389637616054 + - type: mrr_at_100 + value: 9.19843044441297 + - type: mrr_at_1000 + value: 9.263870046532622 + - type: mrr_at_20 + value: 8.991124756997893 + - type: mrr_at_3 + value: 7.637017070979335 + - type: mrr_at_5 + value: 8.209793351302785 + - type: ndcg_at_1 + value: 6.132 + - type: ndcg_at_10 + value: 10.407 + - type: ndcg_at_100 + value: 12.959000000000001 + - type: ndcg_at_1000 + value: 14.991 + - type: ndcg_at_20 + value: 11.324 + - type: ndcg_at_3 + value: 8.117 + - type: ndcg_at_5 + value: 9.146 + - type: precision_at_1 + value: 6.132 + - type: precision_at_10 + value: 1.584 + - type: precision_at_100 + value: 0.28600000000000003 + - type: precision_at_1000 + value: 0.045 + - type: precision_at_20 + value: 0.9740000000000001 + - type: precision_at_3 + value: 3.167 + - type: precision_at_5 + value: 2.399 + - type: recall_at_1 + value: 6.132 + - type: recall_at_10 + value: 15.836 + - type: recall_at_100 + value: 28.571 + - type: recall_at_1000 + value: 45.216 + - type: recall_at_20 + value: 19.474 + - type: recall_at_3 + value: 9.501 + - type: recall_at_5 + value: 11.995000000000001 + task: + type: Retrieval + - dataset: + config: default + name: MTEB RARbMath (default) + revision: 2393603c0221ff52f448d12dd75f0856103c6cca + split: test + type: RAR-b/math-pooled + metrics: + - type: main_score + value: 23.658 + - type: map_at_1 + value: 19.686999999999998 + - type: map_at_10 + value: 22.178 + - type: map_at_100 + value: 22.765 + - type: map_at_1000 + value: 22.844 + - type: map_at_20 + value: 22.462 + - type: map_at_3 + value: 21.29 + - type: map_at_5 + value: 21.787 + - type: mrr_at_1 + value: 19.686659281531888 + - type: mrr_at_10 + value: 22.177553460588754 + - type: mrr_at_100 + value: 22.7654510715158 + - type: mrr_at_1000 + value: 22.843891574167113 + - type: mrr_at_20 + value: 22.46217836587706 + - type: mrr_at_3 + value: 21.290288547765996 + - type: mrr_at_5 + value: 21.787202616447765 + - type: ndcg_at_1 + value: 19.686999999999998 + - type: ndcg_at_10 + value: 23.658 + - type: ndcg_at_100 + value: 27.0 + - type: ndcg_at_1000 + value: 29.509999999999998 + - type: ndcg_at_20 + value: 24.715 + - type: ndcg_at_3 + value: 21.817 + - type: ndcg_at_5 + value: 22.711000000000002 + - type: precision_at_1 + value: 19.686999999999998 + - type: precision_at_10 + value: 2.844 + - type: precision_at_100 + value: 0.45199999999999996 + - type: precision_at_1000 + value: 0.066 + - type: precision_at_20 + value: 1.633 + - type: precision_at_3 + value: 7.781000000000001 + - type: precision_at_5 + value: 5.102 + - type: recall_at_1 + value: 19.686999999999998 + - type: recall_at_10 + value: 28.438000000000002 + - type: recall_at_100 + value: 45.165 + - type: recall_at_1000 + value: 65.865 + - type: recall_at_20 + value: 32.663 + - type: recall_at_3 + value: 23.342 + - type: recall_at_5 + value: 25.509999999999998 + task: + type: Retrieval + - dataset: + config: default + name: MTEB SCIDOCS (default) + revision: f8c2fcf00f625baaa80f62ec5bd9e1fff3b8ae88 + split: test + type: mteb/scidocs + metrics: + - type: main_score + value: 13.222999999999999 + - type: map_at_1 + value: 2.9979999999999998 + - type: map_at_10 + value: 7.475 + - type: map_at_100 + value: 8.903 + - type: map_at_1000 + value: 9.16 + - type: map_at_20 + value: 8.136000000000001 + - type: map_at_3 + value: 5.329 + - type: map_at_5 + value: 6.411 + - type: mrr_at_1 + value: 14.7 + - type: mrr_at_10 + value: 22.86599206349206 + - type: mrr_at_100 + value: 24.016847471793167 + - type: mrr_at_1000 + value: 24.09878143285336 + - type: mrr_at_20 + value: 23.487873612455665 + - type: mrr_at_3 + value: 19.850000000000016 + - type: mrr_at_5 + value: 21.385000000000005 + - type: ndcg_at_1 + value: 14.7 + - type: ndcg_at_10 + value: 13.222999999999999 + - type: ndcg_at_100 + value: 19.725 + - type: ndcg_at_1000 + value: 24.723 + - type: ndcg_at_20 + value: 15.215 + - type: ndcg_at_3 + value: 12.073 + - type: ndcg_at_5 + value: 10.707 + - type: precision_at_1 + value: 14.7 + - type: precision_at_10 + value: 7.049999999999999 + - type: precision_at_100 + value: 1.6650000000000003 + - type: precision_at_1000 + value: 0.28600000000000003 + - type: precision_at_20 + value: 4.68 + - type: precision_at_3 + value: 11.3 + - type: precision_at_5 + value: 9.48 + - type: recall_at_1 + value: 2.9979999999999998 + - type: recall_at_10 + value: 14.277999999999999 + - type: recall_at_100 + value: 33.772000000000006 + - type: recall_at_1000 + value: 58.15 + - type: recall_at_20 + value: 18.956999999999997 + - type: recall_at_3 + value: 6.883 + - type: recall_at_5 + value: 9.613 + task: + type: Retrieval + - dataset: + config: default + name: MTEB SIQA (default) + revision: 4ed8415e9dc24060deefc84be59e2db0aacbadcc + split: test + type: RAR-b/siqa + metrics: + - type: main_score + value: 0.517 + - type: map_at_1 + value: 0.307 + - type: map_at_10 + value: 0.42700000000000005 + - type: map_at_100 + value: 0.511 + - type: map_at_1000 + value: 0.5640000000000001 + - type: map_at_20 + value: 0.477 + - type: map_at_3 + value: 0.358 + - type: map_at_5 + value: 0.384 + - type: mrr_at_1 + value: 0.3070624360286591 + - type: mrr_at_10 + value: 0.42667868921707197 + - type: mrr_at_100 + value: 0.5106947806513232 + - type: mrr_at_1000 + value: 0.5637065102894676 + - type: mrr_at_20 + value: 0.476660924594987 + - type: mrr_at_3 + value: 0.35823950870010235 + - type: mrr_at_5 + value: 0.3838280450358239 + - type: ndcg_at_1 + value: 0.307 + - type: ndcg_at_10 + value: 0.517 + - type: ndcg_at_100 + value: 0.9570000000000001 + - type: ndcg_at_1000 + value: 3.83 + - type: ndcg_at_20 + value: 0.69 + - type: ndcg_at_3 + value: 0.372 + - type: ndcg_at_5 + value: 0.416 + - type: precision_at_1 + value: 0.307 + - type: precision_at_10 + value: 0.082 + - type: precision_at_100 + value: 0.03 + - type: precision_at_1000 + value: 0.029 + - type: precision_at_20 + value: 0.074 + - type: precision_at_3 + value: 0.136 + - type: precision_at_5 + value: 0.10200000000000001 + - type: recall_at_1 + value: 0.307 + - type: recall_at_10 + value: 0.819 + - type: recall_at_100 + value: 2.968 + - type: recall_at_1000 + value: 29.017 + - type: recall_at_20 + value: 1.484 + - type: recall_at_3 + value: 0.409 + - type: recall_at_5 + value: 0.512 + task: + type: Retrieval + - dataset: + config: default + name: MTEB SciFact (default) + revision: 0228b52cf27578f30900b9e5271d331663a030d7 + split: test + type: mteb/scifact + metrics: + - type: main_score + value: 59.348 + - type: map_at_1 + value: 45.306000000000004 + - type: map_at_10 + value: 54.547000000000004 + - type: map_at_100 + value: 55.535000000000004 + - type: map_at_1000 + value: 55.582 + - type: map_at_20 + value: 55.242000000000004 + - type: map_at_3 + value: 51.763000000000005 + - type: map_at_5 + value: 53.27499999999999 + - type: mrr_at_1 + value: 47.0 + - type: mrr_at_10 + value: 55.67632275132276 + - type: mrr_at_100 + value: 56.50056008171798 + - type: mrr_at_1000 + value: 56.54270500751058 + - type: mrr_at_20 + value: 56.277193298903846 + - type: mrr_at_3 + value: 53.22222222222223 + - type: mrr_at_5 + value: 54.70555555555556 + - type: ndcg_at_1 + value: 47.0 + - type: ndcg_at_10 + value: 59.348 + - type: ndcg_at_100 + value: 63.42100000000001 + - type: ndcg_at_1000 + value: 64.534 + - type: ndcg_at_20 + value: 61.622 + - type: ndcg_at_3 + value: 54.117000000000004 + - type: ndcg_at_5 + value: 56.669000000000004 + - type: precision_at_1 + value: 47.0 + - type: precision_at_10 + value: 8.1 + - type: precision_at_100 + value: 1.027 + - type: precision_at_1000 + value: 0.11199999999999999 + - type: precision_at_20 + value: 4.583 + - type: precision_at_3 + value: 21.221999999999998 + - type: precision_at_5 + value: 14.2 + - type: recall_at_1 + value: 45.306000000000004 + - type: recall_at_10 + value: 72.95 + - type: recall_at_100 + value: 90.533 + - type: recall_at_1000 + value: 99.1 + - type: recall_at_20 + value: 81.389 + - type: recall_at_3 + value: 58.9 + - type: recall_at_5 + value: 65.261 + task: + type: Retrieval + - dataset: + config: default + name: MTEB SciFact (default) + revision: 0228b52cf27578f30900b9e5271d331663a030d7 + split: train + type: mteb/scifact + metrics: + - type: main_score + value: 61.812999999999995 + - type: map_at_1 + value: 45.328 + - type: map_at_10 + value: 56.464000000000006 + - type: map_at_100 + value: 57.282 + - type: map_at_1000 + value: 57.312 + - type: map_at_20 + value: 57.019 + - type: map_at_3 + value: 53.447 + - type: map_at_5 + value: 55.452999999999996 + - type: mrr_at_1 + value: 47.83683559950556 + - type: mrr_at_10 + value: 58.03147134420309 + - type: mrr_at_100 + value: 58.65513617901087 + - type: mrr_at_1000 + value: 58.680986977449564 + - type: mrr_at_20 + value: 58.47209594120791 + - type: mrr_at_3 + value: 55.418211784095575 + - type: mrr_at_5 + value: 57.222908941079474 + - type: ndcg_at_1 + value: 47.837 + - type: ndcg_at_10 + value: 61.812999999999995 + - type: ndcg_at_100 + value: 65.254 + - type: ndcg_at_1000 + value: 66.116 + - type: ndcg_at_20 + value: 63.634 + - type: ndcg_at_3 + value: 56.239 + - type: ndcg_at_5 + value: 59.550000000000004 + - type: precision_at_1 + value: 47.837 + - type: precision_at_10 + value: 8.554 + - type: precision_at_100 + value: 1.043 + - type: precision_at_1000 + value: 0.11199999999999999 + - type: precision_at_20 + value: 4.697 + - type: precision_at_3 + value: 22.579 + - type: precision_at_5 + value: 15.476 + - type: recall_at_1 + value: 45.328 + - type: recall_at_10 + value: 76.667 + - type: recall_at_100 + value: 91.801 + - type: recall_at_1000 + value: 98.578 + - type: recall_at_20 + value: 83.531 + - type: recall_at_3 + value: 61.988 + - type: recall_at_5 + value: 69.868 + task: + type: Retrieval + - dataset: + config: default + name: MTEB SpartQA (default) + revision: 9ab3ca3ccdd0d43f9cd6d346a363935d127f4f45 + split: test + type: RAR-b/spartqa + metrics: + - type: main_score + value: 15.065999999999999 + - type: map_at_1 + value: 2.94 + - type: map_at_10 + value: 9.815999999999999 + - type: map_at_100 + value: 11.514000000000001 + - type: map_at_1000 + value: 11.578 + - type: map_at_20 + value: 11.016 + - type: map_at_3 + value: 6.734999999999999 + - type: map_at_5 + value: 8.337 + - type: mrr_at_1 + value: 5.982192543127435 + - type: mrr_at_10 + value: 13.466734681258858 + - type: mrr_at_100 + value: 15.23447840689604 + - type: mrr_at_1000 + value: 15.301367917746896 + - type: mrr_at_20 + value: 14.71801235668512 + - type: mrr_at_3 + value: 9.98423298089408 + - type: mrr_at_5 + value: 11.894360971990297 + - type: ndcg_at_1 + value: 5.982 + - type: ndcg_at_10 + value: 15.065999999999999 + - type: ndcg_at_100 + value: 22.867 + - type: ndcg_at_1000 + value: 24.808 + - type: ndcg_at_20 + value: 19.363 + - type: ndcg_at_3 + value: 8.397 + - type: ndcg_at_5 + value: 11.453000000000001 + - type: precision_at_1 + value: 5.982 + - type: precision_at_10 + value: 4.413 + - type: precision_at_100 + value: 0.9809999999999999 + - type: precision_at_1000 + value: 0.123 + - type: precision_at_20 + value: 3.415 + - type: precision_at_3 + value: 5.9270000000000005 + - type: precision_at_5 + value: 5.659 + - type: recall_at_1 + value: 2.94 + - type: recall_at_10 + value: 26.951999999999998 + - type: recall_at_100 + value: 58.11500000000001 + - type: recall_at_1000 + value: 71.777 + - type: recall_at_20 + value: 42.042 + - type: recall_at_3 + value: 9.915000000000001 + - type: recall_at_5 + value: 16.814999999999998 + task: + type: Retrieval + - dataset: + config: default + name: MTEB StackOverflowQA (default) + revision: db8f169f3894c14a00251061f957b2063eef2bd5 + split: test + type: CoIR-Retrieval/stackoverflow-qa + metrics: + - type: main_score + value: 55.907 + - type: map_at_1 + value: 46.991 + - type: map_at_10 + value: 52.763000000000005 + - type: map_at_100 + value: 53.386 + - type: map_at_1000 + value: 53.432 + - type: map_at_20 + value: 53.141000000000005 + - type: map_at_3 + value: 51.044999999999995 + - type: map_at_5 + value: 51.98500000000001 + - type: mrr_at_1 + value: 46.99097291875627 + - type: mrr_at_10 + value: 52.76291175112643 + - type: mrr_at_100 + value: 53.386278433480506 + - type: mrr_at_1000 + value: 53.431881088094414 + - type: mrr_at_20 + value: 53.140779558381865 + - type: mrr_at_3 + value: 51.04480106987634 + - type: mrr_at_5 + value: 51.985122032765055 + - type: ndcg_at_1 + value: 46.991 + - type: ndcg_at_10 + value: 55.907 + - type: ndcg_at_100 + value: 59.019 + - type: ndcg_at_1000 + value: 60.416000000000004 + - type: ndcg_at_20 + value: 57.269999999999996 + - type: ndcg_at_3 + value: 52.337 + - type: ndcg_at_5 + value: 54.053 + - type: precision_at_1 + value: 46.991 + - type: precision_at_10 + value: 6.595 + - type: precision_at_100 + value: 0.807 + - type: precision_at_1000 + value: 0.092 + - type: precision_at_20 + value: 3.566 + - type: precision_at_3 + value: 18.689 + - type: precision_at_5 + value: 12.056000000000001 + - type: recall_at_1 + value: 46.991 + - type: recall_at_10 + value: 65.948 + - type: recall_at_100 + value: 80.692 + - type: recall_at_1000 + value: 92.07600000000001 + - type: recall_at_20 + value: 71.314 + - type: recall_at_3 + value: 56.068 + - type: recall_at_5 + value: 60.281 + task: + type: Retrieval + - dataset: + config: default + name: MTEB SyntheticText2SQL (default) + revision: 686b87296c3a0191b5d9415a00526c62db9fce09 + split: test + type: CoIR-Retrieval/synthetic-text2sql + metrics: + - type: main_score + value: 35.068 + - type: map_at_1 + value: 2.547 + - type: map_at_10 + value: 26.267000000000003 + - type: map_at_100 + value: 27.162999999999997 + - type: map_at_1000 + value: 27.229 + - type: map_at_20 + value: 26.789 + - type: map_at_3 + value: 23.717 + - type: map_at_5 + value: 25.237 + - type: mrr_at_1 + value: 21.329687232951635 + - type: mrr_at_10 + value: 36.61716352922974 + - type: mrr_at_100 + value: 37.472046007482554 + - type: mrr_at_1000 + value: 37.53687081095924 + - type: mrr_at_20 + value: 37.10337925231122 + - type: mrr_at_3 + value: 34.30752577906894 + - type: mrr_at_5 + value: 35.66712242921432 + - type: ndcg_at_1 + value: 2.547 + - type: ndcg_at_10 + value: 35.068 + - type: ndcg_at_100 + value: 39.708 + - type: ndcg_at_1000 + value: 41.454 + - type: ndcg_at_20 + value: 36.943 + - type: ndcg_at_3 + value: 29.837000000000003 + - type: ndcg_at_5 + value: 32.583 + - type: precision_at_1 + value: 2.547 + - type: precision_at_10 + value: 6.165 + - type: precision_at_100 + value: 0.84 + - type: precision_at_1000 + value: 0.098 + - type: precision_at_20 + value: 3.451 + - type: precision_at_3 + value: 15.769 + - type: precision_at_5 + value: 10.798 + - type: recall_at_1 + value: 2.547 + - type: recall_at_10 + value: 61.648 + - type: recall_at_100 + value: 84.03699999999999 + - type: recall_at_1000 + value: 97.77799999999999 + - type: recall_at_20 + value: 69.014 + - type: recall_at_3 + value: 47.308 + - type: recall_at_5 + value: 53.991 + task: + type: Retrieval + - dataset: + config: default + name: MTEB TRECCOVID (default) + revision: bb9466bac8153a0349341eb1b22e06409e78ef4e + split: test + type: mteb/trec-covid + metrics: + - type: main_score + value: 44.603 + - type: map_at_1 + value: 0.128 + - type: map_at_10 + value: 1.012 + - type: map_at_100 + value: 4.585999999999999 + - type: map_at_1000 + value: 11.622 + - type: map_at_20 + value: 1.611 + - type: map_at_3 + value: 0.35500000000000004 + - type: map_at_5 + value: 0.586 + - type: mrr_at_1 + value: 50.0 + - type: mrr_at_10 + value: 61.15555555555555 + - type: mrr_at_100 + value: 61.673274833274824 + - type: mrr_at_1000 + value: 61.70384154456443 + - type: mrr_at_20 + value: 61.43174603174602 + - type: mrr_at_3 + value: 59.33333333333333 + - type: mrr_at_5 + value: 60.73333333333333 + - type: ndcg_at_1 + value: 45.0 + - type: ndcg_at_10 + value: 44.603 + - type: ndcg_at_100 + value: 32.218 + - type: ndcg_at_1000 + value: 28.721999999999998 + - type: ndcg_at_20 + value: 40.752 + - type: ndcg_at_3 + value: 45.641999999999996 + - type: ndcg_at_5 + value: 45.903 + - type: precision_at_1 + value: 50.0 + - type: precision_at_10 + value: 48.4 + - type: precision_at_100 + value: 33.339999999999996 + - type: precision_at_1000 + value: 13.794 + - type: precision_at_20 + value: 43.3 + - type: precision_at_3 + value: 51.333 + - type: precision_at_5 + value: 51.6 + - type: recall_at_1 + value: 0.128 + - type: recall_at_10 + value: 1.226 + - type: recall_at_100 + value: 7.185999999999999 + - type: recall_at_1000 + value: 27.279999999999998 + - type: recall_at_20 + value: 2.088 + - type: recall_at_3 + value: 0.40299999999999997 + - type: recall_at_5 + value: 0.69 + task: + type: Retrieval + - dataset: + config: default + name: MTEB TempReasonL1 (default) + revision: 9097e99aa8c9d827189c65f2e11bfe756af439f6 + split: test + type: RAR-b/TempReason-l1 + metrics: + - type: main_score + value: 1.8399999999999999 + - type: map_at_1 + value: 0.0 + - type: map_at_10 + value: 1.0370000000000001 + - type: map_at_100 + value: 1.275 + - type: map_at_1000 + value: 1.322 + - type: map_at_20 + value: 1.16 + - type: map_at_3 + value: 0.512 + - type: map_at_5 + value: 0.767 + - type: mrr_at_1 + value: 0.0 + - type: mrr_at_10 + value: 1.036884920634922 + - type: mrr_at_100 + value: 1.274587207544468 + - type: mrr_at_1000 + value: 1.3215562619414125 + - type: mrr_at_20 + value: 1.1597194843769947 + - type: mrr_at_3 + value: 0.5125 + - type: mrr_at_5 + value: 0.7674999999999993 + - type: ndcg_at_1 + value: 0.0 + - type: ndcg_at_10 + value: 1.8399999999999999 + - type: ndcg_at_100 + value: 3.206 + - type: ndcg_at_1000 + value: 4.7940000000000005 + - type: ndcg_at_20 + value: 2.2800000000000002 + - type: ndcg_at_3 + value: 0.7000000000000001 + - type: ndcg_at_5 + value: 1.167 + - type: precision_at_1 + value: 0.0 + - type: precision_at_10 + value: 0.45199999999999996 + - type: precision_at_100 + value: 0.11399999999999999 + - type: precision_at_1000 + value: 0.025 + - type: precision_at_20 + value: 0.313 + - type: precision_at_3 + value: 0.41700000000000004 + - type: precision_at_5 + value: 0.48 + - type: recall_at_1 + value: 0.0 + - type: recall_at_10 + value: 4.5249999999999995 + - type: recall_at_100 + value: 11.450000000000001 + - type: recall_at_1000 + value: 24.75 + - type: recall_at_20 + value: 6.25 + - type: recall_at_3 + value: 1.25 + - type: recall_at_5 + value: 2.4 + task: + type: Retrieval + - dataset: + config: default + name: MTEB TempReasonL2Context (default) + revision: f2dc4764024ae93cc42d9c09bc53a31da1af84b2 + split: test + type: RAR-b/TempReason-l2-context + metrics: + - type: main_score + value: 10.853 + - type: map_at_1 + value: 4.317 + - type: map_at_10 + value: 8.261000000000001 + - type: map_at_100 + value: 9.324 + - type: map_at_1000 + value: 9.443 + - type: map_at_20 + value: 8.738999999999999 + - type: map_at_3 + value: 6.658 + - type: map_at_5 + value: 7.475 + - type: mrr_at_1 + value: 4.317213266629609 + - type: mrr_at_10 + value: 8.260541570713873 + - type: mrr_at_100 + value: 9.324279575140654 + - type: mrr_at_1000 + value: 9.44289907211348 + - type: mrr_at_20 + value: 8.738944029983301 + - type: mrr_at_3 + value: 6.658019887591863 + - type: mrr_at_5 + value: 7.475140510159946 + - type: ndcg_at_1 + value: 4.317 + - type: ndcg_at_10 + value: 10.853 + - type: ndcg_at_100 + value: 17.087 + - type: ndcg_at_1000 + value: 20.593 + - type: ndcg_at_20 + value: 12.598999999999998 + - type: ndcg_at_3 + value: 7.467 + - type: ndcg_at_5 + value: 8.931000000000001 + - type: precision_at_1 + value: 4.317 + - type: precision_at_10 + value: 1.934 + - type: precision_at_100 + value: 0.51 + - type: precision_at_1000 + value: 0.079 + - type: precision_at_20 + value: 1.313 + - type: precision_at_3 + value: 3.273 + - type: precision_at_5 + value: 2.672 + - type: recall_at_1 + value: 4.317 + - type: recall_at_10 + value: 19.344 + - type: recall_at_100 + value: 50.973 + - type: recall_at_1000 + value: 79.35900000000001 + - type: recall_at_20 + value: 26.255 + - type: recall_at_3 + value: 9.82 + - type: recall_at_5 + value: 13.358999999999998 + task: + type: Retrieval + - dataset: + config: default + name: MTEB TempReasonL2Fact (default) + revision: 13758bcf978613b249d0de4d0840f57815122bdf + split: test + type: RAR-b/TempReason-l2-fact + metrics: + - type: main_score + value: 6.4430000000000005 + - type: map_at_1 + value: 1.538 + - type: map_at_10 + value: 4.381 + - type: map_at_100 + value: 5.583 + - type: map_at_1000 + value: 5.755 + - type: map_at_20 + value: 4.888 + - type: map_at_3 + value: 3.042 + - type: map_at_5 + value: 3.6929999999999996 + - type: mrr_at_1 + value: 1.5378914211599035 + - type: mrr_at_10 + value: 4.380615627141467 + - type: mrr_at_100 + value: 5.582481340163152 + - type: mrr_at_1000 + value: 5.7543358690140085 + - type: mrr_at_20 + value: 4.888341522384855 + - type: mrr_at_3 + value: 3.041813353097401 + - type: mrr_at_5 + value: 3.693101105552468 + - type: ndcg_at_1 + value: 1.538 + - type: ndcg_at_10 + value: 6.4430000000000005 + - type: ndcg_at_100 + value: 14.17 + - type: ndcg_at_1000 + value: 18.682000000000002 + - type: ndcg_at_20 + value: 8.291 + - type: ndcg_at_3 + value: 3.5709999999999997 + - type: ndcg_at_5 + value: 4.749 + - type: precision_at_1 + value: 1.538 + - type: precision_at_10 + value: 1.329 + - type: precision_at_100 + value: 0.541 + - type: precision_at_1000 + value: 0.09 + - type: precision_at_20 + value: 1.0290000000000001 + - type: precision_at_3 + value: 1.7049999999999998 + - type: precision_at_5 + value: 1.5970000000000002 + - type: recall_at_1 + value: 1.538 + - type: recall_at_10 + value: 13.285 + - type: recall_at_100 + value: 54.14099999999999 + - type: recall_at_1000 + value: 89.642 + - type: recall_at_20 + value: 20.586 + - type: recall_at_3 + value: 5.114 + - type: recall_at_5 + value: 7.986 + task: + type: Retrieval + - dataset: + config: default + name: MTEB TempReasonL2Pure (default) + revision: 27668949b97bfb178901e0cf047cbee805305dc1 + split: test + type: RAR-b/TempReason-l2-pure + metrics: + - type: main_score + value: 0.211 + - type: map_at_1 + value: 0.055999999999999994 + - type: map_at_10 + value: 0.14100000000000001 + - type: map_at_100 + value: 0.259 + - type: map_at_1000 + value: 0.337 + - type: map_at_20 + value: 0.179 + - type: map_at_3 + value: 0.10200000000000001 + - type: map_at_5 + value: 0.11 + - type: mrr_at_1 + value: 0.055586436909394105 + - type: mrr_at_10 + value: 0.14145865869045413 + - type: mrr_at_100 + value: 0.2592058422890182 + - type: mrr_at_1000 + value: 0.33682005085387395 + - type: mrr_at_20 + value: 0.17880657618592516 + - type: mrr_at_3 + value: 0.10190846766722254 + - type: mrr_at_5 + value: 0.11024643320363164 + - type: ndcg_at_1 + value: 0.055999999999999994 + - type: ndcg_at_10 + value: 0.211 + - type: ndcg_at_100 + value: 1.088 + - type: ndcg_at_1000 + value: 3.7859999999999996 + - type: ndcg_at_20 + value: 0.356 + - type: ndcg_at_3 + value: 0.11800000000000001 + - type: ndcg_at_5 + value: 0.134 + - type: precision_at_1 + value: 0.055999999999999994 + - type: precision_at_10 + value: 0.044000000000000004 + - type: precision_at_100 + value: 0.053 + - type: precision_at_1000 + value: 0.027999999999999997 + - type: precision_at_20 + value: 0.052 + - type: precision_at_3 + value: 0.055999999999999994 + - type: precision_at_5 + value: 0.041 + - type: recall_at_1 + value: 0.055999999999999994 + - type: recall_at_10 + value: 0.445 + - type: recall_at_100 + value: 5.299 + - type: recall_at_1000 + value: 27.979 + - type: recall_at_20 + value: 1.038 + - type: recall_at_3 + value: 0.167 + - type: recall_at_5 + value: 0.20400000000000001 + task: + type: Retrieval + - dataset: + config: default + name: MTEB TempReasonL3Context (default) + revision: 3c42539652de3d787cecfb897d3b20905e5c7250 + split: test + type: RAR-b/TempReason-l3-context + metrics: + - type: main_score + value: 8.985999999999999 + - type: map_at_1 + value: 2.915 + - type: map_at_10 + value: 6.546 + - type: map_at_100 + value: 7.591 + - type: map_at_1000 + value: 7.71 + - type: map_at_20 + value: 7.015000000000001 + - type: map_at_3 + value: 5.065 + - type: map_at_5 + value: 5.779999999999999 + - type: mrr_at_1 + value: 2.914595571622232 + - type: mrr_at_10 + value: 6.546229351810003 + - type: mrr_at_100 + value: 7.590639752125733 + - type: mrr_at_1000 + value: 7.7100662080438696 + - type: mrr_at_20 + value: 7.014767909905029 + - type: mrr_at_3 + value: 5.064768790480501 + - type: mrr_at_5 + value: 5.7798614249133955 + - type: ndcg_at_1 + value: 2.915 + - type: ndcg_at_10 + value: 8.985999999999999 + - type: ndcg_at_100 + value: 15.088 + - type: ndcg_at_1000 + value: 18.618000000000002 + - type: ndcg_at_20 + value: 10.708 + - type: ndcg_at_3 + value: 5.825 + - type: ndcg_at_5 + value: 7.116 + - type: precision_at_1 + value: 2.915 + - type: precision_at_10 + value: 1.699 + - type: precision_at_100 + value: 0.479 + - type: precision_at_1000 + value: 0.076 + - type: precision_at_20 + value: 1.192 + - type: precision_at_3 + value: 2.681 + - type: precision_at_5 + value: 2.237 + - type: recall_at_1 + value: 2.915 + - type: recall_at_10 + value: 16.991 + - type: recall_at_100 + value: 47.876000000000005 + - type: recall_at_1000 + value: 76.48 + - type: recall_at_20 + value: 23.836 + - type: recall_at_3 + value: 8.043 + - type: recall_at_5 + value: 11.184 + task: + type: Retrieval + - dataset: + config: default + name: MTEB TempReasonL3Fact (default) + revision: 4b70e90197901da24f3cfcd51d27111292878680 + split: test + type: RAR-b/TempReason-l3-fact + metrics: + - type: main_score + value: 6.9239999999999995 + - type: map_at_1 + value: 0.9490000000000001 + - type: map_at_10 + value: 4.482 + - type: map_at_100 + value: 5.673 + - type: map_at_1000 + value: 5.831 + - type: map_at_20 + value: 4.97 + - type: map_at_3 + value: 2.862 + - type: map_at_5 + value: 3.673 + - type: mrr_at_1 + value: 0.9489380930863082 + - type: mrr_at_10 + value: 4.4841269841269815 + - type: mrr_at_100 + value: 5.672705016400986 + - type: mrr_at_1000 + value: 5.831192204277269 + - type: mrr_at_20 + value: 4.970528777573662 + - type: mrr_at_3 + value: 2.8618767886729892 + - type: mrr_at_5 + value: 3.6729929206205796 + - type: ndcg_at_1 + value: 0.9490000000000001 + - type: ndcg_at_10 + value: 6.9239999999999995 + - type: ndcg_at_100 + value: 14.408000000000001 + - type: ndcg_at_1000 + value: 18.734 + - type: ndcg_at_20 + value: 8.698 + - type: ndcg_at_3 + value: 3.512 + - type: ndcg_at_5 + value: 4.978 + - type: precision_at_1 + value: 0.9490000000000001 + - type: precision_at_10 + value: 1.496 + - type: precision_at_100 + value: 0.541 + - type: precision_at_1000 + value: 0.08800000000000001 + - type: precision_at_20 + value: 1.098 + - type: precision_at_3 + value: 1.7999999999999998 + - type: precision_at_5 + value: 1.794 + - type: recall_at_1 + value: 0.9490000000000001 + - type: recall_at_10 + value: 14.957 + - type: recall_at_100 + value: 54.089 + - type: recall_at_1000 + value: 88.47699999999999 + - type: recall_at_20 + value: 21.961 + - type: recall_at_3 + value: 5.4 + - type: recall_at_5 + value: 8.97 + task: + type: Retrieval + - dataset: + config: default + name: MTEB TempReasonL3Pure (default) + revision: 68fba138e7e63daccecfbdad0a9d2714e56e34ff + split: test + type: RAR-b/TempReason-l3-pure + metrics: + - type: main_score + value: 3.9510000000000005 + - type: map_at_1 + value: 0.045 + - type: map_at_10 + value: 2.464 + - type: map_at_100 + value: 3.159 + - type: map_at_1000 + value: 3.2649999999999997 + - type: map_at_20 + value: 2.7439999999999998 + - type: map_at_3 + value: 1.5779999999999998 + - type: map_at_5 + value: 2.075 + - type: mrr_at_1 + value: 0.045187528242205156 + - type: mrr_at_10 + value: 2.4634465173326463 + - type: mrr_at_100 + value: 3.1587507355194098 + - type: mrr_at_1000 + value: 3.2645570477304884 + - type: mrr_at_20 + value: 2.7446874866280555 + - type: mrr_at_3 + value: 1.5777978611236638 + - type: mrr_at_5 + value: 2.07486067178792 + - type: ndcg_at_1 + value: 0.045 + - type: ndcg_at_10 + value: 3.9510000000000005 + - type: ndcg_at_100 + value: 8.116 + - type: ndcg_at_1000 + value: 11.599 + - type: ndcg_at_20 + value: 4.977 + - type: ndcg_at_3 + value: 2.11 + - type: ndcg_at_5 + value: 3.005 + - type: precision_at_1 + value: 0.045 + - type: precision_at_10 + value: 0.877 + - type: precision_at_100 + value: 0.3 + - type: precision_at_1000 + value: 0.059000000000000004 + - type: precision_at_20 + value: 0.642 + - type: precision_at_3 + value: 1.22 + - type: precision_at_5 + value: 1.166 + - type: recall_at_1 + value: 0.045 + - type: recall_at_10 + value: 8.766 + - type: recall_at_100 + value: 30.005 + - type: recall_at_1000 + value: 58.925000000000004 + - type: recall_at_20 + value: 12.833 + - type: recall_at_3 + value: 3.66 + - type: recall_at_5 + value: 5.829 + task: + type: Retrieval + - dataset: + config: default + name: MTEB TopiOCQAHardNegatives (default) + revision: b4cc09fb8bb3a9e0ce0f94dc69c96397a2a47c18 + split: validation + type: mteb/TopiOCQA_validation_top_250_only_w_correct-v2 + metrics: + - type: main_score + value: 10.59 + - type: map_at_1 + value: 4.3 + - type: map_at_10 + value: 8.134 + - type: map_at_100 + value: 8.967 + - type: map_at_1000 + value: 9.154 + - type: map_at_20 + value: 8.498999999999999 + - type: map_at_3 + value: 6.550000000000001 + - type: map_at_5 + value: 7.385 + - type: mrr_at_1 + value: 4.3 + - type: mrr_at_10 + value: 8.133968253968261 + - type: mrr_at_100 + value: 8.966515392145464 + - type: mrr_at_1000 + value: 9.153606984319149 + - type: mrr_at_20 + value: 8.49921878517081 + - type: mrr_at_3 + value: 6.550000000000002 + - type: mrr_at_5 + value: 7.385000000000004 + - type: ndcg_at_1 + value: 4.3 + - type: ndcg_at_10 + value: 10.59 + - type: ndcg_at_100 + value: 15.728 + - type: ndcg_at_1000 + value: 21.025 + - type: ndcg_at_20 + value: 11.944 + - type: ndcg_at_3 + value: 7.282 + - type: ndcg_at_5 + value: 8.797 + - type: precision_at_1 + value: 4.3 + - type: precision_at_10 + value: 1.8599999999999999 + - type: precision_at_100 + value: 0.45199999999999996 + - type: precision_at_1000 + value: 0.087 + - type: precision_at_20 + value: 1.2 + - type: precision_at_3 + value: 3.1329999999999996 + - type: precision_at_5 + value: 2.62 + - type: recall_at_1 + value: 4.3 + - type: recall_at_10 + value: 18.6 + - type: recall_at_100 + value: 45.2 + - type: recall_at_1000 + value: 87.4 + - type: recall_at_20 + value: 24.0 + - type: recall_at_3 + value: 9.4 + - type: recall_at_5 + value: 13.100000000000001 + task: + type: Retrieval + - dataset: + config: default + name: MTEB Touche2020 (default) + revision: a34f9a33db75fa0cbb21bb5cfc3dae8dc8bec93f + split: test + type: mteb/touche2020 + metrics: + - type: main_score + value: 22.398 + - type: map_at_1 + value: 2.067 + - type: map_at_10 + value: 8.579 + - type: map_at_100 + value: 15.012 + - type: map_at_1000 + value: 16.706 + - type: map_at_20 + value: 10.653 + - type: map_at_3 + value: 3.909 + - type: map_at_5 + value: 6.077 + - type: mrr_at_1 + value: 28.57142857142857 + - type: mrr_at_10 + value: 43.01830255911888 + - type: mrr_at_100 + value: 43.96547082263703 + - type: mrr_at_1000 + value: 43.96547082263703 + - type: mrr_at_20 + value: 43.71604198403339 + - type: mrr_at_3 + value: 37.41496598639456 + - type: mrr_at_5 + value: 41.496598639455776 + - type: ndcg_at_1 + value: 24.490000000000002 + - type: ndcg_at_10 + value: 22.398 + - type: ndcg_at_100 + value: 36.604 + - type: ndcg_at_1000 + value: 48.111 + - type: ndcg_at_20 + value: 23.369999999999997 + - type: ndcg_at_3 + value: 21.378 + - type: ndcg_at_5 + value: 23.685000000000002 + - type: precision_at_1 + value: 28.571 + - type: precision_at_10 + value: 21.224 + - type: precision_at_100 + value: 8.408 + - type: precision_at_1000 + value: 1.59 + - type: precision_at_20 + value: 16.735 + - type: precision_at_3 + value: 23.128999999999998 + - type: precision_at_5 + value: 26.122 + - type: recall_at_1 + value: 2.067 + - type: recall_at_10 + value: 15.182 + - type: recall_at_100 + value: 50.768 + - type: recall_at_1000 + value: 86.29299999999999 + - type: recall_at_20 + value: 22.32 + - type: recall_at_3 + value: 4.865 + - type: recall_at_5 + value: 9.24 + task: + type: Retrieval + - dataset: + config: default + name: MTEB Touche2020Retrieval.v3 (default) + revision: 431886eaecc48f067a3975b70d0949ea2862463c + split: test + type: mteb/webis-touche2020-v3 + metrics: + - type: main_score + value: 52.575 + - type: map_at_1 + value: 2.026 + - type: map_at_10 + value: 15.136 + - type: map_at_100 + value: 31.539 + - type: map_at_1000 + value: 34.672 + - type: map_at_20 + value: 21.477 + - type: map_at_3 + value: 5.931 + - type: map_at_5 + value: 9.476999999999999 + - type: mrr_at_1 + value: 63.26530612244898 + - type: mrr_at_10 + value: 77.57045675413023 + - type: mrr_at_100 + value: 77.75598551108757 + - type: mrr_at_1000 + value: 77.75598551108757 + - type: mrr_at_20 + value: 77.75598551108757 + - type: mrr_at_3 + value: 75.85034013605441 + - type: mrr_at_5 + value: 77.27891156462586 + - type: ndcg_at_1 + value: 54.081999999999994 + - type: ndcg_at_10 + value: 52.575 + - type: ndcg_at_100 + value: 55.051 + - type: ndcg_at_1000 + value: 67.027 + - type: ndcg_at_20 + value: 46.561 + - type: ndcg_at_3 + value: 58.48799999999999 + - type: ndcg_at_5 + value: 57.115 + - type: precision_at_1 + value: 63.26500000000001 + - type: precision_at_10 + value: 56.531 + - type: precision_at_100 + value: 18.898 + - type: precision_at_1000 + value: 3.084 + - type: precision_at_20 + value: 44.082 + - type: precision_at_3 + value: 68.027 + - type: precision_at_5 + value: 65.714 + - type: recall_at_1 + value: 2.026 + - type: recall_at_10 + value: 19.494 + - type: recall_at_100 + value: 59.349 + - type: recall_at_1000 + value: 89.84 + - type: recall_at_20 + value: 29.953000000000003 + - type: recall_at_3 + value: 6.819999999999999 + - type: recall_at_5 + value: 11.386000000000001 + task: + type: Retrieval + - dataset: + config: default + name: MTEB WinoGrande (default) + revision: f74c094f321077cf909ddfb8bccc1b5912a4ac28 + split: test + type: RAR-b/winogrande + metrics: + - type: main_score + value: 61.800999999999995 + - type: map_at_1 + value: 29.044999999999998 + - type: map_at_10 + value: 51.514 + - type: map_at_100 + value: 51.896 + - type: map_at_1000 + value: 51.897000000000006 + - type: map_at_20 + value: 51.842 + - type: map_at_3 + value: 46.803 + - type: map_at_5 + value: 49.957 + - type: mrr_at_1 + value: 29.36069455406472 + - type: mrr_at_10 + value: 51.617619423460056 + - type: mrr_at_100 + value: 52.01415572431666 + - type: mrr_at_1000 + value: 52.014274589675495 + - type: mrr_at_20 + value: 51.95996098874236 + - type: mrr_at_3 + value: 46.86924493554325 + - type: mrr_at_5 + value: 50.04209418574064 + - type: ndcg_at_1 + value: 29.044999999999998 + - type: ndcg_at_10 + value: 61.800999999999995 + - type: ndcg_at_100 + value: 63.315 + - type: ndcg_at_1000 + value: 63.324000000000005 + - type: ndcg_at_20 + value: 62.986 + - type: ndcg_at_3 + value: 52.525 + - type: ndcg_at_5 + value: 58.160999999999994 + - type: precision_at_1 + value: 29.044999999999998 + - type: precision_at_10 + value: 9.361 + - type: precision_at_100 + value: 0.9990000000000001 + - type: precision_at_1000 + value: 0.1 + - type: precision_at_20 + value: 4.913 + - type: precision_at_3 + value: 23.02 + - type: precision_at_5 + value: 16.527 + - type: recall_at_1 + value: 29.044999999999998 + - type: recall_at_10 + value: 93.607 + - type: recall_at_100 + value: 99.921 + - type: recall_at_1000 + value: 100.0 + - type: recall_at_20 + value: 98.264 + - type: recall_at_3 + value: 69.06099999999999 + - type: recall_at_5 + value: 82.636 + task: + type: Retrieval --- # Static Embeddings with BERT uncased tokenizer finetuned on various datasets @@ -1165,8 +8630,42 @@ You can finetune this model on your own dataset. | cosine_mrr@10 | 0.5482 | | cosine_map@100 | 0.4203 | +We've evaluated [sentence-transformers/static-retrieval-mrl-en-v1](https://huggingface.co/sentence-transformers/static-retrieval-mrl-en-v1) on NanoBEIR and plotted it against the inferenec speed computed on my [hardware](#hardware-details). For the inference speed tests, we calculated the number of computed query embeddings of the [GooAQ dataset](https://huggingface.co/datasets/sentence-transformers/gooaq) per second, either on CPU or GPU. + +We evaluate against 3 types of models: +1. Attention-based dense embedding models, e.g. traditional Sentence Transformer models like [`all-mpnet-base-v2`](https://huggingface.co/sentence-transformers/all-mpnet-base-v2), [`bge-base-en-v1.5`](https://huggingface.co/BAAI/bge-base-en-v1.5), and [`gte-large-en-v1.5`](https://huggingface.co/Alibaba-NLP/gte-large-en-v1.5). +2. Static Embedding-based models, e.g. [`static-retrieval-mrl-en-v1`](https://huggingface.co/sentence-transformers/static-retrieval-mrl-en-v1), [`potion-base-8M`](https://huggingface.co/minishlab/potion-base-8M), [`M2V_base_output`](https://huggingface.co/minishlab/M2V_base_output), and [`glove.6B.300d`](https://huggingface.co/sentence-transformers/average_word_embeddings_glove.6B.300d). +3. Sparse bag-of-words model, BM25, often a difficult baseline. + +
Click to expand BM25 implementation details + + I relied on the highly efficient [bm25s](https://github.com/xhluca/bm25s) implementation, using `model.get_scores()` on tokens after tokenization and stemming with the English `PyStemmer`. + +
+ +> **NOTE:** Many of the attention-based dense embedding models are finetuned on the training splits of the (Nano)BEIR evaluation datasets. This gives the models an unfair advantage in this benchmark and can result in lower downstream performance on real retrieval tasks. +> +> [static-retrieval-mrl-en-v1](https://huggingface.co/sentence-transformers/static-retrieval-mrl-en-v1) is purposefully not trained on any of these datasets. + +##### GPU +![NanoBEIR performance vs inference speed](img/nano_beir_vs_speed_gpu.png) + +##### CPU +![NanoBEIR performance vs inference speed](img/nano_beir_vs_speed_cpu.png) + +We can draw some notable conclusions from these figures: +1. [`static-retrieval-mrl-en-v1`](https://huggingface.co/sentence-transformers/static-retrieval-mrl-en-v1) outperforms all other Static Embedding models. +2. [`static-retrieval-mrl-en-v1`](https://huggingface.co/sentence-transformers/static-retrieval-mrl-en-v1) is the only Static Embedding model to outperform BM25. +3. [`static-retrieval-mrl-en-v1`](https://huggingface.co/sentence-transformers/static-retrieval-mrl-en-v1) is + * **87.4%** as performant as the commonly used [`all-mpnet-base-v2`](https://huggingface.co/sentence-transformers/all-mpnet-base-v2), + * **24x** faster on GPU, + * **397x** faster on CPU. +4. [`static-retrieval-mrl-en-v1`](https://huggingface.co/sentence-transformers/static-retrieval-mrl-en-v1) is quicker on CPU than on GPU: This model can run extraordinarily quickly everywhere, including consumer-grade PCs, tiny servers, phones, or in-browser. + #### Matryoshka Evaluations +We experimented with the results on NanoBEIR performance when we performed Matryoshka-style dimensionality reduction by truncating the output embeddings to a lower dimensionality. + | Dimensionality | NanoBEIR_mean | NanoArguAna | NanoClimateFEVER | NanoDBPedia | NanoFEVER | NanoFiQA2018 | NanoHotpotQA | NanoMSMARCO | NanoNFCorpus | NanoNQ | NanoQuoraRetrieval | NanoSCIDOCS | NanoSciFact | NanoTouche2020 | |----------------|---------------|-------------|------------------|-------------|-----------|--------------|--------------|-------------|--------------|--------|--------------------|-------------|-------------|----------------| | 1024 | **0.5031** | 0.4077 | 0.3308 | 0.5681 | 0.6921 | 0.3651 | 0.6547 | 0.4040 | 0.3241 | 0.4533 | 0.8950 | 0.2642 | 0.6111 | 0.5702 | @@ -1176,6 +8675,10 @@ You can finetune this model on your own dataset. | 64 | **0.4176** | 0.3424 | 0.2809 | 0.5022 | 0.5480 | 0.2831 | 0.4680 | 0.3739 | 0.2153 | 0.3845 | 0.8525 | 0.1680 | 0.5045 | 0.5050 | | 32 | **0.3532** | 0.2866 | 0.1870 | 0.4292 | 0.4193 | 0.2292 | 0.3602 | 0.3587 | 0.1444 | 0.3525 | 0.8325 | 0.1525 | 0.3983 | 0.4408 | +![NanoBEIR performance vs Matryoshka dimensionality reduction](img/nano_beir_matryoshka.png) + +These findings show that reducing the dimensionality by e.g. 2x only has a 1.47% reduction in performance (0.5031 NDCG@10 vs 0.4957 NDCG@10), while realistically resulting in a 2x speedup in retrieval speed. +