Choose an embedding model with your data, not the leaderboard
The top-ranked model came fourth on procurement documents. Leaderboards measure something else.
Leaderboards measure something, but not necessarily something that applies to your documents. The benchmark is often generic news articles or Wikipedia. Your data might be technical specifications, procurement documents or policy manuals. What ranks first on MTEB might rank fourth on your content.
We built a set of 120 real questions with their correct documents and ran it through five models. Some candidates we'd never heard of. None of them was the leaderboard champion. We measured mean reciprocal rank at positions one, five and ten.
The results were humbling. The reigning champion came fourth on our dataset. A smaller, less popular model landed in second. The cheapest option was only two points behind the best.
The order shifted against the public ranking, and the cheapest option landed two points behind the best. For our client that two-point gap was acceptable at a quarter the cost of the leader.
That evaluation set stayed in the client's repository. Today it's what they use to approve any change to the system. Every time they consider a new embedding model or a retrieval change, they run it against those 120 questions. It's become their canary for regression.
The lesson is portable: build a small evaluation set from your actual use cases, then measure on that before committing. The leaderboard is a starting point, never the destination.