Get the best result from your prompt
Nobody knows which AI model will give the best result to your prompt before you compare them
Current LLM performance rankings, such as those found on lmsys.org or livebench.ai, estimate the probability of one model outperforming another one in specific categories. Notably, the top-ranked LLM only has a slightly better than 50% chance of outperforming the second-ranked one.
Even with sophisticated algorithms that consider category-specific rankings (e.g. reasoning, coding, writing, languages, …), the difference in performance is not significant.
Therefore, the best approach is to reprompt and compare the models:
- Reprompting with a second LLM to verify an AI assertion.
- Having multiple AI proposals help select the best outcome.
Empirical data from Mammouth
Reprompting Data from Mammouth AI
| Number of Text models solicited per prompts | % of total prompts |
|---|---|
| > = 4 | 7% |
| >= 3 | 12% |
| >= 2 | 34% |
| = 1 | 66% |
| Number of Image Models solicited per prompts | % of total prompts |
|---|---|
| >= 3 | 19% |
| >= 2 | 41% |
| = 1 | 59% |
For 66% of daily text queries, users solicit one model
- 66% of user’s queries do not need a reprompting (based on Mammouth.ai data).
For 34% of daily text queries, users solicit more than one model
- As a consequence, 34% of total queries benefit from reprompting. Those 34% correspond to the high-value queries. Those queries are more creative and more complex.
- 24% of total prompts are reprompted with exactly 1 model
- 12% of total prompts are reprompted with 2 models or more.
- 7% of total prompts are reprompted with more than 3 models
Reprompting is even more popular with Image generation tools
- Indeed, 41% of image queries are reprompted with at least one model.
- 19% of those queries are reprompted with more than two models.
As AI models are getting more performant, the definition of the best result is becoming more subjective and less objective
There are two reasons to favor one model result to another:
- The objective reason : User favors the model that respects the rule of his prompt and provide the correct answer.
- The subjective reason : When both AIs give an objectively correct answer, one model can be favored by the user for subjective reason.
—> As AI performance improves, the differentiation progressively moves from objective to subjective. It makes reprompting even more relevant. Hence this LLM popularity Index.