Acquire a greater understanding of the varied LLM benchmarks and scores and achieve an intuitive understanding of when they might be useful on your functions.
It looks like virtually each week a brand new large-scale language mannequin (LLM) is launched to the general public. Every time an LLM is introduced, these suppliers tout very spectacular efficiency numbers. The problem I’ve discovered is the wide selection of efficiency metrics referenced all through these press releases. There are some metrics that seem extra typically than others, however sadly, there is not only one or two “go-to” metrics. If you wish to see a concrete instance of this, Check out GPT-4’s performance page.. We reference varied benchmarks and scores.
The primary pure query to ask is, “Why cannot we merely agree to make use of a single metric?” In brief, there isn’t any clear technique to consider LLM efficiency, so every efficiency metric makes an attempt to supply a quantitative evaluation of 1 targeted area.. Moreover, many of those efficiency metrics have “sub-metrics” that calculate the metric in a barely completely different method than the unique metric. After I initially began researching for this weblog submit, I meant to cowl all of those benchmarks and scores, however rapidly realized that doing so would imply protecting over 50 completely different metrics.
What I’ve found is which you can break down these varied benchmarks and scores into classes of what you are usually making an attempt to guage, since it isn’t precisely possible to guage every particular person metric. The rest of this submit will focus on these completely different classes and likewise present particular examples of frequent metrics that fall into every of those classes. The aim of this submit is to permit you to go away this submit with a high-level understanding of what efficiency metrics you might be evaluating on your particular use case.
The six classes evaluated on this submit embody: Please be aware: There isn’t any particular “trade normal” for a way these classes are created. These classes have been created based mostly on the strategies we heard referred to most frequently.
- Normal data benchmark
