Evaluating LLMs: BLEU, ROUGE, and AI as a Judge
- Generative AI
- H2O Eval Studio
This lesson walks through a hands on lab that programmatically compares Llama 13B and Vicuna 13B by having each model generate LinkedIn style posts from sample instructions. It computes BLEU and ROUGE scores against the original posts, then shows why those metrics struggle to capture which model is actually more creative or more accurate in context. The lesson introduces the alternative used on H2O's own evaluation leaderboard, prompting a third model such as GPT-3.5 to act as an impartial judge, score each response from one to ten, and explain its reasoning in a structured format.
Key takeaways
- Why BLEU and ROUGE scores are computed the same way but do not reliably compare two LLMs
- How prompting a third model to act as an AB tester judge produces a score and a reasoning behind it
- What comparing Llama 13B and Vicuna 13B on the same generation task reveals about model style
- Why the same judging approach used in this lab matches how H2O's own evaluation leaderboard works
Start owning your AI