Applied AI
By the end you'll know the difference between benchmark scores and production reliability, why most evaluation setups are silently broken, and what to measure when your AI feature ships to real users.
6 steps · ~25 minutes of reading total
RAG vs fine-tuning: how to choose
EmpiricaFoundational — both improve LLM quality but solve different problems. Pick the wrong one and the rest of evaluation is meaningless.
Evaluation before deployment
EmpiricaThe full pre-launch evaluation loop: offline benchmarks, shadow traffic, guardrails, what to measure that benchmarks miss.
Benchmark scores vs production reliability
EmpiricaWhy a model topping MMLU can still fail your users — the gap between leaderboard accuracy and the behaviour your pipeline actually requires.
Milestone: you can articulate your eval setup in five sentences
MilestoneIf the eval can't be stated briefly, it's almost certainly testing the wrong thing. Use this as the gate before any model ships.
The real cost of LLM API calls
EmpiricaCost is part of evaluation, not separate from it. A model that's 2% better but 8× more expensive isn't actually better at deployment scale.
LMSYS Chatbot Arena (public leaderboard)
LMSYS ↗Pairwise human preference ranking across major LLMs. Useful as a sanity check against vendor-supplied benchmarks — and free.