Why your RAG system needs an evaluation harness before it needs a better model

A request teams reach for early: 'Can we switch to the newest model? Our RAG answers aren't good enough.' The usual finding once the system is instrumented: the model was fine, retrieval was starving it.

Without an evaluation harness, teams debug RAG by vibes — read a few bad answers, form a theory, change something, hope. With a harness — a few hundred representative questions with graded reference answers — the same debugging takes an afternoon and produces numbers instead of anecdotes.

Build the harness before you build the features. Grade retrieval and generation separately: retrieval recall tells you whether the right context ever reached the model; groundedness tells you whether the model used it faithfully. These two numbers alone resolve most 'the AI is wrong' complaints into actionable engineering work.

Once the harness exists, every improvement becomes an experiment: chunking strategy, embedding model, reranking, prompt changes. Some will surprise you — reranking can beat a model upgrade by a wide margin, at a tenth of the cost.

A better model is sometimes the answer. But you only know that when you can measure it — and by then, you'll usually have found three cheaper wins first.