Best RAG Evaluation Tools for QA Teams Compared
Compare Ragas, Confident AI, Arize Phoenix and Galileo for testing RAG apps: retrieval vs faithfulness metrics, a worked citation check and a pilot plan.

Which RAG evaluation tool should your QA team use?
Ragas, Confident AI (with its open-source DeepEval library), Arize Phoenix and Galileo are all credible RAG evaluation tools, and the right one depends on where your team works. Pick Ragas for a lightweight, open-source library of RAG metrics you call from Python. Pick Confident AI and DeepEval when you want RAG checks to run as pytest-style tests that fail the build. Pick Arize Phoenix when you want to see each retrieval step in a trace and score it there. Pick Galileo when you want a managed platform with named retrieval and generation metrics.
RAG (retrieval-augmented generation) is a design where an application first searches your documents, then gives the best matches to a language model to write the answer. A RAG evaluation tool measures two things separately: did the search find the right material, and did the answer stay true to it.
This comparison uses each vendor's official documentation, checked in November 2026. We did not run a hands-on benchmark of the four products. The policy-assistant example later on is illustrative. For background, see our guide to LLMs in QA testing, and for agent workflows, our comparison of AI agent testing platforms.
Test retrieval and generation as two separate systems
A RAG answer can go wrong in two very different places. If retrieval fails, the model never sees the right document, so even a perfect writer produces a wrong answer. If generation fails, the right document was there but the model ignored it, added facts, or left out an important detail. One overall quality score hides which half broke, and that makes the bug hard to assign.
Four terms come up in every tool:
- Context precision: of the chunks retrieved, how many are actually relevant, and are the relevant ones ranked near the top?
- Context recall: did retrieval find all the information needed to answer?
- Faithfulness (also called groundedness or context adherence): is every claim in the answer supported by the retrieved chunks?
- Answer relevancy: does the answer actually address the question asked?
The DeepEval RAG quickstart makes the same point: evaluate the retriever and generator as separate components, because the output is only as good as the context fed to the model. Note that an answer can be perfectly faithful to an outdated policy and still be wrong for the user, so faithfulness alone is not correctness.
How we compared the tools
- Retrieval metrics: are precision, recall and relevance of retrieved chunks available?
- Generation metrics: is there a documented faithfulness or groundedness check?
- Test workflow: can checks run in CI and fail a build?
- Debugging: can you see which chunk caused a bad answer?
- Customisation: can you add your own rules and judge prompts?
- Deployment: open-source library, hosted platform, or both?
We left out prices because plans change and depend on volume; check each vendor directly.
Comparison table: RAG evaluation tools at a glance
ACCELQ is listed first as an adjacent functional test automation platform and a past SoftwareTestPilot partner. We do not claim it scores retrieval or faithfulness; it covers the web and API flows around a RAG feature, such as login, search screens and document upload.
| Tool | Documented RAG strength | Format | Validate in a pilot |
|---|---|---|---|
| ACCELQ | Functional regression around the AI feature | Platform | Do not assume RAG metric coverage |
| Ragas | Context precision, context recall, noise sensitivity, response relevancy, faithfulness | Open-source Python library | Judge model choice, score stability |
| Confident AI / DeepEval | Retriever and generator tested separately; pytest integration that fails builds | Open-source library plus hosted platform | Metric inputs, report sharing |
| Arize Phoenix | Retrieval relevance per retrieval step in a trace, faithfulness evaluator | Open-source with tracing | Instrumentation, span coverage |
| Galileo | Chunk relevance, context precision, Precision @ K, context adherence, completeness | Managed platform | Agreement with your reviewers |
None is best for every team. Match the tool to how your team already writes and runs tests.
Ragas: an open-source metric library for RAG
Ragas publishes a list of available metrics grouped by task. Under retrieval-augmented generation it lists context precision, context recall, context entities recall, noise sensitivity, response relevancy and faithfulness, alongside non-LLM string comparisons such as exact match and BLEU.
Its faithfulness page explains the calculation clearly: split the response into claims, check each claim against the retrieved context, and divide supported claims by total claims to get a score from 0 to 1. Understanding that formula matters for testers. A long answer with one invented sentence can still score fairly high, so set your pass threshold from real examples rather than guessing.
Strengths: free, transparent formulas, broad RAG metric set.
Limitations: many metrics call a judge model, so scores cost money per run and can vary; you build reporting and CI wiring yourself.
Choose it when a Python team wants well-documented RAG metrics without adopting a platform.
Confident AI and DeepEval: RAG checks as unit tests
DeepEval is an open-source evaluation library, and Confident AI is the platform built around it. The DeepEval RAG quickstart shows how it plugs into pytest through assert_test() and deepeval test run, so failing metrics fail the build locally and on every push or pull request. It also covers testing the retriever and generator separately and evaluating multi-turn RAG conversations.
The Confident AI quickstart describes running evaluations locally with DeepEval or remotely on the platform, with datasets, monitoring and team reports. This split is useful: engineers keep tests in code, while reviewers and managers read shared results.
Strengths: feels familiar to automation testers who know pytest; clear CI story.
Limitations: judge-based metrics still need calibration; non-Python teams lean on the hosted side.
Choose it when you want RAG quality to gate releases the same way unit tests do.
Arize Phoenix: score retrieval where it happens
Phoenix ties evaluation to tracing. Its retrieval relevance evaluator judges whether information retrieved during a step is relevant to the request it served. The documentation stresses that it is independent of the final answer and points to a separate faithfulness evaluator for groundedness. It applies to retriever, reranker and information-returning tool steps, so it also works when retrieval happens through a tool or web search rather than a vector database.
The Phoenix Evals reference lists built-in evaluators for faithfulness, correctness and relevance, batch evaluation over DataFrames, and custom evaluators. Its Evaluate RAG cookbook walks through building and evaluating a pipeline end to end.
Strengths: you can see which step and which chunk caused a bad answer; open source.
Limitations: value depends on instrumenting your pipeline so retrieval steps appear in traces.
Choose it when debugging why answers go wrong matters as much as scoring them.
Galileo: named retrieval and generation metrics on a managed platform
Galileo's RAG metrics overview splits metrics into retrieval quality (chunk relevance, context relevance, context precision, Precision @ K) and generation quality (context adherence and completeness). It also maps symptoms to metrics, for example using Precision @ K when the top results contain a lot of noise and completeness when answers are grounded but miss details.
Its context adherence page defines the metric as a measure of closed-domain hallucination: claims the model made that were not in the provided context. The page reports benchmark figures from Galileo's own internal dataset and notes that performance may vary by use case, so treat them as vendor figures, not independent results.
Strengths: clear symptom-to-metric guidance; managed setup.
Limitations: proprietary platform; judge-based scores still need checking against your reviewers.
Choose it when you want a hosted RAG evaluation workflow with ready-made metrics.
Worked example: deterministic checks for an HR policy assistant
This is an illustrative scenario. An internal assistant answers employee questions from HR policy documents and must cite the document it used. Before adding any judge model, write cheap checks that need no AI at all: did the expected document appear in the top three results, does every citation point to a retrieved document, and is any cited document marked as deprecated?
# Illustrative retrieval and citation checks for a policy assistant (no vendor SDK)
cases = [
{"id": "Q1-leave", "expected_docs": {"leave-policy-v3"},
"retrieved": ["leave-policy-v3", "holiday-list", "travel-policy"],
"cited": ["leave-policy-v3"]},
{"id": "Q2-old-version", "expected_docs": {"travel-policy-v4"},
"retrieved": ["travel-policy-v2", "expense-faq", "holiday-list"],
"cited": ["travel-policy-v2"]},
{"id": "Q3-invented-cite", "expected_docs": {"expense-policy-v1"},
"retrieved": ["expense-policy-v1", "expense-faq", "travel-policy-v4"],
"cited": ["expense-policy-v9"]},
{"id": "Q4-deep-rank", "expected_docs": {"security-policy-v2"},
"retrieved": ["holiday-list", "expense-faq", "travel-policy-v4", "security-policy-v2"],
"cited": ["security-policy-v2"]},
]
K = 3
DEPRECATED = {"travel-policy-v2"}
def check(case):
problems = []
top_k = case["retrieved"][:K]
found = case["expected_docs"] & set(top_k)
recall = len(found) / len(case["expected_docs"])
if recall < 1:
problems.append(f"recall@{K}={recall:.2f}")
for doc in case["cited"]:
if doc not in case["retrieved"]:
problems.append(f"cites {doc} which was never retrieved")
if doc in DEPRECATED:
problems.append(f"cites deprecated {doc}")
return problems
failed = 0
for c in cases:
p = check(c)
failed += bool(p)
print(c["id"], "PASS" if not p else "FAIL: " + "; ".join(p))
print(f"{failed} of {len(cases)} cases fail")
We ran this script locally with Python 3. It printed:
Q1-leave PASS
Q2-old-version FAIL: recall@3=0.00; cites deprecated travel-policy-v2
Q3-invented-cite FAIL: cites expense-policy-v9 which was never retrieved
Q4-deep-rank FAIL: recall@3=0.00
3 of 4 cases fail| Case | Defect | Which half broke | Who fixes it |
|---|---|---|---|
| Q1 | None | - | - |
| Q2 | Outdated policy retrieved and cited | Retrieval and indexing | Whoever maintains the document index |
| Q3 | Invented citation | Generation | Prompt or model owner |
| Q4 | Right document ranked fourth | Retrieval ranking | Search or reranker owner |
Why this works: each failure names a specific half of the system, so the defect report goes to the right person. These checks give the same result every run and cost nothing. Then use Ragas, DeepEval, Phoenix or Galileo for what code cannot judge well, such as whether every sentence in a free-text answer is supported.
Practice exercise: Q2 failed two rules. Which one would still fail if you re-indexed and removed travel-policy-v2, but the retriever then returned no travel document at all? Answer: the recall@3 check, because the expected travel-policy-v4 would still be missing from the top three. For hands-on practice writing assertions against real responses, try our free API Playground.
How to build a RAG test set that finds real bugs
A RAG test set needs more than questions. For each case, record the question, the document or documents that should be retrieved, the facts a correct answer must contain, and anything the answer must not say. Without the expected documents you cannot measure recall, and without required facts you cannot tell a vague answer from a complete one.
Cover these categories deliberately rather than sampling random questions:
- Direct lookups: the answer sits in one clear paragraph. These should almost always pass.
- Multi-document questions: the answer needs two sources, such as a policy and its latest amendment.
- Version traps: an older and a newer document disagree. The assistant must use the current one.
- Unanswerable questions: nothing in the documents answers them. A good assistant says it does not know.
- Wording variation: the same question asked with synonyms, typos or Hinglish phrasing, which often breaks keyword-heavy retrieval.
- Permission boundaries: a user asks about documents they should not see.
Keep the set small at first, around 20 to 30 cases, and grow it every time a real failure is reported. A failure that becomes a saved test case cannot quietly return after the next index rebuild or model change. Store the test set in version control next to your other automated tests so changes are reviewed like code.
Also record the configuration each run used: chunk size, number of results retrieved, embedding model and prompt version. When a score drops, that record tells you what changed.
Common RAG evaluation mistakes
- Only scoring the final answer. You will not know whether to fix search or the prompt.
- Treating faithfulness as correctness. An answer faithful to an outdated document is still wrong.
- No labelled expected documents. Without knowing which document should be found, recall cannot be measured.
- Trusting judge scores blindly. Compare them with human labels on a few dozen cases first.
- Ignoring permissions. Test that one team's documents never appear in another team's answers; this is a security check, not a quality score.
- Forgetting index changes. Re-run the suite whenever documents, chunk size or the embedding model change.
- Too few unanswerable questions. Include questions the documents cannot answer and check the assistant says so.
A practical two-week pilot
- Days 1-3: write 20-30 questions with the expected document for each, including outdated, unanswerable and permission-restricted cases.
- Days 4-5: add deterministic retrieval and citation checks like the example above.
- Days 6-9: run faithfulness and relevance metrics in two shortlisted tools on the same cases.
- Days 10-11: have two people label 30 answers and compare with the tool scores.
- Days 12-14: agree which failures block a release and wire them into CI.
Choose the tool whose failures your team can explain and fix fastest.
Next step
Good RAG testing rests on the same skills as any test design: clear expected results, controlled data and repeatable checks. Build those foundations with our free software testing courses with certificates, then explore automation and AI testing courses when you want structured depth.
Which RAG failure has been hardest for your team to catch? Share it in the SoftwareTestPilot QA Community.
References
- 1Ragas metric list— docs.ragas.io
- 2Ragas faithfulness— docs.ragas.io
- 3DeepEval RAG quickstart— deepeval.com
- 4Confident AI quickstart— www.confident-ai.com
- 5Phoenix retrieval relevance— arize.com
- 6Phoenix Evals reference— arize.com
- 7Phoenix Evaluate RAG cookbook— arize.com
- 8Galileo RAG metrics— docs.galileo.ai
- 9Galileo context adherence— docs.galileo.ai
