AI query rewriting should improve retrieval without changing the user's question or broadening access. Evaluate the rewritten query, retrieved evidence and final answer separately. A more fluent search phrase is not necessarily a more faithful one, especially when the original includes dates, exclusions or domain-specific identifiers.
Let’s plan your retrieval evaluation.
Bring your workflow, constraints and open questions. We’ll help you define the requirements and work through a practical next step.
Define why rewriting is needed
Identify the failure in the baseline search: abbreviations, conversational follow-ups, terminology differences or missing context. Do not add rewriting simply because it is available.
For a hypothetical internal assistant, Who approved it? may need a referenced request from the conversation. The rewrite should use authorised context and preserve ambiguity when the referent is uncertain.
Use a rewrite evaluation matrix
| Case | What must be preserved | Failure to detect |
|---|---|---|
| Exact identifier | Original reference | Identifier paraphrased incorrectly |
| Date constraint | Intended period | Current data substituted for historical scope |
| Negative condition | Exclusion or exception | Not removed or reversed |
| Follow-up question | Correct permitted context | Wrong previous record selected |
| Domain abbreviation | Approved meaning | Common expansion replaces business meaning |
| Ambiguous input | Need for clarification | Confident invented intent |
Check access enforcement after rewriting. Expanding a query must not expand the user's authorised document population.
Compare with a baseline
Use the same task set with and without rewriting. Review whether relevant evidence appears, whether important evidence disappears and whether the final answer remains faithful.
Avoid judging only retrieval count. More documents can mean more distraction. Include questions where the collection cannot answer and where asking a clarifying question is the appropriate outcome.
Keep investigation evidence proportionate
Record the original and rewritten forms where the data policy permits, with source and configuration versions. Restrict sensitive query content and avoid retaining it indefinitely without a purpose.
When results regress, determine whether rewriting, retrieval ranking or answer synthesis caused the failure. Changing all three at once makes the effect harder to assess.
Query-rewriting questions
Should the rewrite always replace the original query?
Compare approaches appropriate to the search system, including retaining the original as evidence or an additional query. Evaluate the complete outcome.
Can users' intent always be inferred from conversation history?
No. Context can be stale, restricted or ambiguous. Provide a clarification path rather than inventing certainty.
Pair the evaluation with citation support checks, and review retrieval behaviour with Drema's generative AI team.