Your answer model can’t recover evidence your retriever never found. RAG query rewriting gives small SaaS search a way to translate customer language into terms your knowledge base actually contains.
A user may ask about a “frozen account,” while your help center refers to failed entitlement checks. Or they may ask about refunds through Apple, while the relevant policy uses terms such as third-party billing and subscription cancellation. This vocabulary mismatch causes retrieval failures, not generation failures.
I treat rewriting as a controlled retrieval layer that preserves user intent, rather than a way to make questions sound smarter.
Key Takeaways
- Treat RAG query rewriting as a controlled retrieval layer that preserves the user’s intent, constraints, protected terms, and permissions.
- Route queries based on the likely retrieval failure: skip rewriting for precise exact-match requests, and use expansion, sub-query decomposition, or HyDE when vocabulary gaps, ambiguity, or multiple tasks reduce recall.
- Combine original and rewritten queries with hybrid search, metadata filters, and semantic reranking; generated terms should widen candidate retrieval without becoming mandatory evidence.
- Evaluate retrieval separately from answer generation using Recall@k, MRR, NDCG, constraint retention, citation support, and latency across labeled real-world queries.
- Control the latency and cost tax by using small models, limiting variants, running searches in parallel, caching safe reformulations, and applying rewriting only where it measurably improves evidence retrieval.
Why Small SaaS Search Misses the Right Passage
Most small SaaS teams have a compact knowledge base with a large vocabulary problem. Product documentation uses internal feature names, technical conditions, plan rules, and release-specific language. Customers use their own shorthand, creating a vocabulary mismatch.
Vector search helps bridge meaning, but broadly related chunks can still hurt search relevance. Lexical search has the opposite problem. It can find an exact error code but miss a plain-language description of the same issue.
The first fix is not a larger model. In document retrieval, I start with the document pipeline and inspect how lexical search sees each document.
Check whether PDF extraction preserved headings, tables, lists, and reading order. Store source IDs, document versions, section paths, update dates, product names, and access permissions with each chunk. Query rewriting cannot repair bad chunking or document structure, such as separating a policy exception from its rule.
A multi-tenant product has another requirement. Retrieval must filter for tenant, role, department, plan, and document status before semantic ranking begins. A relevant passage from another customer’s knowledge base is still a serious failure.
A useful chunk contains the rule and the condition that changes the rule. Shared keywords alone are not enough.
RAG Query Rewriting Changes Retrieval, Not User Intent
In a retrieval augmented generation (RAG) pipeline, query rewriting creates one or more retrieval-focused versions of the user’s question before search. A structured prompt template generates those variants and guides this query transformation while keeping the original question intact.
That distinction matters. A rewrite should make the search index easier to query without changing the customer’s intent, constraints, dates, named entities, or permissions.
These approaches overlap, but they aren’t interchangeable. Lexical search favors exact terms, while semantic methods such as vector search can match related language.
| Method | What it produces | Best use case | Main failure mode |
|---|---|---|---|
| Query paraphrasing | A differently worded version of the same question | Vocabulary variation in retrieval | Little improvement if the question is underspecified |
| Query expansion | The original query plus related terms | Synonyms, acronyms, product vocabulary | Noise from loosely related terms |
| Query rewriting | A new search-oriented query or query set | Ambiguous, multi-part, or domain-heavy questions | Semantic drift or dropped constraints |
For example, a retrieval-focused rewrite might turn “Why can’t I invite another admin?” into “admin seat limit, role permissions, pending invitations, account owner settings.” That is more useful to a retriever because it names likely evidence without claiming an answer.
Research on robust query rewriting for information retrieval involving large language models also points to a practical rule I follow: more rewrites aren’t automatically better. Query expansion can improve recall performance, but every additional variant can multiply cost, latency, and irrelevant candidates.
Choose a Query Transformation Based on the Failure
A single query rewriting prompt for every request looks clean in a diagram. It usually wastes calls in production. I route queries based on likely failure, using a prompt template for structured variants and query expansion for missing terms.
Split questions that contain separate retrieval tasks with sub-query decomposition
Sub-query decomposition helps when a customer asks several things at once. Consider a question that combines plan cancellation rules, billing eligibility, and administrator access. One broad retrieval query may find a little evidence for each part, but not enough to answer any of them safely.
I break that request into separate retrieval tasks:
- Find the policy for the billing or cancellation condition.
- Find the account-role rule that affects administrator access.
- Retrieve any document that explains how those rules interact.
The answer model then receives grouped evidence rather than a mixed pile of vaguely related chunks. This sub-query decomposition keeps evidence tied to each task. That is safer for support bots, internal IT search, onboarding assistants, and product documentation search.
Step-back prompting works differently. Instead of restating the exact question, it asks for the broader principle behind it. A narrow question about an exception may need a general subscription policy, a permissions model, or an API authentication rule before the retriever can find the right section.
Use step-back prompting as an additional path, not a replacement for the original wording. Broad context is useful only when the final evidence still answers the narrow question.
Use HyDE carefully when language differs from documentation
Hypothetical document embeddings, often called HyDE, use pseudo-answer generation. Large language models create a short hypothetical answer or document, which an embedding model converts into a vector for vector search.
HyDE can help when users describe an outcome while the documentation explains a mechanism. A user may describe an integration as “stuck,” while the technical guide refers to webhook retries, failed signatures, or delivery logs.
The risk is obvious. Pseudo-answer generation can invent a product feature, policy condition, or technical cause. I never treat it as evidence. It is only a retrieval aid, and the final answer must cite retrieved source material.
Keep the original query in the search set, and preserve error codes, product names, dates, plan names, and billing providers as protected terms for lexical search. If query rewriting changes one of those terms, reject it or fall back to the original search. This pseudo-answer generation remains only a retrieval aid, so the final answer must cite retrieved source material.
Build a Small, Observable Rewrite Pipeline
A small SaaS product doesn’t need five agents for query rewriting. It needs a pipeline where each routing decision is visible and testable, so teams can improve search relevance. I track those decisions in an evaluation pipeline.
I use a sequence like this:
- Classify the query before rewriting. I use a lightweight classifier built with small language models. Skip rewriting for clear exact-match requests. Lexical search handles known HTTP status codes, SKUs, API endpoints, and unique feature names well.
- Create a structured query rewriting artifact for document retrieval. Record the detected intent, protected terms, query transformation, rewrite variants, embedding model, and confidence score. Teams using LangChain can pass the artifact through a prompt template, but don’t overwrite the original query.
- Run the original and rewritten queries together. Use vector search and hybrid search for candidate generation. In Elasticsearch, keep the original as a control candidate query. This protects against a rewrite that expands the search in the wrong direction.
- Apply metadata filters in Elasticsearch before ranking. Tenant boundaries, permissions, product version, document status, and locale should narrow the candidate set before similarity scoring.
- Apply reranking to the candidate set before generation with a semantic ranker. I usually retrieve 20 to 50 candidates. The semantic ranker then selects the strongest five to eight chunks for the answer model.
This is also where I check whether a rewrite is worth keeping. If the baseline query already retrieves the correct passage at rank one, a rewrite adds little value. If the baseline misses it completely, the transformation may be useful.
Combine Rewriting With Hybrid Search and Reranking
Query rewriting works best as part of a retrieval stack. For an Elasticsearch backend, it is not a substitute for hybrid search, filters, or candidate scoring.
Hybrid search combines lexical search, often BM25, with vector search. Lexical search protects exact-match terms such as error codes, plan names, legal language, IDs, and API parameters.
For an Elasticsearch backend, I keep hard constraints separate from LLM-generated terms:
- In Elasticsearch, put tenant, permission, active-document, and product-version requirements in
filterclauses. - In Elasticsearch, put an exact error code or required named entity in a
mustclause when it must appear. - In Elasticsearch, put generated synonyms, related concepts, and pseudo-answer terms in lower-boosted
shouldclauses, where lexical search can contribute optional matches. Treat these terms as query expansion, not hard constraints. - In Elasticsearch, use
minimum_should_matchcarefully. A strict setting can erase the recall benefit of expansion.
Don’t make a model-generated phrase mandatory through lexical search or a must clause unless you’ve validated it. A good-looking rewrite can create an Elasticsearch hard false negative when it adds a term your documents don’t use.
Elasticsearch’s overview of LLM-generated query terms and pseudo-answers explains how pseudo-answer generation supplies hypothetical terms. The practical point is simple: pseudo-answer generation should widen candidate retrieval without becoming mandatory evidence.
After hybrid retrieval, reranking compares the query and each candidate chunk more directly. An Elasticsearch semantic ranker scores candidates for query-passage relevance, not just term overlap. The semantic ranker can move the passage that answers the question above chunks that merely share vocabulary.
Microsoft reports that its Azure AI Search stack can generate 10 rewrites for a 32-token query in 147 milliseconds, then rerank 50 documents in 158 milliseconds, according to its AI Search performance whitepaper. Those figures aren’t Elasticsearch benchmarks, and a semantic ranker on your deployment may have different timing.
Practical takeaway: use query rewriting to widen candidate retrieval, then measure retrieval accuracy across the full Elasticsearch path.
Evaluate Retrieval Before You Judge the Answer
A polished answer can hide a poor retrieval result, even when query rewriting produced a fluent question. I separate retrieval evaluation from generation evaluation because they fail for different reasons.
Start with a labeled query set from real support tickets, product searches, internal questions, failed chatbot conversations, and high-risk policy requests. Include easy requests, ambiguous requests, exact-match lexical search queries, and multi-part questions. Also test sub-query decomposition, vocabulary variation in lexical search, and tight constraints against a hybrid search baseline.
For each test case, record the approved evidence source and the specific chunk or section supporting the answer. Then compare baseline retrieval against each query rewriting strategy, focusing on retrieval accuracy for approved evidence.
Track these metrics by query class:
- Recall@k shows whether the correct evidence reached the candidate set.
- MRR and NDCG show whether the semantic ranker placed the right evidence near the top.
- Constraint retention checks whether rewrites kept names, plan types, dates, codes, and product terms.
- Citation precision and coverage show whether the response cites retrieved evidence and supports every material claim.
- P50 and P95 latency expose slow paths that averages hide.
AWS recommends logging separate latency metrics for embedding, retrieval, ranking, and query classes in its agent performance guidance. The evaluation pipeline should log actual query logs from Elasticsearch, rewritten variants, router decisions, and pseudo-answer generation outputs. It should capture metadata filters from Elasticsearch, chunk IDs from Elasticsearch, reranker scores from Elasticsearch, reranking decisions, semantic ranker scores, citations, and benchmark comparisons with Elasticsearch.
That trace is how you find semantic drift. Automated checks should compare the original query, rewritten variants, and pseudo-answer generation output. They should flag dropped protected terms, added product names, and rewrites that retrieve worse evidence than the baseline.
Run new prompts and rewrite models through the evaluation pipeline against the held-out set before deployment. Release them to a small traffic slice first. If recall performance or citation support falls for a high-risk class, route those requests back to the original query path.
Keep the Latency Tax Under Control
Query rewriting adds an extra model call before retrieval. Multi-query generation can also add more search requests. Meilisearch makes the same production point in its guide to query rewriting for RAG: better retrieval can come with higher latency and cost.
I don’t spend that budget on every query.
Use small language models for rewrites and routing. Limit output to two or three variants, then run independent elasticsearch queries in parallel. Cache stable, low-risk reformulations in elasticsearch when privacy and document freshness allow it. Most importantly, bypass rewriting when the query is already precise.
A request such as HTTP 401 webhook signature needs lexical search and the right documentation version. The HTTP status code should bypass generative expansion.
Set a p95 latency budget for each route. Use an evaluation pipeline to track the full latency path, including the semantic ranker and time to first token, not search time alone. If rewriting improves recall but pushes support chat beyond its response target, apply it only to query classes where the improvement is measurable.
Search for Evidence, Not Better-Sounding Questions
The strongest retrieval augmented generation (RAG) systems don’t use query rewriting for every query. They identify when customer language and document language are misaligned, then apply a bounded transformation to close that gap.
For a small SaaS team, the durable sequence is clear: protect user intent, retrieve with hybrid search, rerank the candidates, validate citations, and monitor every stage. Better evidence retrieval is more valuable than a more elaborate prompt.
FAQ
Does every RAG chatbot need query rewriting?
No. Skip it for exact error codes, product IDs, clear API terms, and questions where baseline retrieval already finds the right passage. Add rewriting when evaluation reveals recurring vocabulary gaps, ambiguity, or multi-part requests.
What is the difference between query expansion and query rewriting?
Query expansion adds related terms to the original question. The broader approach can restructure a request, split it into sub-queries, retrieve wider context, or create a hypothetical document embedding. Expansion is one possible part of a broader rewriting strategy.
How can a team detect semantic drift in rewritten queries?
Compare the original and rewritten query for protected terms, named entities, dates, billing providers, plan names, and product constraints. Then measure whether the rewrite improves recall and citation support against a labeled test set. A rewrite that sounds reasonable but retrieves weaker evidence should fail evaluation.
How many chunks should reach the answer model?
There is no universal number. I start by retrieving 20 to 50 candidates, reranking them, and sending five to eight high-quality chunks to the model. The right count depends on chunk size, document structure, answer complexity, and context limits.