How to Enhance RAG Pipelines with Reasoning Using NVIDIA Llama Nemotron Models

A key challenge for retrieval-augmented generation (RAG) systems is handling user queries that lack explicit clarity or carry implicit intent. Users often phrase questions imprecisely. For instance, consider the user query, “Tell me about the latest update in NVIDIA NeMo model training.” It’s possible that the user is implicitly interested in advancements in NeMo large language model (LLM) customization features, rather than its speech models. However, this preference is not expressed explicitly, which can potentially lead to suboptimal results.

Overcoming these limitations and unlocking the true potential of RAG requires moving beyond basic techniques. This post introduces the AI reasoning capabilities of NVIDIA Nemotron LLMs that significantly enhance RAG pipelines. We walk through a real-life example of how we applied advanced strategies, such as query analysis and rewriting, to refine the search capabilities of a query engine.  

What is query rewriting in RAG?

Query rewriting in RAG is a crucial step that transforms a user’s initial prompt into a more optimized query to improve information retrieval. This process is vital for boosting RAG performance because it bridges the semantic gap between how a user asks a question and how information is structured in the knowledge base. By refining the query, the system can overcome issues like vagueness or excessive complexity, leading to the retrieval of more precise and relevant documents. This higher-quality context directly enables the language model to generate more accurate, comprehensive, and factually grounded answers. 

Several techniques have emerged for effective query rewriting, particularly leveraging LLMs:

By employing such techniques, RAG systems can restructure poorly formed questions, introduce vital keywords, and anchor user queries more closely to the semantics of the corpus—substantially elevating both search and answer quality. 

To incorporate the query rewriting technique into RAG, the prompt would need to be tailored specifically to the RAG use cases. Check out a few sample prompts for each method:

Q2E prompt

Q2D prompt

CoT query rewriting prompt

How do NVIDIA Nemotron models advance RAG?

The NVIDIA Nemotron family of reasoning and multimodal models builds on the Meta Llama family to deliver a suite of LLMs optimized for efficiency, performance, and advanced applications like RAG and agentic systems. Nemotron models are an open family of advanced AI models designed to deliver strong reasoning capabilities, high efficiency, and flexible deployment for enterprise AI agents. Available in Nano, Super, and Ultra sizes, these models combine Meta Llama architecture with NVIDIA extensive post-training techniques to achieve top accuracy on industry benchmarks. 

Among the Nemotron family of models, we found that the Llama 3.3 Nemotron Super 49B v1  model suits the use case for driving advancements in RAG the most, particularly considering inference latency and appropriate reasoning ability. Results on the Natural Questions (NQ) dataset clearly show query rewriting significantly improves accuracy of retrieval. Accuracy@K indicates the fraction of questions where a correct answer is found in the top-K retrieved passages. 

Architecture for RAG pipeline with Llama Nemotron

Figure 1 shows the architecture of an enhanced RAG process with Llama 3.3 Nemotron Super 49B v1. 

In the architecture, the Llama Nemotron model is used as a query extractor with the following functions: 

Slack is integrated with the backend with Slack to enable integration with additional applications and to eliminate the need for developing and maintaining a traditional frontend. Several key components ensure seamless communication between the Slack users and the backend, including:  

For the purposes of this post, the architecture shown in Figure 1 is applied to help improve search results for NVIDIA GTC 2025 sessions. Query rewrite ensures the semantic similarity search retrieves a more focused set of sessions. This is further explained with examples in the next section.

How to refine a search query engine with reasoning capabilities

A key challenge that highlights the necessity of query rewriting in a RAG workflow is the semantic gap between the users’ language and the vocabulary of the content. For instance, consider the user query, “Sessions for training an LLM for low-resourced language.” The challenge in this query is the phrase “low-resourced language.” 

With this query, the user is looking for sessions about sessions related to training multilingual LLMs or Sovereign AI. While numerous GTC 2025 sessions discuss this topic, none of them use the key phrase “low-resourced language.” Rather, more common phrases include “multilingual,” “non-English,” “Sovereign AI,” or specific languages such as “Korean” or “French.” For this reason, using the original query to retrieve and rank the relevant sessions is not likely to produce a satisfying result.

To tackle this problem, we adopted the Q2E techniques to rewrite the queries. In this user case, Q2D and COT query rewriting are not appropriate because the user query will be domain specific and general purpose LLM lacks knowledge on creating pseudo documents or context to the user query, leading to a high chance of LLM hallucination. A sample Q2E prompt for this use case is shown below.

For the sample query, “Sessions for training an LLM for low-resourced language,” query expansion can significantly increase the ranking of the most relevant sessions returned by a semantic similarity based retriever. Table 2 provides more details. 

Moreover, query expansion helps the reranker to focus on a broader but still highly relevant scope during the ranking process. For example, the truncated logical thinking token of the Llama Nemotron model with a different query:

Note that with query expansion, the reranker is better equipped to identify sessions that discuss related concepts, even if they don’t use the exact original query terms. This broader perspective enables the reranker to create a more comprehensive and user-centric ranking, surfacing sessions that provide a deeper understanding of the user’s overall information need. 

What are the benefits of query rewriting?

By improving search results through query rewriting, the enhanced pipeline offers a compelling advantage over traditional approaches to RAG. The primary advantage comes from intelligently reformulating user queries. This adds crucial context and details. This step is responsible for creating a high-quality, highly relevant candidate pool, which is the biggest factor in the system’s improved performance.

What are the challenges of this approach?

Query rewriting requires AI inference, which is resource-intensive and slower than traditional methods, limiting scalability. Furthermore, LLMs can only process a limited number of documents at one time, necessitating sliding window strategies for large candidate sets. This increases complexity and can hinder global ranking quality.

When to optimize a RAG pipeline

This enhanced RAG pipeline is especially valuable in domains where accuracy and precision matter more than speed, as detailed in Table 3.  

Get started enhancing your RAG pipelines

In this post, we introduced an innovative approach to improve RAG pipelines using the reasoning capabilities of the NVIDIA Llama Nemotron family of models. By addressing the limitations of traditional methods, this enhanced architecture enables more effective and user-centric information access, particularly in scenarios demanding high precision and nuanced understanding. 

To learn more about the full capabilities of the Llama Nemotron collection of LLM models, see Build Enterprise AI Agents with Advanced Open NVIDIA Llama Nemotron Reasoning Models. You can experiment with NVIDIA NIM models on the NVIDIA API Catalog. Further enhance and accelerate your RAG pipelines with NVIDIA NeMo Retriever and the NVIDIA RAG blueprint.