Run Multimodal Extraction for More Efficient AI Pipelines Using One GPU

As enterprises generate and consume increasing volumes of diverse data, extracting insights from multimodal documents, like PDFs and presentations, has become a major challenge. Traditional text-only extraction and basic retrieval-augmented generation (RAG) pipelines fall short, failing to capture the full value of these complex documents. The result? Missed insights, inefficient workflows, and rising operational costs.

In this blog post, we’ll dive into the key components of building an effective multimodal extraction pipeline, with code examples and one GPU, using NVIDIA NeMo Retriever microservices.

NeMo Retriever extraction is an example architecture for a multimodal document processing pipeline. It uses microservices to efficiently extract information from different file types across millions of documents. Paired with NeMo Retriever embedding and reranking models, it forms a complete, scalable RAG solution as showcased in NVIDIA AI Blueprint for RAG (described in Figure 1).

Figure 1. A diagram of the extraction pipeline as part of the Build an Enterprise RAG NVIDIA AI Blueprint

 In the next section, we’ll take a simple use case that shows the step-by-step NeMo Retriever extraction pipeline, together with other NeMo Retriever components, to use the ingested data.

Completing a business task with multimodal document understanding

For this purpose, we’ll consider an organizational shared folder containing product resources about NVIDIA Blackwell GPUs. The files include text, images, charts, and tables in different types such as PDFs, PPTXs, and JPEGs.

In this example, a customer support engineer is asking for an NVIDIA Blackwell performance comparison to support a partner.

This type of request can be sent by any application, such as a chat user interface or an automatic content generator. Here, we’ll demonstrate it using direct prompting with the pipeline’s Python client.

Step 1: Spin up the pipeline with one GPU

Deploy the blueprint using Docker on an on-prem or cloud machine. See the deployment guide available in the NeMo Retriever extraction quickstart guide on NVIDIA docs. 

In this example, the entire pipeline was deployed on an AWS g6e.xlarge machine (L40S GPU, 48 GB). Verify all deployed services for your desired profile are up and running.

Figure 2. NVIDIA NeMo Retriever extraction deployed services

The pipeline services include the visual elements recognition and OCR (optical character recognition) models, the embedding model, Milvus DB, and the observability tools (Prometheus and Grafana, Attu, Zipkin, and more).

Note: for prototyping purposes, the pipeline source code can be accessed at the blueprint page, Build an Enterprise RAG pipeline. 

Step 2: Submit ingestion job for files in storage

Once all services are up, we can submit ingestion jobs either by Python client or through CLI (command line interface).

We’ll show the Python client usage. 

In this example, we pass the path for the files in our collection and define the tasks we want to be included in the job (i.e., extract, split, and embed‌). We set the extraction task to include all modalities types, and the split task to chunk text to a size of 1,024 tokens.

Step 3: Analyze the job results 

Once the ingestion job is completed, we can analyze the structure of the results (Figure 3):

Figure 3. Extraction job results

The job extraction resulted in several objects from different modalities for each of our documents. These include text, images, and structured objects that refer to charts and tables.

For example, we can see a text object that was extracted:

The text can also be split into smaller chunks. We can control the chunking strategy in our split configuration of the Ingestor.

This is an example of  a random table that was extracted:

Figure 4. A table extracted by the ingestion job

In addition to the visual object extraction, the textual content of it is saved as well:

These objects were chunked and embedded automatically by the job. We can track the vector embeddings in the Milvus collection created automatically by the pipeline through the Milvus client or Attu (web user interface for Milvus) service that was deployed with the rest of the services bundle in step 1.

Figure 5. The Milvus collection was automatically created by the NVIDIA NeMo Retriever extraction pipeline

Step 4: Retrieval

We’ll demonstrate building a retrieval component based on the NeMo Retriever extraction pipeline, ingested data, and NeMo Retriever embedding. First, define the NVIDIA client for the embedding and the generator LLM microservices.

Embed the user query (using the same embedding model used in the ingestion)

Get top similar results to the user’s query using NeMo Retriever extraction Python client retriever:

Create a relevant prompt for the generator LLM and get the response:

We retrieved a highly relevant chunk that didn’t require any direct search and review of the original file.

This simple use case shows how, with a quick deployment setup, we can perform automatic contextual understanding of multimodal enterprise source files. 

Conclusion

The NeMo Retriever extraction pipeline addresses the challenges of multimodal document processing by automatically handling different file types, such as PDFs, presentations, and spreadsheets.t extracts meaningful content from text, images, tables, and charts, changing previously siloed information into accessible, structured data. This enables organizations to unlock deeper insights from their existing knowledge repositories.

The architecture behind this solution brings together advanced components like object detection, chart parsing, and vector embeddings to enable efficient, context-aware retrieval. By preserving relationships across modalities and surfacing them through semantic search, the pipeline delivers a comprehensive approach to document understanding. Implementing this end-to-end pipeline with NeMo Retriever marks a major advancement in enterprise knowledge management, turning static, underutilized documents into high-value assets that can fuel generative AI applications and smarter decision-making.

By continuously extracting and using new data, NeMo Retriever can also help organizations create a data flywheel, where improved data quality leads to better AI models, which in turn generate even more valuable data.

Get started with the NeMo Retriever extraction pipeline using the NVIDIA AI blueprint for RAG, or try the individual NeMo Retriever microservices for extraction, embedding, and reranking on build.nvidia.com.