Vision Language Model Prompt Engineering Guide for Image and Video Understanding

Vision language models (VLMs) are evolving at a breakneck speed. In 2020, the first VLMs revolutionized the generative AI landscape by bringing visual understanding to large language models (LLMs) through the use of a vision encoder. These initial VLMs were limited in their abilities, only able to understand text and single image inputs. 

Fast-forward a few years and VLMs are now capable of understanding multi-image and video inputs to perform advanced vision-language tasks such as visual question-answering (VQA), captioning, search, and summarization.

Figure 1. Suite of model customization tools in NVIDIA NeMo

VLM accuracy can be improved for specific use cases by tuning the prompts and model weights. Fine-tuning the model weights can be done with efficient techniques such as PEFT but still requires sufficient data and compute. 

However, prompt engineering and in context learning can also be used to improve the output quality and is as simple as adjusting the text input to the model at runtime. 

This post walks you through the evolution of VLMs and how to effectively prompt a VLM for single image, multi-image, and video understanding use cases. For more information about VLMs and visual AI agents, register for the upcoming Vision for All: Unlocking Video Analytics with AI Agents webinar.

Single-image understanding

From a single image, a VLM is capable of identifying, describing, classifying, and reasoning over what is in the image. Some VLMs have advanced capabilities such as providing bounding box coordinates for certain objects, and translating text in the image. This works best for use cases where an image needs to be classified, captioned or described in text. 

Single-image capability can also be used to detect basic events occurring in a livestream. This can be accomplished by sampling one frame at a time from a livestream and providing it to the VLM along with a prompt describing what to look for. This works especially well when the types of events to detect are large, clearly visible, and slow-moving.

The VLM can also be prompted to output in a structured format such as JSON so that the response can easily be parsed and sent to a database or a notification service. 

For example, we provided Llama-3.2-90b-vision-instruct with the following prompt and image (Figure 2). 

Figure 2. Input image to the VLM for question and answering

Prompt: Is there a fire truck? Is there a fire? Are there firefighters? Output the answer to each question in JSON format.

VLM response: 

For each of the questions, the VLM was able to accurately respond and output it in a structured format to be used for downstream tasks. 

While this method enables the VLM to operate on video input by sampling one frame at a time, the VLM itself does not understand the full context of the video, limiting it to simple use cases. Sampling one frame at a time does not give it the temporal understanding needed to detect actions or trends that occur over time. This can be improved with multi-image and video VLMs. 

Multi-image understanding

Many VLMs can accept multiple images as input allowing the model to compare, contrast, and learn from input images. 

VLMs generally have good zero-shot capability across a wide range of use cases. However, when trying to obtain more precise answers from a VLM on domain-specific use cases, VLMs can struggle because they lack the full context required to accurately respond to the user prompts. 

To better understand this, we used a VLM to estimate the stock level of products. This is an important retail use case as it could autonomously monitor and alert employees when products are running low in a store to avoid lost revenues from stock-outs. 

We provided a single image of a table (Figure 3) that should be fully stocked with several snack products to Cosmos Nemotron 34B and asked it to estimate the stock level. 

Figure 3. Single-image VLM input for retail stock level detection

Prompt: Estimate the stock level of the snack table on a scale of 0–100. 

VLM response: 

The model rates it as 80–90% stocked even though it’s quite clear to the human eye that the stock level is low. From the model’s perspective, it does not know what the table looked like beforehand when looking at a single image. 

Using the Cosmos Nemotron 34B multi-image capability, we provided another image of the snack table at a 100% stock level (Figure 4), giving  the model extra context to produce a more accurate estimate. 

Figure 4. In-context learning input to VLM for retail stock level detection

Prompt: First compare and contrast the stock level of the two images. Then generate an estimate for each image of the stock level on a scale of 0–100%.

VLM response:

By providing the additional image as a reference, the VLM now estimates the stock level of the second image at 20–30%, which is significantly more accurate. 

Multi-image input can also be used in more complex multimodal RAG pipelines where example images are dynamically added to the prompt based on the user’s input. Multi-image VLMs can also be used for basic video understanding by sampling a small set of frames from a video and providing it to the VLM. 

However, many VLMs that accept multi-image input have limited context windows (10–20 frames) and may not have been trained on video data. To truly understand video input, we recommend using a VLM trained on video-specific datasets and a long context window. 

Video understanding 

The latest VLMs have full video understanding capabilities with large context lengths, frame compression and video training data. This enables VLMs to ingest longer videos and have improved temporal understanding: the ability to understand actions and trends over time. 

The first example showed how to use a single frame from a livestream to detect a fire but that same method wouldn’t be able to tell you if the fire is growing or getting smaller because it lacks the temporal context. 

A VLM with long context and video understanding processes many frames across time to understand if the fire is growing or not. Here are some examples from the Cosmos Nemotron model families.

Sequential visual understanding

VLMs with sequential understanding relies on processing frames in a sequence, enabling them to capture the progression of actions and events over time. Unlike multi-image models, which process a limited number of frames without explicit temporal connections, sequential video models incorporate temporal dynamics by analyzing frames in an orderly sequence.

This is helpful in situations where you want to capture the causality of events in a video, as it enables the model to track the progression of actions over time and establish cause-effect relationships. 

The next few examples are based on the following video where a worker drops a box in a warehouse. We discuss a few prompts and responses.

Figure 5. Input warehouse video to the VLM for scene understanding

Prompt: What happened in this video?

VLM response:

VLM response:

The VLM is now able to deduce that the worker might have dropped the box. This suggests that generic prompts might not capture all the nuances correctly unless you direct the VLM to look for something specific. Modify the prompt a little.

Prompt: Elaborate on the visual and narrative elements of this warehouse video in detail. Highlight all anomalies or hazards if present.

VLM response:

The model is now capturing all the important details of the scene along with important events, such as workers dropping a box. The detailed response is useful if you want to perform Q&A with the video later. VLMs with video understanding are also able to address directional questions, as they can understand actions over time. 

Prompt: From which direction did the worker enter the scene?

VLM response:

However, these models often lack precise temporal localization, which can limit their ability to pinpoint when specific events occur. 

Prompt: When did the worker drop the box?

VLM response:

Temporal localization

VLMs incorporating Localization Interpretable Temporal Attention (LITA) or similar temporal localization techniques elevate video understanding by explicitly learning when and where critical events occur. The models achieve this by introducing temporal attention mechanisms that focus on key segments of a video, aligning visual and textual modalities more effectively over time. This helps answer the “when” questions in the videos.

Prompt: When did the worker drop the box?

VLM response:

The optimal prompt format for a VLM depends on the model’s architecture and the nature of the caption pairs used during training. Different training datasets influence how a VLM interprets the prompt.

Conclusion 

This post walked through how VLMs have evolved from supporting only single-image input to being capable of complex temporal reasoning on long video inputs. To get started with VLMs, visit build.nvidia.com and try out some of the prompts shown in this post. For technical questions, see the Visual AI Agent forum. 

VLMs can be used to build various visual AI agents. Explore one of the sample vision AI workflows on the /NVIDIA/metropolis-nim-workflows GitHub repo with no GPU required. To build a video analytics AI agent, try the AI blueprint for video search and summarization.

For more information about VLMs and visual AI agents, register for the upcoming Vision for All: Unlocking Video Analytics with AI Agents webinar.

For more information about LLM prompting, see An Introduction to Large Language Models: Prompt Engineering and P-Tuning.