US What a vision language model is
A vision language model (VLM) is a generative AI model that understands images and text together. You give it a picture, a set of frames or a video clip plus an instruction such as “Is anyone in the loading bay without a hi-vis vest?”, and it replies in natural language. It is a kind of multimodal AI: the language model does the reasoning, and a vision encoder lets it see.
That makes a VLM very different from a classic computer vision model. A detector trained on ten classes only ever finds those ten classes, and adding an eleventh means collecting and labeling new images and retraining. A VLM is open-ended. Change the prompt and the same model can caption a scene, count items, read a gauge or explain why a part looks wrong. Well-known open models include LLaVA, NVIDIA’s VILA, Meta’s Llama 3.2 Vision and Alibaba’s Qwen2.5-VL; GPT-4o and Gemini are closed models with the same abilities.
How a VLM works
Most open VLMs share a three-part design. Each part can be swapped, which is why so many variants appear each year.
-
//
Vision encoder. Usually a vision transformer trained on very large sets of image and caption pairs, in the style of OpenAI’s CLIP. It turns each image into a grid of feature vectors.
-
//
Projector. A small set of layers that maps the image features into the language model’s token space, so the image looks to the LLM like a sequence of extra tokens. Some models use cross-attention layers instead.
-
//
Language model. A pretrained large language model reads the image tokens and the text prompt together and generates the answer one token at a time.
-
//
Prompt and context. Users can interleave several images with text and hold a multi-turn conversation, and a VLM can also be built into an agent that calls tools or other models.
How VLMs are trained and benchmarked
Training usually happens in stages. The vision encoder is pre-trained on image and text pairs so that matching pictures and descriptions land close together. The projector is then aligned on captioned images while the LLM stays frozen. Finally, the whole model is instruction-tuned on visual question and answer data so that it follows prompts. Teams adapting a VLM to their own site often fine-tune it further with a small labeled set, typically using parameter-efficient methods such as LoRA.
Published benchmarks such as MMMU, DocVQA and TextVQA compare general reasoning, document reading and text-in-image skills. They are a starting point only. A model that tops a leaderboard can still struggle with your camera angles and lighting, so test candidates on your own footage.
VLMs vs traditional computer vision models
VLMs do not replace detectors and classifiers; they solve different problems. Many deployed systems use both.
| Aspect | Detector or classifier | Vision language model |
|---|---|---|
| Output | Fixed labels, boxes or masks | Free text: answers, captions, reports |
| New object or question | Collect, label and retrain | Change the prompt |
| Typical model size | Millions to tens of millions of parameters | Typically billions of parameters |
| Speed | Every frame in real time on modest hardware | Fractions of a second to seconds per query |
| Behavior | Predictable and easy to measure | Flexible, but can produce wrong answers confidently |
| Best at | Counting, measuring, high-rate detection | Describing, reasoning, search and summaries |
A common pattern at the edge is to let a fast detector watch every frame and pass only the interesting moments to the VLM, which then describes the event, answers a question or writes the alert text.
Where vision language models are used
The most mature uses are those where a plain-language answer is worth more than a box on a screen.
-
//
Video search and summarisation. Operators ask “show every forklift near the fire exit this week” instead of scrubbing footage. It extends classic video analytics from fixed rules to open questions.
-
//
Inspection and reporting. A VLM describes an anomaly in words, so a maintenance ticket arrives with context rather than a bare alarm code.
-
//
Document and gauge reading. Delivery notes, labels, analog dials and HMI screens can be read and turned into structured data.
-
//
Robotics. Robots take natural language instructions and ground them in what their cameras see.
Running a vision language model at the edge
Memory is the first constraint. A VLM’s weights, its image tokens and its text context all sit in memory at once, so module capacity often decides what will fit before compute does. Quantising weights to 8-bit or 4-bit precision shrinks the footprint, typically with a small loss in quality, and smaller models of a few billion parameters are designed with edge devices in mind.
NVIDIA’s Jetson AI Lab publishes guides for running open VLMs on Jetson Orin, and Jetson AGX Orin computers with up to 64 GB of memory are a practical entry point for several camera streams. Jetson Thor systems, with up to 128 GB and a Blackwell GPU, are built for generative and physical AI on the machine itself. For larger models or many concurrent users, edge AI servers with a full-size NVIDIA GPU keep inference on site.
Quick answers on vision language models
Vision language models: common questions
What does VLM stand for?
Vision language model. It is a model that combines image understanding with a large language model, so it can take pictures and text as input and answer in text.
What is the difference between a VLM and an LLM?
Input. An LLM reads and writes text only. A VLM adds a vision encoder so the same kind of language model can also take images or video as input.
Is CLIP a vision language model?
Partly. CLIP links images and text in a shared embedding space and is widely used as the vision encoder inside VLMs, but on its own it matches and ranks rather than generating answers.
Is GPT-4o a vision language model?
Yes, and more. GPT-4o accepts images and text, which makes it a VLM, and it also handles audio, so it is usually described as a multimodal model.
Can a VLM run on a Jetson?
Yes, within memory limits. Smaller, quantised VLMs run on Jetson Orin modules, and Jetson Thor handles larger ones. Throughput is lower than for a detector, so most systems sample frames or trigger the VLM on events.
We build Jetson and GPU edge AI computers for VLM projects with the module, memory, storage and camera inputs fitted and tested as standard. Share your model size, camera count and response time target, and we will size the system.