US What a local LLM is
A local LLM is a large language model that runs entirely on hardware you own or control, from a desktop GPU to a rack server or an embedded edge AI computer. Nothing is sent to an outside service: the model file sits on local storage, inference runs on the local processor or accelerator, and the answers never leave the machine.
Running a model locally depends on open-weight models, whose trained weights the developer publishes for download. Families such as Meta’s Llama, Mistral, Google’s Gemma, Alibaba’s Qwen and Microsoft’s Phi come in sizes from about one billion to hundreds of billions of parameters. Closed models, including those behind ChatGPT, Claude and Gemini, are only offered as cloud services, so they cannot run locally.
Why run an LLM locally
Cloud LLMs are convenient, but a local model suits sites where data, connectivity or cost rule the cloud out.
-
//
Data stays on site. Prompts, documents and outputs never leave the building, which matters for defense, healthcare, legal and proprietary engineering data.
-
//
Works offline. A local model keeps working on a ship, a mine, a vehicle or an air-gapped network where there is no reliable connection.
-
//
Predictable latency. Response time depends on your hardware, not on network round trips or a shared service’s queue.
-
//
Predictable cost. You pay once for hardware and power rather than per token, which suits steady, high-volume use.
-
//
Version control. The model only changes when you update it, so a validated system behaves the same way next year.
Local LLM hardware: memory comes first
The first question for any local LLM hardware is whether the model fits in memory. On a discrete GPU that means the card’s own memory (VRAM); on a Jetson module, the CPU and GPU share one pool of memory. The weights need roughly 2 bytes per parameter at 16-bit precision, 1 byte at 8-bit and around 0.5 to 0.6 bytes in common 4-bit formats. Add headroom for the KV cache, which grows with the context length and the number of users.
| Model size | Weights at 4-bit | Weights at FP16 | Example hardware from our range |
|---|---|---|---|
| 1–3B | Around 1–2 GB | Around 2–6 GB | Hailo-10H module (8 GB), Jetson Orin Nano 8GB |
| 7–8B | Around 4–5 GB | Around 14–16 GB | Jetson Orin NX 16GB, NVIDIA L4 (24 GB) |
| 13–14B | Around 8–9 GB | Around 26–28 GB | Jetson AGX Orin, NVIDIA L40S (48 GB) |
| 30–34B | Around 18–20 GB | Around 60–68 GB | Jetson Thor T4000 (64 GB), RTX PRO 6000 Blackwell (96 GB) |
| 70B | Around 40 GB | Around 140 GB | Jetson Thor T5000 (128 GB) at 4-bit; H200 NVL (141 GB) or several GPUs at FP16 |
The figures cover weights only and are rounded; check the model card and your runtime’s requirements. Once the model fits, memory bandwidth sets how many tokens per second it produces, because every new token reads through the weights. That is why data centre GPUs with HBM memory answer faster than an embedded module holding the same model. Quantization is what makes the smaller rows practical: 4-bit versions of most open models lose little quality in everyday use, but test them on your own prompts.
Software for running LLMs locally
A local LLM needs a runtime that loads the weights and serves prompts. Most are free and open source.
-
//
llama.cpp. A C/C++ inference engine that runs quantized models in its GGUF format on CPUs and GPUs, including Jetson.
-
//
Ollama and LM Studio. Desktop and server tools built on llama.cpp that download a model and expose a chat window or local API in a few steps.
-
//
vLLM. A serving engine designed for many concurrent users on data centre GPUs, with batching and efficient KV cache handling.
-
//
NVIDIA TensorRT-LLM. NVIDIA’s library for compiling LLMs into optimized engines for its GPUs and Jetson modules, building on TensorRT.
Local LLMs on edge AI computers
On industrial sites, local LLMs usually sit next to the equipment they serve. A panel PC answers maintenance questions from the machine’s manuals, a robot turns spoken instructions into tasks, a vessel searches its own technical library offline, and a security system writes plain-text summaries of alarms. Most of these need a model of a few billion to a few tens of billions of parameters.
For single-user, low-power deployments, a Jetson module is the usual choice. Our Jetson Thor computers carry 64 GB (T4000) or 128 GB (T5000), enough for large quantized models alongside camera pipelines. Where many people query one model, or a larger model must run at higher precision, edge AI servers take data centre GPUs such as the RTX PRO 6000 Blackwell with 96 GB or the H200 NVL with 141 GB. Our guide to choosing the right Jetson module compares memory across the range.
Quick answers on local LLMs
Local LLMs: common questions
How do you run an LLM locally?
Pick a model, a runtime and hardware with enough memory. Download an open-weight model in a quantized format, load it with a runtime such as llama.cpp, Ollama or vLLM, and query it through the chat window or local API.
What is the best hardware for a local LLM?
The one with enough fast memory for your model. A GPU or Jetson module whose memory holds the quantized weights plus context. For many users, choose a GPU server; for low power and harsh sites, a Jetson.
What LLM can I run locally?
It depends on your memory. As a rough guide at 4-bit, 8 GB runs models up to about 8 billion parameters, 24 GB up to about 30 billion and 48 GB or more a 70-billion-parameter model.
Is a local LLM as good as ChatGPT?
For focused tasks it can be close. The largest cloud models still lead on broad reasoning. A smaller local model with your own documents via retrieval often matches them on a narrow, well-defined job.
Can a Jetson run an LLM?
Yes. Jetson Orin and Jetson Thor modules run quantized open-weight models with llama.cpp, Ollama or TensorRT-LLM. Model size is limited by the module’s memory.
We build Jetson and GPU systems for local LLMs with the operating system, drivers and inference runtime installed as standard, and test each one with your chosen model before it ships. Tell us the model, the number of users and the site, and we will size the memory and GPU to match.