Login Register

How to Run a Local LLM: Ollama, llama.cpp, LM Studio, and vLLM Compared

DaDesktop Team ·

There are several ways to run an LLM locally. Some tools are designed to make getting started easy, while others give you more control or are built to serve many users at once. The right choice depends on whether you want a simple local chat, a configurable inference engine, or a production API.

Ollama

Ollama is one of the easiest ways to start running models locally. Install it, download a model, and run it from the command line. It also provides a local API for applications and other tools.

Pros:

  • Simple installation and model management
  • Easy command-line workflow
  • OpenAI-compatible API
  • Supports NVIDIA, AMD, Apple Silicon, and Vulkan-based GPU acceleration
  • Modelfiles let you customise models and parameters
  • Supports concurrent requests when enough memory is available

Cons:

  • Less low-level control than llama.cpp
  • Its model management is built around the Ollama ecosystem
  • Not the first choice when you need maximum serving throughput or distributed inference

Difficulty: Low. A good choice if you want to get a model running quickly without dealing with many inference settings.

llama.cpp

llama.cpp is a lightweight C/C++ inference engine focused on running models efficiently across a wide range of hardware. It uses GGUF models and gives you detailed control over how the model is loaded and run.

Pros:

  • Fine-grained control over context, GPU offloading, batching, threads, quantization, and other inference settings
  • Broad hardware support, including CUDA, HIP, Metal, Vulkan, and SYCL
  • Supports many quantization levels from low-bit formats through 8-bit
  • Can split models across GPUs
  • Can use CPU and GPU together when a model is larger than available VRAM
  • Includes llama-server for an OpenAI-compatible API

Cons:

  • More configuration than Ollama or LM Studio
  • GGUF models are generally downloaded and managed separately
  • Many useful settings require understanding inference parameters

Difficulty: Medium. A good choice if you want to control exactly how a model runs or want to experiment with performance and quantization.

LM Studio

LM Studio is a desktop application for downloading, configuring, and running local LLMs. It provides a graphical interface for finding models and managing things such as GPU offloading and context size.

Pros:

  • Simple graphical interface
  • Searches and downloads models through Hugging Face
  • Shows model and resource information before loading
  • OpenAI-compatible API server
  • Can run models headlessly through its llmster server
  • Supports GGUF through llama.cpp and MLX models on Apple Silicon

Cons:

  • Less low-level control than using llama.cpp directly
  • The desktop application is less suited to some server deployments
  • Not primarily designed for large-scale multi-user serving

Difficulty: Low. A good choice if you want to experiment with local models without spending much time in the command line.

vLLM

vLLM is designed for serving LLMs to applications and multiple users. Its main advantage is efficient serving at higher concurrency, using techniques such as PagedAttention, continuous batching, prefix caching, and distributed inference.

Pros:

  • High throughput for multiple concurrent requests
  • Continuous batching and efficient KV-cache management
  • OpenAI-compatible API server
  • Works directly with many Hugging Face models
  • Supports quantization including FP8, INT4, GPTQ, AWQ, GGUF, and others
  • Supports tensor, pipeline, expert, and other forms of parallelism
  • Designed for production inference and serving

Cons:

  • More complicated setup and configuration
  • Primarily aimed at Linux environments
  • Usually unnecessary for a single person running one model interactively
  • Hardware and model compatibility need to be checked before deployment

Difficulty: High. Best suited to people deploying an inference service rather than simply running a model on a personal computer.

Which one should you pick?

  • Just want to run a model easily: Ollama or LM Studio. Choose Ollama for the command line and simple API, or LM Studio for a graphical interface.
  • Want control over inference: llama.cpp. It gives you direct control over model loading, quantization, context, GPU offloading, and other settings.
  • Need a local API: Ollama, llama.cpp, or LM Studio. All three provide OpenAI-compatible APIs.
  • Need to serve many users: vLLM. Its continuous batching and distributed inference features are designed for this use case.
  • Want to experiment with different quantizations: llama.cpp or LM Studio.

Run it on DaDesktop

If you do not have enough GPU hardware locally, you can run these tools on a DaDesktop cloud desktop. Choose a GPU with enough VRAM for the model you want to run, launch the desktop, and install the inference software you prefer.

Ollama and LM Studio are useful when you want a simple local environment. llama.cpp gives you more control over the hardware and inference settings. vLLM is an option when you need to expose a model as a higher-throughput API.

View available GPUs to compare VRAM and other specifications.