Skip to content
Obedience ART
← All Open-Source AI Picks
GitHub preview card for ggml-org/llama.cpp

Run models locally

llama.cpp

The fast C/C++ engine that made it possible to run large language models on ordinary laptops and phones.

Rank
#5 in the top 6
Stars
134k
On GitHub since
10 Mar 2023
Licence
MIT
Checked
8 Nov 2026

What it does

llama.cpp runs language models stored in the GGUF format on CPUs and GPUs, using quantisation (storing the model's numbers with fewer bits) so a model that needed a data-centre GPU fits in a laptop's memory. Many friendlier tools are built on top of it. It ships a command-line chat and a server that speaks the same API format as OpenAI's.

How to use it

Shell: PowerShell (Windows 11)

  1. Install the prebuilt binaries with winget (or download a release zip from GitHub).

    winget install llama.cpp

  2. Chat with a small model downloaded straight from Hugging Face.

    llama-cli -hf ggml-org/gemma-3-1b-it-GGUF

  3. Or start a local server with a web chat page at http://127.0.0.1:8080.

    llama-server -hf ggml-org/gemma-3-1b-it-GGUF --host 127.0.0.1 --port 8080

Best practices

  • Start with a Q4_K_M quantised file. It is the usual balance of size and quality; go higher only if you have spare memory.
  • Offload layers to your GPU with -ngl (for example -ngl 99) when you have one; it is many times faster than CPU only.
  • Download GGUF files only from publishers you trust, and prefer official ggml-org or model-maker repos.
  • Bind llama-server to 127.0.0.1 unless you add an API key with --api-key.

In the news

View ggml-org/llama.cpp on GitHub