Running a model locally

Step 2 of 8. Step 1 was the map: what ragd will do, the three fixed decisions (candle, all-MiniLM-L6-v2, SQLite), and the order the pieces get built in. This step starts the project and ends at the track’s first tag, llm-ch02. By the end you have a model running on your own machine, the arithmetic that says which models fit in memory, and a candle project that compiles and runs — still without weights, on purpose.

First, a model from the shell

The service gets a Rust inference stack in a moment. Before wrapping a model in code, it is worth seeing one raw: the token stream, the load time, and what the weights do to memory. The fastest way in is Ollama — a command-line runner that pulls quantized weights and serves them locally. It is not what the finished service uses; it is the evening’s observation equipment.

Install it (Homebrew also works: brew install ollama; this machine took the release tarball), start the server, and pull a small model:

curl -LO https://github.com/ollama/ollama/releases/latest/download/ollama-darwin.tgz
mkdir -p ~/.local/ollama
tar -xzf ollama-darwin.tgz -C ~/.local/ollama
export PATH="$HOME/.local/ollama:$PATH"

ollama serve        # in one terminal
ollama pull qwen2.5:0.5b   # in another

The pull prints a progress line per blob and a success when done. ollama list shows what landed:

NAME              ID              SIZE      MODIFIED
qwen2.5:0.5b      a8b0c5157701    397 MB    2 minutes ago

Qwen2.5 0.5B is the instruct-tuned half-billion-parameter model from Alibaba, in the Q4_K_M 4-bit quantization — the default ollama serves. Then the first run:

ollama run --verbose qwen2.5:0.5b "What is retrieval-augmented generation? Answer in two sentences."

Retrieval-augmented generation is a technique where the model is trained to
retrieve key sentences from a large, annotated corpus, and then the model
generates new text that closely matches the retrieved sentences. This method
is particularly useful for generating structured, coherent text by leveraging
previously learned knowledge.

total duration:       1.258769583s
load duration:        781.558667ms
prompt eval count:    42 token(s)
prompt eval duration: 234.347ms
prompt eval rate:     179.22 tokens/s
eval count:           58 token(s)
eval duration:        236.316ms
eval rate:            245.43 tokens/s

(The spinner frames that animate over the response are dropped here.) Two things are worth reading out of those numbers. The load duration is the cost of reading 379 MB of weights off disk into memory — paid once per model per session, and the reason a service keeps the model loaded instead of starting a process per question. The eval rate is the generation speed; 245 tokens a second is a fast answer to a two-sentence question.

The weights

ollama pull downloaded a few files into ~/.ollama/models/blobs/, and one of them is nearly all of the 397 MB: the weights, in a 379 MB file whose name is its content hash. Weights are just tensors — the matrices of a transformer — stored in a model-specific binary format. Two formats matter in practice:

safetensors — what Hugging Face publishes for a model like Qwen2.5, and what candle reads directly. The numbers are stored in training precision, two bytes per value (bfloat16 for Qwen2.5).
GGUF — the format llama.cpp (and therefore ollama) runs, where the weights have been quantized: each value squeezed into fewer bits, four on average in Q4_K_M, plus per-block scale factors.

Quantization is the whole reason a model fits on a laptop. The ideal arithmetic is cheap:

parameters × bits ÷ 8 = bytes

494,000,000 × 4 ÷ 8  ≈ 247 MB   # Qwen2.5-0.5B at 4 bits, ideal
494,000,000 × 2      ≈ 988 MB   # the same model at bfloat16

The observed file is 379 MB, above the 247 MB ideal, because the ideal ignores the per-block scales that quantization adds, and because half a billion parameters undercounts the file: the 152,000-entry vocabulary has its own embedding table, and the tokenizer vocabulary lives in the same file. The ideal is still the right first tool — it is what tells you a 7B model at 4 bits is about 4 GB before you spend an evening downloading it.

The memory it needs

Disk size is the download. What the model needs at run time is more, and it has three parts:

loaded memory = weights + KV cache + compute buffers

The weights you know. The KV cache is the part people forget. To generate token n, the model attends over every previous token, and caching the attention keys and values for tokens 0…n−1 is what makes generation linear instead of quadratic. The cache grows with the context window, and it is computable from the model’s config.json (fetched from the model’s Hugging Face page; these are Qwen2.5-0.5B’s real values — 24 layers, 2 key-value heads, 64-dimensional head, and 2 bytes per value at half precision):

2 × layers × kv_heads × head_dim × 2 bytes per token
2 × 24  ×  2       ×  64       × 2      = 12,288 B ≈ 12 KiB per token

Twelve KiB per token sounds like nothing until you multiply by a real context: at the default 32,768, the cache alone is 384 MiB. Ask the model for a longer context and the cache scales with it — memory is where long contexts are paid for.

Watch the same arithmetic land in ollama ps after a run:

NAME              ID              SIZE      PROCESSOR    CONTEXT    UNTIL
qwen2.5:0.5b      a8b0c5157701    936 MB    100% GPU     32768      4 minutes from now

936 MB loaded for a 379 MB file: 379 of weights, ~384 of KV cache at that context length, the rest compute buffers. The 100% GPU says this run went to the Mac’s Metal device — ollama helps itself to the GPU where one exists. The track’s service is CPU-only by contract, and the same API call with the GPU layers turned to zero gives the CPU story:

curl -s http://127.0.0.1:11434/api/generate \
  -d '{"model":"qwen2.5:0.5b","prompt":"What is retrieval-augmented generation? Answer in two sentences.","stream":false,"options":{"num_gpu":0}}'

The JSON response’s response field carries the answer; ollama ps now reports 100% CPU at 1.1 GB loaded, and the streamed generation ran at about 211 tokens a second on this machine (an Apple M2 Max; a 0.5B model is small enough that a modern CPU keeps respectable speed). The step-1 arithmetic said a 1–2B model in 4-bit fits an ordinary laptop: 1.5B parameters is about 0.9 GB of weights at 4 bits, its KV cache is 28 KiB per token (28 layers, 2 KV heads, 128-dimensional head), so a 4,096-token context adds 112 MiB — comfortably inside 8 GB of RAM, with the GPU version unnecessary.

The runtime, in the project

That is the behaviour. Now the project, at the state this step’s tag freezes: llm-apps/, a cargo binary whose only dependency is the runtime itself.

cargo new llm-apps --name llm-apps
cd llm-apps
cargo add candle-core --no-default-features

[package]
name = "llm-apps"
version = "0.1.0"
edition = "2024"

[dependencies]
# The runtime the whole track runs on. `default-features = false` keeps the
# CPU path only - no Metal, no CUDA - which is everything this track needs
# and the shortest dependency tree candle has.
candle-core = { version = "0.11.0", default-features = false }

candle is Hugging Face’s pure-Rust tensor library: no Python, no subprocess, weights loaded straight out of safetensors files. default-features = false turns off every accelerator — no Metal, no CUDA — which matches the track’s contract and leaves the shortest dependency tree candle has. CPU only, and a half-billion-parameter model still moves at the speed the shell run just showed.

The first program does not touch a model. It proves the runtime works before the weights exist: build two tensors on the CPU, multiply, print.

use candle_core::{Device, Tensor};

fn main() -> candle_core::Result<()> {
    let device = Device::Cpu;

    let a = Tensor::arange(0f32, 6f32, &device)?.reshape((2, 3))?;
    let b = Tensor::arange(0f32, 6f32, &device)?.reshape((3, 2))?;
    let c = a.matmul(&b)?;

    println!("running on: {device:?}");
    println!("a = {a}");
    println!("a x b = {c}");

    Ok(())
}

running on: Cpu
a = [[0., 1., 2.],
 [3., 4., 5.]]
Tensor[[2, 3], f32]
a x b = [[10., 13.],
 [28., 40.]]
Tensor[[2, 2], f32]

Two details in ten lines are load-bearing for the rest of the track. Every candle operation returns a Result, so the program ends in ? all the way up — a model that fails to load is an error to handle, not a crash. And the Device is a value, not a global: step 3 will pass the same Device::Cpu into an embedding model, and the hosted-API option in step 7 will sit behind a trait the same way. Nothing here changes when the weights arrive.

The tag

The step’s end state is tagged in the examples repository:

git checkout llm-ch02

At that tag the project is a candle binary that compiles and runs on CPU — the runtime, before the weights. Step 3 loads the first real model into it: all-MiniLM-L6-v2, and the 384-number vectors it produces. The track overview keeps the full list current as articles land.

What we’re building: a local RAG service — the map, if this is your first step in the track
Serving HTTP — where step 7’s streaming API picks up