LLMs with Local Models

Running language models on your own hardware gives you privacy, zero per-token cost, and the ability to work offline. The tradeoff is that local models are generally smaller and less capable than the frontier models available through cloud APIs, and running larger models requires significant GPU memory or Apple Silicon unified memory.

In this chapter we focus on Ollama, the most popular tool for running local models. Ollama handles model downloading, quantization, GPU acceleration, and exposes a simple API; you can go from zero to a running local LLM in minutes. We also briefly mention alternative tools at the end of the chapter.

If you want to go deeper into Ollama, including tool use, agents, RAG, and advanced configuration, see my book Ollama in Action.

The examples for this chapter are in the directory source-code/llm_local_models.

Ollama is available for macOS, Linux, and Windows. On macOS:

Or download the installer from ollama.com. After installation, start the Ollama service:

This starts a local server on port 11434. The service runs in the background and manages model loading, GPU memory, and request handling.

Ollama uses a Docker-like model for pulling and running models. To download a model:

This downloads Meta’s Llama 3.2 with 3 billion parameters, quantized to about 2 GB. You can interact with it immediately from the command line:

Some recommended models to start with:

As in the previous chapter, we drive the model through litellm, the open-source library that puts a single OpenAI-format interface on more than 100 providers. The local server is simply another provider: ollama_chat/llama3.2:3b names the model, and switching to a cloud model means changing that string rather than the code.

Two demos are the exception. The vision example and the prompt-caching benchmark use Ollama’s own Python SDK, because they depend on native features that litellm’s normalized response does not carry: the images field on a message, and the prompt_eval_duration / prompt_eval_count fields that make a cache hit measurable. We will point out the switch in each section.

The simplest use of litellm: send a prompt and print the response:

This is the same shape as the cloud examples from the previous chapter — a provider-prefixed model string, a message list, and a chat completion response — but the request never leaves your machine.

For interactive applications, streaming lets users see output as it’s generated rather than waiting for the complete response:

Each chunk carries an incremental piece of the response in chunk.choices[0].delta.content. The flush=True argument ensures text appears immediately rather than being buffered.

Some local models support explicit chain-of-thought reasoning, where the model shows its thinking process before providing a final answer. DeepSeek-R1 is particularly good at this.

First pull the model:

Here is an example that extracts both the reasoning trace and the final answer:

The code checks two places for the reasoning trace. Models served by Ollama usually leave it inline in the content as a <think>...</think> block, which we split out by hand; providers that return the trace as a separate field populate reasoning_content on the message, and litellm surfaces that for you via getattr(response.choices[0].message, "reasoning_content", None). Checking both is the portable habit: the model’s reasoning shows each step of its thinking, making the output more transparent and debuggable than a black-box answer. This is especially valuable for math, logic, and planning tasks.

Cloud APIs handle conversation history by passing the full message list with each request. With local models the same pattern applies, but since there are no per-token costs, you can maintain longer conversations without worrying about expense.

Here is an example that maintains a conversation with memory across multiple exchanges and uses a system prompt to shape the assistant’s personality:

Note that unlike cloud APIs, keeping long conversation histories in local models is free; there are no per-token costs. The main constraint is the model’s context window size, which varies by model (typically 4K to 128K tokens).

When you send the same long context (a document, a knowledge base, or a detailed system prompt) with multiple questions, Ollama can cache the prompt processing to dramatically speed up subsequent requests. This happens automatically when the prefix of the prompt is identical across requests.

This is the one demo that uses Ollama’s own Python SDK instead of litellm. The measurement depends on prompt_eval_duration and prompt_eval_count, which are fields of Ollama’s native response; litellm normalizes every provider into the OpenAI response shape and does not carry them, so here the native client earns its keep. Here is the example:

A run looks like this: both requests carry the same long context, and the warm one evaluates almost none of it:

The key settings for prompt caching:

• keep_alive: passed to ollama.generate(...), this keeps the model and its KV cache in memory between requests. A long duration like "60m" avoids paying the load cost again.
• num_ctx: pins the context window size so both requests are measured against the same window.
• Identical prefix: the cached portion must match exactly. If even one character of the context changes, the cache is invalidated — which is also why the example prepends a fresh nonce on each run, so the first request really is cold.
• Measure prompt evaluation, not the whole call: prompt_eval_duration covers only the tokens Ollama had to evaluate, so it isolates the thing caching improves. The second request reports the same 2,500-token prompt as the first, but evaluates it in a fraction of the time.

Prompt caching is especially valuable for applications like document Q&A, where you load a long document once and then answer many questions about it.

Ollama also supports vision models, allowing you to pass an image along with your text prompt so the model can analyze the visual content. You just need to ensure you’re using a vision-capable model (like llava or qwen3.5). Ollama’s own Python SDK takes the image as the images field on a message — a path, a URL, or raw bytes — and handles the encoding for you, so this is the second demo where we use the native client instead of litellm.

Here is the sample image we will use for this example:

Here is an example of asking a vision model to describe an image:

Here is abbreviated output from running this example:

Even this small 0.8B-parameter vision model extracts detailed structured information from the ticket image: event details, pricing, seating, and layout elements. This simple capability makes it easy to add image understanding to your local applications without needing complex computer vision pipelines.

Ollama exposes an OpenAI-compatible API endpoint — the same protocol litellm speaks to every provider. The local server is therefore not a special case; it is simply the ollama_chat provider, and the code below runs unchanged against a cloud API:

Set LLM_MODEL to a cloud model to send the same request elsewhere:

That one-variable switch is the whole point of routing every example through litellm: you can prototype against a local model and move to a hosted one — for speed, capability, or a larger context window — without rewriting the code.

While Ollama is the system I usually use for running local models, several alternatives exist:

• llama.cpp: The C++ inference engine that Ollama is built on. Use it directly if you need maximum control over quantization, batching, or want to embed inference in a C/C++ application. Available at github.com/ggerganov/llama.cpp.

• LM Studio: A desktop application with a graphical interface for downloading, managing, and chatting with local models. Good for non-programmers or for quickly trying different models. Available at lmstudio.ai.

• vLLM: A high-performance inference server optimized for throughput. Best suited for serving models to multiple users in production. Requires more GPU memory but can handle many concurrent requests efficiently. Available at github.com/vllm-project/vllm.

• Hugging Face Transformers: The transformers Python library can load and run models directly. This gives you the most flexibility for fine-tuning and custom inference pipelines, but requires more setup and GPU memory management. Best for researchers and advanced users.

For most developers getting started with local models, Ollama provides the best balance of simplicity and capability.

The amount of memory you need depends on the model size:

On macOS with Apple Silicon (M1/M2/M3/M4), models run on the GPU using unified memory, which means your total system RAM is also your GPU memory. A MacBook with 16 GB of RAM can comfortably run 7-8B parameter models, and 32 GB or more enables larger models.

On Linux and Windows, a dedicated NVIDIA GPU with sufficient VRAM provides the best performance. Models can also run on CPU only, but inference is significantly slower (roughly 5-10x slower than GPU for most models).

Running LLMs locally with Ollama gives you a private, cost-free, offline-capable alternative to cloud APIs. The setup is straightforward: install Ollama, pull a model, and start making API calls from Python. Features like streaming, conversation memory, prompt caching, and reasoning models make local models practical for many real applications.

The main tradeoff is capability: the largest models that run locally (7-14B parameters on typical hardware) are less capable than frontier cloud models with hundreds of billions of parameters. For many tasks (code assistance, text summarization, data extraction, conversational interfaces), local models perform well enough, and the privacy and cost benefits make them the better choice.

Here are some exercises to help you apply the concepts from this chapter and extend the existing code examples.

• Objective: Extend ollama_streaming.py to create a command-line chat interface. Instead of a single hardcoded query, prompt the user for input in a loop, stream the model’s responses to the console in real-time, and exit when the user types exit or quit.
• Key Concepts: Input loops, real-time output streaming with flush=True, basic text generation.

• Objective: Modify the LocalAssistant class in ollama_memory.py to handle context limits. Implement a maximum history size (e.g., messages). When the conversation history exceeds this limit, the assistant should discard the oldest user/assistant exchanges. However, make sure that the original system prompt is always preserved at the beginning of the message history.
• Key Concepts: System prompt preservation, sliding window list management, message history truncation.

• Objective: Modify image_to_text_description.py to ask the local vision model to extract structured data from ticket.png as a raw JSON block. Instruct the model to return keys like event_name, date, time, venue, and price. In your Python code, parse the model’s output using Python’s built-in json module, print the resulting dictionary, and handle any parsing errors.
• Key Concepts: Vision model prompting, structured output format directives, output parsing with JSON.

• Objective: Create a Python script that implements a multi-model validation pipeline. First, use reason_about from ollama_reasoning.py with ollama_chat/deepseek-r1:7b to solve a logic puzzle (such as a word riddle or math problem). Then, extract the <think> reasoning trace and final answer. Finally, query a smaller general-purpose model like ollama_chat/llama3.2:3b with litellm, passing it both the original question and the reasoning trace, and ask it to verify whether the final answer is logically correct based on the reasoning trace.
• Key Concepts: Multi-model collaboration, chain-of-thought verification, automated self-correction/grading.