Running LLMs Locally Using Ollama
In the last two chapters we used cloud services to run LLMs. For my personal research and experimentation I prefer running smaller models locally on my Mac Mini that has 32G of memory. We will use the Ollama system that runs on macOS, Linux, and Windows. Here we use the Llama 3.2 3B LLM (llama3.2:3b). If you prefer another model, you can substitute its name (such as mistral or qwen3:0.6b) using set-model! or the :model option.
You need to install Ollama on your system: https://ollama.ai
You then need to install the llama3.2:3b model (this takes a while the first time, but the model file is cached so future startups are fast):
ollama run llama3.2:3b
Running the Example Tests
Before we look at the code, you might want to run the example tests first:
In one console, run the Ollama REST API service:
ollama serve
Then run the tests in another console:
lein test
Clojure Client Library for the Ollama Service REST APIs
The following Clojure code (ollama/src/ollama_api/core.clj) defines a client library for interacting with the Ollama REST API text generation and chat services. It includes clojure.spec.alpha input validation, synchronous and streaming completions and chat, and helper functions:
Dependencies:
(:require [clj-http.client :as client])for HTTP requests.(:require [clojure.data.json :as json])for JSON processing.(:require [clojure.spec.alpha :as s])for validating prompts and chat message maps.
ollama-helper is a private function (indicated by - prefix) that makes a POST request to the configured Ollama endpoint (/api/generate or /api/chat), checks for a successful status, and parses the JSON response into a keywordized Clojure map.
The completions and chat functions validate their inputs, merge any caller options with *default-model* ("llama3.2:3b"), and call ollama-helper. Streaming variants (completions-stream and chat-stream) accept an on-chunk callback.
The summarize and answer-question functions wrap completions with task-specific prompt prefixes.
1 (ns ollama-api.core
2 (:require [clj-http.client :as client]
3 [clojure.data.json :as json]
4 [clojure.spec.alpha :as s]))
5
6 ;; Dynamic vars for configuration
7 (def ^:dynamic *base-url* "http://localhost:11434")
8 (def ^:dynamic *default-model* "llama3.2:3b")
9
10 ;; Spec definitions for validation
11 (s/def ::role #(or (#{:user :assistant :system} %)
12 (and (string? %) (#{"user" "assistant" "system"} %))))
13 (s/def ::content string?)
14 (s/def ::message (s/keys :req-un [::role ::content]))
15 (s/def ::messages (s/coll-of ::message :gen-gen-validator true))
16 (s/def ::model string?)
17 (s/def ::prompt string?)
18 (s/def ::stream boolean?)
19 (s/def ::opts (s/keys :opt-un [::model ::stream]))
20
21 ;; Helper functions for spec validation
22 (defn validate-prompt [prompt]
23 (when (or (nil? prompt) (not (string? prompt)))
24 (throw (ex-info "Invalid prompt: must be a non-nil string"
25 {:prompt prompt})))
26 true)
27
28 (defn validate-messages [messages]
29 (when (or (nil? messages) (not (sequential? messages)))
30 (throw (ex-info "Invalid messages: must be a collection"
31 {:messages messages})))
32 (when-not (s/valid? ::messages messages)
33 (throw (ex-info "Invalid messages format"
34 {:messages messages
35 :explain (s/explain-data ::messages messages)})))
36 true)
37
38 (defn- ollama-helper [endpoint body]
39 (let [response (client/post
40 (str *base-url* endpoint)
41 {:accept :json
42 :headers {"Content-Type" "application/json"}
43 :body (json/write-str body)
44 :throw-exceptions false})]
45 (if (client/success? response)
46 (json/read-str (:body response) :key-fn keyword)
47 (throw (ex-info "Ollama API error"
48 {:status (:status response)
49 :body (:body response)})))))
50
51 (defn completions
52 "Use the Ollama API for text completions.
53 Options can include :model, :stream, etc.
54 Throws ExceptionInfo on validation or API errors."
55 ([prompt-text]
56 (completions prompt-text {}))
57 ([prompt-text opts]
58 (validate-prompt prompt-text)
59 (let [body (merge {:model *default-model*
60 :prompt prompt-text
61 :stream false}
62 opts)
63 result (ollama-helper "/api/generate" body)]
64 (:response result))))
65
66 (defn chat
67 "Use the Ollama API for chat conversations.
68 Messages should be a collection of maps like {:role \"user\" :content \"Hello\"}."
69 ([messages]
70 (chat messages {}))
71 ([messages opts]
72 (validate-messages messages)
73 (let [body (merge {:model *default-model*
74 :messages messages
75 :stream false}
76 opts)
77 result (ollama-helper "/api/chat" body)]
78 (get-in result [:message :content]))))
79
80 (defn summarize
81 "Use the Ollama API for text summarization"
82 [prompt-text]
83 (completions (str "Summarize the following text: " prompt-text)))
84
85 (defn answer-question
86 "Use the Ollama API for question answering"
87 [prompt-text]
88 (completions (str "Answer the following question: " prompt-text)))
Here is a representative excerpt of the unit tests in ollama/test/ollama_api/core_test.clj, which use with-redefs to test the client deterministically and also validate error handling and clojure.spec checks:
1 (ns ollama-api.core-test
2 (:require [clojure.test :refer :all]
3 [ollama-api.core :as ollama]))
4
5 (def some-text
6 "Jupiter is the fifth planet from the Sun and the largest in the Solar System. It is a gas giant with a mass one-thousandth that of the Sun, but two-and-a-half times that of all the other planets in the Solar System combined. Jupiter is one of the brightest objects visible to the naked eye in the night sky, and has been known to ancient civilizations since before recorded history. It is named after the Roman god Jupiter.[19] When viewed from Earth, Jupiter can be bright enough for its reflected light to cast visible shadows,[20] and is on average the third-brightest natural object in the night sky after the Moon and Venus.")
7
8 (deftest completions-test
9 (testing "ollama completions API with mocked HTTP"
10 (with-redefs [clj-http.client/post (fn [_ _]
11 {:status 200
12 :body "{\"response\": \"Mocked completions result\"}"})]
13 (let [results (ollama/completions "He walked to the river")]
14 (is (string? results))
15 (is (= "Mocked completions result" results))))))
16
17 (deftest summarize-test
18 (testing "ollama summarize API with mocked HTTP"
19 (with-redefs [clj-http.client/post (fn [_ _]
20 {:status 200
21 :body "{\"response\": \"Mocked summary\"}"})]
22 (let [results (ollama/summarize some-text)]
23 (is (= "Mocked summary" results))))))
24
25 (deftest question-answering-test
26 (testing "ollama question-answering API with mocked HTTP"
27 (with-redefs [clj-http.client/post (fn [_ _]
28 {:status 200
29 :body "{\"response\": \"Mocked answer\"}"})]
30 (let [results (ollama/answer-question "Where is the Valley of Kings?")]
31 (is (= "Mocked answer" results))))))
32
33 (deftest chat-test
34 (testing "ollama chat API with mocked HTTP"
35 (with-redefs [clj-http.client/post (fn [_ _]
36 {:status 200
37 :body "{\"message\": {\"content\": \"Mocked chat response\"}}"})]
38 (let [messages [{:role "user" :content "Hello, who are you?"}]
39 results (ollama/chat messages)]
40 (is (= "Mocked chat response" results))))))
The output will change each time you run the tests. The output from Large Language Models (LLMs) usually change each time you run the same tests or prompts. This variability stems from several key factors:
- Stochasticity: LLMs are complex neural networks trained on massive datasets. During training these networks develop internal representations and probabilistic weights that influence their outputs. When generating text LLMs sample from these weights, introducing an inherent element of randomness. This means even with identical inputs and identical LLM weights the same prompt used repeatedly can lead to slightly different outputs.
- Temperature: Some LLMs use a temperature parameter to control the “randomness” of their outputs. A higher temperature encourages exploration of less likely but potentially more creative responses, while a lower temperature leads to more deterministic and consistent outputs.
- Beam Search vs. Sampling: Different decoding strategies can also impact output variability. Beam search focuses on the most likely sequences, gradually refining them, while sampling explores a wider range of possibilities. Beam search typically leads to less variation compared to sampling, but even within beam search, there can be randomness in how ties are broken when choosing the next word.
I edited the following output shortening the output from the text completion test and by adding labels for each test:
1 $ lein test
2
3 COMPLETION TEST:
4
5 lein test ollama-api.core-test
6 He walked to the river, feeling the cool breeze on his skin and the gentle rustling of leaves in the trees. The sun was just beginning to set, casting a warm orange glow across the sky. As he approached the water's edge, he could see the ripples created by the gentle flow of the river. He stopped for a moment to take it all in, feeling a sense of peace and tranquility wash over him.
7
8 As he continued on his journey, he couldn't help but think about how much the world had changed in recent times. The pandemic had brought with it so much uncertainty and fear, but now, as he stood by the river, he felt grateful to be alive and able to enjoy such simple pleasures. He made a mental note to come back here more often, to take time for himself and to appreciate the beauty of nature....
9
10 SUMMARIZATION TEST:
11
12 Jupiter is a massive gas giant planet located in our solar system. It's the fifth planet from the sun and is known for being one of the brightest objects visible to the naked eye at night. With a mass one-thousandth that of the sun, it has a unique place in the cosmos.
13
14 QUESTION ANSWERING TEST:
15
16 The Valley of Kings is located in Egypt, specifically in the region of Giza. It is a valley on the west bank of the Nile River and is known for its abundance of royal tombs dating back to ancient Egypt's New Kingdom period.
Optional Practice Problems
- Model Switching: Connect
source-code/ollamato a different lightweight model (e.g.,phi3orgemma2:2b) and print responses to a series of prompts. - Token Streaming: Implement a streaming response handler so that generated tokens are printed to the stdout console in real time.
- Stop Sequence Config: Customize model parameters such as stop sequences in the request body to force early response termination.