Using Local LLMs Using Ollama in Java Applications

Note: May 15, 2024: this chapter is only 90% complete.

Using local Large Language Models (LLMs) with Ollama offers a range of advantages and applications that significantly enhance the accessibility and functionality of these powerful AI tools in various settings. Ollama is like the Docker system, but for easily downloading, running, and managing LLMs on your local computer. Ollama was originally written to support Apple Silicon Macs, but now supports Intel Macs, Linux, and Windows.

Advantages of Using Local LLMs with Ollama

Accessibility and Ease of Use

Ollama democratizes the use of sophisticated LLMs by making them accessible to users of all technical backgrounds. You don’t need to be an AI expert to leverage the capabilities of LLMs when using Ollama. The platform’s user-friendly interface and simple text-based interaction make it intuitive and straightforward for anyone to start using LLMs locally.

Privacy and Data Security

Running LLMs locally on your system via Ollama ensures that your data does not leave your device, which is crucial for maintaining privacy and security, especially when handling sensitive information. This setup prevents data from being sent to third-party servers, thus safeguarding it from potential misuse or breaches.

Cost-Effectiveness

Using Ollama to run LLMs locally eliminates the need for costly cloud computing resources. This can be particularly advantageous for users who require extensive use of LLMs, as it avoids the recurring costs associated with cloud services.

Customization and Control

Local deployment of LLMs through Ollama allows users to have greater control over the models and the computational environment. This includes the ability to choose which models to run and to configure settings to optimize performance according to specific hardware capabilities.

Applications of Local LLMs with Ollama

Personalized AI Applications

For hobbyists and personal use, Ollama allows the exploration of LLMs’ capabilities such as text generation, language translation, and more, all within the privacy of one’s own computer. This can be particularly appealing for those interested in building personalized AI tools or learning more about AI without making significant investments.

Development and Testing

Ollama is well-suited for developers who need to integrate LLMs into their applications but wish to do so in a controlled and cost-effective manner. It is particularly useful in development environments where frequent testing and iterations are required. The local setup allows for quick changes and testing without the need to interact with external servers.

Educational and Research Purposes

Educators and researchers can benefit from the local deployment of LLMs using Ollama. It provides a platform for experimenting with AI models without the need for extensive infrastructure, making it easier to teach AI concepts and conduct research in environments with limited resources.

In summary, using local LLMs with Ollama not only makes powerful AI tools more accessible and easier to use but also ensures privacy, reduces costs, and provides users with greater control over their AI applications. Whether for professional development, research, or personal use, Ollama offers a versatile and user-friendly platform for exploring the potential of LLMs locally.

Java Library to Use Ollama’s REST API

Te library defined in the directory ** Java-AI-Book-Code/ollama-llm-client** defines a class named OllamaLlmClient with a method getCompletion that sends a JSON payload to a server and reads the response. Here’s an explanation of what each significant part of the method does:

Create JSON Payload: It constructs a JSON object (message) containing three key-value pairs: prompt (a string provided by the caller), model (also a string provided by the caller indicating the model name), and stream (a boolean value set to false).
Prepare HTTP Connection: It creates a URI object pointing to the server’s URL (http://localhost:11434/api/generate), converts it to a URL object, and opens a connection to it. The connection is configured to send output and to use application/json as the content type.
Send JSON Payload: It converts the JSON object to a byte array using UTF-8 encoding and sends it to the server through the connection’s output stream.
Read Server Response: It reads the server’s response using a BufferedReader that wraps the connection’s input stream, appending each line of the response to a StringBuilder object.
Disconnect: It explicitly disconnects the HTTP connection.
Process Server Response: It converts the response string back into a JSON object and extracts the value associated with the key response. This value is then returned by the method.

In summary, this method sends a JSON payload containing a prompt and model name to a specified server endpoint, reads the JSON response from the server, extracts a specific field from the JSON response, and returns that field’s value.

 1 package com.markwatson.openai;
 2 
 3 import java.io.BufferedReader;
 4 import java.io.InputStreamReader;
 5 import java.io.OutputStream;
 6 import java.net.HttpURLConnection;
 7 import java.net.URI;
 8 import java.net.URL;
 9 import java.net.URLConnection;
10 
11 import org.json.JSONArray;
12 import org.json.JSONObject;
13 
14 public class OllamaLlmClient {
15 
16     public static void main(String[] args) throws Exception {
17         String prompt = "Translate the following English text to French: 'Hello, how are you?'";
18         String completion = getCompletion(prompt, "mistral");
19         System.out.println("completion: " + completion);
20     }
21 
22     public static String getCompletion(String prompt, String modelName) throws Exception {
23         System.out.println("prompt: " + prompt + ", modelName: " + modelName);
24  
25         // New JSON message format
26         JSONObject message = new JSONObject();
27         message.put("prompt", prompt);
28         message.put("model", modelName);
29         message.put("stream", false);
30         URI uri = new URI("http://localhost:11434/api/generate");
31         URL url = uri.toURL();
32         //System.out.println("jsonBody: " + jsonBody);
33         URLConnection connection = url.openConnection();
34         connection.setDoOutput(true);
35         connection.setRequestProperty("Content-Type", "application/json");
36         // Send the JSON payload
37         try (OutputStream os = connection.getOutputStream()) {
38             byte[] input = message.toString().getBytes("utf-8");
39              os.write(input, 0, input.length);
40         }
41 
42         StringBuilder response;
43         // Read the response from the server
44         try (BufferedReader br = new BufferedReader(
45                 new InputStreamReader(connection.getInputStream(), "utf-8"))) {
46             response = new StringBuilder();
47             String responseLine;
48             while ((responseLine = br.readLine()) != null) {
49                 response.append(responseLine.trim());
50             }
51             System.out.println(response.toString());
52         }
53 
54         ((HttpURLConnection) connection).disconnect();
55 
56         JSONObject jsonObject = new JSONObject(response.toString());
57         String s = jsonObject.getString("response");
58         return s;
59     }
60 
61 }

Example Using the Library

The Java library for getting local LLM text completions using Ollama contains a unit test that contains an example showing how to call the API:

1 String r =
2   OllamaLlmClient.getCompletion(
3     "Translate the following English text to French: 'Hello, how are you?'",
4     "mistral");
5 System.out.println("completion: " + r);

The output looks like:

1 prompt: Translate the following English text to French: 'Hello, how are you?', modelName: mistral
2 completion:  In French, "Hello, how are you?" can be translated as "Bonjour, comment allez-vous?" or simply "Comment allez-vous?" depending on the context.

For reference the JSON response object from the API call looks like this:

1 {"model":"mistral","created_at":"2024-05-05T19:38:26.893374Z","response":" In French, \"Hello, how are you?\" can be translated as \"Bonjour, comment allez-vous?\" or simply \"Comment allez-vous?\" depending on the context.","done":true,"context":[733,16289,28793, ...],"total_duration":1777944500,"load_duration":563601792,"prompt_eval_count":25,"prompt_eval_duration":133415000,"eval_count":41,"eval_duration":1079766000}

Extraction of Facts and Relationships from Text Data

Traditional methods for extracting email addresses, names, addresses, etc. from text included the use of hand-crafted regular expressions and custom software. LLMs are text processing engines with knowledge of grammar, sentence structure, and some real world embedded knowledge. Using LLMs can reduce the development time of information extraction systems.

There are sample text prompts in the directory Java-AI-Book-Code/prompts and we will specifically use the file ** two-shot-2-var.txt** that is listed here:

 1 Given the two examples below, extract the names, addresses, and email addresses of individuals mentioned later as Process Text. Format the extracted information in JSON, with keys for "name", "address", and "email". If any information is missing, use "null" for that field. Be very concise in your output by providing only the output JSON.
 2 
 3 Example 1:
 4 Text: "John Doe lives at 1234 Maple Street, Springfield. His email is johndoe@example.com."
 5 Output: 
 6 {
 7   "name": "John Doe",
 8   "address": "1234 Maple Street, Springfield",
 9   "email": "johndoe@example.com"
10 }
11 
12 Example 2:
13 Text: "Jane Smith has recently moved to 5678 Oak Avenue, Anytown. She hasn't updated her email yet."
14 Output: 
15 {
16   "name": "Jane Smith",
17   "address": "5678 Oak Avenue, Anytown",
18   "email": null
19 }
20 
21 Process Text: "{input_text}"
22 Output:

The example code is a test method in OllamaLlmClientTest. testTwoShotTemplate() that is shown here:

1 String input_text = "Mark Johnson enjoys living in Berkeley California at 102 Dunston Street and use mjess@foobar.com for contacting him.";
2 String prompt0 = OllamaLlmClient.readFileToString("../prompts/two-shot-2-var.txt");
3 System.out.println("prompt0: " + prompt0);
4 String prompt = OllamaLlmClient.promptVar(prompt0, "{input_text}", input_text);
5 System.out.println("prompt: " + prompt);
6 String r =
7   OllamaLlmClient.getCompletion(prompt, "llama3:instruct");
8 System.out.println("two shot extraction completion:\n" + r);

The output is (edited for brevity):

1 two shot extraction completion:
2 {
3   "name": "Mark Johnson",
4   "address": "102 Dunston Street, Berkeley, California",
5   "email": "mjess@foobar.com"
6 }

Using LLMs to Summarize Text

LLMs bring a new level of ability to text summarization tasks. With their ability to process massive amounts of information and “understand” natural language, they’re able to capture the essence of lengthy documents and distill them into concise summaries.

Here is a listing or the prompt file:

1 Summarize the following text: "{input_text}"
2 Output:

The example code is in the test OllamaLlmClientTest. testSummarization() listed here:

1 String input_text = "Jupiter is the fifth planet from the Sun and the largest in the Solar System. It is a gas giant with a mass one-thousandth that of the Sun, but two-and-a-half times that of all the other planets in the Solar System combined. Jupiter is one of the brightest objects visible to the naked eye in the night sky, and has been known to ancient civilizations since before recorded history. It is named after the Roman god Jupiter. When viewed from Earth, Jupiter can be bright enough for its reflected light to cast visible shadows, and is on average the third-brightest natural object in the night sky after the Moon and Venus.";
2 
3 String prompt0 = OllamaLlmClient.readFileToString("../prompts/summarization_prompt.txt");
4 System.out.println("prompt0: " + prompt0);
5 String prompt = OllamaLlmClient.promptVar(prompt0, "{input_text}", input_text);
6 System.out.println("prompt: " + prompt);
7 String r =
8   OllamaLlmClient.getCompletion(prompt, "llama3:instruct");
9 System.out.println("summarization completion:\n" + r);

The output is (edited for brevity):

1 summarization completion:
2 
3 Here is a summary of the text:
4 
5 Jupiter is the 5th planet from the Sun and the largest gas giant in our Solar System, with a mass 1/1000 that of the Sun and 2.5 times that of all other planets combined. It's one of the brightest objects visible to the naked eye and has been known since ancient times. On average, it's the 3rd-brightest natural object in the night sky after the Moon and Venus.