# Part IV - Fast and Efficient Rag Document System

Here we develop an application that is practical to run that uses a local vector database and an inexpensive cloud based model.

",{"_267":268,"_43":44,"_47":-5,"_45":46,"_64":65,"_269":270,"_321":265},"previewContent","

\nPreface\n
- \nChoosing Which “Small AI” Tools to Use in This Book\n
\n
\nPart I - Advantages of Small AI With Examples\n
\nAdvantages of Small Models with Examples That Might Surprise You\n
\n
\nDrawbacks of Small AI\n
- \nReduced Reasoning Depth and Nuance\n
- \nHigher Propensity for Hallucination\n
- \nLimited Context Window and Memory Management\n
- \nFragility in Prompt Following\n
- \nSecurity and Support Disparities\n
- \nWrap Up for Drawbacks of Small AI\n
\n
\nPart II - Defining Metrics for Success Using LLMs\n
\nDefining Metrics for Success Using LLMs\n
- \nOperational Metrics: The Small AI Advantage\n
- \nQuality Metrics: The “LLM-as-a-Judge” Pattern\n
- \nRAG Metrics: The Trinity of Trust\n
- \nDeterministic Guardrails\n
- \nThe Golden Ratio: Accuracy per Watt\n
- \nCode Example: The Judge Pattern\n
- \nWrap Up for LLM Metrics\n
\n
\nA Fair Shootout: Evaluating Large vs. Small Models for Task Fitness\n
\n
\nPart III - Python Examples for Switching Between Models\n
\nUsing the LiteLLM Library to Easily Switch Between Models and Model Providers\n
- \nTool Use with LiteLLM\n
- \nLiteLLM Wrap Up\n
\n
\nAutomatically Routing to the ‘Best’ Model Using RouteLLM Library\n
- \nImplementation of a Command Line Query Utility Using RouteLLM\n
\n
\nWrap Up for Winning Big With Small AI\n
- \nThe Engineering Reality: From Vibes to Metrics\n
- \nThe Architecture of Hybrid Intelligence\n
- \nThe “Fitness for Purpose” Philosophy\n
- \nSovereignty, Solvency, and Sustainability\n
- \nFinal Recommendations for Your Journey\n
\n
\n\\# Part IV - Fast and Efficient Rag Document System\n
\nRAG Using zvec Vector Datastore and Gemini-3-flash Model\n
- \nDesign Notes for Example Program\n
- \nExample zvec RAG Application\n
- \nSample Example Output\n
- \nWrap Up for the zvex Based RAG Application\n
\n

Preface

Using AI models does not always mean using Super Scalers like Google and OpenAI. This book covers use cases where using smaller models that meet system requirements can save money, can be run locally for privacy and security, more control of systems using smaller LLMs, help preserve the environment, and control costs.

Dear reader, I first used neural network technology starting in the 1980s (DARPA advisory panel for neural network (NN) tools, commercial software and NN hardware products), continued for a decade from 2014 through 2019 using deep learning models (I managed a deep learning team at Capital One), and since 2020 primarily used LLMs for application development and research.

As I write this in February 2026 many people I interact with are very strident in wanting to use the best and usually most expensive available AI models. If this, dear reader, is your philosophy then I hope to at least slightly change your mind in this book. Personally I prefer using only strong models like Google Gemini 3 Pro or OpenAI’s GPT-5.2 for research and learning but for practical engineering projects I start by defining acceptable performance metrics for embedded LLMs and using the fastest and least expensive options that meet the target performance.

Choosing Which “Small AI” Tools to Use in This Book

Mostly due to my own preferences, I have chosen a very narrow range of tools for use in this book:

\nLocal open weight/open source Large Language Models (LLMs) run locally using Ollama.\n
\nOPTIONAL: running open weight/open source LLMs on Ollama Cloud (free for low volume use, otherwise $20/month).\n
\nGoogle’s very low cost (or free for limited monthly use) gemini-3-flash API.\n
\nFor Python examples: the uv tool and various libraries we will install as we need them.\n

I have left several commercial vendors like Anthropic and OpenAI off the table in my tool selection for book examples but the examples using the Gemini APIs can be easily converted to other model providers because we use the model portability library LiteLLM.

I asked Gemini to estimate cost per 1 million input tokens (some models are more ‘token efficient’ so take the following as approximate data) for commercial LLM APIs:

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n

Model Name	Energy Use (per 1M tokens)
Gemini 3 Flash	~0.6 kWh
Gemini 3 Pro	~1.2 kWh
GLM-4.7	~2.2 kWh
Qwen-3-Coder-480B	~2.5 kWh
Claude 4 Opus	~3.5 kWh
OpenAI GPT-5	~5.0 – 8.0 kWh
OpenAI o3	~33.0 kWh

Gemini 3 Flash is an excellent all-around model and I will use it in this book.

Energy use is much lower running very small models with Ollama (or LLM Studio) on a local Mac, Windows, or Linux laptop. Here are some estimates for running on a MacBook M4:

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n

Model Name	Energy Use (per 1M tokens)
Gemma 3 1B	~0.02 kWh
Qwen 3 1.7B	~0.04 kWh
Qwen 3 4B	~0.08 kWh

We will use several small Google Gemma models in examples, as well as occasionally using Alibaba’s Qwen models. For comparison of power use, a modern LED light bulb uses about 0.009 kW and a classic incandescent bulb uses 0.06kW.

Obviously large commercial models are more capable but smaller models can also be very effective. It is a theme in this book to select models that are adequate for the task at hand.

Resources

\nI have written separate books on using Ollama and LM Studio (also for running models on a personal computer) that you can read online for free: https://leanpub.com/ollama and https://leanpub.com/LMstudio by clicking on the links labeled Free To Read Online.\n
\nYou will want a Google Gemini API key. The free version is fine for casual use and is available on the Google AI Studio https://aistudio.google.com by clicking the link Get API Key on the left menu panel.\n

Part I - Advantages of Small AI With Examples

\nAdvantages of Small Models with examples that might surprise you.\n
\nDisadvantages of Small Models and recognizing possible problems before they happen.\n

Advantages of Small Models with Examples That Might Surprise You

Often dismissed as mere cost-saving alternatives, small models have matured into powerful engines of innovation that challenge our fundamental assumptions about the capability of LLM-based AIs. By decoupling intelligence from massive scale, these compact systems are enabling a new class of applications that prioritize speed, privacy, and ubiquity over brute force.

The Great Divergence: Redefining Intelligence in the Post-Scale Era

The history of artificial intelligence has shifted dramatically in late 2025. While the industry previously focused on a singular Scaling Hypothesis where performance correlated directly with massive parameter counts, a profound divergence has occurred. Efficiency has transitioned from a cost-saving measure to a primary capability enabler. The release of models like Google’s Gemini 3 Flash and Alibaba’s Qwen 3 model series demonstrated that sophisticated reasoning and agentic behavior could be distilled into packages a fraction of the size of their predecessors. This shift dismantles the assumption that smaller inherently means less intelligent. We are witnessing the democratization of high-level intelligence, moving from centralized clouds to local edge devices like smartphones and single-board computers. Small models are evolving into specialized tools capable of knowledge compression that defy traditional scaling laws. Furthermore, the rise of small models is linked to the urgency of energy efficiency and data sovereignty, especially with the European Union’s AI Act enforcing strict data governance in 2025.

The Energy and Latency Wall

By mid-2024, the limits of massive models became apparent in deployment physics rather than intelligence. The Time to First Token for trillion-parameter models introduced latency that broke the illusion of conversation, and energy costs drew scrutiny. The research community responded by pivoting toward inference efficiency and knowledge distillation. This process compresses the reasoning algorithms of giant models into denser neural pathways, stripping away the noise of the internet while retaining the logic of deduction.

The Cloud Efficiency Champion: Anatomy of Gemini 3 Flash

The release of Gemini 3 Flash in December 2025 redefined the performance-per-watt equation for cloud AI. It delivers frontier-class performance at costs previously associated with basic autocomplete engines. The architecture integrates Pro-grade reasoning into a lightweight framework using a dynamic routing mechanism known as Thinking Levels. When a query arrives, the model assesses complexity, routing simple tasks through a fast path and complex logic through a Deep Think path that explores multiple hypotheses.

The Economics of Latency and Throughput: Gemini 3 Flash

Smaller and more efficient models push the Pareto frontier of quality versus cost. Input tokens cost $0.50 per million, and output tokens are $3.00 per million. Processing a standard novel or other book now costs roughly six cents. The model requires approximately 30% fewer tokens to complete tasks compared to the Gemini 2.5 Pro series, effectively compounding savings. Latency improvements are equally critical, with benchmarks indicating speeds three times faster than previous generations, enabling real-time applications like voice translation.

Agentic Capabilities and Green AI

A surprising advantage is the model’s performance in agentic coding workflows, achieving a SWE-bench Verified score of 78%. This anomaly of a smaller model outperforming larger ones is attributed to its specialist training for tool use and multimodal interaction. Environmentally, the model leverages specialized Tensor Processing Units (TPU) v5p and v6e to optimize matrix operations, significantly reducing the Thermal Design Power required per query compared to dense architectures.

The Local Insurgency: Qwen 3 and other Local Models Provide Democratization of Reasoning

Parallel to cloud advancements, smaller models such as Alibaba’s Qwen 3 series has altered what can be achieved with more limited hardware capabilities. The most significant advancement is the democratization of Thinking Mode down to models as small as 1.7 billion parameters. Previously considered an emergent property of massive models, Qwen 3 demonstrates that reasoning can be distilled. The model can toggle between a deliberate Thinking Mode, where it generates internal monologues to verify logic, and a high-speed Non-thinking Mode for conversational fluency.

Benchmarking the Pocket AI

The Qwen 3 1.7B model achieves a score of 57% on the MMLU-Pro benchmark, placing it in competition with much larger models. For developers, this means a model capable of understanding Python syntax and debugging errors can reside permanently in the RAM of a local machine. The series also includes Mixture-of-Experts (MoE) models, such as the Qwen3-30B-A3B, which offers the knowledge base of a 30 billion parameter model while only activating 3 billion parameters per token generation.

Silicon at the Edge: The Hardware Reality of Small AI

The true advantage of small models is realized when decoupled from the grid. The Raspberry Pi 5 has become a standard-bearer for very low-cost local AI. Using optimized inference engines like Ollama, a quantized Qwen 3 1.7B model achieves 5 to 7 tokens per second on the Pi 5, which is sufficient for real-time reading.

The Role of NPUs and Energy Efficiency

For higher performance, Single Board Computers featuring the Rockchip RK3588 chipset and commercial computers using based on technologies like Apple Silicon offer a glimpse into the future. These devices include Neural Processing Units (NPUs) that can run 3B models at speeds exceeding 20 tokens per second. A critical metric here is Tokens per Watt. While a cloud GPU cluster might produce 0.14 tokens per watt, an Orange Pi 5 NPU can achieve 2.10 tokens per watt, and a modern smartphone up to 3.00 tokens per watt. This suggests that for personal workloads, edge inference is significantly more energy-efficient than cloud inference.

The Physics of Thought: Theoretical Mechanisms

The Information Bottleneck principle suggests that deep learning aims to compress input data into a representation that retains only relevant information. Massive models are inefficient compressors that memorize noise, whereas small models are forced to be efficient by finding underlying patterns. When trained via distillation, small models learn the distilled concepts of larger models rather than learning just from raw data. This validates the Data Quality hypothesis, where training on reasoning traces allows small models to emulate multi-step logic without massive capacity.

Sovereignty and Solvency: The Economic and Legal Case

The shift to small models is also driven by the EU AI Act of 2025, which mandates strict governance. For enterprises, the choice between cloud and local often comes down to Total Cost of Ownership. While cloud APIs offer instant scalability, local deployments eliminate marginal costs per token by using existing on-location computer hardware. Anecdotal comments on social media indicates that organizations with high volumes, such as 8,000 daily conversations, break even on local hardware within 6 to 12 months. Furthermore, local models offer compliance by design, processing data on-device to eliminate interception risks and bypass complex cross-border transfer regulations.

Surprising Realities: Case Studies

The practical power of these models is best illustrated through real-world scenarios that defy conventional wisdom about model size.

The Philosopher in the Pocket

A graduate student writing a dissertation on Kantian ethics might use a Qwen 3 1.7B model on a Raspberry Pi 5. One would expect a model of this size to hallucinate complex concepts, but due to the high representation of public-domain philosophy in the training corpus and the model’s Thinking Mode, it maintains coherent argumentation. The student engages the model to reason step-by-step about the Categorical Imperative, receiving a private, offline Socratic tutor that costs nothing to query.

The Real-Time Coding Assistant

A field technician debugging legacy code in a remote factory with no signal can rely on a smartphone running a quantized Qwen 3 4B model and a local coding app like Terminus. Contrary to the assumption that coding requires massive models, the 4B model possesses a dense understanding of syntax. The technician photographs an error log, and the local model suggests a script patch in minutes, functioning as a specialized coding brain that fits in 2GB of RAM.

The Sustainable Green Search Engine

An educational non-profit in a developing nation uses a local cluster of Orange Pi 5 boards to provide AI tutoring. Replacing a cloud-based LLM with local models reduces power infrastructure requirements by an order of magnitude. The local server handles 90% of queries using a 1.7B model, routing only the most ambiguous questions to the Gemini 3 Flash API via satellite. This hybrid architecture delivers high-quality education with a minimal carbon footprint. Note: We will look at Python examples of routing between models in a later chapter.

Horizon 2026: The Future of Distributed Intelligence

The future of AI is a mosaic rather than a monolith. While massive models will continue to push scientific boundaries, small efficient models like Gemini 3 Flash (running in Google data centers) and smaller models like Qwen 3 (running on local hardware) will serve as the workhorses of the economy. They offer speed, economic viability, privacy, and sustainability. As we move toward 2026, we expect adaptive precision and NPU-first designs to become standard, blurring the line between local and cloud processing. In the fall of 2025 Apple released mobile AI models that use a local model for simpler queries and a secure cloud enclave for a more robust model that handles more difficult prompts or queries - I wrote an iOS/iPadOS app that uses this hybrid model. The most useful AI may not be the one that knows everything, but the one small enough to be everywhere.

Drawbacks of Small AI

While small-scale AI using local execution via Ollama and LM Studio, low-cost open-source models (often from China or France), and “Flash” or “Mini” API tiers from Google and OpenAI offers significant advantages in speed and cost, it introduces several technical and operational trade offs.

Reduced Reasoning Depth and Nuance

The most immediate disadvantage of smaller models is their limited cognitive capacity compared to their “frontier” counterparts. Because these models have significantly fewer parameters, they often struggle with complex, multi-step reasoning and subtle linguistic nuances. In tasks requiring high-level abstraction, such as sophisticated architectural design or deep philosophical analysis, small models are more prone to “reasoning shortcuts” or logical fallacies. They lack the vast internal world, i.e., that allows larger models to connect disparate concepts, often resulting in outputs that are technically correct on the surface but lack the depth required for professional-grade creative or analytical work.

Higher Propensity for Hallucination

Small AI models typically exhibit worse performance for hallucination: the generation of confident but factually incorrect information. With a smaller compressed knowledge base, these models have less “buffer” to distinguish between closely related facts, leading to frequent confabulations when pushed beyond their specific training data. While techniques like Retrieval-Augmented Generation (RAG) can mitigate this, the internal logic of a 3B or 7B parameter model is inherently less capable of self-correcting for identifying internal contradictions than a trillion-parameter model. This necessitates much more rigorous human-in-the-loop verification, which can offset the initial productivity gains or cost savings.

Limited Context Window and Memory Management

Although some smaller models claim large context windows, their effective “needle-in-a-haystack” performance—the ability to accurately retrieve a specific fact from a large block of text tends to degrade faster than in flagship models. Local inference using tools like Ollama is often constrained by the user’s hardware (VRAM), leading to forced context compression or truncation. This makes Small AI less reliable for analyzing massive codebases or long legal documents, as the model may lose track of earlier instructions or key details, leading to inconsistent outputs over the course of a long conversation.

Fragility in Prompt Following

Smaller models are significantly more sensitive to prompt engineering. While a frontier model like GPT-5 or Gemini 3 Pro can often interpret a poorly phrased instruction through sheer “intelligence,” a small model frequently requires highly specific, rigid prompt templates to function correctly. This “prompt fragility” means that a minor change in wording can lead to a total failure in output format (e.g., failing to return valid JSON). For developers, this creates a maintenance burden, as prompts must be meticulously tuned and “unit-tested” for each specific small model version, reducing the flexibility of the AI system.

Security and Support Disparities

Finally, there are some risks regarding long term reliability and security. Low cost open source models, particularly those from newer or less transparent providers, may not undergo the same rigorous safety alignment or red teaming as flagship models. Furthermore, the “Small AI” ecosystem moves at a breakneck pace; a model that is a “state-of-the-art” open source leader today may be abandoned by its developers three months later. This lack of long-term support may be a risk for enterprise applications that require stability, consistent updates, and clear provenance of the training data. On the other hand, when you host your own models you are in control of updates.

Wrap Up for Drawbacks of Small AI

While Small AI provides an accessible path toward local privacy and cost efficiency, it inherently trades away the robust reasoning and reliable instruction following found in frontier models. For complex professional workflows, these limitations necessitate a rigorous trade-off analysis to ensure that “smaller” doesn’t ultimately result in “lesser” outcomes.

Dear reader, I am writing this book to offer some assurance that the drawbacks listed here often can be controlled, managed, and generally mitigated for a wide variety of applications.

Part II - Defining Metrics for Success Using LLMs

Moving beyond the subjective “vibe check” is the critical step in maturing from a hobbyist experiment to a production-grade “Small AI” system. While operational metrics like Time to First Token (TTFT) and Tokens Per Second (TPS) validate the latency and “snappiness” advantages of running local models on consumer hardware, they must be rigorously balanced against quality standards to ensure utility isn’t sacrificed for speed. By implementing the “LLM-as-a-Judge” pattern, where capable frontier models automatically grade the outputs of efficient local models and monitoring RAG-specific health indicators like context precision and faithfulness, developers can quantify exactly what is gained in privacy and cost versus what is traded in raw capability. This shift to data-driven evaluation allows us to make good choices in model and platform selections.\nWe are not just building models that function, but engineering systems that win on the specific margins that drive business value.

Defining Metrics for Success Using LLMs

In the early days of generative AI, evaluation was often a matter of “vibes.” You would run a prompt, read the output, nod your head, and say, “That looks about right.” For a hobbyist, that is fine. For a software engineer building reliable systems with Small AI, it is unacceptable.

When we work with “Small AI” (i.e., usually local models, quantized weights, and targeted use cases) we are trading the raw, brute-force false-omniscience of massive models for efficiency, privacy, and speed. To make that trade-off intelligent, we must measure exactly what we are losing in generality and what we are gaining in efficiency. We need rigorous metrics.

This chapter defines the scorecard for your Small AI projects. We will look at two categories of metrics: Operational Metrics (is it fast and cheap?) and Quality Metrics (is it smart enough?).

Operational Metrics: The Small AI Advantage

This is where Small AI shines. If you are running a 7-billion parameter model locally versus calling a 2-trillion parameter model in the cloud, these numbers or criteria may justify your architectural choice of Small AI:

\nTime to First Token (TTFT): This measures the latency between the user hitting “Enter” and seeing the first word appear. For interactive applications, this is your “snappiness” metric.\n
\nTarget: < 200ms for a “real-time” feel.\n
\nSmall AI Win: A local model typically eliminates network latency, often beating cloud APIs significantly on TTFT.\n
\nTokens Per Second (TPS): Once the generation starts, how fast does it flow? This is your throughput.\n
\nContext: Human reading speed is roughly 3–5 tokens per second. If your local model generates at 50 TPS, it is effectively instantaneous to the user.\n
\nCost Per 1k Successful Transactions: Notice I said “Successful” transactions. A cheap model that fails 50% of the time is expensive.\n

Quality Metrics: The “LLM-as-a-Judge” Pattern

How do you measure if your local, fine-tuned Llama-3 (or other) model is “smart” enough? You can’t use standard unit tests because the output varies. You could use human review, but that doesn’t scale for some use cases.

The solution is the LLM-as-a-Judge pattern. In this workflow, you use a massive, frontier model (like GPT-5) to act as the teacher grading the homework of your Small AI.

The Workflow is:

\nCreate a Dataset: 50–100 inputs (prompts) typical of your use case.\n
\nGenerate Outputs: Run these through your Small AI model.\n
\nThe Judge Steps In: Send the pair (Input, Small_AI_Output) to the Frontier Model with a grading rubric prompt.\n

Here is a judge prompt example: “Rate the following response on a scale of 1–5 for factual accuracy and brevity. Output ONLY the number.”

This gives you a quantifiable accuracy score (e.g., “We are at 4.2/5 quality relative to GPT-5”) without paying for GPT-5 on every production query. You only pay for it during development and testing.

RAG Metrics: The Trinity of Trust

Many systems implemented using Small AI rely on RAG (Retrieval Augmented Generation) to inject knowledge into a LLM prompt. If your system fails, you need to know why. Was the model stupid, or was the data missing? RAG architectures lend themselves to logging intermediate processing steps that are useful during development and later to track down problems in production.

RAG systems historically use vector embedding databases but in my projects and experiments I usually combine vector embedding databases with classic BM25 text search.

We separate these concerns using three metrics (inspired by the RAGAS framework):

\nContext Precision (The Retrieval Metric): When the user asked about “Project X,” did your vector database actually retrieve documents about Project X? Low Score means: Your embeddings or search strategy are broken. The LLM never stood a chance.\n
\nFaithfulness (The Hallucination Metric): Did the LLM answer using only the retrieved context, or did it make things up? Low Score means: Your Small AI is hallucinating. You may need to adjust the system prompt or lower the temperature.\n
\nAnswer Relevance (The Utility Metric): Did the answer actually help the user? Low Score means: The model might be polite and factual, but it’s answering the wrong question.\n

Deterministic Guardrails

Never underestimate the power of simple, boolean checks. For Small AI, which can be more prone to instruction-drift than massive models, these are safety nets:

\nJSON Validity Rate: If your app expects JSON, what percentage of outputs parse correctly?\n
\nRefusal Rate: How often does the model say “I cannot help with that”? (Too high = over-aligned/useless; Too low = unsafe).\n
\nBrevity Penalty: For Small AI, verbosity is a double cost: it looks bad and it hogs resources. You can measure this by simply counting the average token length of correct answers.\n

The Golden Ratio: Accuracy per Watt

Finally, I propose a metric specifically for this book: Accuracy per Watt.

If a 70B parameter model gives you 98% accuracy but burns 400 watts, and an 8B model gives you 96% accuracy consuming 40 watts, the “Small AI” choice is 10x more efficient for a 2% quality drop.

In engineering, we rarely hunt for “perfect.” We hunt for “optimal.” By using these metrics early, you stop chasing the impossible performance of a trillion-parameter god-model and start building a highly efficient, reliable tools that win on the metrics that actually matter for your business.

Code Example: The Judge Pattern

To implement the “LLM-as-a-Judge” pattern, we can create a Python script that automates the feedback loop between a lightweight target model and a more robust evaluator model. In the following example, we utilize the litellm library to interface with local Ollama instances, establishing a clear separation of duties: a smaller “Student” model generates answers to specific prompts, while a larger “Judge” model consumes those answers along with a strict grading rubric. This setup demonstrates how to programmatically enforce quality control by injecting a system prompt that explicitly defines criteria such as accuracy and brevity, ultimately parsing the judge’s output into a structured JSON format for immediate programmatic analysis.

The small model is 0.8 billion parameters and the judge model has 8 billion parameters.

  1 from litellm import completion\n  2 import json\n  3 \n  4 DEBUG = False\n  5 DEBUG_LENGTH = 300\n  6 \n  7 # --- CONFIGURATION ---\n  8 # The \"Small AI\" we are testing (Student)\n  9 # Using your specific model from the example\n 10 #STUDENT_MODEL = \"ollama_chat/qwen3:1.7b\" \n 11 STUDENT_MODEL = \"ollama_chat/gemma3:1b\" \n 12 \n 13 # The \"Judge\" model. \n 14 # Best Practice: Use a larger model here (e.g., Llama3.1-8B, Mistral-Large, or GPT-4o)\n 15 # for the most reliable grading. \n 16 JUDGE_MODEL = \"ollama_chat/rnj-1:latest\" # 8B model\n 17 OLLAMA_API = \"http://localhost:11434\"\n 18 \n 19 def get_student_response(user_prompt):\n 20     \"\"\"\n 21     Generates a response from the small, target model.\n 22     \"\"\"\n 23     response = completion(\n 24         model=STUDENT_MODEL,\n 25         messages=[{\"role\": \"user\", \"content\": user_prompt}],\n 26         api_base=OLLAMA_API,\n 27         temperature=0.3 # Keep it deterministic for testing\n 28     )\n 29     return response.choices[0].message.content\n 30 \n 31 def judge_response(user_prompt, student_answer):\n 32     \"\"\"\n 33     Uses the stronger Judge model to evaluate the Student's work.\n 34     Returns a JSON object with score and reasoning.\n 35     \"\"\"\n 36     \n 37     # The Rubric Prompt\n 38     # This is the core of the \"LLM-as-a-Judge\" pattern.\n 39     system_rubric = \"\"\"\n 40     You are an impartial AI Judge. You evaluate the quality of AI responses.\n 41     \n 42     Criteria for evaluation:\n 43     1. Accuracy: Is the code or text factually correct?\n 44     2. Brevity: Is the answer concise without unnecessary fluff?\n 45     \n 46     Output Format:\n 47     Return ONLY a JSON object with the following keys:\n 48     {\n 49         \"score\": (int 1-10),\n 50         \"reasoning\": (string, max 40 words),\n 51         \"pass\": (boolean, true if score > 5),\n 52         \"suggested_prompt\": (string, a DIFFERENT and improved version of the user question that would help the AI give a better response - must NOT be identical to the original question)\n 53     }\n 54     \"\"\"\n 55     \n 56     evaluation_prompt = f\"\"\"\n 57     [User Question]: {user_prompt}\n 58     [AI Response]: {student_answer}\n 59     \n 60     Evaluate the AI Response based on the rubric.\n 61     \"\"\"\n 62 \n 63     response = completion(\n 64         model=JUDGE_MODEL,\n 65         messages=[\n 66             {\"role\": \"system\", \"content\": system_rubric},\n 67             {\"role\": \"user\", \"content\": evaluation_prompt}\n 68         ],\n 69         api_base=OLLAMA_API,\n 70         json_mode=True # Enforce JSON output for easy parsing\n 71     )\n 72     \n 73     return json.loads(response.choices[0].message.content)\n 74 \n 75 # --- MAIN EXECUTION ---\n 76 if __name__ == \"__main__\":\n 77     test_cases = [\n 78         \"Write a Python function to calculate the Fibonacci sequence.\",\n 79         \"What are the names of President Lincoln's children?\",\n 80         \"Compare Quantum Entanglement with the spirituality of meditation in two sentences.\"\n 81     ]\n 82 \n 83     print(f\"👨‍🎓 Student Model: {STUDENT_MODEL}\")\n 84     print(f\"⚖️  Judge Model:   {JUDGE_MODEL}\\n\")\n 85     print(\"-\" * 60)\n 86 \n 87     for i, test_prompt in enumerate(test_cases, 1):\n 88         print(f\"Test #{i}: {test_prompt}\")\n 89         \n 90         # 1. Run the Student\n 91         student_ans = get_student_response(test_prompt)\n 92         if DEBUG:\n 93             print(f\"\\n>> Student Answer: {student_ans}...\")\n 94         else:\n 95             print(f\"\\n>> Partial Student Answer: {student_ans[:DEBUG_LENGTH]}...\")\n 96         \n 97         # 2. Run the Judge\n 98         try:\n 99             evaluation = judge_response(test_prompt, student_ans)\n100             \n101             # 3. Report Card\n102             status = \"✅ PASS\" if evaluation.get('pass', evaluation['score'] > 5) else \"❌ FAIL\"\n103             print(f\">> Judge Rating: {evaluation['score']}/10 | {status}\")\n104             print(f\">> Critique: {evaluation['reasoning']}\")\n105             if 'suggested_prompt' in evaluation:\n106                 suggested = evaluation['suggested_prompt']\n107                 print(f\">> Suggested Prompt: {suggested}\")\n108                 \n109                 # 4. Re-run Student with the improved prompt (only if different)\n110                 if suggested.strip().lower() != test_prompt.strip().lower():\n111                     improved_ans = get_student_response(suggested)\n112                     if DEBUG:\n113                         print(f\">> Improved Student Answer : {improved_ans}\")\n114                     else:\n115                         print(f\">> Partial Improved Student Answer Using New Prompt: {improved_ans[:DEBUG_LENGTH]}...\")\n116                 else:\n117                     print(\">> (Skipped re-run: suggested prompt is same as original)\")\n118         except Exception as e:\n119             print(f\">> Judge Error: {e}\")\n120             \n121         print(\"-\" * 60)\n

\n\n

The core logic resides within the judge_response function, where the “System Rubric” acts as the unit test definition. By setting json_mode=True in the litellm completion call, we force the Judge model to output structured data rather than free-form text, eliminating the fragile string parsing often required when dealing with conversational AI outputs. This ensures that the returned score and reasoning can be immediately utilized by downstream application logic, effectively treating subjective quality assessments as deterministic data points.

This architecture also highlights the strategic advantage of model stratification. The “Student” model is intentionally selected to be small (e.g., Gemma 3 1B), optimizing for inference speed and low resource consumption, while the “Judge” model is significantly larger to ensure nuanced reasoning and adherence to the prompt’s evaluation criteria. By decoupling generation from evaluation, developers can iterate rapidly on the smaller model’s prompts or fine-tuning, using the larger model as a consistent baseline for regression testing without incurring the latency of using a large model for every user interaction.

The following output is edited for brevity:

 1 $ uv run judge.py\n 2 👨‍🎓 Student Model: ollama_chat/gemma3:1b\n 3 ⚖️  Judge Model:   ollama_chat/rnj-1:latest\n 4 \n 5 ------------------------------------------------------------\n 6 Test #1: Write a Python function to calculate the Fibonacci sequence.\n 7 \n 8 >> Partial Student Answer:\n 9 def fibonacci(n):\n10   \"\"\"\n11   Calculates the nth Fibonacci number.\n12 \n13   Args:\n14     n: The index of the desired Fibonacci number (non-negative integer).\n15 \n16   Returns:\n17     The nth Fibonacci number.  Returns 0 if n is 0.\n18     Returns 1 if n is 1.\n19   \"\"\"\n20   if n <= 0:\n21     return 0\n22   elif n == 1:\n23     retur...\n24 >> Judge Rating: 8/10 | ✅ PASS\n25 >> Critique: The AI response provides a well-structured, efficient, and correct implementation of the Fibonacci sequence calculation. It includes a clear docstring, handles edge cases, uses an iterative approach for efficiency, and is well-documented. The only minor improvement could be adding a brief explanation of the iterative approach efficiency in the docstring.\n26 >> Suggested Prompt: Write a Python function to calculate the Fibonacci sequence. Explain the efficiency of the iterative approach compared to a recursive approach.\n27 >> Partial Improved Student Answer Using New Prompt: ```python\n28 def fibonacci_iterative(n):\n29   \"\"\"\n30   Calculates the nth Fibonacci number using an iterative approach.\n31 \n32   Args:\n33     n: The index of the desired Fibonacci number (non-negative integer).\n34 \n35   Returns:\n36     The nth Fibonacci number.\n37   \"\"\"\n38   if n <= 1:\n39     return n\n40 ------------------------------------------------------------\n41 Test #2: What are the names of President Lincolns children?\n42 \n43 >> Partial Student Answer: Abraham Lincoln had six children:\n44 \n45 1.  **Robert Todd Lincoln:** He was the eldest and was a prominent figure in the abolitionist movement.\n46 2.  **Edward Lincoln:** He was a lawyer and became a prominent figure in the Illinois legislature.\n47 3.  **William Henry Lincoln:** He was a successful businessman...\n48 >> Judge Rating: 7/10 | ✅ PASS\n49 >> Critique: The response is accurate and provides detailed information about Lincolns children. However, it could be more concise by focusing on the names and brief roles of each child.\n50 >> Suggested Prompt: What were the names and roles of Abraham Lincolns children?\n51 >> Partial Improved Student Answer Using New Prompt: Okay, let's break down Abraham Lincolns children. It is s a significant and often fascinating part of his life and legacy. Here's a list of his children, along with their roles and a little more detail:\n52 \n53 **1. Mary Todd Lincoln (1818-1865)**\n54 \n55 *   **Role:** Wife and First Lady of the United States. She ...\n56 ------------------------------------------------------------\n57 Test #3: Compare Quantum Entanglement with the spirituality of meditation in two sentences.\n58 \n59 >> Partial Student Answer: Quantum entanglement, where two particles become linked regardless of distance, mirrors the meditative experience of aligning with a deeper, interconnected reality, suggesting a unified field of consciousness.  Meditation, through focused awareness and stillness, can create a similar sense of resona...\n60 >> Judge Rating: 8/10 | ✅ PASS\n61 >> Critique: The response accurately draws a meaningful parallel between quantum entanglement and meditation, using concise language. It effectively conveys the concept of interconnectedness in both physics and spirituality.\n62 >> Suggested Prompt: How does the concept of quantum entanglement relate to the spiritual practice of meditation in terms of interconnectedness and unity?\n63 >> Partial Improved Student Answer Using New Prompt: Okay, this is a fascinating and increasingly explored intersection of quantum physics and spirituality. The connection between quantum entanglement and meditation is a complex and evolving area of research and discussion, but here is a breakdown of how it is being viewed, focusing on interconnectednes...\n64 ------------------------------------------------------------\n

\n\n

Wrap Up for LLM Metrics

Ultimately, defining metrics is the act of maturing your project from a fascinating experiment into a reliable software asset. In the realm of “Small AI,” where we deliberately trade the wider-range, sometimes omniscient-seeming capabilities of trillion-parameter models for the targeted efficiency of local systems, these metrics become your navigation tools. They allow you to pinpoint exactly where a smaller model punches above its weight, proving that while a local 8B model might lack the generic poetry writing skills of a frontier model, it can match or exceed it in specific, domain-constrained tasks when measured by accuracy per watt and latency. By moving beyond subjective “vibe checks” to rigorous standards like the LLM-as-a-Judge pattern and precise operational baselines, you gain the confidence to deploy appropriately sized models. You are no longer crossing your fingers and hoping the model works; you are engineering a system with known tolerances, predictable costs, and a clear definition of what “winning” looks like for your specific use case.

A Fair Shootout: Evaluating Large vs. Small Models for Task Fitness

Theory is useful, but engineering is built on data. In the previous chapter, we defined the scorecard: Operational Metrics (speed, cost) and Quality Metrics (smartness, faithfulness). We also covered the “judge pattern” for using a stronger model to help evaluate and tune the use of smaller models. In this chapter, we will “firm up” our methodology for evaluating different models for fitness for RAG tasks.

The Setup for RAG: David vs. Goliath

Let’s consider a “shootout” to compare the two models, large vs. small:

\nThe Goliath (Cloud Baseline): GPT-5 or Gemini-3-pro (via API). Powerful, easy to start, but expensive at scale and slower due to network round-trips.\n
\nThe David (Small AI Candidate): rnj-1-8B, gemma3-1B, or Llama-3-8B, running locally on a consumer-grade GPU (NVIDIA RTX 4090) or on Apple Silicon.\n

If we run a test set of 200 typical user questions through both systems the resource costs will be much lower when we use prompt caching.

We evaluate operational metrics:

\nTTFT (latency)\n
\nCost of 10K requests/day\n
\nPrivacy\n

We also evaluate quality metrics (how “smart” is the model) using RAG testing:

\nContext precision: How well did we find the best (or at least a good) document?\n
\nFaithfulness: Given retrieved documents, his well did the model stick to the facts provided in preparing a summary or answer?\n
\nAnswer relevance: Did the model answer the user’s question?)\n

Fairness: Adjusting Problem Setups for Small vs. Large Models

When conducting a shootout between a massive frontier model and a local Small AI model, a naive “apples-to-apples” comparison can be misleading. Using the exact same prompt and context window for both often biases the results in favor of the larger model, which generally possesses better instruction-following resilience for poorly structured inputs. To conduct a fair engineering assessment, you must adjust the setup to allow each model to perform at its peak potential.

Prompt Divergence

Large models are excellent at interpreting “lazy” or conversational prompts. You can ask GPT-5, “Check this doc and tell me if it’s okay,” and it will likely infer your intent. A 7B parameter model, however, may get confused by the ambiguity:

\nThe Unfair Test: Using a single, vague prompt for both models.\n
\nThe Fair Test: Optimize the prompt for each model. The large model gets the conversational prompt; the Small AI gets a structured, specific instruction (e.g., “Analyze the text below. List any errors in grammar or factual consistency. Output JSON.”). We are comparing the best possible version of the Small AI system against the large one, not their ability to guess intent.\n

Context Window and “Lost in the Middle”

While a frontier model might handle a 128k token context window with ease, smaller models often degrade in performance as the context fills up, suffering from the “Lost in the Middle” phenomenon where information in the center of the prompt is ignored:

\nThe Unfair Test: Stuffing 50 documents into the context window of an 8B model and expecting it to find a needle in the haystack.\n
\nThe Fair Test: Adjusting the RAG (Retrieval Augmented Generation) strategy. For the Small AI, you might retrieve fewer, more relevant chunks (e.g., Top-3 matches instead of Top-10) or use a “reranker” step to ensure the most critical data is at the beginning or end of the context.\n

In summary, fairness does not mean “identical inputs.” It means “identical goals.” If the goal is to answer a user question, you should allow the Small AI architecture to use tighter prompts and focused context retrieval to achieve that goal, rather than penalizing it for not being a generalist giant.

Comparing Small vs. Large Models for NLP and Extracting Structured Data From Text

Now we look at specific text processing tasks—like summarizing text (NLP) or turning documents or emails into clean JSON (Structured Data Extraction). The performance gap between small and large models changes shape entirely. The “best” model depends heavily on whether the task requires world knowledge or mechanical consistency.

NLP Tasks: The Battle for Nuance

For general Natural Language Processing tasks like summarization, sentiment analysis, and tone adjustment, the sheer parameter count of large models often provides a distinct advantage in handling subtlety:

\nLarge models (Abstractive Power): Large models excel at “abstractive” tasks—reading a text and rewriting it with a new flow or voice. If you ask GPT-5 to “summarize this technical report in the style of a distinct 1940s noir detective,” it draws on a vast cultural training set to nail the stylistic nuance.\n
\nSmall models (Extractive Utility): Small models often struggle with high-concept stylistic shifts. However, they are highly effective at “extractive” tasks—identifying the key sentences and condensing them. For a business use case (e.g., “Summarize the three main blockers in this meeting transcript”), a fine-tuned 8B model is often indistinguishable from the frontier model, provided the prompt is direct.\n

Dear reader, for my own work and experiments, I usually use Large AI for reviewing what I write, studying and understanding philosopy, and deep research. I use Small AI for factual summarization, classification, data processing, and routing.

Extracting Structured Data: The Small AI Sweet Spot

Extracting structured data (converting unstructured text into JSON, CSV, or SQL) is arguably a “killer app” for Small AI. This task does not require reasoning about the meaning of the universe; it requires strict adherence to syntax and pattern matching.

Surprisingly, small models often rival or even outperform larger generalist models in this domain when properly constrained, largely because they are less prone to “chatty” deviations:

\nThe Mechanics of Extraction: If you need to parse 10,000 resumes to extract “Name,” “Years of Experience,” and “Education,” a massive model is overkill. The reasoning required to identify a date or a university name is low-level.\n
\nConstrained Decoding (Grammars): The equalizer for Small AI in this arena is constrained decoding (often available in local runtimes like llama.cpp). You can force a local model to output only valid JSON schema tokens.\n

Important technique: In almost all cases, you must dear reader, provide one or two examples of the format and style of the JSON output you need and specify that you only want JSON data returned. You must also provide “guard-rail code” to strip off Markdown formatting and use a JSON parsing library to make sure that each JSON output is legally formatted and log and errors. Here is an example prompt with sample User Input at the end:

 1 System Instruction: You are a helpful assistant that extracts data from text. Output ONLY valid JSON. Do not include Markdown formatting (like ```json), explanations, or conversational filler.\n 2 \n 3 Example 1:\n 4 \n 5 Input: \"I have a meeting with Alice in Engineering on Friday at 3 PM regarding the API migration.\"\n 6 \n 7 Output: { \"event_type\": \"meeting\", \"participants\": [\"Alice\"], \"department\": \"Engineering\", \"time\": \"15:00\", \"topic\": \"API migration\" }\n 8 \n 9 Process this user Input:\n10 \n11 \"Please schedule a lunch with Bob and Mary from HR at 1PM to discuss the new hiring guidelines.\"\n

\n\n

We would expect output like:

1 { \"event_type\": \"lunch\",\n2   \"participants\": [\"Bob\", \"Mary\"],\n3   \"department\": \"HR\", \"time\": \"13:00\",\n4   \"topic\": \"new hiring guidelines\" }\n

\n\n

For evaluating large vs. small models it might be sufficient to define this prompt as a template and test it for a small number of test cases and hand score the outputs from both small and large models you are evaluating.

If you have a large number of test cases, you can automate the small vs. large model evaluation by writing a test harness that captures the percentage of test cases for which the models fails to output valid JSON. Given a set of test cases that fail, spend the time to iterate on the prompt template you are using for your data extraction project.

Verdict: For high-volume data extraction, Small AI is usually superior. It is significantly faster, orders of magnitude cheaper, and when paired with grammar constraints, it offers higher reliability for automated pipelines.

Wrap Up for Evaluating Large vs. Small Models for Task Fitness

Dear reader, this is my opinion: Evaluating AI is no longer about chasing state-of-the-art benchmarks on generic leaderboards; it is about “fitness for purpose” and not any purpose, but specifically fitness to your requirements. By rigorously applying metrics—measuring TTFT for speed, RAGAS scores for quality, and watts for efficiency then you can quantify exactly what different models are trading off. Often evaluations show that a well-prompted local 8B model could deliver a “good enough” experience that was satisfactory in every operational dimension. The lesson for your projects is clear: define your pass/fail criteria first, then optimize downward to the smallest, fastest model that can clear that bar.

Part III - Python Examples for Switching Between Models

In the upcoming chapters, we explore how to use a unified LLM interface that balances cost, performance, and local sovereignty by using the libraries LiteLLM and RouteLLM. LiteLLM provides a consistent completion function to interact with both high-end cloud providers like OpenAI, Claude, and Gemini and also local instances via Ollama. RouteLLM acts as an intelligent gateway: by configuring a “Strong” model (GPT-4) alongside a “Weak” model (Qwen 3 running locally), the script utilizes a Matrix Factorization (MF) router to dynamically assign tasks to different models based on the complexity of user prompts. This architecture allows developers to define a THRESHOLD value, effectively a dial to tune the system’s preference for quality versus economy—ensuring that simple queries stay local while sophisticated reasoning tasks are routed to either the cloud or stronger local models, all while maintaining a singular, standardized codebase for model interaction.

Using the LiteLLM Library to Easily Switch Between Models and Model Providers

You can find the documentation for the litellm library here: https://docs.litellm.ai.

The examples shown below are in the directory Winning-with-Small-AI-book/source-code/litellm_examples.

One of the biggest challenges in modern AI application development is the fragmentation of APIs;\ninteracting with a local model via Ollama usually requires a different library and syntax than\nconnecting to OpenAI or Anthropic. To illustrate this “native” approach, let’s look at how we\ninteract with a local model like on of the Qwen 3 series (similar models, different parameter counts) using the official Ollama Python library. In the\ncode below, we must instantiate a provider-specific client and use its unique chat method to request\na Python script for printing prime numbers.

 1 import ollama\n 2 \n 3 # The Client is initialized with the host/api_base\n 4 client = ollama.Client(host='http://localhost:11434')\n 5 \n 6 response = client.chat(\n 7     model=\"qwen3:1.7b\",\n 8     messages=[{\n 9         \"role\": \"user\",\n10         \"content\": \"Write a Python script to print the first ten prime numbers. Be concise.\"\n11     }]\n12 )\n13 \n14 # Native SDK returns a dictionary-like object (ChatResponse)\n15 print(response['message']['content'])\n

\n\n

LiteLLM simplifies this landscape by abstracting these differences behind a unified,\nOpenAI-compatible interface. Instead of maintaining separate logic for every provider, you can use a\nsingle completion function for everything. Here is how we can rewrite the previous example to use\nLiteLLM with the same local Ollama instance. Notice that we simply prefix the model name with\nollama_chat/ and point to our local API base, while the message structure becomes standardized.

 1 from litellm import completion\n 2 from pprint import pprint\n 3 \n 4 response = completion(\n 5     model=\"ollama_chat/qwen3:1.7b\", \n 6     messages=[{\"content\":\n 7                \"Write a Python script to print the first ten prime numbers. Be concise\",\n 8                \"role\": \"user\"}], \n 9     api_base=\"http://localhost:11434\"\n10 )\n11 #pprint(response)\n12 #print(\"\\n\")\n13 print(response.choices[0].message.content) # text content\n

\n\n

The true utility of this abstraction becomes evident when you need to switch providers entirely,\nperhaps moving from development to production. If you decide to swap your local model for a\ncloud-based one like OpenAI’s gpt-5-mini, you do not need to rewrite your application logic or\nimport a new SDK. You merely change the model parameter in the completion function and ensure your\nenvironment variables are configured. The code below demonstrates how the interaction remains\nvirtually identical despite the underlying provider changing completely.

 1 from litellm import completion\n 2 import os\n 3 \n 4 ## set ENV variables if the env key OPENAI_API_KEY is not set in your environment:\n 5 #os.environ[\"OPENAI_API_KEY\"] = \"your-api-key\"\n 6 \n 7 #REASONING = \"minimal\"\n 8 #REASONING = \"low\"\n 9 REASONING = \"medium\"\n10 #REASONING = \"high\"\n11 \n12 response = completion(\n13   model=\"gpt-5-mini\",\n14   messages=[{ \"content\":\n15               \"What is the capital of France?Write a Python script to print the first ten prime numbers. Be concise.\",\n16               \"role\": \"user\"}]\n17 )\n18 \n19 print(response)\n20 print(response.choices[0].message.content) # text content\n

\n\n

By adopting LiteLLM, you effectively decouple your application logic from specific model vendors.\nThis flexibility allows you to prototype rapidly and cost-effectively with local models before\nseamlessly deploying to more powerful cloud models when necessary. As these examples demonstrate,\nwhether you are running one of the lightweight Qwen 3 models on your laptop or connecting to a cutting-edge\nproprietary model, your core codebase remains consistent, clean, and easy to maintain.

Tool Use with LiteLLM

LiteLLM provides extensive support for tool use (function calling) by acting as a “universal translator” that maps the OpenAI-standard tools and tool_choice parameters to various other model providers like Anthropic, Cohere, Vertex AI, Google AI Studio, and local models running on Ollama.

LiteLLM normalizes the tool-calling interface so you can write your code once using the OpenAI schema. Behind the scenes, it transforms your JSON schema into the specific format required by the target provider (e.g., Anthropic’s tools format or Gemini’s function_declarations).

Dear reader, I cover tool use (function calling) in several of my other recent books and won’t provide examples here except for sharing a code snippet to determine which models support tool use with LiteLLM:

 1 import litellm\n 2 \n 3 supports=litellm.supports_function_calling\n 4 \n 5 print(f'{supports(model=\"anthropic/claude-sonnet-4-5-20250929\")=}')\n 6 print(f'{supports(model=\"claude-sonnet-4-5-20250929\")=}')\n 7 print(f'{supports(model=\"claude-opus-4-5-20251101\")=}')\n 8 print(f'{supports(model=\"gpt-4o\")=}')\n 9 print(f'{supports(model=\"gpt-5.1-mini\")=}')\n10 print(f'{supports(model=\"gpt-5.1\")=}')\n11 print(f'{supports(model=\"gpt-5.2\")=}')\n12 print(f'{supports(model=\"gemini-3-flash-preview\")=}')\n13 print(f'{supports(model=\"gemini-3-pro-preview\")=}')\n14 print(f'{supports(model=\"gemini/gemini-pro\")=}')\n15 \n16 print(f'{supports(model=\"ollama/mistral\")=}')\n17 print(f'{supports(model=\"ollama_chat/mistral\")=}')\n18 print(f'{supports(model=\"ollama/deepseek-r1\")=}')\n19 print(f'{supports(model=\"ollama_chat/deepseek-r1\")=}')\n20 \n21 print(f'{supports(model=\"ollama/gemma3:1b\")=}')\n22 print(f'{supports(model=\"ollama_chat/gemma3:1b\")=}')\n23 \n24 print(f'{supports(model=\"ollama/llama3.1\")=}')\n25 print(f'{supports(model=\"ollama/llama3.2\")=}')\n

\n\n

The output looks like:

 1 $ uv run check_tool_use_support.py\n 2 supports(model=\"anthropic/claude-sonnet-4-5-20250929\")=True\n 3 supports(model=\"claude-sonnet-4-5-20250929\")=True\n 4 supports(model=\"claude-opus-4-5-20251101\")=True\n 5 supports(model=\"gpt-4o\")=True\n 6 supports(model=\"gpt-5.1-mini\")=False\n 7 supports(model=\"gpt-5.1\")=True\n 8 supports(model=\"gpt-5.2\")=True\n 9 supports(model=\"gemini-3-flash-preview\")=True\n10 supports(model=\"gemini-3-pro-preview\")=True\n11 supports(model=\"gemini/gemini-pro\")=True\n12 supports(model=\"ollama/mistral\")=True\n13 supports(model=\"ollama_chat/mistral\")=False\n14 supports(model=\"ollama/deepseek-r1\")=False\n15 supports(model=\"ollama_chat/deepseek-r1\")=False\n16 supports(model=\"ollama/gemma3:1b\")=False\n17 supports(model=\"ollama_chat/gemma3:1b\")=False\n18 supports(model=\"ollama/llama3.1\")=True\n19 supports(model=\"ollama/llama3.2\")=False\n

\n\n

LiteLLM Wrap Up

As useful as LiteLLM abstractions are for switching between models and model providers, I use LiteLLM infrequently and only in cases where I plan on experimenting with a wide variety of models for a new task.

In the next chapter we look at the library RouteLLM that uses LiteLLM in its implementation. RouteLLM chooses between models and providers automatically based on the estimated complexity or difficulty of a LLM prompt.

Automatically Routing to the ‘Best’ Model Using RouteLLM Library

In the pursuit of performance, developers often default to the most powerful models available, yet this “frontier-first” approach frequently leads to unnecessary costs and latency for trivial queries. As we explore the paradigm of “Winning Big With Small AI,” the challenge becomes identifying exactly when a lightweight, local model is sufficient and when the heavy lifting of a GPT-4 or Claude 3.5 Sonnet is truly required. This chapter introduces RouteLLM, an open-source framework designed to automate this decision-making process. By using an intelligent controller to score incoming prompts, RouteLLM acts as a high-speed traffic cop, seamlessly diverting simpler tasks to your local hardware while reserving cloud-based API calls for the most complex reasoning tasks. This hybrid architecture doesn’t just save money—it optimizes the entire lifecycle of an AI application by ensuring every prompt is handled by the most efficient tool for the job.

Here we use the package https://github.com/lm-sys/RouteLLM.

The provided Python script demonstrates how to implement an intelligent routing layer using the routellm library to optimize for both cost and performance by dynamically selecting between a high-capability “strong” model and a local “weak” model. By leveraging the Matrix Factorization (MF) router, the application evaluates the complexity of an incoming user prompt and directs it to either OpenAI’s GPT-4 for nuanced reasoning or a local Ollama instance running Qwen 3 for simpler tasks. This architectural pattern is essential for developers looking to maintain high-quality outputs while drastically reducing API latency and operational expenses, as it ensures that expensive tokens are only consumed when the difficulty of the request justifies the cost. The code covers the initial controller setup, environmental configuration for local hosting, and the execution of a threshold-based routing call that returns both the generated response and metadata identifying which model was ultimately selected to handle the query.

The implementation begins by initializing a Controller that bridges two distinct environments: the OpenAI cloud and a local Ollama server. By setting the OLLAMA_API_BASE environment variable, the script ensures that the “weak” model requests are directed to the local hardware rather than a remote endpoint. The core logic resides in the router-mf-0.1 model string, where the Matrix Factorization algorithm uses a calibrated threshold to decide if the prompt is sufficiently complex to require GPT-4.

Once the query is processed, the script prints the specific model name used for the generation alongside the response content. This transparency is crucial for debugging and fine-tuning the routing threshold, as a lower threshold will lean toward the strong model while a higher one favors the weak model. This approach provides a flexible foundation for building “local-first” AI applications that can seamlessly escalate to the frontier models when local resources reach their reasoning limits.

Implementation of a Command Line Query Utility Using RouteLLM

The following script Winning-with-Small-AI-book/source-code/auto-route/ask.py demonstrates how to implement an intelligent routing layer using the routellm library to optimize for both cost and performance by dynamically selecting between a high-capability “strong” model and a local “weak” model. By leveraging the Matrix Factorization (MF) router, the application evaluates the complexity of an incoming user prompt and directs it to either OpenAI’s GPT-4 for nuanced reasoning or a local Ollama instance running Qwen 3 for simpler tasks. This architectural pattern is essential for developers looking to maintain high-quality outputs while drastically reducing API latency and operational expenses, as it ensures that expensive tokens are only consumed when the difficulty of the request justifies the cost. The code covers the initial controller setup, environmental configuration for local hosting, and the execution of a threshold-based routing call that returns both the generated response and metadata identifying which model was ultimately selected to handle the query.

 1 import sys\n 2 import os\n 3 from routellm.controller import Controller\n 4 import litellm\n 5 \n 6 # 1. SETUP (Hardcode your preferred chain here)\n 7 # This example uses GPT-4 (Strong) and Ollama (Weak/Local)\n 8 #os.environ[\"OPENAI_API_KEY\"] = \"your-key-here\"\n 9 \n10 # Set OLLAMA_API_BASE to ensure the weak model uses localhost,\n11 # while the strong model (OpenAI) continues to use the official API.\n12 os.environ[\"OLLAMA_API_BASE\"] = \"http://localhost:11434\"\n13 \n14 client = Controller(\n15     routers=[\"mf\"],\n16     strong_model=\"openai/gpt-4\", \n17     weak_model=\"ollama_chat/qwen3:1.7b\",  # Connects to your local Ollama\n18 )\n19 \n20 print(client)\n21 print(sys.argv)\n22 \n23 # 2. GET INPUT\n24 if len(sys.argv) < 2:\n25     print(\"Usage: uv run ask.py 'Your query here'\")\n26     sys.exit()\n27 \n28 query = sys.argv[1]\n29 \n30 # 3. RUN ROUTER\n31 # model THRESHOLD of  0.11593 is the default balance (~50% routing split)\n32 # Making THRESHOLD smaller has the effect of perferring the stronger model.\n33 # Making THRESHOLD larger has the effect of perferring the weaker model.\n34 \n35 THRESHOLD = str(0.1) # slight preference for the stronger model\n36 \n37 response = client.chat.completions.create(\n38     model=\"router-mf-\" + THRESHOLD,\n39     messages=[\n40         {\"role\": \"system\", \"content\": \"Your answers are very concise\"},\n41         {\"role\": \"user\", \"content\": query}\n42     ]\n43 )\n44 \n45 # 4. PRINT RESULT\n46 print(f\"[{response.model}]\") \n47 print(response.choices[0].message.content)\n

\n\n

Wrap Up for Winning Big With Small AI

Dear reader, we have traveled a long path together in this short book. We began by challenging the prevailing industry dogma of the “Scaling Hypothesis” which suggests that bigger is always better. We end with a new engineering reality: that the most useful AI is often the one that is small enough to run where you need it, fast enough to feel instantaneous, and efficient enough to sustain both your budget and the environment.

As we conclude, it is essential to synthesize the technical strategies, architectural patterns, and economic philosophies we have explored. The transition from using massive frontier models for everything to a nuanced, hybrid approach using “Small AI” is not just a cost-saving measure; it is a maturity model for the entire discipline of software engineering in the age of artificial intelligence.

The Engineering Reality: From Vibes to Metrics

In the early days of the generative AI boom, evaluation was subjective, often referred to as a “vibe check” where we hoped the model would perform well. Throughout this book, we have dismantled that approach. We established that building reliable systems requires moving from magic to mechanics.

We learned that Operational Metrics are the superpower of Small AI. When we run a quantized 1.7B or 4B parameter model locally, we are not just saving money; we are optimizing for Time to First Token (TTFT) and Tokens Per Second (TPS). By achieving a TTFT of under 200ms, we create applications that feel “real-time,” breaking the latency barriers that plague massive cloud-based models.

However, speed is nothing without intelligence. This is why we adopted the “LLM-as-a-Judge” pattern. We recognized that we cannot trust a small model blindly. By using a frontier model (like GPT-5 or Gemini 3 Pro) to grade the output of our smaller models (like Gemma 3 or Qwen 3), we essentially distill the intelligence of the giant into the efficiency of the dwarf. We codified this into a repeatable workflow: generate a response with the student model, and have the judge evaluate it against a strict rubric for accuracy, brevity, and format compliance.

The Architecture of Hybrid Intelligence

One of the most profound lessons we covered is that the future is not a monolith; it is a mosaic. We do not have to choose strictly between “Local” and “Cloud.” Through the use of libraries like LiteLLM and RouteLLM, we explored the power of hybrid architectures.

In our Python examples, we demonstrated how RouteLLM acts as an intelligent traffic cop. By using Matrix Factorization (MF) routers, we can dynamically assess the complexity of a user prompt. If a user asks for a simple Python script to print prime numbers, the router directs this to a local “weak” model like one of the smaller Qwen 3 models (0.6B, 1.7B, or 4B model parameters), costing us virtually nothing. If the user asks for a nuanced comparison of quantum entanglement and meditation, the router recognizes the complexity and escalates the request to a “strong” model like GPT-4 or Gemini 3 Pro.

This architectural pattern allows us to tune the THRESHOLD variable that is a dial that controls the trade-off between quality and economy. This is the definition of engineering control. It ensures that we are not burning expensive GPU cycles on trivial tasks, nor are we frustrating users with incompetent answers to complex questions.

The “Fitness for Purpose” Philosophy

Throughout Part II, we conducted a “Fair Shootout” between David (Small AI) and Goliath (Cloud Baseline). We learned that fairness does not mean giving both models the same prompt; it means giving them the same goal.

We discovered that Small AI has a specific “Sweet Spot”: Extracting Structured Data. While a small model may struggle to write a noir-style detective novel, it excels at strict syntax adherence. When constrained by grammars or rigid system prompts, a local 7B model can extract JSON from text with close to the same reliability as a frontier model, but orders of magnitude faster and cheaper.

Conversely, we acknowledged the Drawbacks of Small AI. We must remain vigilant against “Prompt Fragility,” where a minor change in wording causes a small model to fail, and “Reduced Reasoning Depth,” where the model takes shortcuts in logic. We also discussed the “Lost in the Middle” phenomenon, where small models struggle to retrieve facts buried in the center of large context windows. The mitigation strategy here is precise engineering: using RAG metrics like Context Precision and Faithfulness to ensure we are feeding the model only the most relevant data, rather than stuffing the context window with noise.

Sovereignty, Solvency, and Sustainability

Beyond code and architecture, we touched upon the economic and legal imperatives driving the shift to Small AI. The EU AI Act of 2025 and other governance frameworks are pushing organizations toward data sovereignty. Running models locally via Ollama or LM Studio ensures compliance by design: data never leaves your personal device or in the case of companies data need not leave your local IT infrastructure.

Furthermore, we cannot ignore the “Golden Ratio”: Accuracy per Watt. As we discussed, if a 70B parameter model offers 98% accuracy but burns 400 watts, and an 8B model offers 96% accuracy at 40 watts, the engineering choice often favors the latter. We saw that Gemini 3 Flash consumes approximately 0.6 kWh per 1 million tokens, while massive reasoning models like OpenAI o3 can consume up to 33.0 kWh for the same volume. In a world increasingly conscious of energy consumption, Small AI is the sustainable choice.

Final Recommendations for Your Journey

Dear reader, as you close this book and return to your IDEs and command lines, I offer this final advice:

\nDefine Your Pass/Fail Criteria First: Do not start by picking a model. Start by defining the metrics (TTFT, RAGAS scores, JSON validity). Once you have the bar set, optimize downward to the smallest model that clears it.\n
\nEmbrace the “Judge” Pattern: Do not rely on your intuition. Automate your evaluation pipelines so that you can swap out models (e.g., from Llama-3.1 to Qwen 3) with confidence, knowing exactly how performance changes.\n
\nStart Local, Scale Hybrid: Begin your development with local tools like Ollama. They are free, private, and fast. As you move to production, introduce routing layers (RouteLLM) to bring in the big guns only when necessary.\n
\nRespect the Physics of Thought: Remember that small models learn differently, often through distillation. They are efficient compressors of knowledge, not omniscient databases. Feed them context, guide them with specific prompts, and restrict their output formats.\n

The era of “one model to rule them all” may be over for specific engineering projects. While Super Scalers like Google, Microsoft, and Amazon AWS would like to offer you huge one size fits all solutions, I recommend not using these as your defaults when trying to solve engineering problems. We have entered the era of distributed, specialized intelligence. The most useful AI is not the one that knows everything, but the one that is small enough to be everywhere.

I hope dear reader that I have convinced you in this short book to consider the use of smaller AI models as appropirate for your projects. I used local models on Ollama and also Google Gemini Flash APIs in the cloud for most of our discussion but I hope you easily use this material for whatever your work environment is.

Thank you for joining me on this exploration. Now, go forth and win big with Small AI.

You can contact me through my web site markwatson.com.

# Part IV - Fast and Efficient Rag Document System

Here we develop an application that is practical to run that uses a local vector database and an inexpensive cloud based model.

RAG Using zvec Vector Datastore and Gemini-3-flash Model

The zvec library implements a lightweight, lightning-fast, in-process vector database. Allibaba released zvec in February 2026. We will see how to use zvec and then build a high performance RAG system. We will use the very low cost gemini-3-flash-preview model.

Note: There is a similar example that uses zvec and a small local model in my “Ollama in Action: Building Safe, Private AI with LLMs, Function Calling and Agents” book. This is a link to read the corresponding chapter online.

Design Notes for Example Program

Building a Retrieval-Augmented Generation (RAG) pipeline using a frontier cloud model offloads heavy compute requirements and provides access to advanced reasoning, high-speed generation, and massive context windows. In this chapter, we construct a high-performance RAG system utilizing the Gemini API for both embeddings and inference (gemini-3-flash-preview), paired with zvec, a lightweight, high-performance local vector database.

The architecture follows a classic two-phase RAG pattern, adding an additional third step to improve the user experience:

\nIngestion: Parse local text files, chunk the content, generate embeddings via the Gemini API (e.g., using text-embedding-004), and index them into zvec.\n
\nRetrieval: Embed the user query via the Gemini API, perform a similarity search in zvec, and extract the top-k most relevant chunks.\n
\nGeneration & Formatting: Pass the retrieved chunks along with the user’s original query to gemini-3-flash-preview. The model synthesizes the provided context to generate a highly accurate, well-formatted response for the user to read.\n

Example zvec RAG Application

This script demonstrates a practical implementation of a Retrieval Augmented Generation (RAG) system, bridging the gap between local document storage and the use of a very inexpensive cloud-based large language models. By leveraging the Google Gemini SDK for both vector embeddings and natural language generation, and the zvec library for high-performance local vector indexing, the code provides a complete pipeline for scanning a directory of text-based files, chunking them into manageable segments, and storing their semantic representations in a searchable database. The configuration management highlights best practices for API security by prioritizing environment variables, while the integration of a custom schema within zvec ensures that both the high-dimensional vectors (3072 dimensions for gemini-embedding-001) and their corresponding text metadata are preserved for quick retrieval during the chat phase.

  1 #!/usr/bin/env python3\n  2 \"\"\"RAG Chat with Google Gemini - Gemini-3-Flash-Preview\"\"\"\n  3 \n  4 import os\n  5 from pathlib import Path\n  6 from google import genai\n  7 from google.genai import types\n  8 import zvec\n  9 \n 10 # Configuration\n 11 config = {\n 12   \"data_dir\": \"../data\",\n 13   \"extensions\": [\".md\", \".txt\", \".pdf\", \".html\"],\n 14   \"chunk_size\": 500,\n 15   \"overlap\": 50,\n 16   \"topk\": 5,\n 17   # Prioritize GEMINI_API_KEY to avoid warning if both are set\n 18   \"gemini_api_key\": (\n 19     os.getenv(\"GEMINI_API_KEY\") or os.getenv(\"GOOGLE_API_KEY\", \"\")\n 20   ),\n 21   \"embedding_model\": \"gemini-embedding-001\",\n 22   \"chat_model\": \"gemini-3-flash-preview\",\n 23 }\n 24 \n 25 # Set up Gemini Client\n 26 if config[\"gemini_api_key\"]:\n 27   # The SDK handles the API key\n 28   client = genai.Client(api_key=config[\"gemini_api_key\"])\n 29 else:\n 30   client = None\n 31   print(\"Warning: GEMINI_API_KEY not set\")\n 32 \n 33 def get_embedding(text, model=config[\"embedding_model\"]):\n 34   \"\"\"Get embedding vector using Gemini embedding model.\"\"\"\n 35   if client is None:\n 36     return [0.0] * 3072\n 37   \n 38   if model is None:\n 39     model = config[\"embedding_model\"]\n 40 \n 41   try:\n 42     # Standard ID is 'gemini-embedding-001'\n 43     result = client.models.embed_content(\n 44       model=model,\n 45       contents=text,\n 46       config=types.EmbedContentConfig(task_type=\"RETRIEVAL_QUERY\")\n 47     )\n 48     return result.embeddings[0].values\n 49 \n 50   except Exception as e:\n 51     print(f\"Error calling Gemini embeddings: {e}\")\n 52     # Return zero vector for gemini-embedding-001 (3072 dims)\n 53     return [0.0] * 3072\n 54 \n 55 def chunk_text(text, chunk_size=500, overlap=50):\n 56   \"\"\"Split text into overlapping chunks.\"\"\"\n 57   chunks = []\n 58   start = 0\n 59   while start < len(text):\n 60     end = start + chunk_size\n 61     chunks.append(text[start:end])\n 62     start = end - overlap\n 63   return chunks\n 64 \n 65 def build_index():\n 66   \"\"\"Index all text files from the data directory into zvec.\"\"\"\n 67   schema = zvec.CollectionSchema(\n 68     name=\"example\",\n 69     vectors=zvec.VectorSchema(\n 70       \"embedding\", zvec.DataType.VECTOR_FP32, 3072\n 71     ),\n 72     fields=zvec.FieldSchema(\"text\", zvec.DataType.STRING),\n 73   )\n 74 \n 75   db_path = \"./temp_zvec_example\"\n 76   if os.path.exists(db_path):\n 77     import shutil\n 78     shutil.rmtree(db_path)\n 79 \n 80   collection = zvec.create_and_open(path=db_path, schema=schema)\n 81 \n 82   docs = []\n 83   doc_count = 0\n 84   data_path = Path(config[\"data_dir\"])\n 85   if not data_path.exists():\n 86     print(f\"Data directory {config['data_dir']} not found.\")\n 87     data_path.mkdir(parents=True, exist_ok=True)\n 88     return collection\n 89 \n 90   for root, _, files in os.walk(config[\"data_dir\"]):\n 91     for file in files:\n 92       exts = config[\"extensions\"]\n 93       if any(file.lower().endswith(ext) for ext in exts):\n 94         try:\n 95           file_path = Path(root) / file\n 96           with open(file_path, \"r\", encoding=\"utf-8\") as f:\n 97             content = f.read()\n 98             chunks = chunk_text(content)\n 99           for i, chunk in enumerate(chunks):\n100             embedding = get_embedding(chunk)\n101             docs.append(zvec.Doc(\n102               id=f\"{file}_{i}\",\n103               vectors={\"embedding\": embedding},\n104               fields={\"text\": chunk},\n105             ))\n106             doc_count += 1\n107         except Exception as e:\n108           print(f\"Error processing {file}: {e}\")\n109 \n110   if docs:\n111     collection.insert(docs)\n112     print(f\"Indexed {doc_count} chunks from {config['data_dir']}\")\n113   else:\n114     print(\"No documents found to index.\")\n115   return collection\n116 \n117 def search(collection, query, topk=5):\n118   \"\"\"Search the zvec collection for relevant chunks.\"\"\"\n119   query_vector = get_embedding(query)\n120   results = collection.query(\n121     zvec.VectorQuery(\"embedding\", vector=query_vector),\n122     topk=topk,\n123   )\n124   chunks = []\n125   for res in results:\n126     text = res.fields.get(\"text\", \"\") if res.fields else \"\"\n127     if text:\n128       chunks.append(text)\n129   return chunks\n130 \n131 def ask_gemini(question, context_chunks):\n132   \"\"\"Send retrieved chunks + question to Gemini model.\"\"\"\n133   if client is None:\n134     return \"Error: Client not initialized (missing API key)\"\n135 \n136   context = \"\\n\\n---\\n\\n\".join(context_chunks)\n137   system_prompt = (\n138     \"You are a helpful assistant. Answer the user's question \"\n139     \"using ONLY the context provided below. If the context does \"\n140     \"not contain enough information, say so. Be concise and \"\n141     \"accurate.\\n\\nContext:\\n\" + context\n142   )\n143 \n144   try:\n145     prompt = f\"{system_prompt}\\n\\nQuestion: {question}\"\n146 \n147     response = client.models.generate_content(\n148       model=config[\"chat_model\"],\n149       contents=prompt\n150     )\n151     if response.text:\n152       return response.text\n153     else:\n154       return \"No response from Gemini\"\n155 \n156   except Exception as e:\n157     return f\"Error calling Gemini chat: {e}\"\n158 \n159 def main():\n160   print(\"Building zvec index from text files …\")\n161   collection = build_index()\n162   print(f\"\\nRAG chat ready (model: {config['chat_model']})\")\n163   print(\"Type your question, or 'quit' to exit.\\n\")\n164 \n165   while True:\n166     try:\n167       question = input(\"You> \").strip()\n168     except (EOFError, KeyboardInterrupt):\n169       print(\"\\nGoodbye!\")\n170       break\n171     if not question or question.lower() in (\"quit\", \"exit\", \"q\"):\n172       print(\"Goodbye!\")\n173       break\n174 \n175     chunks = search(collection, question)\n176     if not chunks:\n177       print(\"No relevant chunks found in the index.\\n\")\n178       continue\n179 \n180     answer = ask_gemini(question, chunks)\n181     print(f\"\\nAssistant> {answer}\\n\")\n182 \n183 if __name__ == \"__main__\":\n184   main()\n

\n\n

The core of this implementation lies in the synergy between the build_index and search functions. During the indexing phase, documents are broken down using a sliding window approach that is defined by chunk_size and overlap to ensure context is preserved across segment boundaries. These segments are then transformed into embeddings using the Gemini API and persisted into a zvec collection. This local vector store allows for near instantaneous similarity searches, finding the top-k most relevant document fragments that match the semantic intent of a user’s query without needing to re-process the entire dataset for every question.

The final stage of the pipeline utilizes the gemini-3-flash-preview model to synthesize an answer based strictly on the retrieved context. By injecting the found text chunks into a structured system prompt, the script constrains the model’s behavior, reducing the likelihood of “hallucinations” and ensuring the assistant remains grounded in the provided data. This modular design—separating ingestion, retrieval, and generation—serves as a robust template for building more complex AI applications that require private data grounding and low-latency response times.

Sample Example Output

The example text data in the directory ../data/ contains a few explicit “facts” I added to demonstrate that this system prioritizes the context on the input text documents, and not general model world knowledge.

 1 $ uv run zvec_RAG_app_gemini.py \n 2 Building zvec index from text files …\n 3 Indexed 27 chunks from ../data\n 4 \n 5 RAG chat ready (model: gemini-3-flash-preview)\n 6 Type your question, or 'quit' to exit.\n 7 \n 8 You> who said that economics is bullshit?\n 9 \n10 Assistant> Pauli Blendergast said that economics is bullshit.\n11 \n12 You> what equipment is common in a chemistry laboratory?\n13 \n14 Assistant> A chemistry laboratory stereotypically uses various forms of laboratory glassware, although it is not considered central to the field.\n15 \n16 You> \n

\n\n

Wrap Up for the zvex Based RAG Application

In this chapter, we have developed a fully functional Retrieval Augmented Generation (RAG) implementation using the gemini-3-flash-preview model and the zvec data store. By constructing a pipeline that handles document ingestion, intelligent text chunking, and high-dimensional embedding generation, we have created a system capable of grounding AI responses in local, private data. This approach effectively mitigates the common issue of model hallucinations by providing the LLM with a specific, retrieved context to analyze before it formulates an answer. We defined a strict schema within zvec to ensure metadata like source text remains linked to its vector representation. As you move forward, the patterns established here, specifically the separation of the indexing phase from the query loop, will serve as the architectural foundation for more scaling intensive AI applications, allowing you to swap out embedding models or adjust chunking strategies as your specific dataset requirements evolve.

The application we developed here is capable of handling large text datasets and is inexpensive to run using the gemini-3-flash-preview model.

\n\n

\n\n","tocStructure",[271,272,273,274,275,276,277,278,279,280,281,282,283,284,285,286,287,288,289,290,291,292,293,294,295,296,297,298,299,300,301,302,303,304,305,306,307,308,309,310,311,312,313,314,315,316,317,318,319,320],{"_74":75,"_43":76,"_69":-5,"_77":78,"_70":79,"_86":-5,"_87":-5,"_88":-5},{"_74":83,"_43":84,"_69":-5,"_77":85,"_70":79,"_86":-5,"_87":-5,"_88":-5},{"_74":75,"_43":90,"_69":-5,"_77":91,"_70":79,"_86":-5,"_87":-5,"_88":-5},{"_74":75,"_43":94,"_69":-5,"_77":95,"_70":79,"_86":-5,"_87":-5,"_88":-5},{"_74":83,"_43":98,"_69":-5,"_77":99,"_70":79,"_86":-5,"_87":-5,"_88":-5},{"_74":83,"_43":101,"_69":-5,"_77":102,"_70":79,"_86":-5,"_87":-5,"_88":-5},{"_74":83,"_43":104,"_69":-5,"_77":105,"_70":79,"_86":-5,"_87":-5,"_88":-5},{"_74":83,"_43":107,"_69":-5,"_77":108,"_70":79,"_86":-5,"_87":-5,"_88":-5},{"_74":83,"_43":110,"_69":-5,"_77":111,"_70":79,"_86":-5,"_87":-5,"_88":-5},{"_74":83,"_43":113,"_69":-5,"_77":114,"_70":79,"_86":-5,"_87":-5,"_88":-5},{"_74":83,"_43":116,"_69":-5,"_77":117,"_70":79,"_86":-5,"_87":-5,"_88":-5},{"_74":75,"_43":119,"_69":-5,"_77":120,"_70":79,"_86":-5,"_87":-5,"_88":-5},{"_74":83,"_43":123,"_69":-5,"_77":124,"_70":79,"_86":-5,"_87":-5,"_88":-5},{"_74":83,"_43":126,"_69":-5,"_77":127,"_70":79,"_86":-5,"_87":-5,"_88":-5},{"_74":83,"_43":129,"_69":-5,"_77":130,"_70":79,"_86":-5,"_87":-5,"_88":-5},{"_74":83,"_43":132,"_69":-5,"_77":133,"_70":79,"_86":-5,"_87":-5,"_88":-5},{"_74":83,"_43":135,"_69":-5,"_77":136,"_70":79,"_86":-5,"_87":-5,"_88":-5},{"_74":83,"_43":138,"_69":-5,"_77":139,"_70":79,"_86":-5,"_87":-5,"_88":-5},{"_74":75,"_43":141,"_69":-5,"_77":142,"_70":79,"_86":-5,"_87":-5,"_88":-5},{"_74":75,"_43":145,"_69":-5,"_77":146,"_70":79,"_86":-5,"_87":-5,"_88":-5},{"_74":83,"_43":149,"_69":-5,"_77":150,"_70":79,"_86":-5,"_87":-5,"_88":-5},{"_74":83,"_43":152,"_69":-5,"_77":153,"_70":79,"_86":-5,"_87":-5,"_88":-5},{"_74":83,"_43":155,"_69":-5,"_77":156,"_70":79,"_86":-5,"_87":-5,"_88":-5},{"_74":83,"_43":158,"_69":-5,"_77":159,"_70":79,"_86":-5,"_87":-5,"_88":-5},{"_74":83,"_43":161,"_69":-5,"_77":162,"_70":79,"_86":-5,"_87":-5,"_88":-5},{"_74":83,"_43":164,"_69":-5,"_77":165,"_70":79,"_86":-5,"_87":-5,"_88":-5},{"_74":83,"_43":167,"_69":-5,"_77":168,"_70":79,"_86":-5,"_87":-5,"_88":-5},{"_74":75,"_43":170,"_69":-5,"_77":171,"_70":79,"_86":-5,"_87":-5,"_88":-5},{"_74":83,"_43":174,"_69":-5,"_77":175,"_70":79,"_86":-5,"_87":-5,"_88":-5},{"_74":83,"_43":177,"_69":-5,"_77":178,"_70":79,"_86":-5,"_87":-5,"_88":-5},{"_74":83,"_43":180,"_69":-5,"_77":181,"_70":79,"_86":-5,"_87":-5,"_88":-5},{"_74":83,"_43":183,"_69":-5,"_77":184,"_70":79,"_86":-5,"_87":-5,"_88":-5},{"_74":75,"_43":186,"_69":-5,"_77":187,"_70":79,"_86":-5,"_87":-5,"_88":-5},{"_74":75,"_43":190,"_69":-5,"_77":191,"_70":79,"_86":-5,"_87":-5,"_88":-5},{"_74":83,"_43":194,"_69":-5,"_77":195,"_70":79,"_86":-5,"_87":-5,"_88":-5},{"_74":83,"_43":197,"_69":-5,"_77":198,"_70":79,"_86":-5,"_87":-5,"_88":-5},{"_74":75,"_43":200,"_69":-5,"_77":201,"_70":79,"_86":-5,"_87":-5,"_88":-5},{"_74":83,"_43":204,"_69":-5,"_77":205,"_70":79,"_86":-5,"_87":-5,"_88":-5},{"_74":75,"_43":207,"_69":-5,"_77":208,"_70":79,"_86":-5,"_87":-5,"_88":-5},{"_74":83,"_43":211,"_69":-5,"_77":212,"_70":79,"_86":-5,"_87":-5,"_88":-5},{"_74":83,"_43":214,"_69":-5,"_77":215,"_70":79,"_86":-5,"_87":-5,"_88":-5},{"_74":83,"_43":217,"_69":-5,"_77":218,"_70":79,"_86":-5,"_87":-5,"_88":-5},{"_74":83,"_43":220,"_69":-5,"_77":221,"_70":79,"_86":-5,"_87":-5,"_88":-5},{"_74":83,"_43":223,"_69":-5,"_77":224,"_70":79,"_86":-5,"_87":-5,"_88":-5},{"_74":75,"_43":226,"_69":-5,"_77":227,"_70":79,"_86":-5,"_87":-5,"_88":-5},{"_74":75,"_43":230,"_69":-5,"_77":231,"_70":79,"_86":-5,"_87":-5,"_88":-5},{"_74":83,"_43":234,"_69":-5,"_77":235,"_70":79,"_86":-5,"_87":-5,"_88":-5},{"_74":83,"_43":237,"_69":-5,"_77":238,"_70":79,"_86":-5,"_87":-5,"_88":-5},{"_74":83,"_43":240,"_69":-5,"_77":241,"_70":79,"_86":-5,"_87":-5,"_88":-5},{"_74":83,"_43":243,"_69":-5,"_77":244,"_70":79,"_86":-5,"_87":-5,"_88":-5},"publicChapterContent","nextChapter","sections",[],"routes/_store+/_layout","routes/_store+/_readers+/_layout","actionData","errors"]

# Part IV - Fast and Efficient Rag Document System

Table of Contents

Preface

Choosing Which “Small AI” Tools to Use in This Book

Resources

Part I - Advantages of Small AI With Examples

Advantages of Small Models with Examples That Might Surprise You

The Great Divergence: Redefining Intelligence in the Post-Scale Era

The Energy and Latency Wall

The Cloud Efficiency Champion: Anatomy of Gemini 3 Flash

The Economics of Latency and Throughput: Gemini 3 Flash

Agentic Capabilities and Green AI

The Local Insurgency: Qwen 3 and other Local Models Provide Democratization of Reasoning

Benchmarking the Pocket AI

Silicon at the Edge: The Hardware Reality of Small AI

The Role of NPUs and Energy Efficiency

The Physics of Thought: Theoretical Mechanisms

Sovereignty and Solvency: The Economic and Legal Case

Surprising Realities: Case Studies

The Philosopher in the Pocket

The Real-Time Coding Assistant

The Sustainable Green Search Engine

Horizon 2026: The Future of Distributed Intelligence

Drawbacks of Small AI

Reduced Reasoning Depth and Nuance

Higher Propensity for Hallucination

Limited Context Window and Memory Management

Fragility in Prompt Following

Security and Support Disparities

Wrap Up for Drawbacks of Small AI

Part II - Defining Metrics for Success Using LLMs

Defining Metrics for Success Using LLMs

Operational Metrics: The Small AI Advantage

Quality Metrics: The “LLM-as-a-Judge” Pattern

RAG Metrics: The Trinity of Trust

Deterministic Guardrails

The Golden Ratio: Accuracy per Watt

Code Example: The Judge Pattern

Wrap Up for LLM Metrics

A Fair Shootout: Evaluating Large vs. Small Models for Task Fitness

The Setup for RAG: David vs. Goliath

Fairness: Adjusting Problem Setups for Small vs. Large Models

Prompt Divergence

Context Window and “Lost in the Middle”

Comparing Small vs. Large Models for NLP and Extracting Structured Data From Text

NLP Tasks: The Battle for Nuance

Extracting Structured Data: The Small AI Sweet Spot

Wrap Up for Evaluating Large vs. Small Models for Task Fitness

Part III - Python Examples for Switching Between Models

Using the LiteLLM Library to Easily Switch Between Models and Model Providers

Tool Use with LiteLLM

LiteLLM Wrap Up

Automatically Routing to the ‘Best’ Model Using RouteLLM Library

Implementation of a Command Line Query Utility Using RouteLLM

Wrap Up for Winning Big With Small AI

The Engineering Reality: From Vibes to Metrics

The Architecture of Hybrid Intelligence

The “Fitness for Purpose” Philosophy

Sovereignty, Solvency, and Sustainability

Final Recommendations for Your Journey

# Part IV - Fast and Efficient Rag Document System

RAG Using zvec Vector Datastore and Gemini-3-flash Model

Design Notes for Example Program

Example zvec RAG Application

Sample Example Output

Wrap Up for the zvex Based RAG Application