Conditions, Restarts, and Resilient LLM Clients

Most languages handle errors with exceptions. When code throws an exception, the stack unwinds to the nearest matching catch, and the code that failed is gone. Common Lisp has the condition system, which can do that and more. It splits error handling into three parts:

  1. Signaling: low-level code reports a problem by signaling a condition.
  2. Restarts: low-level code also lists ways to recover, called restarts.
  3. Handlers: high-level code decides which restart to use.

The key point: a handler runs before the stack unwinds. It can look at the problem and then choose a restart, and the program continues from the point the low-level code chose, with all its state intact. If no handler takes charge, the debugger shows the restarts and you choose one by hand.

The first half of this chapter covers the condition system with a small example that has nothing to do with AI. The second half applies it to a daily problem: calls to LLM services fail. Servers rate-limit you, local servers aren’t running, a model name has a typo, and small models return broken JSON. We build a small library, resilient-llm, that offers restarts for each failure, plus three handler policies: retry with backoff, fall back to another model, and ask the model to repair its JSON.

The LLM code builds on the litelm library from the previous chapter. The live examples need a local Ollama server with the model llama3.2:3b. The tests need no server.

Note: The source code is in the directory src/resilient-llm.

Project Structure

File Purpose
package.lisp Package definition and exports
basics.lisp Condition system fundamentals
resilient-llm.lisp Restarts for LLM calls, and retry, fallback, and JSON repair policies
tests.lisp Offline tests that use fake models
examples.lisp Live examples that use Ollama
resilient-llm.asd ASDF system definitions
 1 ;;;; resilient-llm.asd -- conditions and restarts for robust LLM calls.
 2 
 3 (asdf:defsystem #:resilient-llm
 4   :description "Condition system tutorial, plus retry, fallback, and JSON repair policies for LLM calls."
 5   :author "Mark Watson"
 6   :license "Apache 2"
 7   :depends-on (#:litelm #:usocket #:com.inuoe.jzon)
 8   :serial t
 9   :components ((:file "package")
10                (:file "basics")
11                (:file "resilient-llm")))
12 
13 (asdf:defsystem #:resilient-llm/tests
14   :description "Offline tests for resilient-llm. They use fake models, so no LLM server is needed."
15   :depends-on (#:resilient-llm)
16   :components ((:file "tests")))

We depend on usocket because the HTTP client dexador reports network failures with usocket conditions, and we need to name those types. The jzon library parses JSON replies from models.

Condition System Fundamentals

Defining a Condition

A condition is an object that describes a situation. You define a condition type with define-condition, which looks like defclass:

1 (define-condition bad-age (error)
2   ((text :initarg :text :reader bad-age-text))
3   (:report (lambda (condition stream)
4              (format stream "Not a valid age: ~S" (bad-age-text condition)))))

bad-age inherits from error. The :report function writes the message that the debugger shows and that princ prints.

The standard condition types form a hierarchy. The ones you will use most:

Type Use
condition Root of the hierarchy
warning Something looks wrong, but the program can go on
serious-condition Parent of error and of conditions such as storage exhaustion
error The program cannot go on without help

Three Ways to Signal

Function If no handler takes charge
signal Returns nil, and the program continues
warn Prints the warning to *error-output* and returns nil
error Enters the debugger

signal is useful for conditions that are not errors at all. Code can announce an event, and any caller who cares can listen. Later in this chapter we signal a retrying condition before each retry so callers can log it.

handler-case: Unwind, Then Handle

handler-case works like try/catch in other languages. When a matching condition is signaled, the stack unwinds to the handler-case form, and the clause’s value becomes the value of the form:

1 (handler-case (parse-integer "abc")
2   (parse-error () 0))
3 ;; => 0

The code that signaled is gone by the time the clause runs, so it can’t continue. ignore-errors is a shorthand for a handler-case that returns nil and the condition on any error.

handler-bind: Handle Without Unwinding

handler-bind runs its handler function at the point of the signal, on top of the stack. The handler can do one of two things:

  1. Transfer control somewhere else, for example by invoking a restart. This is how it takes charge.
  2. Return normally. This declines the condition, and the search continues with the next handler out.

Handlers run innermost first. A handler that logs and returns lets an outer handler make the real decision.

restart-case: Offer Ways to Recover

restart-case runs a form and names ways to recover if something goes wrong inside it. Here is parse-age, which offers two restarts:

 1 (defun parse-age (text)
 2   "Return TEXT as an integer age from 0 to 150. On bad input, signal BAD-AGE
 3 with two restarts: USE-VALUE supplies a replacement age, and SKIP-ENTRY
 4 returns NIL."
 5   (restart-case
 6       (let ((age (ignore-errors (parse-integer text))))
 7         (if (and age (<= 0 age 150))
 8             age
 9             (error 'bad-age :text text)))
10     (use-value (value)
11       :report "Supply an age to use instead."
12       :interactive (lambda ()
13                      (format *query-io* "Age to use: ")
14                      (finish-output *query-io*)
15                      (list (read *query-io*)))
16       value)
17     (skip-entry ()
18       :report "Skip this entry."
19       nil)))
20 
21 (defun parse-ages (texts)
22   "Parse each string in TEXTS. Entries skipped with SKIP-ENTRY are dropped."
23   (loop for text in texts
24         for age = (parse-age text)
25         when age collect age))

Each restart clause has a name, a lambda list, options, and a body. When a handler invokes a restart, the stack unwinds to the restart-case form, and the restart’s body supplies the value of the form. :report describes the restart in the debugger. :interactive is a function that prompts for the restart’s arguments when you pick it in the debugger.

use-value is one of the standard restart names in Common Lisp. Each has a function of the same name that finds and invokes it: use-value, store-value, continue, muffle-warning, and abort. skip-entry is our own name; you invoke it with (invoke-restart 'skip-entry).

Choosing a Policy

parse-age never decides what to do about bad input. Callers decide by binding a handler that picks a restart. Here are three policies for the same low-level code:

 1 (defun parse-ages-skipping (texts)
 2   (handler-bind ((bad-age (lambda (condition)
 3                             (declare (ignore condition))
 4                             (invoke-restart 'skip-entry))))
 5     (parse-ages texts)))
 6 
 7 (defun parse-ages-with-default (texts default)
 8   (handler-bind ((bad-age (lambda (condition)
 9                             (use-value default condition))))
10     (parse-ages texts)))
11 
12 (defun parse-ages-logging (texts)
13   "Log each bad entry, then skip it. The first handler returns normally,
14 which declines the condition, so the next handler out gets its turn."
15   (handler-bind ((bad-age (lambda (condition)
16                             (declare (ignore condition))
17                             (invoke-restart 'skip-entry))))
18     (handler-bind ((bad-age (lambda (condition)
19                               (format t "~&Log: ~A~%" condition))))
20       (parse-ages texts))))
1 (parse-ages-skipping '("42" "abc" "7" "200"))      ;; => (42 7)
2 (parse-ages-with-default '("42" "abc" "7" "200") 0) ;; => (42 0 7 0)
3 (parse-ages-logging '("1" "x"))
4 ;; Log: Not a valid age: "x"
5 ;; => (1)

Notice that parse-ages keeps looping after a bad entry. With handler-case, the first bad entry would unwind out of the loop and lose the ages already parsed. With handler-bind and restarts, the loop continues from the bad entry.

parse-ages-logging nests two handlers. The inner one prints and returns, which declines the condition, so the outer one runs and invokes skip-entry.

Restarts in the Debugger

When no handler takes charge, the debugger lists the restarts, and you pick one by number or by name. Here is a session in the SBCL REPL:

 1 * (parse-ages (list "42" "abc" "7"))
 2 
 3 debugger invoked on a BAD-AGE in thread
 4 #<THREAD tid=259 "main thread" RUNNING {80052904A3}>:
 5   Not a valid age: "abc"
 6 
 7 Type HELP for debugger help, or (SB-EXT:EXIT) to exit from SBCL.
 8 
 9 restarts (invokable by number or by possibly-abbreviated name):
10   0: [USE-VALUE ] Supply an age to use instead.
11   1: [SKIP-ENTRY] Skip this entry.
12   2: [ABORT     ] Exit debugger, returning to top level.
13 
14 (PARSE-AGE "abc")
15    source: (ERROR 'BAD-AGE :TEXT TEXT)
16 0] use-value
17 Age to use: 30
18 (42 30 7)

The program asked for a value, used it, and finished the loop. In other languages you would fix the data and run the program again from the start. In Lisp you fix it inside the running program. This matters most for long jobs, such as a batch of a thousand LLM calls that fails on item 950.

Resilient LLM Calls

Now we apply the same pattern to LLM calls. The low-level function complete offers restarts. Separate policy functions choose among them.

Which Failures Can We Expect?

I ran three calls through litelm to see which conditions they signal:

Situation Condition signaled
Unknown model name (HTTP 404) litelm:not-found-error
Too many requests (HTTP 429) litelm:rate-limit-error
Server not running usocket:invalid-argument-error, a subtype of usocket:socket-error

litelm maps HTTP status codes to its own conditions, all subtypes of litelm:api-error, which holds the status code and the response body. Network failures come from the socket library and never pass through litelm.

The refused connection surprised me. I expected usocket:connection-refused-error, but SBCL on macOS reported usocket:invalid-argument-error. I don’t know why; the socket library maps the operating system error that way. This is a good reason to test against usocket:socket-error, the common parent, rather than a narrow subtype.

The Low-Level Call

complete sends a prompt and returns the reply text. It offers three restarts:

 1 (defvar *default-model* "ollama/llama3.2:3b")
 2 
 3 (defun litelm-completion (model prompt)
 4   (litelm:response-content (litelm:completion model :messages prompt)))
 5 
 6 (defvar *completion-function* 'litelm-completion
 7   "Function of (MODEL PROMPT) that returns the reply text. Tests bind this
 8 to a fake model.")
 9 
10 (defvar *sleep-function* 'sleep
11   "Function of one argument, the number of seconds to wait. Tests bind
12 this to a function that only records the delay.")
13 
14 (defun complete (prompt &key (model *default-model*))
15   "Send PROMPT to MODEL and return the reply text. If the call fails, these
16 restarts are available: RETRY, USE-MODEL, and USE-VALUE."
17   (loop
18     (restart-case (return (funcall *completion-function* model prompt))
19       (retry ()
20         :report (lambda (stream) (format stream "Call ~A again." model)))
21       (use-model (new-model)
22         :report "Call a different model."
23         :interactive (lambda ()
24                        (format *query-io* "Model (for example \"ollama/qwen3.5:2b\"): ")
25                        (finish-output *query-io*)
26                        (list (read *query-io*)))
27         (setf model new-model))
28       (use-value (reply)
29         :report "Return a reply of your choice instead."
30         :interactive (lambda ()
31                        (format *query-io* "Reply string: ")
32                        (finish-output *query-io*)
33                        (list (read *query-io*)))
34         (return reply)))))

The loop and restart-case work together. On success, return exits the loop with the reply. When a handler invokes retry, the empty restart body returns, and the loop calls the model again. use-model changes the model variable and loops. use-value returns its argument from the loop.

The two special variables, *completion-function* and *sleep-function*, exist for testing. Tests bind them to fake functions so they run fast and need no server.

Before we write any policy, the restarts already help at the REPL:

 1 * (complete "Reply with only the word yes." :model "ollama/no-such-model")
 2 
 3 debugger invoked on a LITELM:NOT-FOUND-ERROR in thread
 4 #<THREAD tid=259 "main thread" RUNNING {80052904A3}>:
 5   LLM API error 404: {"error":{"message":"model 'no-such-model' not found","type":"not_found_error","param":null,"code":null}}
 6 
 7 Type HELP for debugger help, or (SB-EXT:EXIT) to exit from SBCL.
 8 
 9 restarts (invokable by number or by possibly-abbreviated name):
10   0: [RETRY    ] Call ollama/no-such-model again.
11   1: [USE-MODEL] Call a different model.
12   2: [USE-VALUE] Return a reply of your choice instead.
13   3: [ABORT    ] Exit debugger, returning to top level.
14 
15 (LITELM::%MAP-HTTP-ERROR 404 "{\"error\":{\"message\":\"model 'no-such-model' not found\",\"type\":\"not_found_error\",\"param\":null,\"code\":null}}
16 ")
17    source: (ERROR CONDITION :STATUS STATUS :BODY BODY)
18 0] use-model
19 Model (for example "ollama/qwen3.5:2b"): "ollama/llama3.2:3b"
20 "yes"

If you forget to start Ollama, you can start it in another terminal and pick retry. The call continues as if it had worked the first time.

Which Errors Are Worth Retrying?

Some failures go away if you wait, and some never will. Retrying a bad API key wastes time. transient-error-p lists the ones worth retrying:

1 (defun transient-error-p (condition)
2   "True for errors that may go away if we wait: rate limits, server errors
3 \(HTTP 5xx), and network failures such as a refused connection."
4   (or (typep condition 'litelm:rate-limit-error)
5       (and (typep condition 'litelm:api-error)
6            (integerp (litelm:api-error-status condition))
7            (>= (litelm:api-error-status condition) 500))
8       (typep condition 'usocket:socket-error)))

I did not test timeouts. litelm sets a 60 second connect timeout and a 120 second read timeout. If a timeout in your setup signals a condition type that this function does not list, add that type here.

Exponential Backoff with Jitter

When a server says “too many requests”, retrying at once makes things worse. The standard fix is exponential backoff: wait 1 second, then 2, then 4, up to a cap. Jitter adds randomness so that many clients that failed at the same moment don’t all retry at the same moment:

 1 ;; SBCL starts every process with the same *RANDOM-STATE*, so without
 2 ;; this seed every client would pick the same "random" delays.
 3 (defvar *backoff-random-state* (make-random-state t))
 4 
 5 (defun backoff-delay (attempt &key (base-delay 1.0) (max-delay 30.0))
 6   "Exponential backoff with jitter. The ceiling doubles with each ATTEMPT
 7 \(1, 2, 3, ...) up to MAX-DELAY, and the delay is a random value from half
 8 the ceiling to the full ceiling. The randomness keeps many clients that
 9 failed at the same moment from retrying at the same moment."
10   (let ((ceiling (min max-delay (* base-delay (expt 2 (1- attempt))))))
11     (+ (/ ceiling 2) (random (/ ceiling 2.0) *backoff-random-state*))))

The comment about *random-state* comes from a bug I found while testing. My first version called (random n) with the default random state, and two separate runs printed the same three delays: 0.46, 0.72, and 1.26 seconds. Every SBCL process starts with the same random state, so the jitter did nothing to spread out separate clients. (make-random-state t) seeds a new state from a source of randomness.

The Retry Policy

The policy is a handler-bind that invokes the retry restart for transient errors:

 1 (define-condition retrying (condition)
 2   ((attempt :initarg :attempt :reader retrying-attempt)
 3    (delay :initarg :delay :reader retrying-delay)
 4    (cause :initarg :cause :reader retrying-cause))
 5   (:report (lambda (condition stream)
 6              (format stream "Retry ~D in ~,2F seconds after: ~A"
 7                      (retrying-attempt condition)
 8                      (retrying-delay condition)
 9                      (retrying-cause condition)))))
10 
11 (defun call-with-retries (thunk &key (max-retries 3) (base-delay 1.0) (max-delay 30.0))
12   "Call THUNK. When a transient error reaches us and a RETRY restart is
13 available, wait and invoke RETRY, at most MAX-RETRIES times in total for
14 the whole call. Other errors pass through untouched."
15   (let ((retries 0))
16     (handler-bind
17         ((error (lambda (condition)
18                   (let ((restart (find-restart 'retry condition)))
19                     (when (and restart
20                                (transient-error-p condition)
21                                (< retries max-retries))
22                       (incf retries)
23                       (let ((delay (backoff-delay retries :base-delay base-delay
24                                                           :max-delay max-delay)))
25                         ;; SIGNAL returns NIL when no one handles RETRYING,
26                         ;; so this costs nothing unless a caller listens.
27                         (signal 'retrying :attempt retries :delay delay
28                                           :cause condition)
29                         (funcall *sleep-function* delay)
30                         (invoke-restart restart)))))))
31       (funcall thunk))))
32 
33 (defmacro with-retries ((&rest options &key max-retries base-delay max-delay)
34                         &body body)
35   "Run BODY, retrying transient LLM errors with exponential backoff."
36   (declare (ignore max-retries base-delay max-delay))
37   `(call-with-retries (lambda () ,@body) ,@options))

Three details matter:

  1. The handler calls find-restart before it acts. If the error came from code that offers no retry restart, the handler declines.
  2. When the retry budget runs out, the handler returns normally. That declines the condition, and an outer handler or the debugger gets it.
  3. Before each retry, the handler signals retrying. signal returns nil when no one listens, so callers who don’t care pay nothing. Callers who want a log bind a handler for retrying.

The pair of a call-with- function and a with- macro is a common Lisp idiom. The function holds the logic, so you can redefine it at the REPL without recompiling every caller. The macro only wraps the body in a lambda.

The retry budget covers everything inside with-retries. If the body makes several calls, they share the budget.

The Fallback Policy

When a model fails for good, try another one. This policy invokes use-model with the next model from a list:

 1 (defun call-with-fallback-models (models thunk)
 2   "Call THUNK. When an LLM error reaches us, switch to the next model in
 3 MODELS by invoking the USE-MODEL restart."
 4   (let ((remaining models))
 5     (handler-bind
 6         ((error (lambda (condition)
 7                   (let ((restart (find-restart 'use-model condition)))
 8                     (when (and restart
 9                                remaining
10                                (or (typep condition 'litelm:litelm-error)
11                                    (transient-error-p condition)))
12                       (invoke-restart restart (pop remaining)))))))
13       (funcall thunk))))
14 
15 (defmacro with-fallback-models ((&rest models) &body body)
16   "Run BODY. If a model call fails, try the next of MODELS."
17   `(call-with-fallback-models (list ,@models) (lambda () ,@body)))

Policies combine by nesting, and the order matters. Put with-retries inside with-fallback-models:

1 (with-fallback-models ("ollama/llama3.2:3b")
2   (with-retries ()
3     (complete "..." :model "ollama/no-such-model")))

The inner handler sees each error first. It retries transient errors until its budget runs out, then declines. The outer handler then switches models. A not-found-error is not transient, so the retry handler declines at once and the fallback handler switches models without waiting.

Since the retry budget is shared, a transient error on the fallback model gets no retries if the first model used up the budget. For a separate budget per model, write a policy that resets its counter when it invokes use-model.

JSON Replies and Repair

We often ask a model for JSON. Small models wrap it in Markdown code fences, add a sentence before it, or write invalid JSON such as {a: 1} with an unquoted key. We handle the common case, code fences, in plain code, and we use a restart for the rest:

 1 (define-condition malformed-json (error)
 2   ((text :initarg :text :reader malformed-json-text)
 3    (cause :initarg :cause :reader malformed-json-cause))
 4   (:report (lambda (condition stream)
 5              (format stream "Model reply is not valid JSON (~A): ~S"
 6                      (malformed-json-cause condition)
 7                      (malformed-json-text condition)))))
 8 
 9 (defun strip-code-fences (text)
10   "Remove a Markdown code fence such as ```json ... ``` around TEXT. Small
11 models add these even when told not to."
12   (let* ((trimmed (string-trim '(#\Space #\Tab #\Newline #\Return) text))
13          (start (and (>= (length trimmed) 3)
14                      (string= "```" trimmed :end2 3)
15                      (position #\Newline trimmed)))
16          (end (and start (search "```" trimmed :from-end t))))
17     (if (and start end (> end start))
18         (string-trim '(#\Space #\Tab #\Newline #\Return)
19                      (subseq trimmed (1+ start) end))
20         trimmed)))
21 
22 (defun parse-json-reply (text)
23   "Parse TEXT as JSON, or signal MALFORMED-JSON."
24   (handler-case (jzon:parse (strip-code-fences text))
25     (error (condition)
26       (error 'malformed-json :text text :cause condition))))

parse-json-reply uses handler-case to turn any parse error from jzon into our own malformed-json condition, which carries the bad text. Callers can then handle one condition type no matter which JSON library we use.

complete-json asks for JSON and offers a repair-json restart that sends the bad text and the parse error back to the model:

 1 (defun repair-prompt (text condition)
 2   (format nil "This text should be valid JSON, but parsing failed with: ~A~%~%~A~%~%~
 3 Reply with only the corrected JSON, no explanation and no Markdown."
 4           (malformed-json-cause condition) text))
 5 
 6 (defun complete-json (prompt &key (model *default-model*))
 7   "Ask MODEL for JSON and return it parsed with jzon: objects become hash
 8 tables and arrays become vectors. If the reply is not valid JSON, these
 9 restarts are available: REPAIR-JSON and USE-VALUE."
10   (let ((text (complete prompt :model model)))
11     (loop
12       (restart-case (return (parse-json-reply text))
13         (repair-json (condition)
14           :report "Ask the model to fix its JSON."
15           (setf text (complete (repair-prompt text condition) :model model)))
16         (use-value (value)
17           :report "Return a value of your choice instead."
18           :interactive (lambda ()
19                          (format *query-io* "Value: ")
20                          (finish-output *query-io*)
21                          (list (read *query-io*)))
22           (return value))))))
23 
24 (defmacro with-json-repair ((&key (max-repairs 2)) &body body)
25   "Run BODY. When a model reply is not valid JSON, ask the model to repair
26 it, at most MAX-REPAIRS times in total."
27   (let ((repairs (gensym "REPAIRS")))
28     `(let ((,repairs 0))
29        (handler-bind
30            ((malformed-json
31               (lambda (condition)
32                 (let ((restart (find-restart 'repair-json condition)))
33                   (when (and restart (< ,repairs ,max-repairs))
34                     (incf ,repairs)
35                     (invoke-restart restart condition))))))
36          ,@body))))

The repair-json restart takes the condition as its argument, so the repair prompt can quote the parse error. The model then knows what to fix.

The repair call goes through complete, so it has its own retry and use-model restarts. If the server rate-limits the repair call, an outer with-retries handles it like any other call.

Since with-json-repair uses a gensym for its counter, the macro’s variable can’t clash with a variable in the body.

Testing with Fake Models

Since complete calls the model through *completion-function*, tests can bind a fake model. This helper returns a fake that fails a set number of times with a rate-limit error, then succeeds:

 1 (defun rate-limit ()
 2   (error 'litelm:rate-limit-error :status 429 :body "slow down"))
 3 
 4 (defun make-flaky-model (failures reply)
 5   "A fake model that signals a rate limit FAILURES times, then returns REPLY.
 6 The second value of the result counts calls."
 7   (let ((calls 0))
 8     (values (lambda (model prompt)
 9               (declare (ignore model prompt))
10               (incf calls)
11               (if (<= calls failures) (rate-limit) reply))
12             (lambda () calls))))
13 
14 (defmacro with-fake-model ((function) &body body)
15   `(let ((*completion-function* ,function)
16          (*sleep-function* (lambda (seconds) (declare (ignore seconds)))))
17      ,@body))

with-fake-model also binds *sleep-function* to a function that does nothing, so the retry tests take no time. Here are two of the tests from tests.lisp:

 1   (multiple-value-bind (model calls) (make-flaky-model 2 "ok")
 2     (let ((delays '()))
 3       (check "retry twice, then succeed"
 4              (with-fake-model (model)
 5                (handler-bind ((retrying (lambda (condition)
 6                                           (push (retrying-attempt condition) delays))))
 7                  (with-retries () (complete "hi"))))
 8              "ok")
 9       (check "three calls in total" (funcall calls) 3)
10       (check "RETRYING signaled for attempts 1 and 2" (reverse delays) '(1 2))))
 1   (let ((prompts '()))
 2     (flet ((model (model prompt)
 3              (declare (ignore model))
 4              (push prompt prompts)
 5              (if (= (length prompts) 1) "{a: 1}" "{\"a\": 1}")))
 6       (check "no repair policy: MALFORMED-JSON escapes"
 7              (with-fake-model (#'model) (complete-json "give me json"))
 8              '(:error malformed-json))
 9       (setf prompts '())
10       (check "repair malformed JSON"
11              (with-fake-model (#'model)
12                (with-json-repair ()
13                  (gethash "a" (complete-json "give me json"))))
14              1)
15       (check "the repair prompt quotes the bad reply"
16              (and (search "{a: 1}" (first prompts)) t)
17              t)))

Run all 23 tests with:

1 (ql:quickload :resilient-llm/tests)
2 (resilient-llm::run-tests)
 1 pass  skip bad ages
 2 pass  replace bad ages with a default
 3 pass  log, then skip
 4 pass  no handler: the error escapes
 5 pass  retry twice, then succeed
 6 pass  three calls in total
 7 pass  RETRYING signaled for attempts 1 and 2
 8 pass  give up after max-retries
 9 pass  one call plus three retries
10 pass  no retry for errors that are not transient
11 pass  called once
12 pass  a refused connection is transient
13 pass  HTTP 503 is transient
14 pass  HTTP 401 is not transient
15 pass  backoff stays between half the ceiling and the ceiling
16 pass  fall back to the next model
17 pass  models tried in order
18 pass  a handler can supply a canned reply with USE-VALUE
19 pass  strip a ```json fence
20 pass  parse a fenced reply
21 pass  no repair policy: MALFORMED-JSON escapes
22 pass  repair malformed JSON
23 pass  the repair prompt quotes the bad reply
24 All tests passed.

Live Examples with Ollama

The file examples.lisp runs three examples against a local Ollama server. To simulate a server that is down, it registers a litelm provider on a port where nothing listens:

 1 (litelm:define-provider :ollama-down "http://localhost:11999/v1" :requires-key nil)
 2 
 3 (defun log-retries (condition)
 4   (format t "~&~A~%" condition))
 5 
 6 (defun example-fallback ()
 7   (format t "~&--- Missing model, fall back to llama3.2:3b~%")
 8   (with-fallback-models ("ollama/llama3.2:3b")
 9     (with-retries ()
10       (complete "In one sentence, what is Common Lisp?"
11                 :model "ollama/no-such-model"))))
12 
13 (defun example-retries-then-fallback ()
14   (format t "~&--- Server down: retry with backoff, then fall back~%")
15   (handler-bind ((retrying #'log-retries))
16     (with-fallback-models ("ollama/llama3.2:3b")
17       (with-retries (:max-retries 3 :base-delay 0.5)
18         (complete "Name one Common Lisp implementation. Reply with only its name."
19                   :model "ollama-down/llama3.2:3b")))))
20 
21 (defun example-json ()
22   (format t "~&--- JSON reply~%")
23   (let ((result
24           (with-json-repair ()
25             ;; This handler runs first and returns normally, so it only
26             ;; logs; WITH-JSON-REPAIR then picks the REPAIR-JSON restart.
27             (handler-bind ((malformed-json
28                              (lambda (condition)
29                                (format t "~&Repairing: ~A~%" condition))))
30               (complete-json
31                "Return a JSON object with keys \"name\" and \"year\" for the
32 programming language Common Lisp, where year is the year ANSI approved
33 the standard. Reply with only the JSON.")))))
34     (format t "name: ~A, year: ~A~%"
35             (gethash "name" result) (gethash "year" result))
36     result))
1 (ql:quickload :resilient-llm)
2 (load "examples.lisp")
1 --- Missing model, fall back to llama3.2:3b
2 Common Lisp is a general-purpose, high-level programming language based on the ANSI standard for Lisp, known for its macro system, strong type inference, and extensive standard library, making it a powerful and flexible language for both rapid prototyping and long-term development.
3 --- Server down: retry with backoff, then fall back
4 Retry 1 in 0.31 seconds after: Condition USOCKET:INVALID-ARGUMENT-ERROR was signalled.
5 Retry 2 in 0.94 seconds after: Condition USOCKET:INVALID-ARGUMENT-ERROR was signalled.
6 Retry 3 in 1.36 seconds after: Condition USOCKET:INVALID-ARGUMENT-ERROR was signalled.
7 SBCL
8 --- JSON reply
9 name: Common Lisp, year: 1994

In the second example the refused connection is transient, so the retry policy waited three times, with delays that grow, then declined. The fallback policy took over and switched to the working model. Your delays will differ because of the jitter.

In the third example llama3.2:3b returned valid JSON on the first try, so no “Repairing” line appeared. The repair path runs in the offline tests, where a fake model returns {a: 1} first.

Wrap Up

The condition system separates three jobs that most languages tie together: reporting a problem, offering ways to recover, and choosing one. In this chapter:

  • complete and complete-json report problems and offer the retry, use-model, use-value, and repair-json restarts, but decide nothing.
  • with-retries, with-fallback-models, and with-json-repair each pick restarts for one kind of problem, and they combine by nesting.
  • With no policy in place, the debugger offers the same restarts, so you can fix a failed call by hand and keep the program’s state.

The same pattern applies to other code in this book. A RAG pipeline that embeds thousands of documents could offer a skip-document restart, and a web scraper could offer retry and skip-url restarts.