Recursive Language Models: Searching Wikidata from a Persistent Python REPL

Most LLM agents use JSON tool calls. The model asks for a tool, the harness runs it, and the harness pastes the raw result back into the chat. That works for small results. It fails when a single Wikidata entity returns hundreds of claims, and a question needs three or four hops through the graph. Each hop adds a large blob of JSON to the context, and the model must join the results in its head.

This chapter builds a different kind of harness, a Recursive Language Model (RLM) harness. The model does not call tools with JSON. It writes Python. The harness runs that Python in a REPL whose variables persist from turn to turn. Graph data lands in Python variables, not in the chat, and the model sees only what it prints. When the model needs to read text and judge it, it calls a small, stateless sub-LLM from inside its own code. That call from code back into an LLM is the “recursive” part of the name.

The example follows the paper Wikidata Search Traces: A Dataset for Training Knowledge Graph Search Agents by Chenene, Rosas-Hinostroza, Stasenko, Evenstein Sigalov, and Langlais (arxiv.org/html/2610.06650v3). The authors released an RLM harness that keeps graph results in persistent Python state and uses sub-calls to interpret selected evidence. They report that on 100 questions the harness beats direct tool calling for both models they test. Their conclusion is that much long-horizon search failure comes from how an agent manages the evidence it retrieves, not only from the model itself.

Our version is small enough to read in one sitting: one module for the harness and one for a benchmark. It runs against any OpenAI-compatible endpoint, so you can test it with local models served by Ollama or oMLX.

The Program at a Glance

The code lives in source-code/RecursiveLanguageModels_persistent_python_repl:

File Role
rlm_wikidata_bidirectional.py SQLite cache, Wikidata graph primitives, the REPL harness and its loop
benchmark_suite.py Four multi-hop tasks, a cache seeder, an answer checker, and a summary table
ARCHITECTURE.txt ASCII diagrams that compare JSON tool calling with the RLM harness
DESIGN_NOTES.md Notes on the benchmark and seeder design
SAMPLE_OUTPUT.txt A full benchmark run with a small local model
Makefile create_db, run_omlx, and clean targets

The project needs Python 3.14 or newer and two packages, openai and requests. It uses uv:

1 [project]
2 name = "persistent-python-repl"
3 version = "0.1.0"
4 requires-python = ">=3.14"
5 dependencies = [
6     "openai>=3.31.0",
7     "requests>=2.34.2",
8 ]

Two Agent Designs Compared

The file ARCHITECTURE.txt draws the two designs side by side. Here is the standard JSON tool-calling agent:

 1   +-----------------------+       JSON Tool Call        +-------------------+
 2   |      Primary LLM      | --------------------------> |  Wikidata API /   |
 3   |                       |                             |  SPARQL Endpoint  |
 4   |  [Message History]    | <-------------------------- |                   |
 5   |  - Turn 1 Raw JSON    |     Massive Raw Payload     +-------------------+
 6   |  - Turn 2 Raw JSON    |     (Appended directly
 7   |  - Turn 3 Raw JSON    |      to LLM context!)
 8   +-----------------------+
 9    * Bottleneck: Multi-hop joins happen in token space; hub entities quickly
10      exhaust context limits and degrade attention.

And here is the RLM harness:

 1   +-----------------------+     1. Python Code Block    +-------------------+
 2   |      Primary LLM      | --------------------------> |  Persistent REPL  |
 3   |   (Reasoning Loop)    |                             |  (`repl_globals`) |
 4   |                       | <-------------------------- |                   |
 5   |  [Clean History]      |   4. Concise `stdout` only  |  Variables:       |
 6   |  - Compact Code       |      (Truncated <= 10k)     |  `films = [...]`  |
 7   |  - Printed Summaries  |                             |  `matches = ...`  |
 8   +-----------------------+                             +---------+---------+
 9                                                                   |
10                          2. Graph Primitives & Sub-Calls          | 3. Grounding
11               +---------------------------------------------------+    Check
12               |                         |                         |
13               v                         v                         v
14     +-------------------+     +-------------------+     +-------------------+
15     |   SQLite Cache    |     | Stateless Sub-LLM |     | `visited_entities`|
16     | (`wikidata_api`,  |     |  (`llm_query` /   |     |   Set Verifier    |
17     |  `sparql`, `llm`) |     |    `llm_map`)     |     |   on `FINAL()`    |
18     +---------+---------+     +-------------------+     +-------------------+
19               |
20               | (Cache Miss)
21               v
22     +-------------------+
23     | Live Wikidata API |
24     | & SPARQL Endpoint |
25     +-------------------+

The key change: a join between two result sets is now one line of Python. Suppose the model must find people who won both the Turing Award and the Nobel Memorial Prize in Economic Sciences. With JSON tools, it must read two long lists in its context and spot the shared IDs. In the REPL it writes:

1 turing = set(edges("Q185667", "P166")["P166"])
2 nobel = set(edges("Q47170", "P166")["P166"])
3 both = turing & nobel
4 print(both)

Python computes Code Test exactly, and the context grows by one short line of output, not two lists of a hundred IDs.

The SQLite Cache

Every Wikidata API response, every SPARQL result, and every sub-LLM answer goes through a SQLite cache. The cache makes runs repeatable, lets you rerun a benchmark without the network, and spares the public Wikidata servers.

The class also defines FinalAnswerException, the signal that ends a run. We cover it with FINAL below.

 1 """
 2 rlm_wikidata_bidirectional.py
 3 Recursive Language Model (RLM) Knowledge Graph Search Harness
 4 with bidirectional traversal (Action API + SPARQL reverse edges)
 5 and deterministic SQLite caching.
 6 """
 7 
 8 import ast
 9 import hashlib
10 import io
11 import json
12 import re
13 import sqlite3
14 import sys
15 import threading
16 import time
17 import traceback
18 from concurrent.futures import ThreadPoolExecutor
19 from pathlib import Path
20 from typing import Any, Dict, List, Optional, Set
21 import requests
22 from openai import OpenAI
23 
24 
25 class FinalAnswerException(Exception):
26     """Signal raised inside the REPL when the agent calls FINAL(answer)."""
27     def __init__(self, answer: Any):
28         self.answer = answer
29 
30 
31 class SQLiteCache:
32     """
33     Thread-safe persistent SQLite cache for Wikidata Action API,
34     WDQS SPARQL queries, and stateless LLM sub-calls.
35     """
36     def __init__(self, db_path: str = "wikidata_rlm_cache.sqlite"):
37         self.db_path = Path(db_path)
38         self._lock = threading.Lock()
39         self.conn = sqlite3.connect(str(self.db_path), check_same_thread=False)
40         self._init_db()
41 
42     def _init_db(self):
43         with self._lock:
44             self.conn.execute("PRAGMA journal_mode=WAL;")
45             self.conn.execute("""
46                 CREATE TABLE IF NOT EXISTS cache (
47                     cache_key TEXT PRIMARY KEY,
48                     namespace TEXT NOT NULL,
49                     payload TEXT NOT NULL,
50                     created_at REAL NOT NULL
51                 )
52             """)
53             self.conn.commit()
54 
55     @staticmethod
56     def make_key(namespace: str, data: Any) -> str:
57         canonical = json.dumps(data, sort_keys=True, ensure_ascii=True)
58         digest = hashlib.sha256(canonical.encode("utf-8")).hexdigest()
59         return f"{namespace}:{digest}"
60 
61     def get(self, namespace: str, key_data: Any) -> Optional[Any]:
62         cache_key = self.make_key(namespace, key_data)
63         with self._lock:
64             cur = self.conn.execute(
65                 "SELECT payload FROM cache WHERE cache_key = ?", (cache_key,)
66             )
67             row = cur.fetchone()
68         if row is not None:
69             return json.loads(row[0])
70         return None
71 
72     def set(self, namespace: str, key_data: Any, value: Any):
73         cache_key = self.make_key(namespace, key_data)
74         payload = json.dumps(value, ensure_ascii=False)
75         with self._lock:
76             self.conn.execute(
77                 """
78                 INSERT OR REPLACE INTO cache (cache_key, namespace, payload, created_at)
79                 VALUES (?, ?, ?, ?)
80                 """,
81                 (cache_key, namespace, payload, time.time())
82             )
83             self.conn.commit()
84 
85     def stats(self) -> Dict[str, int]:
86         with self._lock:
87             cur = self.conn.execute(
88                 "SELECT namespace, COUNT(*) FROM cache GROUP BY namespace"
89             )
90             return dict(cur.fetchall())
91 
92     def close(self):
93         with self._lock:
94             self.conn.close()

The key for each entry is a namespace plus a SHA-256 hash of the request, serialized as JSON with sort_keys=True. Two requests with the same parameters in a different order hash to the same key. The namespace (wikidata_api, wikidata_sparql, or llm_subcall) keeps the three kinds of data apart and lets stats() report a count for each.

The check_same_thread=False flag plus an explicit lock lets llm_map call the cache from worker threads. WAL journal mode lets readers and a writer work at the same time.

The Wikidata Environment

WikidataEnvironment holds the functions the model may call. It talks to two Wikidata services:

  • The Action API (www.wikidata.org/w/api.php) for search, labels, and the outgoing claims of an entity.
  • The SPARQL endpoint (query.wikidata.org/sparql) for incoming edges: which entities point at this one?

The Action API can only answer “what does Q25188 point to?” It cannot answer “which films point to Q25188 as their director?” without a SPARQL query. Supporting both directions is why the module name ends in bidirectional.

Rate Limiting and Retries

Wikidata asks clients to be polite, and the public servers return HTTP 429 when you are not. All network calls go through one class method that enforces a minimum gap between requests across all threads and backs off on 429 and 5xx errors:

 1 class WikidataEnvironment:
 2     """
 3     Provides bidirectional graph navigation primitives (outgoing `claims` via Action API
 4     and incoming `edges` via SPARQL), caches responses in SQLite, and enforces grounding.
 5     """
 6     API_URL = "https://www.wikidata.org/w/api.php"
 7     SPARQL_URL = "https://query.wikidata.org/sparql"
 8     HEADERS = {
 9         "User-Agent": "RLMBookExample/1.2 (Educational Python Agent)",
10         "Accept": "application/sparql-results+json, application/json"
11     }
12 
13     # Rate limiting: minimum seconds between network requests (shared across
14     # all instances and threads), plus retry/backoff on 429 / 5xx responses.
15     MIN_REQUEST_INTERVAL = 2.0
16     MAX_RETRIES = 6
17     _throttle_lock = threading.Lock()
18     _last_request_time = 0.0
19 
20     @classmethod
21     def _throttled_request(cls, url: str, params: Dict[str, Any], timeout: int) -> requests.Response:
22         backoff = 5.0
23         for attempt in range(cls.MAX_RETRIES + 1):
24             with cls._throttle_lock:
25                 wait = cls.MIN_REQUEST_INTERVAL - (time.monotonic() - cls._last_request_time)
26                 if wait > 0:
27                     time.sleep(wait)
28                 try:
29                     resp = requests.get(url, params=params, headers=cls.HEADERS, timeout=timeout)
30                 finally:
31                     cls._last_request_time = time.monotonic()
32 
33             if resp.status_code in (429, 500, 502, 503, 504) and attempt < cls.MAX_RETRIES:
34                 retry_after = resp.headers.get("Retry-After")
35                 try:
36                     delay = float(retry_after) if retry_after else backoff
37                 except ValueError:
38                     delay = backoff
39                 print(f"    [rate-limit] HTTP {resp.status_code}; sleeping {delay:.0f}s "
40                       f"(retry {attempt + 1}/{cls.MAX_RETRIES})", file=sys.stderr)
41                 time.sleep(delay)
42                 backoff = min(backoff * 2, 120.0)
43                 continue
44 
45             resp.raise_for_status()
46             return resp
47         raise RuntimeError("unreachable")

The lock and _last_request_time live on the class, not the instance, so every environment in the process shares one request budget. The method honors a Retry-After header when the server sends one and otherwise doubles the wait, capped at two minutes. Wikidata requires a descriptive User-Agent; change it to name your own project and contact address.

Cached Fetches

The constructor sets up counters and the visited_entities set. _get and _sparql check the cache first, refuse to touch the network in offline mode, and store each fresh response:

 1     def __init__(self, client: OpenAI, model: str, cache: SQLiteCache, offline_only: bool = False):
 2         self.client = client
 3         self.model = model
 4         self.cache = cache
 5         self.offline_only = offline_only
 6 
 7         self.visited_entities: Set[str] = set()
 8         self.call_log: List[str] = []
 9         self.api_cache_hits = 0
10         self.api_network_calls = 0
11         self.subcall_count = 0
12         self.subcall_cache_hits = 0
13 
14     def _get(self, params: Dict[str, Any]) -> Dict[str, Any]:
15         params["format"] = "json"
16         cached = self.cache.get("wikidata_api", params)
17         if cached is not None:
18             self.api_cache_hits += 1
19             return cached
20 
21         if self.offline_only:
22             raise RuntimeError(f"Offline mode enabled and cache miss for params: {params}")
23 
24         self.api_network_calls += 1
25         resp = self._throttled_request(self.API_URL, params, timeout=15)
26         data = resp.json()
27         self.cache.set("wikidata_api", params, data)
28         return data
29 
30     def _sparql(self, query: str) -> Dict[str, Any]:
31         cached = self.cache.get("wikidata_sparql", {"query": query})
32         if cached is not None:
33             self.api_cache_hits += 1
34             return cached
35 
36         if self.offline_only:
37             raise RuntimeError(f"Offline mode enabled and SPARQL cache miss for query: {query}")
38 
39         self.api_network_calls += 1
40         resp = self._throttled_request(
41             self.SPARQL_URL,
42             {"query": query, "format": "json"},
43             timeout=25
44         )
45         data = resp.json()
46         self.cache.set("wikidata_sparql", {"query": query}, data)
47         return data

Search and Metadata Primitives

Four small functions find entities and properties by name and fetch labels:

 1     def search_entity(self, query: str, limit: int = 5) -> List[Dict[str, str]]:
 2         """Search Wikidata entities by label/alias."""
 3         self.call_log.append(f"search_entity({query!r})")
 4         data = self._get({
 5             "action": "wbsearchentities",
 6             "search": query,
 7             "language": "en",
 8             "limit": limit,
 9             "type": "item"
10         })
11         results = []
12         for item in data.get("search", []):
13             qid = item["id"]
14             self.visited_entities.add(qid)
15             results.append({
16                 "id": qid,
17                 "label": item.get("label", ""),
18                 "description": item.get("description", "")
19             })
20         return results
21 
22     def search_property(self, query: str, limit: int = 5) -> List[Dict[str, str]]:
23         """Search Wikidata properties (P-IDs) by label."""
24         self.call_log.append(f"search_property({query!r})")
25         data = self._get({
26             "action": "wbsearchentities",
27             "search": query,
28             "language": "en",
29             "limit": limit,
30             "type": "property"
31         })
32         return [
33             {
34                 "id": item["id"],
35                 "label": item.get("label", ""),
36                 "description": item.get("description", "")
37             }
38             for item in data.get("search", [])
39         ]
40 
41     def describe(self, entity_id: str) -> Dict[str, str]:
42         """Return the English label and description for a Q-ID or P-ID."""
43         self.call_log.append(f"describe({entity_id!r})")
44         self.visited_entities.add(entity_id)
45         data = self._get({
46             "action": "wbgetentities",
47             "ids": entity_id,
48             "props": "labels|descriptions",
49             "languages": "en"
50         })
51         ent = data.get("entities", {}).get(entity_id, {})
52         return {
53             "id": entity_id,
54             "label": ent.get("labels", {}).get("en", {}).get("value", entity_id),
55             "description": ent.get("descriptions", {}).get("en", {}).get("value", "")
56         }
57 
58     def label(self, entity_id: str) -> str:
59         """Return just the English label for an entity or property ID."""
60         return self.describe(entity_id)["label"]

Note that every function which sees a Q-ID adds it to visited_entities. That set drives the grounding check at the end of a run.

Forward and Reverse Traversal

claims follows outgoing edges. Raw Wikidata statements nest the value deep inside a mainsnak structure, with a different shape for each data type. claims flattens all of that to a plain dict that maps a property ID to a list of values:

 1     def claims(self, entity_id: str, property_id: str = None) -> Dict[str, List[Any]]:
 2         """
 3         Fetch OUTGOING statements (claims) from `entity_id`.
 4         Optionally filter by `property_id` (e.g., 'P31').
 5         """
 6         self.call_log.append(f"claims({entity_id!r}, {property_id!r})")
 7         self.visited_entities.add(entity_id)
 8         data = self._get({
 9             "action": "wbgetentities",
10             "ids": entity_id,
11             "props": "claims"
12         })
13         raw_claims = data.get("entities", {}).get(entity_id, {}).get("claims", {})
14         if property_id:
15             raw_claims = {property_id: raw_claims.get(property_id, [])}
16 
17         parsed: Dict[str, List[Any]] = {}
18         for pid, stmt_list in raw_claims.items():
19             values = []
20             for stmt in stmt_list:
21                 mainsnak = stmt.get("mainsnak", {})
22                 datavalue = mainsnak.get("datavalue", {})
23                 dtype = datavalue.get("type")
24                 val = datavalue.get("value")
25 
26                 if dtype == "wikibase-entityid" and isinstance(val, dict):
27                     target_id = val.get("id")
28                     if target_id:
29                         self.visited_entities.add(target_id)
30                         values.append(target_id)
31                 elif dtype == "time" and isinstance(val, dict):
32                     values.append(val.get("time"))
33                 elif dtype == "quantity" and isinstance(val, dict):
34                     values.append(val.get("amount"))
35                 elif dtype == "monolingualtext" and isinstance(val, dict):
36                     values.append(val.get("text"))
37                 elif val is not None:
38                     values.append(val)
39             if values:
40                 parsed[pid] = values
41         return parsed

Entity values become bare Q-IDs, times become strings such as +2016-11-11T00:00:00Z, and quantities become their amount. Since claims(qid) and claims(qid, pid) send the same Action API request, the second call is always a cache hit.

edges follows incoming edges with SPARQL. With a property it asks “which subjects have property_id pointing to entity_id?” Without one it returns every incoming edge, grouped by property:

 1     def edges(
 2         self,
 3         entity_id: str,
 4         property_id: str = None,
 5         limit: int = 100
 6     ) -> Dict[str, List[str]]:
 7         """
 8         Fetch INCOMING links (reverse edges) where other entities point to `entity_id`.
 9         Returns a dict mapping {property_id: [subject_QIDs]}.
10         If `property_id` is provided (e.g., 'P57'), filters only incoming links for that property.
11         """
12         if not re.match(r"^Q\d+$", entity_id):
13             raise ValueError(f"Invalid entity_id {entity_id!r}; expected Q-ID.")
14         if property_id is not None and not re.match(r"^P\d+$", property_id):
15             raise ValueError(f"Invalid property_id {property_id!r}; expected P-ID.")
16 
17         self.call_log.append(f"edges({entity_id!r}, {property_id!r}, limit={limit})")
18         self.visited_entities.add(entity_id)
19 
20         if property_id:
21             sparql_query = f"""
22             SELECT ?subject WHERE {{
23               ?subject wdt:{property_id} wd:{entity_id} .
24               FILTER(STRSTARTS(STR(?subject), "http://www.wikidata.org/entity/Q"))
25             }}
26             LIMIT {int(limit)}
27             """
28             data = self._sparql(sparql_query)
29             subjects = []
30             for row in data.get("results", {}).get("bindings", []):
31                 uri = row["subject"]["value"]
32                 qid = uri.rsplit("/", 1)[-1]
33                 self.visited_entities.add(qid)
34                 subjects.append(qid)
35             return {property_id: subjects} if subjects else {}
36         else:
37             sparql_query = f"""
38             SELECT ?prop ?subject WHERE {{
39               ?subject ?p wd:{entity_id} .
40               ?prop wikibase:directClaim ?p .
41               FILTER(STRSTARTS(STR(?subject), "http://www.wikidata.org/entity/Q"))
42             }}
43             LIMIT {int(limit)}
44             """
45             data = self._sparql(sparql_query)
46             grouped: Dict[str, List[str]] = {}
47             for row in data.get("results", {}).get("bindings", []):
48                 pid = row["prop"]["value"].rsplit("/", 1)[-1]
49                 qid = row["subject"]["value"].rsplit("/", 1)[-1]
50                 self.visited_entities.add(qid)
51                 grouped.setdefault(pid, []).append(qid)
52             return grouped
53 
54     def has(self, entity_id: str, property_id: str, target_id: str) -> bool:
55         """Check whether `entity_id` has an outgoing claim `property_id` pointing to `target_id`."""
56         prop_claims = self.claims(entity_id, property_id).get(property_id, [])
57         return target_id in prop_claims

The model writes the arguments to edges, and those arguments go straight into a SPARQL string. The two regular expressions at the top make sure the IDs are plain Q and P numbers, so model-written code cannot inject extra SPARQL. The wikibase:directClaim pattern in the second query maps a direct-claim predicate such as wdt:P57 back to its property entity, so we can report the property ID. The FILTER drops subjects that are not items, such as lexemes.

Recursive Sub-Calls

llm_query is the recursive part of the harness. The model passes some text and an instruction, and a fresh LLM call with no chat history answers it. The model can use this to classify film genres from descriptions, pick the right entity from a list of ten, or pull a year out of a block of text, without pulling that text into its own context:

 1     def llm_query(self, text: str, instruction: str) -> str:
 2         """
 3         Delegate reading/filtering of retrieved evidence to a stateless sub-call.
 4         Caches results by (model, instruction, text).
 5         """
 6         self.subcall_count += 1
 7         key_payload = {"model": self.model, "instruction": instruction, "text": text}
 8         cached = self.cache.get("llm_subcall", key_payload)
 9         if cached is not None:
10             self.subcall_cache_hits += 1
11             return cached
12 
13         resp = self.client.chat.completions.create(
14             model=self.model,
15             messages=[
16                 {
17                     "role": "system",
18                     "content": "You are a precise analytical assistant. Follow the instruction strictly using only the provided text."
19                 },
20                 {
21                     "role": "user",
22                     "content": f"Instruction: {instruction}\n\nEvidence:\n{text}"
23                 }
24             ],
25             temperature=0.0
26         )
27         answer = resp.choices[0].message.content.strip()
28         self.cache.set("llm_subcall", key_payload, answer)
29         return answer
30 
31     def llm_map(self, items: List[str], instruction: str, max_workers: int = 4) -> List[str]:
32         """Run `llm_query` concurrently across a list of text items."""
33         with ThreadPoolExecutor(max_workers=max_workers) as pool:
34             return list(pool.map(lambda item: self.llm_query(str(item), instruction), items))

The sub-call uses temperature 0 and caches on model, instruction, and text, so a rerun returns the same answer. llm_map fans one instruction out over a list with a thread pool. Here the sub-LLM is the same model as the main agent. You could point it at a smaller, cheaper model.

Grounding and the Namespace

An agent can make up a plausible Q-ID from memory. FINAL blocks that. It finds every Q-ID in the answer and rejects the answer if any of them never came back from a graph call during this run:

 1     def FINAL(self, answer: Any):
 2         """
 3         Terminate the search and return `answer`.
 4         Verifies grounding if the answer is a Q-ID or contains Q-IDs in a list/dict.
 5         """
 6         qids_to_check = re.findall(r"\bQ\d+\b", str(answer))
 7         unvisited = [q for q in qids_to_check if q not in self.visited_entities]
 8         if unvisited:
 9             raise ValueError(
10                 f"Grounding Error: Entity ID(s) {unvisited} were never visited or retrieved "
11                 f"from the graph during this session. Verify them using describe(), claims(), or edges() first."
12             )
13         raise FinalAnswerException(answer)
14 
15     def bind_namespace(self) -> Dict[str, Any]:
16         """Return the persistent dictionary of globals exposed to the agent's REPL."""
17         return {
18             "search_entity": self.search_entity,
19             "search_property": self.search_property,
20             "describe": self.describe,
21             "label": self.label,
22             "claims": self.claims,
23             "edges": self.edges,
24             "has": self.has,
25             "llm_query": self.llm_query,
26             "llm_map": self.llm_map,
27             "FINAL": self.FINAL,
28         }

A grounding failure raises a normal ValueError. The harness catches it like any other exception and shows the traceback to the model, which can then verify the ID and try again. A good answer raises FinalAnswerException, which unwinds out of the model’s code and ends the loop.

bind_namespace returns the dict that becomes the REPL’s globals. The model’s code can see these ten functions and nothing else from the harness. Variables the model creates go into the same dict, which is how they persist across turns.

An icon indicating this blurb contains a warning

The REPL runs model-written code with exec in your own Python process. The namespace limits what the model sees, not what it can do: code can still import os. Run this example on a machine and account you trust, or move the REPL into a container or subprocess before you use it with untrusted models or prompts.

The System Prompt

The prompt lists the functions with their return shapes and gives three rules of strategy:

 1 SYSTEM_PROMPT = """You are a Recursive Language Model (RLM) agent exploring Wikidata through a persistent Python REPL.
 2 Reply with ONE Python code block (```python ... ```) per turn.
 3 Variables you define persist across turns. Do not re-fetch data you have already stored in variables.
 4 
 5 Available functions in your global namespace:
 6 - search_entity(query: str, limit: int = 5) -> list[dict]  # [{'id': 'Q...', 'label': '...', 'description': '...'}]
 7 - search_property(query: str, limit: int = 5) -> list[dict] # [{'id': 'P...', 'label': '...', 'description': '...'}]
 8 - describe(entity_id: str) -> dict                         # {'id': 'Q...', 'label': '...', 'description': '...'}
 9 - label(entity_id: str) -> str                             # English label for Q-ID or P-ID
10 - claims(entity_id: str, property_id: str = None) -> dict  # OUTGOING statements: {pid: [values_or_target_QIDs]}
11 - edges(entity_id: str, property_id: str = None, limit: int = 100) -> dict # INCOMING reverse links: {pid: [subject_QIDs]}
12 - has(entity_id: str, property_id: str, target_id: str) -> bool
13 - llm_query(text: str, instruction: str) -> str            # Stateless LLM sub-call for semantic filtering
14 - llm_map(items: list[str], instruction: str) -> list[str] # Parallel stateless sub-calls
15 - FINAL(answer: Any)                                       # Call when certain to finish and return the answer
16 
17 Strategy:
18 1. Use `claims(qid)` for outgoing edges and `edges(qid, pid)` for incoming reverse relationships (e.g. finding works by an author/director or items located in a place).
19 2. Keep printed output concise; inspect slices or summaries rather than dumping huge dicts.
20 3. The question has an answer. Do not stop until you find and verify it, then call FINAL(answer).
21 """

The prompt shows the return shape of each function as a comment. Models write better code when they know the shape of the data before they call a function.

The Harness Loop

RLMHarness owns the turn loop. Its constructor takes the limits that bound a run: the number of turns, the size of REPL output shown to the model, and the token budget for each reply.

 1 class RLMHarness:
 2     """Manages the persistent Python namespace, SQLite cache, and multi-turn loop."""
 3 
 4     def __init__(
 5         self,
 6         client: OpenAI,
 7         model: str,
 8         cache: SQLiteCache,
 9         max_turns: int = 12,
10         max_output_chars: int = 10000,
11         offline_only: bool = False,
12         max_tokens: int = 2048
13     ):
14         self.client = client
15         self.model = model
16         self.cache = cache
17         self.max_turns = max_turns
18         self.max_output_chars = max_output_chars
19         self.offline_only = offline_only
20         self.max_tokens = max_tokens

Cleaning Up Model Output

Small local models rarely reply with one clean code block. Some wrap their reply in <think> tags. Some get cut off at the token limit before they close the fence. Some fall back to the native tool-call format they learned in training. Some get stuck and repeat one line over and over. The next three methods turn all of these into runnable Python:

 1     @staticmethod
 2     def _collapse_repeats(code: str) -> tuple:
 3         """Collapse runs of identical consecutive non-blank lines (degenerate LLM loops). Returns (code, n_removed)."""
 4         out: List[str] = []
 5         removed = 0
 6         for line in code.splitlines():
 7             if out and line.strip() and line == out[-1]:
 8                 removed += 1
 9                 continue
10             out.append(line)
11         return "\n".join(out), removed
12 
13     @staticmethod
14     def extract_code(response_text: str) -> str:
15         text = response_text or ""
16         # Drop reasoning blocks emitted by "thinking" models.
17         text = re.sub(r"<think>.*?</think>", "", text, flags=re.DOTALL)
18         text = re.sub(r"^.*?</think>", "", text, flags=re.DOTALL)  # unopened </think>
19 
20         match = re.search(r"```(?:python|py)?\s*(.*?)```", text, re.DOTALL)
21         if match:
22             return match.group(1).strip()
23         # Unclosed fence (e.g. reply truncated at max_tokens).
24         match = re.search(r"```(?:python|py)?[ \t]*\n(.*)", text, re.DOTALL)
25         if match:
26             return match.group(1).strip()
27 
28         # Fallback: native tool-call syntax, e.g. <tool_call>...</tool_call>
29         if "<tool_call>" in text or "</tool_call>" in text:
30             bodies = re.findall(r"<tool_call>(.*?)(?:</tool_call>|$)", text, re.DOTALL)
31             lines = []
32             for body in bodies or [text]:
33                 body = re.sub(r"</?tool_call>", "", body).strip()
34                 lines.append(RLMHarness._tool_call_to_python(body))
35             return "\n".join(l for l in lines if l)
36 
37         return text.strip()
38 
39     @staticmethod
40     def _tool_call_to_python(body: str) -> str:
41         """Convert a JSON tool call ({"name":..., "arguments":{...}}) to a Python call; pass raw code through."""
42         try:
43             obj = json.loads(body)
44         except (json.JSONDecodeError, ValueError):
45             return body
46         if isinstance(obj, dict) and "name" in obj:
47             args = obj.get("arguments") or obj.get("parameters") or {}
48             if isinstance(args, str):
49                 try:
50                     args = json.loads(args)
51                 except (json.JSONDecodeError, ValueError):
52                     args = {}
53             kwargs = ", ".join(f"{k}={v!r}" for k, v in args.items())
54             call = f"{obj['name']}({kwargs})"
55             return call if obj["name"] == "FINAL" else f"print({call})"
56         return body

extract_code tries the clean case first, then an unclosed fence, then tool-call tags. If nothing matches, it treats the whole reply as code and lets the Python parser judge it. _tool_call_to_python turns {"name": "claims", "arguments": {"entity_id": "Q42"}} into print(claims(entity_id='Q42')). It wraps the call in print so the model sees the result, except for FINAL, which returns nothing.

Notebook-Style Echo

Models often write a bare expression such as claims("Q42") and expect to see the result, as in a Jupyter cell. Plain exec throws that value away. _exec_with_echo parses the code, pops the last statement if it is an expression, runs the rest, and then evaluates and prints the popped expression:

 1     @staticmethod
 2     def _exec_with_echo(code: str, repl_globals: Dict[str, Any]):
 3         """Exec `code`; if the final statement is an expression, print its value (like a notebook cell)."""
 4         tree = ast.parse(code, mode="exec")
 5         last_expr = None
 6         if tree.body and isinstance(tree.body[-1], ast.Expr):
 7             last_expr = ast.Expression(tree.body.pop().value)
 8         exec(compile(tree, "<repl>", "exec"), repl_globals)
 9         if last_expr is not None:
10             value = eval(compile(last_expr, "<repl>", "eval"), repl_globals)
11             if value is not None:
12                 print(repr(value))

Both exec and eval receive the same repl_globals dict, so names defined in one turn exist in the next.

The Turn Loop

run ties the parts together. Each turn it asks the model for code, cleans the code, runs it with stdout and stderr captured, and sends the captured text back as the next user message:

 1     def run(self, question: str) -> Dict[str, Any]:
 2         env = WikidataEnvironment(
 3             self.client, self.model, cache=self.cache, offline_only=self.offline_only
 4         )
 5         repl_globals = env.bind_namespace()
 6         messages = [
 7             {"role": "system", "content": SYSTEM_PROMPT},
 8             {"role": "user", "content": f"Question: {question}"}
 9         ]
10 
11         for turn in range(1, self.max_turns + 1):
12             resp = self.client.chat.completions.create(
13                 model=self.model,
14                 messages=messages,
15                 temperature=0.0,
16                 max_tokens=self.max_tokens,
17             )
18             choice = resp.choices[0]
19             assistant_msg = choice.message.content or ""
20             truncated = choice.finish_reason == "length"
21 
22             code = self.extract_code(assistant_msg)
23             code, repeats = self._collapse_repeats(code)
24             if truncated:
25                 # The last line may have been cut off mid-generation; drop it if it breaks parsing.
26                 try:
27                     ast.parse(code)
28                 except SyntaxError:
29                     code = "\n".join(code.splitlines()[:-1]).rstrip()
30 
31             # Store a cleaned, canonical code block in history (not the raw runaway text),
32             # which also reinforces the expected ```python``` format.
33             messages.append({"role": "assistant", "content": f"```python\n{code}\n```"})
34             print(f"\n--- [Turn {turn} Code] ---\n{code}")
35 
36             notes = []
37             if repeats:
38                 notes.append(f"[Harness] Removed {repeats} duplicated consecutive line(s) from your reply. "
39                              "Do not repeat the same call; inspect the result and move on.")
40             if truncated:
41                 notes.append(f"[Harness] Your reply hit the {self.max_tokens}-token limit and was truncated. "
42                              "Keep replies short: one small code block per turn.")
43 
44             stdout_buf = io.StringIO()
45             old_stdout, old_stderr = sys.stdout, sys.stderr
46             sys.stdout = sys.stderr = stdout_buf
47 
48             try:
49                 self._exec_with_echo(code, repl_globals)
50                 output = stdout_buf.getvalue()
51                 if not output.strip():
52                     output = "(Executed successfully with no printed output. Use print() to inspect variables.)"
53             except FinalAnswerException as fin:
54                 sys.stdout, sys.stderr = old_stdout, old_stderr
55                 print(f"\n=== FINAL ANSWER: {fin.answer} ===")
56                 return {
57                     "answer": fin.answer,
58                     "turns": turn,
59                     "graph_calls": len(env.call_log),
60                     "api_cache_hits": env.api_cache_hits,
61                     "api_network_calls": env.api_network_calls,
62                     "subcalls": env.subcall_count,
63                     "subcall_cache_hits": env.subcall_cache_hits,
64                     "visited_entities": len(env.visited_entities),
65                 }
66             except SyntaxError:
67                 output = stdout_buf.getvalue() + "\n" + traceback.format_exc() + (
68                     "\nHint: Reply with plain Python inside a single ```python ... ``` block. "
69                     "Do not use <tool_call> tags or JSON; call the functions directly, e.g.\n"
70                     "```python\nresults = search_entity(\"Guggenheim Museum Bilbao\")\nprint(results)\n```"
71                 )
72             except Exception:
73                 output = stdout_buf.getvalue() + "\n" + traceback.format_exc()
74             finally:
75                 sys.stdout, sys.stderr = old_stdout, old_stderr
76 
77             if len(output) > self.max_output_chars:
78                 output = output[:self.max_output_chars] + "\n...[Output truncated at 10,000 chars]"
79 
80             if notes:
81                 output = output.rstrip() + "\n\n" + "\n".join(notes)
82 
83             print(f"--- [Turn {turn} REPL Output] ---\n{output.strip()}")
84             messages.append({"role": "user", "content": f"REPL Output:\n{output}"})
85 
86         raise RuntimeError(f"Agent exceeded max_turns ({self.max_turns}) without calling FINAL().")

A few choices here matter more than they look:

  • The history holds the cleaned code, not the raw reply. If the model rambled, repeated itself, or used tool-call tags, the next turn sees a tidy python block instead. Each turn then shows the model the format we want.
  • Errors are feedback, not failures. A traceback goes back to the model as REPL output. A SyntaxError also gets a hint with a correct example, since small models often fall back to the wrong format.
  • Output has a hard cap. The model sees at most 10,000 characters per turn. That cap bounds context growth, the problem this whole design exists to solve. Data the model does not print stays in Python variables, where the next turn can still use it.
  • The harness tells the model what it fixed. Notes about removed repeats and truncated replies ride along with the output so the model can change its behavior.

Running One Question

The module’s __main__ block runs one question against a local Ollama server:

 1 if __name__ == "__main__":
 2     client = OpenAI(
 3         base_url="http://localhost:11434/v1",
 4         api_key="local-key"
 5     )
 6 
 7     cache = SQLiteCache("wikidata_rlm_cache.sqlite")
 8     try:
 9         harness = RLMHarness(
10             client=client,
11             model="qwen2.5-coder:32b",
12             cache=cache
13         )
14 
15         # Multi-hop query requiring BOTH reverse edges (`edges`) and forward claims (`claims`):
16         # 1. Reverse edge: Find science fiction films directed by (P57) Denis Villeneuve (Q25188)
17         # 2. Forward claim: Check composer (P86) and publication date (P577)
18         sample_question = (
19             "Which science fiction film directed by Denis Villeneuve and released in 2016 "
20             "had its musical score composed by Jóhann Jóhannsson? Return a dict with its Q-ID and label."
21         )
22         result = harness.run(sample_question)
23         print("\nTrajectory Summary:", json.dumps(result, indent=2))
24         print("Total SQLite Cache Entries:", cache.stats())
25     finally:
26         cache.close()

To run it, pull the model in Ollama and run the module with uv:

1 ollama pull qwen2.5-coder:32b
2 uv run rlm_wikidata_bidirectional.py

The Benchmark Suite

benchmark_suite.py runs four questions, each built to test one kind of graph reasoning:

Task Pattern What the model must do
task_1_forward_2hop claims then claims Museum to architect to birthplace
task_2_reverse_edge_join edges plus claims Films by a director, filtered by composer and year
task_3_set_intersection edges and edges People who won two given awards
task_4_notable_work_filter claims plus describe Pick one book from an author’s notable works

Each task is a dataclass with the question, the expected Q-ID, keywords that must appear in the answer, and a seed function:

1 @dataclass
2 class BenchmarkTask:
3     task_id: str
4     category: str
5     question: str
6     expected_qid: str
7     expected_keywords: List[str]
8     seed_fn: Callable[[WikidataEnvironment], None]

A seed function walks the expected path through the graph with no LLM involved, so every lookup lands in the cache. Here is the seeder for the set intersection task:

 1 def seed_turing_nobel(env: WikidataEnvironment):
 2     env.search_entity("Turing Award")
 3     env.search_entity("Nobel Memorial Prize in Economic Sciences")
 4     env.search_entity("Nobel Prize in Economics")
 5     env.search_property("award received")
 6     turing_winners = set(env.edges("Q185667", "P166", limit=100).get("P166", []))
 7     nobel_winners = set(env.edges("Q47170", "P166", limit=100).get("P166", []))
 8     for qid in turing_winners & nobel_winners:
 9         env.describe(qid)
10         env.claims(qid, "P166")

After seeding, you can run models in --offline mode with no network access, so every model gets the same data. The answer check is a loose string match on the serialized answer, so FINAL may return a string, dict, or list:

1 def verify_answer(answer: Any, task: BenchmarkTask) -> bool:
2     """Check whether the agent's FINAL(answer) contains the target Q-ID and keywords."""
3     serialized = json.dumps(answer, ensure_ascii=False) if isinstance(answer, (dict, list)) else str(answer)
4     serialized_lower = serialized.lower()
5     has_qid = task.expected_qid.lower() in serialized_lower
6     has_keywords = all(kw.lower() in serialized_lower for kw in task.expected_keywords)
7     return has_qid and has_keywords

run_benchmark runs each task through one RLMHarness, catches failures, and prints a Markdown table of turns, graph calls, cache hits, network calls, sub-calls, and time. The command line flags pick the endpoint and model:

Flag Default Meaning
--base-url http://localhost:11434/v1 OpenAI-compatible endpoint
--api-key local-key Key for the endpoint
--model qwen2.5-coder:32b Model name
--db wikidata_rlm_cache.sqlite Cache file
--seed off Seed the cache, then run
--seed-only off Seed the cache and exit
--offline off Use only the cache; no Wikidata requests

The Makefile wraps the common cases:

1 clean:
2     rm -rf *~ *.sqlite __pycache__ .venv
3 
4 create_db:
5     uv run benchmark_suite.py --seed-only
6 
7 run_omlx:
8     uv run benchmark_suite.py --base-url http://127.0.0.1:8000/v1 --model Laguna-XS-2.1-6bit

A Sample Run with a Small Local Model

SAMPLE_OUTPUT.txt holds a full run of make run_omlx, which serves the small Laguna-XS-2.1-6bit model through oMLX. The run is worth a close read because it shows both what the design buys you and where a small model still struggles.

What Went Well

Task 4 is the design working as intended. The model fetched the fifteen notable works, then stored the list in a variable and printed one line per work:

1 --- [Turn 4 REPL Output] ---
2 Q116546: Earthsea series - fantasy fiction series by Ursula K. Le Guin, including 5 novels 1968 to 2001 among other publications
3 Q2008211: The Left Hand of Darkness - 1969 novel by Ursula K. Le Guin
4 Q467720: The Dispossessed - 1974 novel by Ursula K. Le Guin
5 ...

It saw that the list holds the Earthsea series, not the first novel, so it followed the series’ P527 (“has part”) claims one more hop and found the book:

1 --- [Turn 6 REPL Output] ---
2 P527 (has part) items:
3   Q1771810: A Wizard of Earthsea - 1968 novel by Ursula K. Le Guin
4   Q2070380: The Tombs of Atuan - 1971 novel by Ursula K. Le Guin
5 ...
6 --- [Turn 8 Code] ---
7 FINAL({"qid": "Q1771810", "label": "A Wizard of Earthsea", "earliest_publication_year": 1968})

In Task 2 the model first used the wrong property for composer (P571, which is “inception”). Its own code printed No composer found with P571, so it called search_property("composer"), found P86, and confirmed the answer:

1 === Arrival (Q20382729) Details ===
2 Label: Arrival
3 Director: Denis Villeneuve (Q548823)
4 Composer: Jóhann Jóhannsson (Q428347)
5 Release date: ['+2016-11-11T00:00:00Z', '+2016-11-10T00:00:00Z', '+2016-11-24T00:00:00Z']

Both of these answers are correct.

What Went Wrong

The final table reports zero of four solved:

1 | Task ID | Passed | Turns | Graph Calls | Cache Hits | Net Calls | Sub-Calls | Time (s) |
2 | :--- | :---: | :---: | :---: | :---: | :---: | :---: | :---: |
3 | `task_1_forward_2hop` | **FAIL** | 10 | 0 | 0 | 0 | 0 | 23.09s |
4 | `task_2_reverse_edge_join` | **FAIL** | 9 | 92 | 5 | 87 | 0 | 575.74s |
5 | `task_3_set_intersection` | **FAIL** | 10 | 0 | 0 | 0 | 0 | 47.22s |
6 | `task_4_notable_work_filter` | **FAIL** | 8 | 28 | 19 | 9 | 0 | 42.5s |

There are four separate lessons in that table.

1. The benchmark’s expected IDs do not match live Wikidata. The model returned Q20382729 for Arrival and Q1771810 for A Wizard of Earthsea. Both IDs came from the live graph, and both passed the grounding check. The benchmark expects Q20168408 and Q320435, so verify_answer scored two correct answers as failures. The seeders have the same problem: they seed Q170040 for the Guggenheim Museum Bilbao, Q6722 for Frank Gehry, and Q25188 for Denis Villeneuve, while live search in this run returned Q179199, Q180374, and Q548823. An offline run would therefore miss the cache on the paths the model actually takes. The paper avoids this class of bug by building its questions on a frozen Wikidata snapshot. The lesson for your own benchmarks: derive expected IDs from the same data source the agent queries, and check them with a script, not from memory.

2. Small models fight the output format. Many turns failed with a SyntaxError because the model ran several calls together on one line, as in describe("Q185667")describe("Q47170"), or left a stray fence or a <|tool_call_begin|> token in its code. The hint in the error message fixed the problem each time, but each fix cost a turn. Task 3 spent five of its ten turns on syntax errors and ran out of turns.

3. Printed loops can make many network calls. In Task 2 the model looped over all of Denis Villeneuve’s claims and called label(pid) for each property. That one turn made dozens of throttled requests and drew repeated HTTP 429 responses, then hit a read timeout. The task took almost ten minutes, with 87 network calls. A second run would be fast, since every one of those calls is now in the cache.

4. The model never used llm_query. The Sub-Calls column is zero for all tasks. These four questions are simple enough that plain graph calls suffice, so this is not a failure. Questions that need judgment over long text, such as “which of these 40 films is science fiction, based on its description?”, give the sub-call a real job.

Also note that the zeros in the Graph Calls column for Tasks 1 and 3 come from the error path in run_benchmark, which records zeros when harness.run raises. They do not mean the model made no calls.

Task 1 shows one more pattern. By turn 9 the model had found Toronto (Q172) as Frank Gehry’s birthplace, but it went on to look up the place of birth property instead of calling FINAL, and ran out of turns. A larger turn budget, or a prompt that tells the model to call FINAL as soon as it has a verified answer, would likely fix this.

Wrap Up

The RLM harness changes where an agent’s working memory lives. With JSON tool calls, every result piles up in the context, and the model must join and filter in token space. With a persistent REPL, results live in Python variables, joins and filters are exact Python operations, and the model reads only short printed summaries. Sub-calls let the model hand off reading and judging tasks to a fresh LLM call that sees only the text it needs.

The rest of the code supports that core idea:

  1. A content-keyed SQLite cache makes runs repeatable, makes reruns fast, and keeps load off public servers.
  2. Bidirectional primitives give the model both outgoing claims (Action API) and incoming edges (SPARQL), which multi-hop questions need.
  3. A grounding check in FINAL rejects Q-IDs the model did not retrieve during the run.
  4. Defensive parsing turns the messy output of small models into runnable code and tells the model what went wrong.

The sample run also shows the limits. A small local model can follow the method and find correct answers, but it wastes turns on format errors, and a benchmark is only as good as its ground truth.

Exercises for the Reader

  1. Fix the benchmark ground truth. Write a script that resolves each expected entity by name with search_entity, prints the live Q-IDs, and updates BENCHMARK_TASKS and the seed functions. Rerun make create_db and then the benchmark with --offline.
  2. Sandbox the REPL. Move _exec_with_echo into a subprocess that holds repl_globals and talks to the harness over a pipe. Expose the graph primitives as stubs that send requests back to the parent process. What does this cost in speed?
  3. Add a turn-budget note. Tell the model in each REPL output how many turns remain. Does Task 1 now call FINAL in time?
  4. Use a smaller sub-model. Add a sub_model argument to WikidataEnvironment so llm_query can use a cheaper model than the main agent. Write a question that needs llm_map, for example one that filters a director’s films by genre from their descriptions, and compare results.
  5. Compare with JSON tool calling. Build a baseline agent that exposes the same ten functions as JSON tools and appends raw results to the chat. Run both agents on the four tasks with the same model and record turns, prompt tokens, and pass rate.
  6. Batch label lookups. The Action API wbgetentities call accepts up to 50 IDs separated by |. Add a labels(ids) primitive that fetches many labels in one request, and document it in the system prompt. How much faster is Task 2 with a cold cache?