GraphRAG and Knowledge Graph Quality

Earlier chapters in this book built a substantial semantic technology stack. In the rdf-sparql chapter, I implemented an in-memory RDF datastore and SPARQL pattern matcher. In kgcreator, I extracted RDF triples from plain text using spaCy and schema.org. In kgn, I queried public SPARQL endpoints to find multi-hop relationships. In NSK, I combined a Datalog-style triplestore and unification engine with a local Ollama neural fallback. And in zvec_RAG, I indexed text chunks into a local vector store for retrieval-augmented generation (RAG).

Those chapters left two practical questions open:

  1. How do you know a generated knowledge graph is valid and consistent? Automated extractors produce duplicate entity names, missing properties, and bad confidence scores. Without validation, entity resolution, and provenance tracking, errors accumulate in the store.
  2. How do you combine structured RDF graphs with unstructured text chunks during RAG? Vector-only and chunk-only retrieval work well when one text passage holds the whole answer. They fail when a question requires connecting facts across documents or resolving aliases.

This chapter solves both problems by building kg-rag, an end-to-end knowledge graph quality and GraphRAG pipeline in Hy. The code for this chapter is in the directory source_code_for_examples/graphrag_kg_quality.

What You Will Build

The kg-rag project processes a bundled set of six short documents describing the software projects in this book (KGCreator, KGN, NSK, zvec_RAG, and kg-rag). It extracts entities and triples, records chunk-level provenance using the W3C PROV-O vocabulary, merges duplicate entities with owl:sameAs, infers implicit triples with RDFS rules, validates the graph with SHACL shapes (pyshacl), supports transactional SPARQL Update with snapshot rollback, and answers questions with both local 2-hop neighborhood expansion and global community summaries.

 1 +-------------------+     +-----------------------+     +------------------------+
 2 |  Bundled Corpus   | --> |   Extractor Layer     | --> | RDF Graph + Provenance |
 3 |  data/doc1..6.txt |     | FakeExtractor / Ollama|     | reified rdf:Statement  |
 4 +-------------------+     +-----------------------+     | prov:wasDerivedFrom    |
 5                                                         +------------------------+
 6                                                                     |
 7         +-----------------------------------------------------------+
 8         v
 9 +------------------------+     +------------------------+     +------------------+
10 |   Entity Resolution    | --> |    RDFS Inference      | --> | SHACL Validation |
11 | Blocking + Sequence    |     | SPARQL CONSTRUCT rules |     | data/shapes.ttl  |
12 | Matcher + owl:sameAs   |     | subClass / subProperty |     | pyshacl report   |
13 +------------------------+     +------------------------+     +------------------+
14                                                                     |
15         +-----------------------------------------------------------+
16         v
17 +--------------------------------------------------------------------------------+
18 |                         Hybrid GraphRAG Retrieval                              |
19 |  * Baseline: Keyword TF-IDF cosine chunk search                                |
20 |  * Local Search: Entity linking -> 1-2 hop SPARQL expansion -> provenance      |
21 |                  chunk boosting -> token-budget context assembly               |
22 |  * Global Search: Connected-component community summaries                      |
23 +--------------------------------------------------------------------------------+
24         |
25         v
26 +--------------------------------------------------------------------------------+
27 |              Grounded Answer Synthesis & Benchmark Evaluation                  |
28 |  * Answers cite [triple: s p o] and [chunk: docX#chunkY]                       |
29 |  * Metrics: Retrieval Recall, Citation Accuracy, Groundedness                  |
30 +--------------------------------------------------------------------------------+

Why It Matters

When you extract triples from text with NLP rules (kgcreator) or a language model (NSK), three failure modes appear right away:

  • Unverified structure: A project node may lack a label or creator, or a confidence score may fall outside [0.0, 1.0]. Shapes Constraint Language (SHACL) catches these violations before bad facts enter the index.
  • Fragmented identities: One document refers to "Mark L. Watson" and "KG-Creator", while another refers to "Mark Watson" and "KGCreator". Until you resolve those aliases to canonical URIs, a multi-hop query stops at the document boundary.
  • Missing provenance: When a RAG system states a fact, the user needs to know which document chunk produced that triple and how confident the extractor was.

At retrieval time, chunk-only RAG (zvec_RAG) scores each passage independently. Consider this multi-hop question from our benchmark:

“Which database stores embeddings for the RAG system built by Mark L. Watson, and which model generates its answers?”

Document 1 states that Mark L. Watson created zvec_RAG. Document 5 states that zvec_RAG stores embeddings in Alibaba zvec and generates answers with qwen3:1.7b, without mentioning Mark L. Watson. A chunk-only retriever under a tight token budget picks either Document 1 or Document 5, missing half the chain. GraphRAG links Mark L. Watson to mark_watson via owl:sameAs, walks one hop to zvec_rag, and walks a second hop to zvec_db and qwen3_1_7b, pulling the exact supporting chunks via prov:wasDerivedFrom.

The Hy Angle

Hy makes RDF manipulation, rule execution, and retrieval scoring concise:

  • Frozen dataclasses with type annotations (#^): Records such as DocChunk, ExtractedTriple, ValidationResult, ResolutionCandidate, and ScoredTriple are immutable values declared in four to eight lines each.
  • Comprehensions (lfor, gfor, dfor): Computing TF-IDF vectors, filtering triples inside a connected component, and building lookup maps take a single expression.
  • Tuple unpacking (#(s p o)): rdflib represents every triple as a 3-tuple. Hy unpacks (for [#(s p o) g] ...) directly in loops and comprehensions.
  • First-class validators for transactions: Our transactional-sparql-update function takes an optional validator lambda (fn [g] ...), runs a SPARQL Update, checks the post-update graph with SHACL, and restores the snapshot if validation fails.

Python Equivalent

To see how Hy maps to Python when working with rdflib, compare our RDFS closure loop in quality.hy with the equivalent Python code.

In Hy:

 1 (defn apply-rdfs-inference [g [max-passes 6]]
 2   (setv added []
 3         pass-num 0)
 4   (while (< pass-num max-passes)
 5     (setv new-in-pass [])
 6     (for [rule RDFS_CONSTRUCT_RULES]
 7       (for [triple (.query g rule)]
 8         (when (not (in triple g))
 9           (.add g triple)
10           (.append new-in-pass triple)
11           (.append added triple))))
12     (when (= (len new-in-pass) 0)
13       (break))
14     (setv pass-num (+ pass-num 1)))
15   added)

In Python:

 1 def apply_rdfs_inference(g, max_passes=6):
 2     added = []
 3     for _ in range(max_passes):
 4         new_in_pass = []
 5         for rule in RDFS_CONSTRUCT_RULES:
 6             for triple in g.query(rule):
 7                 if triple not in g:
 8                     g.add(triple)
 9                     new_in_pass.append(triple)
10                     added.append(triple)
11         if not new_in_pass:
12             break
13     return added

Both compile to the same bytecode and call g.query(rule) on the underlying rdflib.Graph.

Run It

The project runs offline with no API keys or external database servers. If you want to test live extraction with Ollama (OllamaExtractor), start Ollama locally; if Ollama is not running, the extractor prints a short notice and falls back to FakeExtractor.

Run the walkthrough and test suite with uv:

1 cd source_code_for_examples/graphrag_kg_quality
2 uv sync
3 uv run hy demo.hy
4 uv run pytest -q

Here is the actual output from uv run hy demo.hy:

 1 === 1. Corpus & Extraction ===
 2 Loaded 12 text chunks across 6 documents.
 3 Extracted 27 entities and 30 raw triples.
 4 Provenance for (kgcreator usesComponent spacy): [{'chunk_id': 'doc3#chunk0', 'doc_id': 'doc3_kgcreator', 'confidence': 0.95}]
 5 
 6 === 2. Entity Resolution (Blocking + String Similarity) ===
 7   [auto-merge] mark_l_watson (Mark L. Watson) -> mark_watson (Mark Watson) [block=Person:m, sim=0.9167]
 8   [auto-merge] kg_creator (KG-Creator) -> kgcreator (KGCreator) [block=System:k, sim=0.9474]
 9   [review-queue] qwen3_5_4b (qwen3.5:4b) ? qwen3_1_7b (qwen3:1.7b) [block=Component:q, sim=0.7778]
10   [review-queue] mark_l_watson (Mark L. Watson) ? m_watson (M. Watson) [block=Person:m, sim=0.7619]
11   [review-queue] m_watson (M. Watson) ? mark_watson (Mark Watson) [block=Person:m, sim=0.8421]
12 
13 === 3. RDFS Inference & SHACL Validation ===
14 RDFS inference derived 37 implicit triples.
15 Clean graph SHACL conforms: True (violations: 0)
16 Corrupted sample SHACL conforms: False (violations: 4)
17   - focus=http://example.org/kgrag/orphan_agent | Every SoftwareProject must have at least one string rdfs:label.
18   - focus=http://example.org/kgrag/orphan_agent | Every SoftwareProject must specify an ex:createdBy pointing to an ex:Person.
19   - focus=http://example.org/kgrag/stmt_bad_conf | Every reified rdf:Statement must have an xsd:double ex:confidence in [0.0, 1.0].
20   - focus=http://example.org/kgrag/stmt_bad_conf | Every reified rdf:Statement must link to a source chunk IRI via prov:wasDerivedFrom.
21 
22 === 4. SPARQL Update & Snapshot Rollback ===
23 Valid SPARQL Update: ok=True, triples 377 -> 378
24 Invalid SPARQL Update rolled back: ok=False, triples restored to 378
25 
26 === 5. Baseline vs GraphRAG Retrieval (Local & Global) ===
27 Question: Which database stores embeddings for the RAG system built by Mark L. Watson, and which model generates its answers?
28   Baseline chunks: ['doc5#chunk1', 'doc5#chunk0']
29   GraphRAG seeds:  ['mark_watson']
30   GraphRAG chunks: ['doc6#chunk0', 'doc5#chunk1', 'doc5#chunk0', 'doc1#chunk0']
31   GraphRAG cited triples (10): ['zvec_rag usesEmbeddingModel embeddinggemma', 'zvec_rag usesGenerationModel qwen3_1_7b', 'kg_rag createdBy mark_watson', 'zvec_rag createdBy mark_watson']
32   Global search communities: 3
33     * [community_3: KG Quality & GraphRAG Validation] Entities: kg_rag, prov_o, pyshacl, shacl, sparql_2hop. Key relations: kg_rag dependsOn pyshacl; kg_rag dependsOn sparql_2hop; kg_rag tracksProvenanceWith prov_o; kg_rag usesComponent pyshacl; kg_rag usesComponent sparql_2hop.
34     * [community_2: Local Vector RAG Pipeline] Entities: embeddinggemma, qwen3_1_7b, zvec_db, zvec_rag. Key relations: zvec_rag dependsOn zvec_db; zvec_rag usesComponent zvec_db; zvec_rag usesDatastore zvec_db; zvec_rag usesEmbeddingModel embeddinggemma; zvec_rag usesGenerationModel qwen3_1_7b.
35     * [community_1: Symbolic & Neural-Symbolic Graph Systems] Entities: datalog_rules, dbpedia, hy_lang, kgcreator, kgn, nsk, nsk_engine, ollama_fallback, qwen3_5_4b, rdf_datastore, rdflib_store, schema_org, spacy, turtle_rdf, unification. Key relations: kgcreator dependsOn hy_lang; kgcreator dependsOn rdflib_store; kgcreator dependsOn spacy; kgcreator linksEntitiesTo dbpedia; kgcreator usesComponent hy_lang.
36 
37 === 6. Benchmark Evaluation Table ===
38 Case | Mode           | Recall | CiteAcc | Grounded
39 -----+----------------+--------+---------+---------
40 Q1   | Baseline       |   0.67 |    1.00 |     0.67
41 Q1   | GraphRAG       |   1.00 |    1.00 |     1.00
42 Q2   | Baseline       |   0.33 |    1.00 |     0.00
43 Q2   | GraphRAG       |   1.00 |    1.00 |     1.00
44 Q3   | Baseline       |   1.00 |    1.00 |     0.67
45 Q3   | GraphRAG       |   1.00 |    1.00 |     1.00
46 Q4   | Baseline       |   1.00 |    1.00 |     0.33
47 Q4   | GraphRAG       |   1.00 |    1.00 |     1.00
48 Q5   | Baseline       |   1.00 |    1.00 |     0.67
49 Q5   | GraphRAG       |   1.00 |    1.00 |     1.00
50 Q6   | Baseline       |   0.20 |    1.00 |     0.20
51 Q6   | GraphRAG       |   1.00 |    1.00 |     1.00
52 -----+----------------+--------+---------+---------
53 AVG  | Baseline       |   0.70 |    1.00 |     0.42
54 AVG  | GraphRAG       |   1.00 |    1.00 |     1.00

How It Works

1. Corpus Ingestion and Triple Extraction: corpus.hy and extract.hy

The data/ directory holds six short text files (doc1_hy_ecosystem.txt through doc6_shacl_prov.txt). Each document starts with Document: and Source: metadata lines followed by four sentences. corpus.hy reads these files and groups every two sentences into a DocChunk with a stable identifier such as doc1#chunk0:

 1 (defclass [(dataclass :frozen True)] DocChunk []
 2   "A text chunk with provenance identifiers."
 3   #^ str doc-id
 4   #^ str chunk-id
 5   #^ str source-uri
 6   #^ str text)
 7 
 8 (defn chunk-lines [doc-id source-uri lines [sentences-per-chunk 2]]
 9   "Group body lines into fixed-line-count chunks with stable IDs."
10   (setv chunks []
11         idx 0
12         i 0)
13   (while (< i (len lines))
14     (setv group (cut lines i (+ i sentences-per-chunk))
15           short-id (get (.split doc-id "_" 1) 0)
16           chunk-id f"{short-id}#chunk{idx}"
17           text (.join " " group))
18     (.append chunks (DocChunk :doc-id doc-id
19                               :chunk-id chunk-id
20                               :source-uri source-uri
21                               :text text))
22     (setv idx (+ idx 1)
23           i (+ i sentences-per-chunk)))
24   chunks)

In extract.hy, every extracted triple carries its source doc-id, chunk-id, and a floating-point confidence in [0.0, 1.0]:

 1 (defclass [(dataclass :frozen True)] EntityRecord []
 2   "An extracted entity with local slug, human label, and ontology class."
 3   #^ str local-id
 4   #^ str label
 5   #^ str entity-type)
 6 
 7 (defclass [(dataclass :frozen True)] ExtractedTriple []
 8   "A subject-predicate-object assertion tied to a document chunk and confidence."
 9   #^ str subject
10   #^ str predicate
11   #^ str object-val
12   #^ str doc-id
13   #^ str chunk-id
14   #^ float confidence)

FakeExtractor matches deterministic text triggers so tests run in under a second without network calls. OllamaExtractor sends a JSON-mode request to a local Ollama server and falls back to FakeExtractor when Ollama is unreachable.

2. RDF Graph Construction, PROV-O Provenance, and SPARQL Update: kg.hy

When we load an ExtractedTriple into rdflib, asserting (s, p, o) alone loses the chunk ID and confidence score. In kg.hy, add-provenance-triple adds both the domain triple and a reified rdf:Statement node linked to a prov:Entity chunk node via prov:wasDerivedFrom:

 1 (defn add-provenance-triple [g triple-rec]
 2   "Add both the asserted triple and its reified rdf:Statement provenance node."
 3   (setv s-ref (get EX triple-rec.subject)
 4         p-ref (get EX triple-rec.predicate)
 5         o-ref (get EX triple-rec.object-val)
 6         c-ref (chunk-uri triple-rec.chunk-id)
 7         stmt (statement-uri triple-rec.subject
 8                             triple-rec.predicate
 9                             triple-rec.object-val
10                             triple-rec.chunk-id))
11   (.add g #(s-ref p-ref o-ref))
12   (.add g #(c-ref RDF.type PROV.Entity))
13   (.add g #(c-ref EX.chunkId (Literal triple-rec.chunk-id :datatype XSD.string)))
14   (.add g #(c-ref EX.docId (Literal triple-rec.doc-id :datatype XSD.string)))
15   (.add g #(stmt RDF.type RDF.Statement))
16   (.add g #(stmt RDF.subject s-ref))
17   (.add g #(stmt RDF.predicate p-ref))
18   (.add g #(stmt RDF.object o-ref))
19   (.add g #(stmt PROV.wasDerivedFrom c-ref))
20   (.add g #(stmt EX.fromDocument (Literal triple-rec.doc-id :datatype XSD.string)))
21   (.add g #(stmt EX.confidence (Literal (float triple-rec.confidence) :datatype XSD.double)))
22   stmt)

At retrieval time, provenance-for-triple runs a SPARQL query over the reified statements to recover every chunk and confidence score supporting (s, p, o).

Knowledge graphs also change over time as documents are edited or retracted. transactional-sparql-update clones the graph before running a SPARQL Update (INSERT DATA, DELETE WHERE, or DELETE/INSERT), runs an optional validation callback afterward, and restores the snapshot if either the update or validation fails:

 1 (defn transactional-sparql-update [g update-query [validator None]]
 2   "Run a SPARQL Update on g; roll back to a snapshot if execution or validator fails."
 3   (setv snapshot (clone-graph g))
 4   (try
 5     (.update g update-query)
 6     (when (not (is validator None))
 7       (setv #(ok reason) (validator g))
 8       (when (not ok)
 9         (raise (ValueError f"Post-update validation failed: {reason}"))))
10     #(True "committed" (len g))
11     (except [e Exception]
12       (.remove g #(None None None))
13       (for [triple snapshot]
14         (.add g triple))
15       #(False (str e) (len g)))))

3. Entity Resolution: Blocking, Similarity, and owl:sameAs in resolve.hy

Comparing every entity pair in an N-node graph takes O(N^2) string comparisons. resolve.hy avoids quadratic blowup by grouping entities into blocks that share a coarse ontology class (Person, System, Component, Standard) and the first character of their normalized label:

 1 (defn normalize-label [label]
 2   "Lowercase label, strip periods, and collapse whitespace."
 3   (setv s (.replace (.lower label) "." "")
 4         s (re.sub r"\s+" " " (.strip s)))
 5   s)
 6 
 7 (defn blocking-key [entity]
 8   "Group entities by coarse ontology class and first character of normalized label."
 9   (setv norm (normalize-label entity.label)
10         first-ch (if (> (len norm) 0) (get norm 0) "_"))
11   f"{(coarse-type entity.entity-type)}:{first-ch}")
12 
13 (defn label-similarity [label-a label-b]
14   "Compute string similarity in [0.0, 1.0] using difflib.SequenceMatcher."
15   (setv na (normalize-label label-a)
16         nb (normalize-label label-b))
17   (.ratio (SequenceMatcher None na nb)))

Within each block, resolve-entities applies two thresholds:

  • Auto-merge (>= 0.88): High-similarity pairs such as "Mark L. Watson" vs "Mark Watson" (0.9167) and "KG-Creator" vs "KGCreator" (0.9474) emit an owl:sameAs link and call canonicalize-uri-in-graph, which rewrites triples (including provenance rdf:subject and rdf:object pointers) to use the canonical URI.
  • Human review queue (0.72 to 0.88): Ambiguous pairs such as "M. Watson" vs "Mark Watson" (0.8421) and "qwen3.5:4b" vs "qwen3:1.7b" (0.7778) are not merged automatically. Notice that the two Qwen model names have high character overlap (0.7778) even though they are distinct models; routing this band to a review list prevents false merges.

4. RDFS Inference and SHACL Validation: quality.hy and data/shapes.ttl

Before validating the graph with SHACL, apply-rdfs-inference runs six SPARQL CONSTRUCT rules to materialize the RDFS closure:

  1. Transitive rdfs:subClassOf chains.
  2. Instance typing across rdfs:subClassOf (so an ex:NeuralSymbolicSystem such as ex:nsk becomes an ex:SoftwareProject and an ex:AISoftware).
  3. Transitive rdfs:subPropertyOf chains.
  4. Property inheritance across rdfs:subPropertyOf (so (ex:nsk ex:authoredBy ex:mark_watson) derives (ex:nsk ex:createdBy ex:mark_watson)).
  5. rdfs:domain typing.
  6. rdfs:range typing.

Once RDFS inference completes, validate-graph checks the graph against data/shapes.ttl using pyshacl:

 1 ex:SoftwareProjectShape
 2     a sh:NodeShape ;
 3     sh:targetClass ex:SoftwareProject ;
 4     sh:property [
 5         sh:path rdfs:label ;
 6         sh:minCount 1 ;
 7         sh:datatype xsd:string ;
 8         sh:message "Every SoftwareProject must have at least one string rdfs:label." ;
 9     ] ;
10     sh:property [
11         sh:path ex:createdBy ;
12         sh:minCount 1 ;
13         sh:class ex:Person ;
14         sh:message "Every SoftwareProject must specify an ex:createdBy pointing to an ex:Person." ;
15     ] .
16 
17 ex:ProvenanceStatementShape
18     a sh:NodeShape ;
19     sh:targetClass rdf:Statement ;
20     sh:property [
21         sh:path prov:wasDerivedFrom ;
22         sh:minCount 1 ;
23         sh:nodeKind sh:IRI ;
24         sh:message "Every reified rdf:Statement must link to a source chunk IRI via prov:wasDerivedFrom." ;
25     ] ;
26     sh:property [
27         sh:path ex:confidence ;
28         sh:minCount 1 ;
29         sh:datatype xsd:double ;
30         sh:minInclusive 0.0 ;
31         sh:maxInclusive 1.0 ;
32         sh:message "Every reified rdf:Statement must have an xsd:double ex:confidence in [0.0, 1.0]." ;
33     ] .

Our resolved and inferred graph passes with zero violations. To show how SHACL catches bad extractions, make-invalid-graph-sample in quality.hy injects an unattributed ex:orphan_agent project and a ex:stmt_bad_conf statement with ex:confidence 1.45. Running validate-graph on that sample returns conforms=False and extracts each sh:ValidationResult focus node and message via SPARQL.

5. Property Graphs vs RDF: A Short Neo4j and Cypher Comparison

Caution: this chapter is not a Neo4j manual. Our priority is trustworthy retrieval, and rdflib + pyshacl runs in-process with zero external services. Still, many engineering teams store knowledge graphs in property graph databases such as Neo4j, so it helps to see how the two models compare.

Feature RDF + SPARQL (rdflib) Property Graph + Cypher (Neo4j)
Core unit Triple (subject, predicate, object) with global IRIs Labeled nodes and directed relationships with key-value maps
Edge attributes Reification (rdf:Statement) or RDF-star Native properties stored directly on relationships
Schema validation W3C SHACL (pyshacl) and OWL/RDFS entailment Property existence/type constraints; no built-in RDFS reasoner
Federation & standards W3C standards (Turtle, JSON-LD, SPARQL 1.1, PROV-O) GQL / openCypher; database-specific storage

Here are two SPARQL queries from our pipeline alongside their Cypher equivalents:

Query A: Fetch provenance chunk ID and confidence for a triple.

In SPARQL (using our reified rdf:Statement model):

 1 PREFIX ex:   <http://example.org/kgrag/>
 2 PREFIX prov: <http://www.w3.org/ns/prov#>
 3 PREFIX rdf:  <http://www.w3.org/1999/02/22-rdf-syntax-ns#>
 4 SELECT ?chunkId ?conf WHERE {
 5   ?stmt a rdf:Statement ;
 6         rdf:subject ex:kgcreator ;
 7         rdf:predicate ex:usesComponent ;
 8         rdf:object ex:spacy ;
 9         prov:wasDerivedFrom ?c ;
10         ex:confidence ?conf .
11   ?c ex:chunkId ?chunkId .
12 }

In Cypher (where chunk ID and confidence live as relationship properties):

1 MATCH (:SoftwareProject {id: 'kgcreator'})-[r:USES_COMPONENT]->(:Component {id: 'spacy'})
2 RETURN r.chunk_id AS chunkId, r.confidence AS conf

Query B: Two-hop traversal from an author to components used by that author’s projects.

In SPARQL:

1 PREFIX ex: <http://example.org/kgrag/>
2 SELECT ?project ?component WHERE {
3   ?project ex:createdBy ex:mark_watson ;
4            ex:usesComponent ?component .
5 }

In Cypher:

1 MATCH (p:SoftwareProject)-[:CREATED_BY]->(:Person {id: 'mark_watson'}),
2       (p)-[:USES_COMPONENT]->(c:Component)
3 RETURN p.id AS project, c.id AS component

Choose RDF when you need standard vocabularies (schema.org, PROV-O), RDFS/OWL inference, or SHACL validation. Choose a property graph when edge-heavy analytics and relationship properties dominate and you do not need semantic web ontologies.

6. Hybrid Retrieval, Token Budgets, and Community Summaries: retrieve.hy and graphrag.hy

retrieve.hy implements two retrievers:

  1. KeywordRetriever (Baseline): Computes IDF-weighted bag-of-words vectors over chunks and ranks chunks by cosine similarity against the query, packing top chunks up to token-budget. This stands in for a chunk-only vector baseline without requiring a multi-gigabyte embedding model in CI.
  2. GraphRetriever (Local and Global GraphRAG):
    • Entity linking (link-entities): Matches query terms against rdfs:label values, resolves owl:sameAs aliases to canonical URIs, and maps class phrases such as "AI software" to instances of ex:AISoftware.
    • Neighborhood expansion (expand-neighborhood): Walks 1-hop and 2-hop outgoing and incoming edges around seed entities, attaching provenance chunk IDs and confidence scores to every ScoredTriple.
    • Provenance chunk boosting and token budgeting (local-search): Ranks neighborhood triples by non-seed query token overlap, allocates up to 35% of token-budget to structured triples, boosts the retrieval score of source chunks linked via prov:wasDerivedFrom to those triples, and fills the remaining token budget with the highest-scoring chunks:
 1     ;; Assemble top triples within 35% of the token budget first
 2     (setv kept-triples []
 3           used 0
 4           triple-budget (int (* token-budget 0.35)))
 5     (for [st ranked-triples]
 6       (setv t-cost 6)
 7       (when (<= (+ used t-cost) triple-budget)
 8         (.append kept-triples st)
 9         (setv used (+ used t-cost))))
10 
11     ;; Boost chunks referenced via prov:wasDerivedFrom by kept triples
12     (setv chunk-boost {}
13           kw-scores (dfor #(sim ch) (.score-chunks self.keyword query) ch.chunk-id sim))
14     (for [st kept-triples]
15       (setv weight (if (= st.hop 1) 0.45 0.35))
16       (for [cid st.chunk-ids]
17         (setv (get chunk-boost cid)
18               (+ (.get chunk-boost cid 0.0) (* weight st.confidence)))))

For broad thematic questions (such as “Compare the major knowledge graph and RAG architectures in this corpus”), build-communities removes high-degree author hub edges (createdBy, authoredBy) and finds connected components in the technical dependency graph. This partitions the graph into three communities (KG Quality & GraphRAG Validation, Local Vector RAG Pipeline, and Symbolic & Neural-Symbolic Graph Systems) and builds a CommunitySummary with member entities, key triples, and supporting chunk IDs for global-search.

In graphrag.hy, synthesize-grounded-answer formats an AnswerResult that cites every used triple ([triple: s p o]) and chunk ([chunk: docX#chunkY]).

7. Evaluating Retrieval and Groundedness: evaluate.hy

evaluate.hy defines six benchmark questions (Q1 through Q6) and scores both baseline and graphrag_local on three metrics in [0.0, 1.0]:

  • Retrieval Recall: Fraction of gold source chunks present in the retrieved chunks or triple provenance links.
  • Citation Accuracy: Fraction of [triple: ...] and [chunk: ...] citations in the answer that exist in the retrieved context.
  • Groundedness: Fraction of required target facts present in the context-backed answer.

Across the six benchmark questions, the chunk-only baseline averages 0.70 recall and 0.42 groundedness because multi-hop questions (Q1, Q2, Q6) split facts across documents or use entity aliases (Mark L. Watson, KG-Creator). GraphRAG reaches 1.00 recall, 1.00 citation accuracy, and 1.00 groundedness while staying inside its 180-token context budget.

Tests

The test suite lives in test_graphrag.hy and runs offline in one second:

1 $ uv run pytest -q
2 ..........                                                               [100%]
3 10 passed in 1.00s

The 10 tests verify:

  1. test-corpus-loading: 12 chunks load with stable docX#chunkY IDs.
  2. test-extraction-and-provenance: provenance-for-triple returns doc3#chunk0 and confidence 0.95 for (kgcreator usesComponent spacy).
  3. test-ollama-extractor-graceful-fallback: OllamaExtractor falls back cleanly to FakeExtractor when pointed at an closed port.
  4. test-entity-resolution-blocking-and-canonicalization: mark_l_watson and kg_creator auto-merge via owl:sameAs and rewrite triples, while m_watson enters the human review queue.
  5. test-rdfs-inference-and-shacl-validation: RDFS CONSTRUCT rules infer ex:SoftwareProject and ex:createdBy on ex:nsk; the clean graph passes SHACL and the corrupted sample fails with 4 violations.
  6. test-sparql-update-and-rollback: A valid SPARQL Update commits, while an update that violates SHACL rolls back to the snapshot.
  7. test-retrieval-token-budget-and-modes: Local and global GraphRAG respect token-budget and recover 2-hop facts.
  8. test-blocking-and-similarity-primitives: Unit checks on blocking-key and label-similarity.
  9. test-sparql-syntax-error-rollback: Malformed SPARQL Update syntax rolls back cleanly.
  10. test-evaluation-metrics-graphrag-outperforms-baseline: Confirms GraphRAG outperforms the baseline on both recall and groundedness with 1.0 citation accuracy.

Wrap Up

A knowledge graph is only as useful as its quality controls and its retrieval path. By combining PROV-O reification, entity resolution with owl:sameAs, RDFS CONSTRUCT inference, SHACL validation, and 2-hop provenance-boosted retrieval, kg-rag turns raw text extractions into a verifiable GraphRAG system in about 650 lines of Hy.

Exercises

  1. Confidence filtering in retrieval (small): Add a :min-confidence keyword parameter (default 0.90) to GraphRetriever.expand-neighborhood in retrieve.hy so that triples whose provenance confidence is below :min-confidence are excluded during 1-to-2 hop traversal. Write a pytest test showing that lowering or raising :min-confidence changes the retrieved triple set.
  2. RDF-star / named-graph provenance and LLM judge (open-ended): Refactor kg.hy to store each document’s assertions in a separate named graph within an rdflib.Dataset instead of reified rdf:Statement nodes, update shapes.ttl to validate the named graphs, and connect GraphRAGPipeline to a local Ollama model that grades answer faithfulness on a 1-to-5 rubric.

Further Reading

  • Aidan Hogan et al., Knowledge Graphs (Springer, 2021) for formal coverage of RDF, RDFS, OWL entailment, and SHACL validation shapes.
  • Tomaž Bratanič and Oskar Hane, Essential GraphRAG (Manning) for hybrid vector + graph retrieval patterns and community summarization.
  • W3C Shapes Constraint Language (SHACL) Recommendation (https://www.w3.org/TR/shacl/) and W3C PROV-O: The PROV Ontology (https://www.w3.org/TR/prov-o/).
  • The rdflib documentation (https://rdflib.readthedocs.io/) and pyshacl repository (https://github.com/RDFLib/pySHACL).