Macros and Metaprogramming: The Lisp Advantage

Most chapters in this book use Hy to call Python libraries with Lisp syntax. That is pleasant and productive, yet it leaves out the deepest reason to choose a Lisp. In Python, the language designers decide what control structures, binding forms, and literal notations exist. In Hy, you can add new syntax yourself and compile it down to standard Python abstract syntax trees (ASTs) before the program runs.

The code for this chapter is in the directory source_code_for_examples/macros.

What You Will Build

In this chapter we build four modules and a test suite that cover the full range of Hy metaprogramming:

  1. Core control and timing macros (core_macros.hy): unless, do-while, swap!, when-let, with-timer, timed, and mini-match, plus helpers that expand macros and translate Hy forms into Python source code.
  2. Macro hygiene examples (hygiene.hy): side-by-side broken and hygienic macros showing multiple evaluation and accidental variable capture, fixed with hy.gensym, def-gensyms, and defmacro!, plus an intentional anaphoric macro (aif).
  3. Reader macros (reader_macros.hy): #env for environment lookups, #json for compile-time JSON validation and literal emission, and #sql for compile-time SQL normalization and placeholder checking backed by SQLite.
  4. A compile-time HTML and SVG DSL (html_dsl.hy): html and svg macros that fold static tags, attributes, and text into string constants at compile time while escaping dynamic values at run time.

We also examine the ask and rule macros from the NSK knowledge graph engine in source_code_for_examples/nsk-neural-symbolic-graph to see how a domain-specific query language works in a larger system.

 1 Source text (.hy)
 2        |
 3        |  1. Hy Reader (hy.read / defreader)
 4        v
 5   Hy Models (Expression, Symbol, Keyword, List, Dict, String, ...)
 6        |
 7        |  2. Macro Expansion (defmacro / hy.macroexpand)
 8        v
 9 Expanded Hy Models
10        |
11        |  3. Hy Compiler (hy.compiler.hy-compile)
12        v
13 Python AST (ast.Module / ast.expr / ast.stmt)
14        |
15        |  4. Python Bytecode Compiler + VM
16        v
17 Running Program

Why It Matters

Functions operate on values at run time. Before a Python or Hy function starts executing, Python evaluates every argument expression from left to right. Because of that strict evaluation rule, a regular function cannot:

  • Delay or skip evaluating an argument (as if, when, and unless must do).
  • Evaluate an argument repeatedly in a loop (as while and do-while must do).
  • Bind a new local variable in the caller’s scope (as setv, for, and when-let must do).
  • Inspect the symbolic structure of an expression before it turns into a value (as swap!, ask, and html must do).

Macros run at compile time (Step 2 in the pipeline above). A macro is a regular Hy function whose arguments are unevaluated Hy models representing source code, and whose return value is a new Hy model that replaces the macro call. By the time Python executes the compiled bytecode, the macro call is gone; only the expanded Python AST remains.

The Hy Angle and the Python Equivalent

Suppose we want swap!, which exchanges two assignable targets such as local variables or list elements:

1 (swap! left right)
2 (swap! (get items 0) (get items 1))

In Python, you cannot write a function swap(a, b) that swaps two caller variables in place, because Python passes object references by value. Rebinding a and b inside swap only mutates the function’s local frame. A Python programmer writes tuple unpacking at every call site (left, right = right, left), or for timing blocks writes a context manager class or a decorator:

 1 # Python context manager for timing a block
 2 from contextlib import contextmanager
 3 import time
 4 
 5 @contextmanager
 6 def with_timer(label: str):
 7     start = time.perf_counter()
 8     try:
 9         yield
10     finally:
11         print(f"{label} took {time.perf_counter() - start:.4f} s")

Python’s with statement works well when you do not need the block’s return value as an expression. In Hy, a with-timer macro wraps any sequence of expressions, prints the elapsed time in a finally clause, and returns the value of the body directly:

1 (setv total (with-timer "sum-squares"
2               (sum (gfor x (range 10000) (* x x)))))

Even better, when we build the HTML DSL in html_dsl.hy, the html macro folds static tags and attributes into a single string literal at compile time. A Python template library must parse templates or concatenate tag strings at run time unless it adds a separate code-generation build step.

Run It

The project directory layout is:

 1 macros/
 2 ├── core_macros.hy    # unless, do-while, swap!, when-let, with-timer, timed, mini-match
 3 ├── hygiene.hy        # Broken vs hygienic macros (gensym, def-gensyms, defmacro!, aif)
 4 ├── reader_macros.hy  # Reader macros: #env, #sql (with SQLite runner), and #json
 5 ├── html_dsl.hy       # Compile-time HTML and SVG DSL with static string folding
 6 ├── demo.hy           # Guided tour script that prints macro expansions and outputs
 7 ├── test_macros.hy    # Pytest suite for core macros, hygiene, namespaces, and readers
 8 ├── test_html_dsl.hy  # Pytest suite for the HTML and SVG DSL
 9 ├── conftest.py       # Pytest hook to collect test_*.hy files
10 └── pyproject.toml    # Dependencies: hy>=1.3.1, hyrule>=1.0.0, pytest>=9.2.0

Sync the environment, run the guided tour, and run the test suite:

1 cd source_code_for_examples/macros
2 uv sync
3 uv run hy demo.hy
4 uv run pytest -q

Running uv run hy demo.hy produces this output:

 1 === 1. Core Macros and Expansion ===
 2 unless result: ['initialized']
 3 do-while steps: [0, 1, 2]
 4 after swap!: omega alpha
 5 when-let found version: 1.3.1
 6 HTTP 200 -> OK
 7 HTTP 404 -> Not Found
 8 HTTP 503 -> Server Error
 9 
10 Macroexpansion of (unless ready (boot)):
11   '(when (not ready) (boot))
12 Python AST via form-to-python:
13   [a, b] = [b, a]
14 
15 === 2. Hygiene: Multiple Evaluation and Capture ===
16 square-bad(bump):  result=16, calls=2 (evaluated twice!)
17 square-good(bump): result=16, calls=1
18 square-o!(bump):   result=16, calls=1
19 sum-steps-bad  with outer step=10: 3 (captured 'step'!)
20 sum-steps-good with outer step=10: 30 (outer step preserved)
21 anaphoric aif: Hello, Mark!
22 
23 === 3. Reader Macros (#env, #json, #sql) ===
24 #env DEMO_ENV_NAME: hy-macros
25 #env with default:  fallback-value
26 #json parsed at read time: {'name': 'hy', 'release': 1.3, 'lisp': True}
27 Compiled SqlQuery: SqlQuery(sql='SELECT name, year FROM languages WHERE lisp = 1 AND year >= ? ORDER BY year', params=(2000,))
28 Query results: [{'name': 'Clojure', 'year': 2007}, {'name': 'Hy', 'year': 2013}]
29 
30 === 4. Compile-Time HTML and SVG DSL ===
31 Rendered HTML (with automatic escaping):
32 <div class="card" data-kind="demo"><h2>Topic Summary</h2><p>Viewer: &lt;script&gt;alert(1)&lt;/script&gt;</p><ul><li>Macros</li><li>Reader Macros</li><li>DSLs</li></ul><hr></div>
33 
34 Macroexpansion of static + dynamic HTML:
35   '(hy.I.html_dsl.RawHtml (.join "" ["<div class=\"box\"><h1>Title</h1><p>" (hy.I.html_dsl.render-child user) "</p></div>"]))
36 
37 Rendered SVG:
38 <svg xmlns="http://www.w3.org/2000/svg" width="140" height="40" viewBox="0 0 140 40"><rect x="2" y="2" width="136" height="36" rx="6" fill="#eef"></rect><text x="70" y="25" text-anchor="middle">Hy 1.3.1</text></svg>

Look closely at the macroexpansion under Section 4 of the output. The input expression had nested :div, :h1, and :p vectors. The html macro merged all static opening tags, attributes, and headings into the single literal "<div class=\"box\"><h1>Title</h1><p>" before the program ran, leaving only (hy.I.html_dsl.render-child user) to run dynamically.

How It Works

1. Code as Models, defmacro, and Quasiquotation: core_macros.hy

When the Hy reader parses source text, it produces models from hy.models:

  • hy.models.Expression for parenthesized calls like (+ 1 2)
  • hy.models.Symbol for identifiers like ready or total
  • hy.models.Keyword for keywords like :else or :class
  • hy.models.List, hy.models.Dict, and hy.models.Tuple for [...], {...}, and #(...)
  • hy.models.String, hy.models.Integer, and hy.models.Float for scalar literals

Every model wraps the underlying Python type (Expression and List subclass tuple; Symbol and String subclass str) and records source line and column numbers so tracebacks point to the right file and line.

You can build models by calling constructors directly, or with quasiquotation (the backtick `):

  • 'form (quote) returns form unevaluated as a model.
  • `form (quasiquote) quotes form as a template, except where ~ or ~@ appears inside it.
  • ~expr (unquote) evaluates expr at macro-expansion time and inserts the resulting value into the template.
  • ~@seq (unquote-splice) evaluates seq and splices its elements flat into the surrounding sequence.

Here is the complete listing of core_macros.hy:

 1 ;; core_macros.hy -- Control-flow macros, timing macros, and AST inspection.
 2 
 3 (import ast)
 4 (import time)
 5 (import hy.compiler [hy-compile])
 6 (import hy.models [Keyword Symbol])
 7 
 8 
 9 (defmacro unless [test #* body]
10   "Evaluate body forms only when test is falsy."
11   `(when (not ~test)
12      ~@body))
13 
14 
15 (defmacro do-while [body-form condition]
16   "Execute body-form at least once, repeating while condition is truthy."
17   `(while True
18      ~body-form
19      (when (not ~condition)
20        (break))))
21 
22 
23 (defmacro swap! [left right]
24   "Swap two assignable places in place."
25   `(setv [~left ~right] [~right ~left]))
26 
27 
28 (defmacro when-let [binding #* body]
29   "Bind a symbol to an expression and run body only when the value is truthy.
30   Usage: (when-let [user (find-user uid)] (greet user))"
31   (when (!= (len binding) 2)
32     (raise (SyntaxError "when-let expects a [symbol expr] binding pair")))
33   (setv [sym expr] binding)
34   `(do
35      (setv ~sym ~expr)
36      (when ~sym
37        ~@body)))
38 
39 
40 (defmacro with-timer [label #* body]
41   "Run body, print elapsed seconds with label, and return the body's value."
42   (setv start (hy.gensym "start"))
43   `(do
44      (setv ~start (hy.I.time.perf-counter))
45      (try
46        (do ~@body)
47        (finally
48          (print ~label "took"
49                 (round (- (hy.I.time.perf-counter) ~start) 4)
50                 "s")))))
51 
52 
53 (defmacro timed [expr]
54   "Evaluate expr once and return #(result elapsed-seconds)."
55   (setv start (hy.gensym "start")
56         result (hy.gensym "result"))
57   `(do
58      (setv ~start (hy.I.time.perf-counter)
59            ~result ~expr)
60      #(~result (- (hy.I.time.perf-counter) ~start))))
61 
62 
63 (defmacro mini-match [expr #* clauses]
64   "Dispatch expr against [pattern result] clauses without re-evaluating expr.
65   Use :else or _ for the wildcard branch."
66   (setv target (hy.gensym "target")
67         branches [])
68   (for [clause clauses]
69     (when (!= (len clause) 2)
70       (raise (SyntaxError "mini-match clauses must be [pattern result] pairs")))
71     (setv [pat result] clause)
72     (if (or (= pat :else) (= pat '_))
73         (.extend branches [True result])
74         (.extend branches [`(= ~target ~pat) result])))
75   `(do
76      (setv ~target ~expr)
77      (cond ~@branches)))
78 
79 
80 (defn expand-once-str [form [module None]]
81   "Return the single-step macro expansion of form as Hy source text."
82   (hy.repr (if (is module None)
83                (hy.macroexpand-1 form)
84                (hy.macroexpand-1 form module))))
85 
86 
87 (defn form-to-python [form [module-name "__main__"]]
88   "Compile a quoted Hy form to formatted Python source code (like hy2py)."
89   (setv expanded (hy.macroexpand form)
90         py-ast (hy-compile expanded module-name)
91         text (.strip (ast.unparse py-ast)))
92   (.removeprefix text "import hy\n"))

Notice three practical patterns in core_macros.hy:

  1. Compile-time validation: when-let and mini-match inspect binding and clause during macro expansion and raise SyntaxError immediately if the caller wrote the wrong shape. Errors surface when the module compiles, not hours later when a cold code branch executes.
  2. Self-contained imports with hy.I: with-timer calls (hy.I.time.perf-counter) instead of (time.perf-counter). The hy.I reader macro compiles to hy.I.time.perf_counter(), which imports time on demand. That way a file that requires with-timer does not crash if it forgot to write (import time).
  3. Splicing pairs into cond: mini-match collects [condition-form result-form] pairs into a flat Python list branches at compile time and splices them into (cond ~@branches).

2. Expansion and Debugging: hy.macroexpand-1, hy2py, and hy --spy

When a macro produces a surprising error, printing its expansion is the fastest way to see what happened. Hy gives you four tools:

  1. (hy.macroexpand-1 form): expands the outer macro of form by one step and returns the resulting Hy model. Wrap it in (hy.repr ...) to print readable Hy syntax.
  2. (hy.macroexpand form): repeatedly expands outer macros until the head is no longer a macro.
  3. hy2py: a command-line tool that compiles a .hy file to a Python AST and prints the generated Python code. Running uv run hy2py core_macros.hy shows how Hy compiles defmacro itself into a Python function registered with hy.macros.macro:
1 def _hy_anon_1(test, *body):
2     """Evaluate body forms only when test is falsy."""
3     return hy.models.Expression([
4         hy.models.Symbol('when', from_parser=True),
5         hy.models.Expression([hy.models.Symbol('not', from_parser=True), test]),
6         *(body or [])
7     ])
8 hy.macros.macro('unless')(_hy_anon_1)
  1. hy --spy: starting the REPL with uv run hy --spy prints the generated Python code before evaluating every REPL input. This lets you watch each macro expand and compile to Python statements in real time.

In our code, form-to-python runs the same pipeline programmatically by calling (hy-compile (hy.macroexpand form) module-name) followed by (ast.unparse py-ast). That lets unit tests assert on the exact Python code a macro generates.

3. Macro Hygiene: hygiene.hy

A macro is hygienic when its expansion never evaluates caller expressions more times than expected and never collides with variable names in the caller’s scope. Lisp macros can fail both ways if you write naive templates.

Here is hygiene.hy, which contrasts the broken versions with their hygienic replacements:

 1 ;; hygiene.hy -- Macro hygiene: multiple evaluation, variable capture, and fixes.
 2 
 3 (require hyrule [defmacro! def-gensyms])
 4 
 5 
 6 ;; ---------- 1. Multiple evaluation ----------
 7 
 8 (defmacro square-bad [x]
 9   "BROKEN: evaluates x twice when x has side effects."
10   `(* ~x ~x))
11 
12 
13 (defmacro square-good [x]
14   "HYGIENIC: binds x to a fresh gensym so x is evaluated once."
15   (setv val (hy.gensym "val"))
16   `(do
17      (setv ~val ~x)
18      (* ~val ~val)))
19 
20 
21 (defmacro! square-o! [o!x]
22   "HYGIENIC (via hyrule defmacro!): o!x evaluates once and binds to g!x."
23   `(* ~g!x ~g!x))
24 
25 
26 ;; ---------- 2. Accidental variable capture ----------
27 
28 (defmacro sum-steps-bad [n #* body]
29   "BROKEN: leaks loop variable 'step' and accumulator 'total' into caller scope."
30   `(do
31      (setv total 0)
32      (for [step (range ~n)]
33        (+= total (do ~@body)))
34      total))
35 
36 
37 (defmacro sum-steps-good [n #* body]
38   "HYGIENIC: uses def-gensyms so caller variables named step or total are safe."
39   (def-gensyms total step limit)
40   `(do
41      (setv ~limit ~n
42            ~total 0)
43      (for [~step (range ~limit)]
44        (+= ~total (do ~@body)))
45      ~total))
46 
47 
48 ;; ---------- 3. Intentional capture (anaphoric macro) ----------
49 
50 (defmacro aif [test-form then-form [else-form None]]
51   "Anaphoric if: intentionally binds 'it' to the value of test-form."
52   `(do
53      (setv it ~test-form)
54      (if it
55          ~then-form
56          ~else-form)))

Let us examine all three cases:

  • Multiple evaluation (square-bad vs square-good and square-o!): (square-bad (bump)) expands to (* (bump) (bump)), calling bump twice. square-good calls (hy.gensym "val") at macro-expansion time to generate a unique symbol (such as _val_1234), evaluates ~x once into ~val, and multiplies ~val by itself. The hyrule library provides defmacro!, inspired by Doug Hoyte’s Let Over Lambda. Any parameter prefixed with o! (once-only, such as o!x) is automatically evaluated once before the macro body and bound to the matching g! gensym (g!x).
  • Accidental variable capture (sum-steps-bad vs sum-steps-good): Because Python and Hy share function-level variable scope, introducing a plain symbol like step or total inside a macro expansion writes directly into the caller’s local scope. When the caller sets (setv step 10) and runs (sum-steps-bad 3 step), the macro’s for [step (range 3)] overwrites the caller’s step with 0, 1, and 2, returning 3 instead of 30. sum-steps-good uses hyrule’s (def-gensyms total step limit) to create three fresh symbols at once.
  • Intentional capture (aif): Sometimes you want a macro to introduce a documented symbol into the caller’s scope. An anaphoric macro like aif binds the pronoun it to the result of test-form so the caller can write (aif (find-user uid) it.email "none"). Because it is part of the macro’s contract, this capture is deliberate rather than a bug.

4. Macro Namespaces: require, export, hy.R, and local-macros

In Python, import runs at run time. Because macros must run while a module is still compiling, Hy separates macro loading from import:

  • (require module [macro-a macro-b]): loads module at compile time and registers macro-a and macro-b in the current module’s macro table.
  • (require module :as m): registers macros under a prefix so you can call (m.unless ready (boot)).
  • (require module :readers [sql env]): loads reader macros from module.
  • hy.R: the compile-time equivalent of hy.I. Writing (hy.R.core_macros.unless False "ok") requires core_macros on the fly and invokes unless without a top-level require statement.
  • Controlling exports with (export ...): by default, (require mod *) brings in every macro defined in mod (and not macros that mod itself required from elsewhere). You can restrict what import * and require * expose using (export :objects [fn1] :macros [mac1]), which compiles to __all__ and _hy_export_macros.

5. Reader Macros: reader_macros.hy

Ordinary macros transform Hy models after the reader has already parsed the source text. Reader macros hook into Step 1 of the pipeline: they run inside the parser when the reader sees #tag.

Inside (defreader name ...), Hy binds the active reader instance to the symbol &reader. Calling (.parse-one-form &reader) reads and returns the next complete Hy form from the input stream.

Here is reader_macros.hy, which defines #sql, #env, and #json:

  1 ;; reader_macros.hy -- Reader macros (#env, #sql, #json) using defreader.
  2 
  3 (import dataclasses [dataclass])
  4 (import json)
  5 (import re)
  6 (import sqlite3)
  7 (import hy.models [Dict List String])
  8 
  9 
 10 (defclass [(dataclass :frozen True)] SqlQuery []
 11   "A parameterized SQL query checked at read time."
 12   #^ str sql
 13   #^ tuple params)
 14 
 15 
 16 (defn normalize-sql [raw-sql]
 17   "Collapse internal whitespace and strip leading/trailing spaces."
 18   (setv cleaned (.strip (re.sub r"\s+" " " (str raw-sql))))
 19   (when (not cleaned)
 20     (raise (SyntaxError "#sql query string cannot be empty")))
 21   cleaned)
 22 
 23 
 24 (defn compile-sql-form [form]
 25   "Validate a #sql form at read time and return code constructing SqlQuery."
 26   (cond
 27     (isinstance form String)
 28       (do
 29         (setv sql-text (normalize-sql form)
 30               placeholders (.count sql-text "?"))
 31         (when (!= placeholders 0)
 32           (raise (SyntaxError
 33                    f"#sql string has {placeholders} '?' placeholders but no params list")))
 34         `(hy.I.reader_macros.SqlQuery ~sql-text #()))
 35     (and (isinstance form List) (>= (len form) 1) (isinstance (get form 0) String))
 36       (do
 37         (setv sql-text (normalize-sql (get form 0))
 38               params (list (cut form 1 None))
 39               placeholders (.count sql-text "?"))
 40         (when (!= placeholders (len params))
 41           (raise (SyntaxError
 42                    f"#sql has {placeholders} '?' placeholders but {(len params)} params")))
 43         `(hy.I.reader_macros.SqlQuery ~sql-text #(~@params)))
 44     True
 45       (raise (SyntaxError
 46                "#sql expects a string literal or [\"SQL ...\" param1 param2 ...]"))))
 47 
 48 
 49 (defreader sql
 50   (setv form (.parse-one-form &reader))
 51   (hy.I.reader_macros.compile-sql-form form))
 52 
 53 
 54 (defn compile-env-form [form]
 55   "Expand #env \"VAR\" or #env [\"VAR\" \"default\"] into os.environ.get."
 56   (cond
 57     (isinstance form String)
 58       `(hy.I.os.environ.get ~(str form))
 59     (and (isinstance form List) (= (len form) 2) (isinstance (get form 0) String))
 60       `(hy.I.os.environ.get ~(str (get form 0)) ~(get form 1))
 61     True
 62       (raise (SyntaxError
 63                "#env expects \"VAR\" or [\"VAR\" default-expr]"))))
 64 
 65 
 66 (defreader env
 67   (setv form (.parse-one-form &reader))
 68   (hy.I.reader_macros.compile-env-form form))
 69 
 70 
 71 (defn json-to-model [val]
 72   "Convert parsed JSON values into Hy models for zero-cost runtime literals."
 73   (cond
 74     (isinstance val dict)
 75       (do
 76         (setv pairs [])
 77         (for [[k v] (.items val)]
 78           (.extend pairs [(String k) (json-to-model v)]))
 79         (Dict pairs))
 80     (isinstance val list)
 81       (List (gfor item val (json-to-model item)))
 82     (isinstance val str)
 83       (String val)
 84     True val))
 85 
 86 
 87 (defn compile-json-form [form]
 88   "Parse a JSON string literal at read time and emit a Hy data literal."
 89   (when (not (isinstance form String))
 90     (raise (SyntaxError "#json expects a string literal")))
 91   (try
 92     (setv parsed (json.loads (str form)))
 93     (except [err json.JSONDecodeError]
 94       (raise (SyntaxError f"Invalid #json literal: {err.msg}"))))
 95   (json-to-model parsed))
 96 
 97 
 98 (defreader json
 99   (setv form (.parse-one-form &reader))
100   (hy.I.reader_macros.compile-json-form form))
101 
102 
103 (defn execute-sql [conn query]
104   "Execute a SqlQuery against a sqlite3 connection and return rows as dicts."
105   (when (not (isinstance query SqlQuery))
106     (raise (TypeError "execute-sql expects a SqlQuery value")))
107   (setv cursor (.execute conn query.sql query.params)
108         columns (if cursor.description
109                     (lfor col cursor.description (get col 0))
110                     []))
111   (lfor row (.fetchall cursor)
112     (dict (zip columns row))))

Notice what #sql and #json accomplish at read time:

  • #sql collapses multi-line whitespace into a single clean SQL string and counts the ? parameter placeholders before the program runs. If a query has two ? placeholders and you pass one parameter expression, #sql raises SyntaxError at compile time.
  • #json calls json.loads at read time and converts the parsed structure into hy.models.Dict and hy.models.List nodes via json-to-model. Malformed JSON fails at compile time, and valid JSON compiles directly to a native Python dict or list literal with zero json.loads call at run time.

6. Case Study: NSK’s ask and rule Macros

Before we build our HTML DSL, let us look at how the NSK knowledge graph chapter (source_code_for_examples/nsk-neural-symbolic-graph) uses macros to embed a Datalog query language inside Hy.

In NSK, a query looks like this:

1 (ask [?author ?lang]
2   [?author :wrote :nsk]
3   [?author ~:codes-in ?lang]
4   :limit 5)

Notice why ask must be a macro rather than a function:

  1. ?author and ?lang are unbound symbols. If ask were a function, Python would try to look up a local variable named hyx_Xquestion_markXauthor and raise NameError.
  2. ~:codes-in uses the unquote prefix ~ to mark a neural predicate. The Hy reader parses ~:codes-in as the two-element expression (unquote :codes-in). Outside a quasiquote or macro that intercepts it, evaluating (unquote :codes-in) is a compile error.
  3. Ordinary Hy variables and function calls inside a pattern (such as [?who :codes-in target-language]) must still be evaluated normally at run time!

NSK solves all three requirements in 30 lines across nsk/terms.hy and nsk/query.hy:

 1 ;; From nsk/terms.hy
 2 (defn var-symbol? [form]
 3   (and (isinstance form Symbol) (> (len form) 1) (.startswith form "?")))
 4 
 5 (defn unquote? [form]
 6   (and (isinstance form Expression) (= (len form) 2)
 7        (= (get form 0) 'unquote)))
 8 
 9 (defn compile-term [form]
10   "Macro helper. Like to-term, but it returns code: ?x and ~p are rewritten and
11   everything else is left for Hy to evaluate, so a pattern can mention ordinary
12   variables and function calls."
13   (cond
14     (var-symbol? form) `(hy.I.nsk/terms.Var ~(cut (str form) 1 None))
15     (unquote? form) `(hy.I.nsk/terms.Neural ~(compile-term (get form 1)))
16     ;; A bare keyword in a call would be read as a keyword argument, so quote it.
17     (isinstance form Keyword) `(quote ~form)
18     True form))
19 
20 (defn compile-pattern [form]
21   (when (not (and (isinstance form List) (= (len form) 3)))
22     (raise (ValueError f"a pattern needs exactly three terms: {(hy.repr form)}")))
23   `#(~@(map compile-term form)))

And in nsk/query.hy:

 1 ;; From nsk/query.hy
 2 (defn split-options [forms]
 3   "Macro helper: separate the [s p o] patterns from trailing :key value pairs."
 4   (setv patterns (lfor form forms :if (isinstance form List) form))
 5   #(patterns (cut forms (len patterns) None)))
 6 
 7 (defmacro ask [select #* body]
 8   (setv [patterns options] (hy.I.nsk/query.split-options body))
 9   `(hy.I.nsk/query.run-query
10      [~@(map hy.I.nsk/terms.compile-pattern patterns)]
11      [~@(map hy.I.nsk/terms.compile-term select)]
12      ~@options))

Look at the fourth branch of compile-term: True form. When a term in a pattern is not a ?variable, not a ~predicate, and not a :keyword, compile-term returns form unchanged! That single line is what lets NSK patterns mix logic variables (?who) with ordinary Python variables (target-language) and function calls seamlessly.

Finally, nsk/__init__.hy includes (require nsk.query [ask rule]) alongside its (import ...) forms so a user of the package can write (require nsk [ask rule]) from the top-level package namespace.

7. Project: A Compile-Time HTML and SVG DSL (html_dsl.hy)

Now we apply the same selective compilation technique to build html_dsl.hy. We want to write HTML and SVG trees as nested Hy vectors starting with a keyword tag name:

1 (html
2   [:div {:class "card" :data-kind "demo"}
3    [:h2 "Topic Summary"]
4    [:p "Viewer: " user]
5    [:ul (lfor item items (html [:li item]))]
6    [:hr]])

If we wrote html as an ordinary function, every tag vector and attribute dict would be allocated at run time on every page render. By writing html as a macro that walks the nested structure at compile time, we get three advantages:

  1. Compile-time string folding: static tags, static attribute dicts, and literal strings or numbers are escaped and concatenated into plain string constants during macro expansion.
  2. Compile-time validation: void HTML elements (<br>, <hr>, <img>, <input>) are checked at compile time; giving children to [:br "oops"] raises SyntaxError before the program runs.
  3. Automatic runtime escaping: any child or attribute value that is not a compile-time literal is wrapped in (hy.I.html_dsl.render-child expr) or (hy.I.html_dsl.render-attr name expr), which escapes <, >, &, and quotes unless the value is already a RawHtml instance produced by (html ...) or (raw ...).

Here is the complete listing of html_dsl.hy:

  1 ;; html_dsl.hy -- Compile-time HTML and SVG DSL with static string folding.
  2 
  3 (import html :as py-html)
  4 (import hy.models [Dict Expression Float Integer Keyword List String])
  5 
  6 
  7 (setv VOID-TAGS #{"area" "base" "br" "col" "embed" "hr" "img"
  8                   "input" "link" "meta" "source" "track" "wbr"})
  9 
 10 
 11 (defclass RawHtml []
 12   "Wrapper for pre-rendered HTML/SVG strings that must not be escaped again."
 13   (defn __init__ [self text]
 14     (setv self.text (str text)))
 15 
 16   (defn __str__ [self]
 17     self.text)
 18 
 19   (defn __repr__ [self]
 20     f"(RawHtml {(hy.repr self.text)})")
 21 
 22   (defn __eq__ [self other]
 23     (cond
 24       (isinstance other RawHtml) (= self.text other.text)
 25       (isinstance other str) (= self.text other)
 26       True False)))
 27 
 28 
 29 (defn raw [text]
 30   "Mark a string as trusted HTML so render-child does not escape it."
 31   (RawHtml text))
 32 
 33 
 34 (defn scalar-to-str [value]
 35   "Convert Hy model scalars (Integer, Float, String) or Python values to str."
 36   (cond
 37     (isinstance value Integer) (str (int value))
 38     (isinstance value Float) (str (float value))
 39     True (str value)))
 40 
 41 
 42 (defn escape-html [value]
 43   "Escape special HTML characters (&, <, >, quotes) in value."
 44   (py-html.escape (scalar-to-str value) :quote True))
 45 
 46 
 47 (defn attr-name [key]
 48   "Convert a keyword or string key into an HTML attribute name."
 49   (if (isinstance key Keyword)
 50       key.name
 51       (str key)))
 52 
 53 
 54 (defn render-attr [key value]
 55   "Render one HTML attribute at run time.
 56   Boolean True renders a bare attribute; False or None omits the attribute."
 57   (setv name (attr-name key))
 58   (cond
 59     (or (is value None) (is value False)) ""
 60     (is value True) f" {name}"
 61     True f" {name}=\"{(escape-html value)}\""))
 62 
 63 
 64 (defn render-child [value]
 65   "Render a dynamic child value at run time with automatic HTML escaping."
 66   (cond
 67     (or (is value None) (is value False)) ""
 68     (isinstance value RawHtml) value.text
 69     (isinstance value #(list tuple))
 70       (.join "" (gfor item value (render-child item)))
 71     (hasattr value "__iter__")
 72       (if (isinstance value #(str bytes))
 73           (escape-html value)
 74           (.join "" (gfor item value (render-child item))))
 75     True (escape-html value)))
 76 
 77 
 78 (defn literal-scalar? [form]
 79   (isinstance form #(String Integer Float)))
 80 
 81 
 82 (defn tag-form? [form]
 83   "True when form is [:tag ...] or (:tag ...) with a Keyword head."
 84   (and (isinstance form #(List Expression))
 85        (> (len form) 0)
 86        (isinstance (get form 0) Keyword)))
 87 
 88 
 89 (defn compile-attrs [attr-dict]
 90   "Compile a Hy Dict model of attributes into a list of segments."
 91   (setv items (list attr-dict))
 92   (when (% (len items) 2)
 93     (raise (SyntaxError "Attribute dictionary must have key/value pairs")))
 94   (setv segments [])
 95   (for [idx (range 0 (len items) 2)]
 96     (setv k (get items idx)
 97           v (get items (+ idx 1))
 98           name (if (isinstance k #(Keyword String)) (attr-name k) None))
 99     (cond
100       (and name (literal-scalar? v))
101         (.append segments f" {name}=\"{(escape-html v)}\"")
102       (and name (= v 'True))
103         (.append segments f" {name}")
104       (and name (or (= v 'False) (= v 'None)))
105         None
106       name
107         (.append segments `(hy.I.html_dsl.render-attr ~(String name) ~v))
108       True
109         (.append segments `(hy.I.html_dsl.render-attr ~k ~v))))
110   segments)
111 
112 
113 (defn compile-node [form]
114   "Compile one DSL node into a list of static strings and runtime expressions."
115   (cond
116     (tag-form? form)
117       (do
118         (setv tag (. (get form 0) name)
119               rest-forms (list (cut form 1 None))
120               has-attrs (and rest-forms (isinstance (get rest-forms 0) Dict))
121               attr-form (if has-attrs (get rest-forms 0) None)
122               children (if has-attrs (cut rest-forms 1 None) rest-forms)
123               segments [f"<{tag}"])
124         (when (is-not attr-form None)
125           (.extend segments (compile-attrs attr-form)))
126         (.append segments ">")
127         (if (in tag VOID-TAGS)
128             (when children
129               (raise (SyntaxError f"Void HTML element <{tag}> cannot have children")))
130             (do
131               (for [child children]
132                 (.extend segments (compile-node child)))
133               (.append segments f"</{tag}>")))
134         segments)
135     (literal-scalar? form)
136       [(escape-html form)]
137     True
138       [`(hy.I.html_dsl.render-child ~form)]))
139 
140 
141 (defn merge-segments [segments]
142   "Coalesce adjacent compile-time string segments into single String models."
143   (setv merged [])
144   (for [seg segments]
145     (if (isinstance seg str)
146         (if (and merged (isinstance (get merged -1) String))
147             (setv (get merged -1) (String (+ (str (get merged -1)) seg)))
148             (.append merged (String seg)))
149         (.append merged seg)))
150   merged)
151 
152 
153 (defn build-html-expr [nodes]
154   "Compile a sequence of nodes into a RawHtml construction form."
155   (setv raw-segments [])
156   (for [node nodes]
157     (.extend raw-segments (compile-node node)))
158   (setv merged (merge-segments raw-segments))
159   (cond
160     (not merged)
161       `(hy.I.html_dsl.RawHtml "")
162     (and (= (len merged) 1) (isinstance (get merged 0) String))
163       `(hy.I.html_dsl.RawHtml ~(get merged 0))
164     True
165       `(hy.I.html_dsl.RawHtml (.join "" [~@merged]))))
166 
167 
168 (defmacro html [#* nodes]
169   "Compile nested [:tag {:attr val} ...] forms into a RawHtml value.
170   Static tags, attributes, and text are folded into string literals at
171   compile time; dynamic expressions are escaped at run time."
172   (hy.I.html_dsl.build-html-expr nodes))
173 
174 
175 (defmacro svg [attrs #* children]
176   "Compile an inline <svg> element with the default SVG xmlns attribute."
177   (when (not (isinstance attrs Dict))
178     (raise (SyntaxError "svg expects an attribute dict as its first argument")))
179   (setv items (list attrs)
180         keys (lfor i (range 0 (len items) 2) (attr-name (get items i)))
181         full-items (if (in "xmlns" keys)
182                        items
183                        [(Keyword "xmlns") (String "http://www.w3.org/2000/svg") #* items])
184         svg-node (List [(Keyword "svg") (Dict full-items) #* children]))
185   (hy.I.html_dsl.build-html-expr [svg-node]))

Two implementation details in html_dsl.hy are worth highlighting for your own macros:

  1. scalar-to-str and Hy model strings: in Hy 1.3.1, calling (str form) on a hy.models.Integer returns "hy.models.Integer(140)" rather than "140". Converting Integer with (int value) and Float with (float value) before calling str produces the clean numeric text "140".
  2. Passing attribute names as strings: in compile-attrs, when an attribute key is a keyword such as :disabled, splicing ~k directly into (render-attr ~k ~v) would emit (render-attr :disabled is-disabled), which Hy compiles as a keyword argument render_attr(disabled=is_disabled)! Converting name to (String name) at compile time emits (render-attr "disabled" is-disabled) as a positional string argument.

8. Macros vs Functions vs Decorators vs __init_subclass__

A good rule in Lisp is to use the least powerful tool that solves the problem cleanly. Functions are easier to pass around as values, inspect in stack traces, and call from Python.

Tool Runs when Receives Best for Avoid when
Function (defn) Run time Evaluated values 90% of program logic, data transformations, composable callbacks You need custom evaluation order, new bindings, or compile-time syntax
Decorator / with Definition or run time Function, class, or runtime context Wrapping functions (caching, auth, retries) or resource cleanup You need to rewrite expressions inside a function body
__init_subclass__ / dataclasses Class creation time Subclass object Registering plugins, validating class attributes The problem is not naturally a class hierarchy
Macro (defmacro) Compile time Unevaluated Hy models Control flow (unless, do-while), binding forms (when-let), embedded DSLs (ask, html), compile-time folding A plain function or context manager would be just as clear
Reader macro (defreader) Parse time Active &reader stream Checked literals (#sql, #json, #env), terse domain notation Ordinary macro syntax (sql ...) is already readable

Also remember Python interoperability: a Python module importing your Hy module can call any defn function or defclass class directly, whereas a macro only exists at Hy compile time (unless a Python caller invokes hy.eval). When you publish a library for both Hy and Python callers, put the runtime engine in ordinary functions (like NSK’s run-query or reader_macros.hy’s execute-sql) and make the macro a thin compile-time layer on top.

Tests

The example includes two pytest files: test_macros.hy and test_html_dsl.hy. Run them with:

1 uv run pytest -q
1 ....................                                                     [100%]
2 20 passed in 0.21s

Here are the key tests from test_macros.hy, which check single-step expansions with expand-once-str, golden Python translations with form-to-python, hygiene guarantees, hy.R namespace resolution, and compile-time reader errors:

 1 ;; From test_macros.hy
 2 
 3 (defn test-unless-expansion-and-behavior []
 4   (assert (= (expand-once-str '(unless ready (launch)))
 5              "'(when (not ready) (launch))"))
 6   (setv log [])
 7   (unless False
 8     (.append log "first")
 9     (.append log "second"))
10   (unless True
11     (.append log "never"))
12   (assert (= log ["first" "second"])))
13 
14 
15 (defn test-hy2py-golden-translation []
16   (setv py-swap (form-to-python '(swap! left right)))
17   (assert (= (.strip py-swap) "[left, right] = [right, left]"))
18 
19   (setv py-unless (form-to-python '(unless done (step))))
20   (assert (in "not done" py-unless))
21   (assert (in "step()" py-unless)))
22 
23 
24 (defn test-multiple-evaluation-hygiene []
25   (setv counter 0)
26   (defn next-int []
27     (nonlocal counter)
28     (+= counter 1)
29     3)
30 
31   (setv bad (square-bad (next-int)))
32   (assert (= bad 9))
33   (assert (= counter 2))
34 
35   (setv counter 0)
36   (setv good (square-good (next-int)))
37   (assert (= good 9))
38   (assert (= counter 1))
39 
40   (setv counter 0)
41   (setv good-o (square-o! (next-int)))
42   (assert (= good-o 9))
43   (assert (= counter 1)))
44 
45 
46 (defn test-variable-capture-hygiene []
47   (setv step 10
48         total 100)
49   ;; Broken macro overwrites caller's 'step' with 0, 1, 2
50   (assert (= (sum-steps-bad 3 step) 3))
51 
52   (setv step 10
53         total 100)
54   ;; Hygienic macro preserves caller's 'step' (10 + 10 + 10 = 30) and 'total'
55   (assert (= (sum-steps-good 3 step) 30))
56   (assert (= step 10))
57   (assert (= total 100)))
58 
59 
60 (defn test-reader-macro-sql []
61   (setv conn (sqlite3.connect ":memory:"))
62   (.execute conn "CREATE TABLE books (title TEXT, pages INT)")
63   (.executemany conn
64     "INSERT INTO books VALUES (?, ?)"
65     [#("On Lisp" 432)
66      #("Practical Common Lisp" 499)
67      #("Brief Guide" 90)])
68 
69   (setv min-pages 200
70         q #sql ["  SELECT title, pages
71                    FROM   books
72                    WHERE  pages >= ?
73                    ORDER BY pages " min-pages])
74   (assert (isinstance q SqlQuery))
75   (assert (= q.sql "SELECT title, pages FROM books WHERE pages >= ? ORDER BY pages"))
76   (assert (= q.params #(200)))
77 
78   (setv rows (execute-sql conn q))
79   (assert (= (len rows) 2))
80   (assert (= (get (get rows 0) "title") "On Lisp"))
81   (.close conn)
82 
83   ;; Mismatched '?' placeholder count fails at compile/read time
84   (with [(pytest.raises Exception)]
85     (hy.eval (hy.read "(do (require reader_macros :readers [sql]) #sql [\"SELECT ? AND ?\" 1])"))))

And here are the core tests from test_html_dsl.hy, which verify that static HTML trees fold into a single literal string at compile time, untrusted input is escaped, and void elements reject children:

 1 ;; From test_html_dsl.hy
 2 
 3 (defn test-static-html-folds-at-compile-time []
 4   (setv exp (hy.repr (hy.macroexpand-1 '(html [:div {:class "card"} [:h1 "Title"] [:hr]]))))
 5   ;; Entire static tree folds into a single string literal with no .join call
 6   (assert (in "\"<div class=\\\"card\\\"><h1>Title</h1><hr></div>\"" exp))
 7   (assert (not-in ".join" exp))
 8 
 9   (setv out (html [:div {:class "card"} [:h1 "Title"] [:hr]]))
10   (assert (isinstance out RawHtml))
11   (assert (= (str out) "<div class=\"card\"><h1>Title</h1><hr></div>")))
12 
13 
14 (defn test-dynamic-escaping-and-raw-bypass []
15   (setv untrusted "<img src=x onerror=\"alert(1)\">"
16         safe-badge (raw "<strong>verified</strong>"))
17   (setv rendered
18     (str (html [:p {:title untrusted}
19                 "User: " untrusted " " safe-badge])))
20   (assert (= rendered
21              (+ "<p title=\"&lt;img src=x onerror=&quot;alert(1)&quot;&gt;\">"
22                 "User: &lt;img src=x onerror=&quot;alert(1)&quot;&gt; "
23                 "<strong>verified</strong></p>"))))
24 
25 
26 (defn test-void-tag-rejects-children-at-compile-time []
27   (with [(pytest.raises Exception)]
28     (hy.eval (hy.read "(do (require html_dsl [html]) (html [:br \"oops\"]))"))))

Wrap Up

Macros are the dividing line between using Lisp syntax and programming in Lisp. With defmacro, quasiquotation (~ and ~@), hy.gensym (or hyrule’s def-gensyms and defmacro!), and defreader, you can move validation, boilerplate, and constant folding from run time to compile time. At the same time, keeping your macros thin and delegating runtime work to ordinary Hy functions ensures that your code remains easy to test with pytest, inspect with hy.macroexpand-1 and hy2py, and call from Python.

Exercises

  1. Small exercise (repeat-until and with-timer-collector): Add a (repeat-until body-form condition) macro to core_macros.hy that loops until condition becomes truthy (the opposite of do-while). Then write a (with-timer-into dict-expr key-expr #* body) macro that records the elapsed seconds into (get dict-expr key-expr) instead of printing to standard output. Be sure to evaluate dict-expr and key-expr only once using hy.gensym.
  2. Open-ended exercise (CSS selector shorthand and #regex reader macro): Extend compile-node in html_dsl.hy so tag keywords support CSS-style #id and .class suffixes at compile time: writing [:div#main.card.featured "Hello"] should compile to "<div id=\"main\" class=\"card featured\">Hello</div>", merging with any :class key in the attribute dictionary. Next, add a #re reader macro in reader_macros.hy that takes a string literal, validates the regular expression syntax at compile time with re.compile, and expands to (hy.I.re.compile pattern).

Further Reading

  • Hy 1.3.1 Macros Manual: official documentation on defmacro, defreader, hy.gensym, require, and export.
  • Hyrule Documentation: Hy’s utility library, including defmacro!, def-gensyms, and macroexpand-all.
  • Paul Graham, On Lisp (Prentice Hall, 1993): classic treatment of bottom-up design, macro pitfalls, and embedded languages.
  • Peter Seibel, Practical Common Lisp (Apress, 2005), Chapters 7, 8, and 30: macro fundamentals and a compile-time HTML generator.
  • Doug Hoyte, Let Over Lambda (2008): reader macros, once-only o!/g! gensyms, and compile-time computation.