Macros and Metaprogramming: The Lisp Advantage
Most chapters in this book use Hy to call Python libraries with Lisp syntax. That is pleasant and productive, yet it leaves out the deepest reason to choose a Lisp. In Python, the language designers decide what control structures, binding forms, and literal notations exist. In Hy, you can add new syntax yourself and compile it down to standard Python abstract syntax trees (ASTs) before the program runs.
The code for this chapter is in the directory source_code_for_examples/macros.
What You Will Build
In this chapter we build four modules and a test suite that cover the full range of Hy metaprogramming:
- Core control and timing macros (
core_macros.hy):unless,do-while,swap!,when-let,with-timer,timed, andmini-match, plus helpers that expand macros and translate Hy forms into Python source code. - Macro hygiene examples (
hygiene.hy): side-by-side broken and hygienic macros showing multiple evaluation and accidental variable capture, fixed withhy.gensym,def-gensyms, anddefmacro!, plus an intentional anaphoric macro (aif). - Reader macros (
reader_macros.hy):#envfor environment lookups,#jsonfor compile-time JSON validation and literal emission, and#sqlfor compile-time SQL normalization and placeholder checking backed by SQLite. - A compile-time HTML and SVG DSL (
html_dsl.hy):htmlandsvgmacros that fold static tags, attributes, and text into string constants at compile time while escaping dynamic values at run time.
We also examine the ask and rule macros from the NSK knowledge graph engine in source_code_for_examples/nsk-neural-symbolic-graph to see how a domain-specific query language works in a larger system.
1 Source text (.hy)
2 |
3 | 1. Hy Reader (hy.read / defreader)
4 v
5 Hy Models (Expression, Symbol, Keyword, List, Dict, String, ...)
6 |
7 | 2. Macro Expansion (defmacro / hy.macroexpand)
8 v
9 Expanded Hy Models
10 |
11 | 3. Hy Compiler (hy.compiler.hy-compile)
12 v
13 Python AST (ast.Module / ast.expr / ast.stmt)
14 |
15 | 4. Python Bytecode Compiler + VM
16 v
17 Running Program
Why It Matters
Functions operate on values at run time. Before a Python or Hy function starts executing, Python evaluates every argument expression from left to right. Because of that strict evaluation rule, a regular function cannot:
- Delay or skip evaluating an argument (as
if,when, andunlessmust do). - Evaluate an argument repeatedly in a loop (as
whileanddo-whilemust do). - Bind a new local variable in the caller’s scope (as
setv,for, andwhen-letmust do). - Inspect the symbolic structure of an expression before it turns into a value (as
swap!,ask, andhtmlmust do).
Macros run at compile time (Step 2 in the pipeline above). A macro is a regular Hy function whose arguments are unevaluated Hy models representing source code, and whose return value is a new Hy model that replaces the macro call. By the time Python executes the compiled bytecode, the macro call is gone; only the expanded Python AST remains.
The Hy Angle and the Python Equivalent
Suppose we want swap!, which exchanges two assignable targets such as local variables or list elements:
1 (swap! left right)
2 (swap! (get items 0) (get items 1))
In Python, you cannot write a function swap(a, b) that swaps two caller variables in place, because Python passes object references by value. Rebinding a and b inside swap only mutates the function’s local frame. A Python programmer writes tuple unpacking at every call site (left, right = right, left), or for timing blocks writes a context manager class or a decorator:
1 # Python context manager for timing a block
2 from contextlib import contextmanager
3 import time
4
5 @contextmanager
6 def with_timer(label: str):
7 start = time.perf_counter()
8 try:
9 yield
10 finally:
11 print(f"{label} took {time.perf_counter() - start:.4f} s")
Python’s with statement works well when you do not need the block’s return value as an expression. In Hy, a with-timer macro wraps any sequence of expressions, prints the elapsed time in a finally clause, and returns the value of the body directly:
1 (setv total (with-timer "sum-squares"
2 (sum (gfor x (range 10000) (* x x)))))
Even better, when we build the HTML DSL in html_dsl.hy, the html macro folds static tags and attributes into a single string literal at compile time. A Python template library must parse templates or concatenate tag strings at run time unless it adds a separate code-generation build step.
Run It
The project directory layout is:
1 macros/
2 ├── core_macros.hy # unless, do-while, swap!, when-let, with-timer, timed, mini-match
3 ├── hygiene.hy # Broken vs hygienic macros (gensym, def-gensyms, defmacro!, aif)
4 ├── reader_macros.hy # Reader macros: #env, #sql (with SQLite runner), and #json
5 ├── html_dsl.hy # Compile-time HTML and SVG DSL with static string folding
6 ├── demo.hy # Guided tour script that prints macro expansions and outputs
7 ├── test_macros.hy # Pytest suite for core macros, hygiene, namespaces, and readers
8 ├── test_html_dsl.hy # Pytest suite for the HTML and SVG DSL
9 ├── conftest.py # Pytest hook to collect test_*.hy files
10 └── pyproject.toml # Dependencies: hy>=1.3.1, hyrule>=1.0.0, pytest>=9.2.0
Sync the environment, run the guided tour, and run the test suite:
1 cd source_code_for_examples/macros
2 uv sync
3 uv run hy demo.hy
4 uv run pytest -q
Running uv run hy demo.hy produces this output:
1 === 1. Core Macros and Expansion ===
2 unless result: ['initialized']
3 do-while steps: [0, 1, 2]
4 after swap!: omega alpha
5 when-let found version: 1.3.1
6 HTTP 200 -> OK
7 HTTP 404 -> Not Found
8 HTTP 503 -> Server Error
9
10 Macroexpansion of (unless ready (boot)):
11 '(when (not ready) (boot))
12 Python AST via form-to-python:
13 [a, b] = [b, a]
14
15 === 2. Hygiene: Multiple Evaluation and Capture ===
16 square-bad(bump): result=16, calls=2 (evaluated twice!)
17 square-good(bump): result=16, calls=1
18 square-o!(bump): result=16, calls=1
19 sum-steps-bad with outer step=10: 3 (captured 'step'!)
20 sum-steps-good with outer step=10: 30 (outer step preserved)
21 anaphoric aif: Hello, Mark!
22
23 === 3. Reader Macros (#env, #json, #sql) ===
24 #env DEMO_ENV_NAME: hy-macros
25 #env with default: fallback-value
26 #json parsed at read time: {'name': 'hy', 'release': 1.3, 'lisp': True}
27 Compiled SqlQuery: SqlQuery(sql='SELECT name, year FROM languages WHERE lisp = 1 AND year >= ? ORDER BY year', params=(2000,))
28 Query results: [{'name': 'Clojure', 'year': 2007}, {'name': 'Hy', 'year': 2013}]
29
30 === 4. Compile-Time HTML and SVG DSL ===
31 Rendered HTML (with automatic escaping):
32 <div class="card" data-kind="demo"><h2>Topic Summary</h2><p>Viewer: <script>alert(1)</script></p><ul><li>Macros</li><li>Reader Macros</li><li>DSLs</li></ul><hr></div>
33
34 Macroexpansion of static + dynamic HTML:
35 '(hy.I.html_dsl.RawHtml (.join "" ["<div class=\"box\"><h1>Title</h1><p>" (hy.I.html_dsl.render-child user) "</p></div>"]))
36
37 Rendered SVG:
38 <svg xmlns="http://www.w3.org/2000/svg" width="140" height="40" viewBox="0 0 140 40"><rect x="2" y="2" width="136" height="36" rx="6" fill="#eef"></rect><text x="70" y="25" text-anchor="middle">Hy 1.3.1</text></svg>
Look closely at the macroexpansion under Section 4 of the output. The input expression had nested :div, :h1, and :p vectors. The html macro merged all static opening tags, attributes, and headings into the single literal "<div class=\"box\"><h1>Title</h1><p>" before the program ran, leaving only (hy.I.html_dsl.render-child user) to run dynamically.
How It Works
1. Code as Models, defmacro, and Quasiquotation: core_macros.hy
When the Hy reader parses source text, it produces models from hy.models:
hy.models.Expressionfor parenthesized calls like(+ 1 2)hy.models.Symbolfor identifiers likereadyortotalhy.models.Keywordfor keywords like:elseor:classhy.models.List,hy.models.Dict, andhy.models.Tuplefor[...],{...}, and#(...)hy.models.String,hy.models.Integer, andhy.models.Floatfor scalar literals
Every model wraps the underlying Python type (Expression and List subclass tuple; Symbol and String subclass str) and records source line and column numbers so tracebacks point to the right file and line.
You can build models by calling constructors directly, or with quasiquotation (the backtick `):
'form(quote) returnsformunevaluated as a model.`form(quasiquote) quotesformas a template, except where~or~@appears inside it.~expr(unquote) evaluatesexprat macro-expansion time and inserts the resulting value into the template.~@seq(unquote-splice) evaluatesseqand splices its elements flat into the surrounding sequence.
Here is the complete listing of core_macros.hy:
1 ;; core_macros.hy -- Control-flow macros, timing macros, and AST inspection.
2
3 (import ast)
4 (import time)
5 (import hy.compiler [hy-compile])
6 (import hy.models [Keyword Symbol])
7
8
9 (defmacro unless [test #* body]
10 "Evaluate body forms only when test is falsy."
11 `(when (not ~test)
12 ~@body))
13
14
15 (defmacro do-while [body-form condition]
16 "Execute body-form at least once, repeating while condition is truthy."
17 `(while True
18 ~body-form
19 (when (not ~condition)
20 (break))))
21
22
23 (defmacro swap! [left right]
24 "Swap two assignable places in place."
25 `(setv [~left ~right] [~right ~left]))
26
27
28 (defmacro when-let [binding #* body]
29 "Bind a symbol to an expression and run body only when the value is truthy.
30 Usage: (when-let [user (find-user uid)] (greet user))"
31 (when (!= (len binding) 2)
32 (raise (SyntaxError "when-let expects a [symbol expr] binding pair")))
33 (setv [sym expr] binding)
34 `(do
35 (setv ~sym ~expr)
36 (when ~sym
37 ~@body)))
38
39
40 (defmacro with-timer [label #* body]
41 "Run body, print elapsed seconds with label, and return the body's value."
42 (setv start (hy.gensym "start"))
43 `(do
44 (setv ~start (hy.I.time.perf-counter))
45 (try
46 (do ~@body)
47 (finally
48 (print ~label "took"
49 (round (- (hy.I.time.perf-counter) ~start) 4)
50 "s")))))
51
52
53 (defmacro timed [expr]
54 "Evaluate expr once and return #(result elapsed-seconds)."
55 (setv start (hy.gensym "start")
56 result (hy.gensym "result"))
57 `(do
58 (setv ~start (hy.I.time.perf-counter)
59 ~result ~expr)
60 #(~result (- (hy.I.time.perf-counter) ~start))))
61
62
63 (defmacro mini-match [expr #* clauses]
64 "Dispatch expr against [pattern result] clauses without re-evaluating expr.
65 Use :else or _ for the wildcard branch."
66 (setv target (hy.gensym "target")
67 branches [])
68 (for [clause clauses]
69 (when (!= (len clause) 2)
70 (raise (SyntaxError "mini-match clauses must be [pattern result] pairs")))
71 (setv [pat result] clause)
72 (if (or (= pat :else) (= pat '_))
73 (.extend branches [True result])
74 (.extend branches [`(= ~target ~pat) result])))
75 `(do
76 (setv ~target ~expr)
77 (cond ~@branches)))
78
79
80 (defn expand-once-str [form [module None]]
81 "Return the single-step macro expansion of form as Hy source text."
82 (hy.repr (if (is module None)
83 (hy.macroexpand-1 form)
84 (hy.macroexpand-1 form module))))
85
86
87 (defn form-to-python [form [module-name "__main__"]]
88 "Compile a quoted Hy form to formatted Python source code (like hy2py)."
89 (setv expanded (hy.macroexpand form)
90 py-ast (hy-compile expanded module-name)
91 text (.strip (ast.unparse py-ast)))
92 (.removeprefix text "import hy\n"))
Notice three practical patterns in core_macros.hy:
- Compile-time validation:
when-letandmini-matchinspectbindingandclauseduring macro expansion and raiseSyntaxErrorimmediately if the caller wrote the wrong shape. Errors surface when the module compiles, not hours later when a cold code branch executes. - Self-contained imports with
hy.I:with-timercalls(hy.I.time.perf-counter)instead of(time.perf-counter). Thehy.Ireader macro compiles tohy.I.time.perf_counter(), which importstimeon demand. That way a file that requireswith-timerdoes not crash if it forgot to write(import time). - Splicing pairs into
cond:mini-matchcollects[condition-form result-form]pairs into a flat Python listbranchesat compile time and splices them into(cond ~@branches).
2. Expansion and Debugging: hy.macroexpand-1, hy2py, and hy --spy
When a macro produces a surprising error, printing its expansion is the fastest way to see what happened. Hy gives you four tools:
(hy.macroexpand-1 form): expands the outer macro offormby one step and returns the resulting Hy model. Wrap it in(hy.repr ...)to print readable Hy syntax.(hy.macroexpand form): repeatedly expands outer macros until the head is no longer a macro.hy2py: a command-line tool that compiles a.hyfile to a Python AST and prints the generated Python code. Runninguv run hy2py core_macros.hyshows how Hy compilesdefmacroitself into a Python function registered withhy.macros.macro:
1 def _hy_anon_1(test, *body):
2 """Evaluate body forms only when test is falsy."""
3 return hy.models.Expression([
4 hy.models.Symbol('when', from_parser=True),
5 hy.models.Expression([hy.models.Symbol('not', from_parser=True), test]),
6 *(body or [])
7 ])
8 hy.macros.macro('unless')(_hy_anon_1)
hy --spy: starting the REPL withuv run hy --spyprints the generated Python code before evaluating every REPL input. This lets you watch each macro expand and compile to Python statements in real time.
In our code, form-to-python runs the same pipeline programmatically by calling (hy-compile (hy.macroexpand form) module-name) followed by (ast.unparse py-ast). That lets unit tests assert on the exact Python code a macro generates.
3. Macro Hygiene: hygiene.hy
A macro is hygienic when its expansion never evaluates caller expressions more times than expected and never collides with variable names in the caller’s scope. Lisp macros can fail both ways if you write naive templates.
Here is hygiene.hy, which contrasts the broken versions with their hygienic replacements:
1 ;; hygiene.hy -- Macro hygiene: multiple evaluation, variable capture, and fixes.
2
3 (require hyrule [defmacro! def-gensyms])
4
5
6 ;; ---------- 1. Multiple evaluation ----------
7
8 (defmacro square-bad [x]
9 "BROKEN: evaluates x twice when x has side effects."
10 `(* ~x ~x))
11
12
13 (defmacro square-good [x]
14 "HYGIENIC: binds x to a fresh gensym so x is evaluated once."
15 (setv val (hy.gensym "val"))
16 `(do
17 (setv ~val ~x)
18 (* ~val ~val)))
19
20
21 (defmacro! square-o! [o!x]
22 "HYGIENIC (via hyrule defmacro!): o!x evaluates once and binds to g!x."
23 `(* ~g!x ~g!x))
24
25
26 ;; ---------- 2. Accidental variable capture ----------
27
28 (defmacro sum-steps-bad [n #* body]
29 "BROKEN: leaks loop variable 'step' and accumulator 'total' into caller scope."
30 `(do
31 (setv total 0)
32 (for [step (range ~n)]
33 (+= total (do ~@body)))
34 total))
35
36
37 (defmacro sum-steps-good [n #* body]
38 "HYGIENIC: uses def-gensyms so caller variables named step or total are safe."
39 (def-gensyms total step limit)
40 `(do
41 (setv ~limit ~n
42 ~total 0)
43 (for [~step (range ~limit)]
44 (+= ~total (do ~@body)))
45 ~total))
46
47
48 ;; ---------- 3. Intentional capture (anaphoric macro) ----------
49
50 (defmacro aif [test-form then-form [else-form None]]
51 "Anaphoric if: intentionally binds 'it' to the value of test-form."
52 `(do
53 (setv it ~test-form)
54 (if it
55 ~then-form
56 ~else-form)))
Let us examine all three cases:
- Multiple evaluation (
square-badvssquare-goodandsquare-o!):(square-bad (bump))expands to(* (bump) (bump)), callingbumptwice.square-goodcalls(hy.gensym "val")at macro-expansion time to generate a unique symbol (such as_val_1234), evaluates~xonce into~val, and multiplies~valby itself. Thehyrulelibrary providesdefmacro!, inspired by Doug Hoyte’s Let Over Lambda. Any parameter prefixed witho!(once-only, such aso!x) is automatically evaluated once before the macro body and bound to the matchingg!gensym (g!x). - Accidental variable capture (
sum-steps-badvssum-steps-good): Because Python and Hy share function-level variable scope, introducing a plain symbol likesteportotalinside a macro expansion writes directly into the caller’s local scope. When the caller sets(setv step 10)and runs(sum-steps-bad 3 step), the macro’sfor [step (range 3)]overwrites the caller’sstepwith0,1, and2, returning3instead of30.sum-steps-gooduseshyrule’s(def-gensyms total step limit)to create three fresh symbols at once. - Intentional capture (
aif): Sometimes you want a macro to introduce a documented symbol into the caller’s scope. An anaphoric macro likeaifbinds the pronounitto the result oftest-formso the caller can write(aif (find-user uid) it.email "none"). Becauseitis part of the macro’s contract, this capture is deliberate rather than a bug.
4. Macro Namespaces: require, export, hy.R, and local-macros
In Python, import runs at run time. Because macros must run while a module is still compiling, Hy separates macro loading from import:
(require module [macro-a macro-b]): loadsmoduleat compile time and registersmacro-aandmacro-bin the current module’s macro table.(require module :as m): registers macros under a prefix so you can call(m.unless ready (boot)).(require module :readers [sql env]): loads reader macros frommodule.hy.R: the compile-time equivalent ofhy.I. Writing(hy.R.core_macros.unless False "ok")requirescore_macroson the fly and invokesunlesswithout a top-levelrequirestatement.- Controlling exports with
(export ...): by default,(require mod *)brings in every macro defined inmod(and not macros thatmoditself required from elsewhere). You can restrict whatimport *andrequire *expose using(export :objects [fn1] :macros [mac1]), which compiles to__all__and_hy_export_macros.
5. Reader Macros: reader_macros.hy
Ordinary macros transform Hy models after the reader has already parsed the source text. Reader macros hook into Step 1 of the pipeline: they run inside the parser when the reader sees #tag.
Inside (defreader name ...), Hy binds the active reader instance to the symbol &reader. Calling (.parse-one-form &reader) reads and returns the next complete Hy form from the input stream.
Here is reader_macros.hy, which defines #sql, #env, and #json:
1 ;; reader_macros.hy -- Reader macros (#env, #sql, #json) using defreader.
2
3 (import dataclasses [dataclass])
4 (import json)
5 (import re)
6 (import sqlite3)
7 (import hy.models [Dict List String])
8
9
10 (defclass [(dataclass :frozen True)] SqlQuery []
11 "A parameterized SQL query checked at read time."
12 #^ str sql
13 #^ tuple params)
14
15
16 (defn normalize-sql [raw-sql]
17 "Collapse internal whitespace and strip leading/trailing spaces."
18 (setv cleaned (.strip (re.sub r"\s+" " " (str raw-sql))))
19 (when (not cleaned)
20 (raise (SyntaxError "#sql query string cannot be empty")))
21 cleaned)
22
23
24 (defn compile-sql-form [form]
25 "Validate a #sql form at read time and return code constructing SqlQuery."
26 (cond
27 (isinstance form String)
28 (do
29 (setv sql-text (normalize-sql form)
30 placeholders (.count sql-text "?"))
31 (when (!= placeholders 0)
32 (raise (SyntaxError
33 f"#sql string has {placeholders} '?' placeholders but no params list")))
34 `(hy.I.reader_macros.SqlQuery ~sql-text #()))
35 (and (isinstance form List) (>= (len form) 1) (isinstance (get form 0) String))
36 (do
37 (setv sql-text (normalize-sql (get form 0))
38 params (list (cut form 1 None))
39 placeholders (.count sql-text "?"))
40 (when (!= placeholders (len params))
41 (raise (SyntaxError
42 f"#sql has {placeholders} '?' placeholders but {(len params)} params")))
43 `(hy.I.reader_macros.SqlQuery ~sql-text #(~@params)))
44 True
45 (raise (SyntaxError
46 "#sql expects a string literal or [\"SQL ...\" param1 param2 ...]"))))
47
48
49 (defreader sql
50 (setv form (.parse-one-form &reader))
51 (hy.I.reader_macros.compile-sql-form form))
52
53
54 (defn compile-env-form [form]
55 "Expand #env \"VAR\" or #env [\"VAR\" \"default\"] into os.environ.get."
56 (cond
57 (isinstance form String)
58 `(hy.I.os.environ.get ~(str form))
59 (and (isinstance form List) (= (len form) 2) (isinstance (get form 0) String))
60 `(hy.I.os.environ.get ~(str (get form 0)) ~(get form 1))
61 True
62 (raise (SyntaxError
63 "#env expects \"VAR\" or [\"VAR\" default-expr]"))))
64
65
66 (defreader env
67 (setv form (.parse-one-form &reader))
68 (hy.I.reader_macros.compile-env-form form))
69
70
71 (defn json-to-model [val]
72 "Convert parsed JSON values into Hy models for zero-cost runtime literals."
73 (cond
74 (isinstance val dict)
75 (do
76 (setv pairs [])
77 (for [[k v] (.items val)]
78 (.extend pairs [(String k) (json-to-model v)]))
79 (Dict pairs))
80 (isinstance val list)
81 (List (gfor item val (json-to-model item)))
82 (isinstance val str)
83 (String val)
84 True val))
85
86
87 (defn compile-json-form [form]
88 "Parse a JSON string literal at read time and emit a Hy data literal."
89 (when (not (isinstance form String))
90 (raise (SyntaxError "#json expects a string literal")))
91 (try
92 (setv parsed (json.loads (str form)))
93 (except [err json.JSONDecodeError]
94 (raise (SyntaxError f"Invalid #json literal: {err.msg}"))))
95 (json-to-model parsed))
96
97
98 (defreader json
99 (setv form (.parse-one-form &reader))
100 (hy.I.reader_macros.compile-json-form form))
101
102
103 (defn execute-sql [conn query]
104 "Execute a SqlQuery against a sqlite3 connection and return rows as dicts."
105 (when (not (isinstance query SqlQuery))
106 (raise (TypeError "execute-sql expects a SqlQuery value")))
107 (setv cursor (.execute conn query.sql query.params)
108 columns (if cursor.description
109 (lfor col cursor.description (get col 0))
110 []))
111 (lfor row (.fetchall cursor)
112 (dict (zip columns row))))
Notice what #sql and #json accomplish at read time:
#sqlcollapses multi-line whitespace into a single clean SQL string and counts the?parameter placeholders before the program runs. If a query has two?placeholders and you pass one parameter expression,#sqlraisesSyntaxErrorat compile time.#jsoncallsjson.loadsat read time and converts the parsed structure intohy.models.Dictandhy.models.Listnodes viajson-to-model. Malformed JSON fails at compile time, and valid JSON compiles directly to a native Python dict or list literal with zerojson.loadscall at run time.
6. Case Study: NSK’s ask and rule Macros
Before we build our HTML DSL, let us look at how the NSK knowledge graph chapter (source_code_for_examples/nsk-neural-symbolic-graph) uses macros to embed a Datalog query language inside Hy.
In NSK, a query looks like this:
1 (ask [?author ?lang]
2 [?author :wrote :nsk]
3 [?author ~:codes-in ?lang]
4 :limit 5)
Notice why ask must be a macro rather than a function:
?authorand?langare unbound symbols. Ifaskwere a function, Python would try to look up a local variable namedhyx_Xquestion_markXauthorand raiseNameError.~:codes-inuses the unquote prefix~to mark a neural predicate. The Hy reader parses~:codes-inas the two-element expression(unquote :codes-in). Outside a quasiquote or macro that intercepts it, evaluating(unquote :codes-in)is a compile error.- Ordinary Hy variables and function calls inside a pattern (such as
[?who :codes-in target-language]) must still be evaluated normally at run time!
NSK solves all three requirements in 30 lines across nsk/terms.hy and nsk/query.hy:
1 ;; From nsk/terms.hy
2 (defn var-symbol? [form]
3 (and (isinstance form Symbol) (> (len form) 1) (.startswith form "?")))
4
5 (defn unquote? [form]
6 (and (isinstance form Expression) (= (len form) 2)
7 (= (get form 0) 'unquote)))
8
9 (defn compile-term [form]
10 "Macro helper. Like to-term, but it returns code: ?x and ~p are rewritten and
11 everything else is left for Hy to evaluate, so a pattern can mention ordinary
12 variables and function calls."
13 (cond
14 (var-symbol? form) `(hy.I.nsk/terms.Var ~(cut (str form) 1 None))
15 (unquote? form) `(hy.I.nsk/terms.Neural ~(compile-term (get form 1)))
16 ;; A bare keyword in a call would be read as a keyword argument, so quote it.
17 (isinstance form Keyword) `(quote ~form)
18 True form))
19
20 (defn compile-pattern [form]
21 (when (not (and (isinstance form List) (= (len form) 3)))
22 (raise (ValueError f"a pattern needs exactly three terms: {(hy.repr form)}")))
23 `#(~@(map compile-term form)))
And in nsk/query.hy:
1 ;; From nsk/query.hy
2 (defn split-options [forms]
3 "Macro helper: separate the [s p o] patterns from trailing :key value pairs."
4 (setv patterns (lfor form forms :if (isinstance form List) form))
5 #(patterns (cut forms (len patterns) None)))
6
7 (defmacro ask [select #* body]
8 (setv [patterns options] (hy.I.nsk/query.split-options body))
9 `(hy.I.nsk/query.run-query
10 [~@(map hy.I.nsk/terms.compile-pattern patterns)]
11 [~@(map hy.I.nsk/terms.compile-term select)]
12 ~@options))
Look at the fourth branch of compile-term: True form. When a term in a pattern is not a ?variable, not a ~predicate, and not a :keyword, compile-term returns form unchanged! That single line is what lets NSK patterns mix logic variables (?who) with ordinary Python variables (target-language) and function calls seamlessly.
Finally, nsk/__init__.hy includes (require nsk.query [ask rule]) alongside its (import ...) forms so a user of the package can write (require nsk [ask rule]) from the top-level package namespace.
7. Project: A Compile-Time HTML and SVG DSL (html_dsl.hy)
Now we apply the same selective compilation technique to build html_dsl.hy. We want to write HTML and SVG trees as nested Hy vectors starting with a keyword tag name:
1 (html
2 [:div {:class "card" :data-kind "demo"}
3 [:h2 "Topic Summary"]
4 [:p "Viewer: " user]
5 [:ul (lfor item items (html [:li item]))]
6 [:hr]])
If we wrote html as an ordinary function, every tag vector and attribute dict would be allocated at run time on every page render. By writing html as a macro that walks the nested structure at compile time, we get three advantages:
- Compile-time string folding: static tags, static attribute dicts, and literal strings or numbers are escaped and concatenated into plain string constants during macro expansion.
- Compile-time validation: void HTML elements (
<br>,<hr>,<img>,<input>) are checked at compile time; giving children to[:br "oops"]raisesSyntaxErrorbefore the program runs. - Automatic runtime escaping: any child or attribute value that is not a compile-time literal is wrapped in
(hy.I.html_dsl.render-child expr)or(hy.I.html_dsl.render-attr name expr), which escapes<,>,&, and quotes unless the value is already aRawHtmlinstance produced by(html ...)or(raw ...).
Here is the complete listing of html_dsl.hy:
1 ;; html_dsl.hy -- Compile-time HTML and SVG DSL with static string folding.
2
3 (import html :as py-html)
4 (import hy.models [Dict Expression Float Integer Keyword List String])
5
6
7 (setv VOID-TAGS #{"area" "base" "br" "col" "embed" "hr" "img"
8 "input" "link" "meta" "source" "track" "wbr"})
9
10
11 (defclass RawHtml []
12 "Wrapper for pre-rendered HTML/SVG strings that must not be escaped again."
13 (defn __init__ [self text]
14 (setv self.text (str text)))
15
16 (defn __str__ [self]
17 self.text)
18
19 (defn __repr__ [self]
20 f"(RawHtml {(hy.repr self.text)})")
21
22 (defn __eq__ [self other]
23 (cond
24 (isinstance other RawHtml) (= self.text other.text)
25 (isinstance other str) (= self.text other)
26 True False)))
27
28
29 (defn raw [text]
30 "Mark a string as trusted HTML so render-child does not escape it."
31 (RawHtml text))
32
33
34 (defn scalar-to-str [value]
35 "Convert Hy model scalars (Integer, Float, String) or Python values to str."
36 (cond
37 (isinstance value Integer) (str (int value))
38 (isinstance value Float) (str (float value))
39 True (str value)))
40
41
42 (defn escape-html [value]
43 "Escape special HTML characters (&, <, >, quotes) in value."
44 (py-html.escape (scalar-to-str value) :quote True))
45
46
47 (defn attr-name [key]
48 "Convert a keyword or string key into an HTML attribute name."
49 (if (isinstance key Keyword)
50 key.name
51 (str key)))
52
53
54 (defn render-attr [key value]
55 "Render one HTML attribute at run time.
56 Boolean True renders a bare attribute; False or None omits the attribute."
57 (setv name (attr-name key))
58 (cond
59 (or (is value None) (is value False)) ""
60 (is value True) f" {name}"
61 True f" {name}=\"{(escape-html value)}\""))
62
63
64 (defn render-child [value]
65 "Render a dynamic child value at run time with automatic HTML escaping."
66 (cond
67 (or (is value None) (is value False)) ""
68 (isinstance value RawHtml) value.text
69 (isinstance value #(list tuple))
70 (.join "" (gfor item value (render-child item)))
71 (hasattr value "__iter__")
72 (if (isinstance value #(str bytes))
73 (escape-html value)
74 (.join "" (gfor item value (render-child item))))
75 True (escape-html value)))
76
77
78 (defn literal-scalar? [form]
79 (isinstance form #(String Integer Float)))
80
81
82 (defn tag-form? [form]
83 "True when form is [:tag ...] or (:tag ...) with a Keyword head."
84 (and (isinstance form #(List Expression))
85 (> (len form) 0)
86 (isinstance (get form 0) Keyword)))
87
88
89 (defn compile-attrs [attr-dict]
90 "Compile a Hy Dict model of attributes into a list of segments."
91 (setv items (list attr-dict))
92 (when (% (len items) 2)
93 (raise (SyntaxError "Attribute dictionary must have key/value pairs")))
94 (setv segments [])
95 (for [idx (range 0 (len items) 2)]
96 (setv k (get items idx)
97 v (get items (+ idx 1))
98 name (if (isinstance k #(Keyword String)) (attr-name k) None))
99 (cond
100 (and name (literal-scalar? v))
101 (.append segments f" {name}=\"{(escape-html v)}\"")
102 (and name (= v 'True))
103 (.append segments f" {name}")
104 (and name (or (= v 'False) (= v 'None)))
105 None
106 name
107 (.append segments `(hy.I.html_dsl.render-attr ~(String name) ~v))
108 True
109 (.append segments `(hy.I.html_dsl.render-attr ~k ~v))))
110 segments)
111
112
113 (defn compile-node [form]
114 "Compile one DSL node into a list of static strings and runtime expressions."
115 (cond
116 (tag-form? form)
117 (do
118 (setv tag (. (get form 0) name)
119 rest-forms (list (cut form 1 None))
120 has-attrs (and rest-forms (isinstance (get rest-forms 0) Dict))
121 attr-form (if has-attrs (get rest-forms 0) None)
122 children (if has-attrs (cut rest-forms 1 None) rest-forms)
123 segments [f"<{tag}"])
124 (when (is-not attr-form None)
125 (.extend segments (compile-attrs attr-form)))
126 (.append segments ">")
127 (if (in tag VOID-TAGS)
128 (when children
129 (raise (SyntaxError f"Void HTML element <{tag}> cannot have children")))
130 (do
131 (for [child children]
132 (.extend segments (compile-node child)))
133 (.append segments f"</{tag}>")))
134 segments)
135 (literal-scalar? form)
136 [(escape-html form)]
137 True
138 [`(hy.I.html_dsl.render-child ~form)]))
139
140
141 (defn merge-segments [segments]
142 "Coalesce adjacent compile-time string segments into single String models."
143 (setv merged [])
144 (for [seg segments]
145 (if (isinstance seg str)
146 (if (and merged (isinstance (get merged -1) String))
147 (setv (get merged -1) (String (+ (str (get merged -1)) seg)))
148 (.append merged (String seg)))
149 (.append merged seg)))
150 merged)
151
152
153 (defn build-html-expr [nodes]
154 "Compile a sequence of nodes into a RawHtml construction form."
155 (setv raw-segments [])
156 (for [node nodes]
157 (.extend raw-segments (compile-node node)))
158 (setv merged (merge-segments raw-segments))
159 (cond
160 (not merged)
161 `(hy.I.html_dsl.RawHtml "")
162 (and (= (len merged) 1) (isinstance (get merged 0) String))
163 `(hy.I.html_dsl.RawHtml ~(get merged 0))
164 True
165 `(hy.I.html_dsl.RawHtml (.join "" [~@merged]))))
166
167
168 (defmacro html [#* nodes]
169 "Compile nested [:tag {:attr val} ...] forms into a RawHtml value.
170 Static tags, attributes, and text are folded into string literals at
171 compile time; dynamic expressions are escaped at run time."
172 (hy.I.html_dsl.build-html-expr nodes))
173
174
175 (defmacro svg [attrs #* children]
176 "Compile an inline <svg> element with the default SVG xmlns attribute."
177 (when (not (isinstance attrs Dict))
178 (raise (SyntaxError "svg expects an attribute dict as its first argument")))
179 (setv items (list attrs)
180 keys (lfor i (range 0 (len items) 2) (attr-name (get items i)))
181 full-items (if (in "xmlns" keys)
182 items
183 [(Keyword "xmlns") (String "http://www.w3.org/2000/svg") #* items])
184 svg-node (List [(Keyword "svg") (Dict full-items) #* children]))
185 (hy.I.html_dsl.build-html-expr [svg-node]))
Two implementation details in html_dsl.hy are worth highlighting for your own macros:
scalar-to-strand Hy model strings: in Hy 1.3.1, calling(str form)on ahy.models.Integerreturns"hy.models.Integer(140)"rather than"140". ConvertingIntegerwith(int value)andFloatwith(float value)before callingstrproduces the clean numeric text"140".- Passing attribute names as strings: in
compile-attrs, when an attribute key is a keyword such as:disabled, splicing~kdirectly into(render-attr ~k ~v)would emit(render-attr :disabled is-disabled), which Hy compiles as a keyword argumentrender_attr(disabled=is_disabled)! Convertingnameto(String name)at compile time emits(render-attr "disabled" is-disabled)as a positional string argument.
8. Macros vs Functions vs Decorators vs __init_subclass__
A good rule in Lisp is to use the least powerful tool that solves the problem cleanly. Functions are easier to pass around as values, inspect in stack traces, and call from Python.
| Tool | Runs when | Receives | Best for | Avoid when |
|---|---|---|---|---|
Function (defn) |
Run time | Evaluated values | 90% of program logic, data transformations, composable callbacks | You need custom evaluation order, new bindings, or compile-time syntax |
Decorator / with |
Definition or run time | Function, class, or runtime context | Wrapping functions (caching, auth, retries) or resource cleanup | You need to rewrite expressions inside a function body |
__init_subclass__ / dataclasses |
Class creation time | Subclass object | Registering plugins, validating class attributes | The problem is not naturally a class hierarchy |
Macro (defmacro) |
Compile time | Unevaluated Hy models | Control flow (unless, do-while), binding forms (when-let), embedded DSLs (ask, html), compile-time folding |
A plain function or context manager would be just as clear |
Reader macro (defreader) |
Parse time | Active &reader stream |
Checked literals (#sql, #json, #env), terse domain notation |
Ordinary macro syntax (sql ...) is already readable |
Also remember Python interoperability: a Python module importing your Hy module can call any defn function or defclass class directly, whereas a macro only exists at Hy compile time (unless a Python caller invokes hy.eval). When you publish a library for both Hy and Python callers, put the runtime engine in ordinary functions (like NSK’s run-query or reader_macros.hy’s execute-sql) and make the macro a thin compile-time layer on top.
Tests
The example includes two pytest files: test_macros.hy and test_html_dsl.hy. Run them with:
1 uv run pytest -q
1 .................... [100%]
2 20 passed in 0.21s
Here are the key tests from test_macros.hy, which check single-step expansions with expand-once-str, golden Python translations with form-to-python, hygiene guarantees, hy.R namespace resolution, and compile-time reader errors:
1 ;; From test_macros.hy
2
3 (defn test-unless-expansion-and-behavior []
4 (assert (= (expand-once-str '(unless ready (launch)))
5 "'(when (not ready) (launch))"))
6 (setv log [])
7 (unless False
8 (.append log "first")
9 (.append log "second"))
10 (unless True
11 (.append log "never"))
12 (assert (= log ["first" "second"])))
13
14
15 (defn test-hy2py-golden-translation []
16 (setv py-swap (form-to-python '(swap! left right)))
17 (assert (= (.strip py-swap) "[left, right] = [right, left]"))
18
19 (setv py-unless (form-to-python '(unless done (step))))
20 (assert (in "not done" py-unless))
21 (assert (in "step()" py-unless)))
22
23
24 (defn test-multiple-evaluation-hygiene []
25 (setv counter 0)
26 (defn next-int []
27 (nonlocal counter)
28 (+= counter 1)
29 3)
30
31 (setv bad (square-bad (next-int)))
32 (assert (= bad 9))
33 (assert (= counter 2))
34
35 (setv counter 0)
36 (setv good (square-good (next-int)))
37 (assert (= good 9))
38 (assert (= counter 1))
39
40 (setv counter 0)
41 (setv good-o (square-o! (next-int)))
42 (assert (= good-o 9))
43 (assert (= counter 1)))
44
45
46 (defn test-variable-capture-hygiene []
47 (setv step 10
48 total 100)
49 ;; Broken macro overwrites caller's 'step' with 0, 1, 2
50 (assert (= (sum-steps-bad 3 step) 3))
51
52 (setv step 10
53 total 100)
54 ;; Hygienic macro preserves caller's 'step' (10 + 10 + 10 = 30) and 'total'
55 (assert (= (sum-steps-good 3 step) 30))
56 (assert (= step 10))
57 (assert (= total 100)))
58
59
60 (defn test-reader-macro-sql []
61 (setv conn (sqlite3.connect ":memory:"))
62 (.execute conn "CREATE TABLE books (title TEXT, pages INT)")
63 (.executemany conn
64 "INSERT INTO books VALUES (?, ?)"
65 [#("On Lisp" 432)
66 #("Practical Common Lisp" 499)
67 #("Brief Guide" 90)])
68
69 (setv min-pages 200
70 q #sql [" SELECT title, pages
71 FROM books
72 WHERE pages >= ?
73 ORDER BY pages " min-pages])
74 (assert (isinstance q SqlQuery))
75 (assert (= q.sql "SELECT title, pages FROM books WHERE pages >= ? ORDER BY pages"))
76 (assert (= q.params #(200)))
77
78 (setv rows (execute-sql conn q))
79 (assert (= (len rows) 2))
80 (assert (= (get (get rows 0) "title") "On Lisp"))
81 (.close conn)
82
83 ;; Mismatched '?' placeholder count fails at compile/read time
84 (with [(pytest.raises Exception)]
85 (hy.eval (hy.read "(do (require reader_macros :readers [sql]) #sql [\"SELECT ? AND ?\" 1])"))))
And here are the core tests from test_html_dsl.hy, which verify that static HTML trees fold into a single literal string at compile time, untrusted input is escaped, and void elements reject children:
1 ;; From test_html_dsl.hy
2
3 (defn test-static-html-folds-at-compile-time []
4 (setv exp (hy.repr (hy.macroexpand-1 '(html [:div {:class "card"} [:h1 "Title"] [:hr]]))))
5 ;; Entire static tree folds into a single string literal with no .join call
6 (assert (in "\"<div class=\\\"card\\\"><h1>Title</h1><hr></div>\"" exp))
7 (assert (not-in ".join" exp))
8
9 (setv out (html [:div {:class "card"} [:h1 "Title"] [:hr]]))
10 (assert (isinstance out RawHtml))
11 (assert (= (str out) "<div class=\"card\"><h1>Title</h1><hr></div>")))
12
13
14 (defn test-dynamic-escaping-and-raw-bypass []
15 (setv untrusted "<img src=x onerror=\"alert(1)\">"
16 safe-badge (raw "<strong>verified</strong>"))
17 (setv rendered
18 (str (html [:p {:title untrusted}
19 "User: " untrusted " " safe-badge])))
20 (assert (= rendered
21 (+ "<p title=\"<img src=x onerror="alert(1)">\">"
22 "User: <img src=x onerror="alert(1)"> "
23 "<strong>verified</strong></p>"))))
24
25
26 (defn test-void-tag-rejects-children-at-compile-time []
27 (with [(pytest.raises Exception)]
28 (hy.eval (hy.read "(do (require html_dsl [html]) (html [:br \"oops\"]))"))))
Wrap Up
Macros are the dividing line between using Lisp syntax and programming in Lisp. With defmacro, quasiquotation (~ and ~@), hy.gensym (or hyrule’s def-gensyms and defmacro!), and defreader, you can move validation, boilerplate, and constant folding from run time to compile time. At the same time, keeping your macros thin and delegating runtime work to ordinary Hy functions ensures that your code remains easy to test with pytest, inspect with hy.macroexpand-1 and hy2py, and call from Python.
Exercises
- Small exercise (
repeat-untilandwith-timer-collector): Add a(repeat-until body-form condition)macro tocore_macros.hythat loops untilconditionbecomes truthy (the opposite ofdo-while). Then write a(with-timer-into dict-expr key-expr #* body)macro that records the elapsed seconds into(get dict-expr key-expr)instead of printing to standard output. Be sure to evaluatedict-exprandkey-expronly once usinghy.gensym. - Open-ended exercise (CSS selector shorthand and
#regexreader macro): Extendcompile-nodeinhtml_dsl.hyso tag keywords support CSS-style#idand.classsuffixes at compile time: writing[:div#main.card.featured "Hello"]should compile to"<div id=\"main\" class=\"card featured\">Hello</div>", merging with any:classkey in the attribute dictionary. Next, add a#rereader macro inreader_macros.hythat takes a string literal, validates the regular expression syntax at compile time withre.compile, and expands to(hy.I.re.compile pattern).
Further Reading
- Hy 1.3.1 Macros Manual: official documentation on
defmacro,defreader,hy.gensym,require, andexport. - Paul Graham, On Lisp (Prentice Hall, 1993): classic treatment of bottom-up design, macro pitfalls, and embedded languages.
- Peter Seibel, Practical Common Lisp (Apress, 2005), Chapters 7, 8, and 30: macro fundamentals and a compile-time HTML generator.
- Doug Hoyte, Let Over Lambda (2008): reader macros, once-only
o!/g!gensyms, and compile-time computation.