Chapter 17: Understanding Spark and the Extension System
Elixir has powerful tools for metaprogramming, which means you can write code that generates or modifies other code. This is possible because Elixir treats code as regular data, specifically, as tuples with three elements.
This chapter explains the basics of Elixir’s metaprogramming, then dives into Spark, a library that powers the extensible DSLs (Domain-Specific Languages) in the Ash Framework. By the end, you’ll see how Ash’s clean, modular design works under the hood.
17.1 The Foundation: Code as Data in Elixir
In Elixir, every piece of code can be represented as simple data structures. This lets you inspect, modify, or generate code programmatically.
The two main tools for this are:
quote do ... end: This “freezes” your code instead of running it. It returns the code’s data representation, called the Abstract Syntax Tree (AST).unquote(expr): Inside a quote block, this evaluates expr right away (at compile time) and inserts the result into the AST. It’s like string interpolation (#{…}) but for code.
These tools enable metaprogramming: generating or transforming code before it runs. This is the magic behind frameworks like Ash, which use a library called Spark to build flexible, extensible DSLs.
17.2 What is the Abstract Syntax Tree (AST)?
The AST is Elixir’s way of turning your source code into a structured, tree-like data format.
When Elixir compiles your program:
- It parses the code.
- It builds the AST (a nested set of tuples).
- It uses the AST to generate bytecode.
You can view the AST using quote:
1 quote do
2 1 + 2 * 3
3 end
4 # Returns: {:+, [context: Elixir, import: Kernel], [1, {:*, [context: Elixir, import: Kernel], [2, 3]}]}
This is a tuple with three parts:
- The operation (:+ for addition).
- Metadata (like context and imports).
- Arguments (the numbers and sub-operations).
Even bigger code blocks become AST:
1 quote do
2 defmodule MyModule do
3 def hello(name), do: IO.puts("Hello #{name}")
4 end
5 end
6 # Returns a nested tuple representing the module, function, and IO call.
Since the AST is just Elixir data, you can:
- Store it in variables.
- Pattern-match on it (e.g., to find specific parts).
- Transform it (e.g., add new code).
- Inject it into other code.
**This “code as data” approach is what powers Ash’s flexibility via Spark. **
17.3 Using quote to Capture Code as Data
quote lets you capture code without running it, turning it into portable AST data:
1 my_code = quote do
2 def ok(socket), do: {:ok, socket}
3 end
4
5 # my_code now holds the AST for that function definition.
You can pass my_code around, store it, or use it in macros to insert it elsewhere.
17.4 Using unquote to Inject Values into Quoted Code
is the opposite: it “thaws” part of the quoted code, evaluates it at compile time, and injects the result:
1 socket = :my_socket
2
3 quote do
4 {:ok, unquote(socket)}
5 end
6 # Returns: {:ok, :my_socket} # The value of socket was inserted.
If you unquote a function call, it runs during compilation:
1 defp generate_greeting do
2 "Hello from compile time!"
3 end
4
5 quote do
6 IO.puts(unquote(generate_greeting()))
7 end
8 # This prints during compilation, not runtime!
With quote and unquote, you can build macros that create sophisticated DSLs. But for large, extensible DSLs (like in Ash), you need something more structured, and that’s where Spark comes in.
17.5 Introducing Spark: The Engine Behind Ash’s DSLs
Spark is a lightweight library (~1,500 lines of code) created by Zach Daniel (Ash’s author) for building extensible DSLs in Elixir. It’s standalone but powers Ash’s modular design and other recent framework such as Beam Bot.
Core ideas in Spark:
- Define a DSL in one module using
use Spark.Dsl. - Break the DSL into sections (blocks like
attributes do ... end), options (key-value settings), and entities (repeatable items like multiple attributes). - Extensions (from third-party packages) add new sections, options, or entities without modifying the core code.
- Extensions are easy to use: Just list them in
use MyDsl, extensions: [...].
This is why Ash feels so natural:
1 defmodule MyApp.Accounts.User do
2 use Ash.Resource,
3 extensions: [AshAuthentication, AshJsonApi.Resource, AshGraphql.Resource]
4
5 # Now you can use blocks like authentication {}, json_api {}, graphql {} — added by extensions!
6 end
17.5.1 Why Spark? Problems It Solves
Building extensible DSLs with plain macros leads to issues:
- Option name clashes between extensions.
- Hard to document or validate extension options.
- No standard place for post-DSL logic (e.g., validation).
- Extensions can’t declare dependencies.
- Users can’t easily discover available options.
Spark fixes these with a declarative, composable system.
17.6 How Spark Works: A Simple Example
Let’s build a tiny DSL to see Spark in action.
Define the core DSL:
1 # lib/my_app/dsl.ex
2 defmodule MyApp.Dsl do
3 use Spark.Dsl, extensions: [] # Start with no extensions
4
5 dsl do
6 # Top-level options
7 option :name, type: :string, doc: "Name of the thing"
8 option :env, type: :atom, default: :dev
9
10 # A section (block) that extensions can extend
11 section :settings do
12 option :log_level, type: :atom, default: :info
13 end
14 end
15 end
Create an extension:
1 # lib/my_app/extensions/logging.ex
2 defmodule MyApp.Extensions.Logging do
3 use Spark.Dsl.Extension,
4 sections: [:settings], # Extend the existing :settings section
5 transformers: [MyApp.Transformers.Logging]
6
7 dsl do
8 section :settings do
9 option :structured_logging, type: :boolean, default: false
10 option :log_requests, type: :boolean, default: true
11 end
12 end
13 end
14
15 # Optional transformer: Runs after DSL parsing for validation or defaults
16 defmodule MyApp.Transformers.Logging do
17 use Spark.Dsl.Transformer
18
19 def transform(dsl_state) do
20 if Spark.Dsl.get_opt(dsl_state, [:settings], :structured_logging) do
21 # Add custom logic here, e.g., validation
22 end
23 {:ok, dsl_state}
24 end
25 end
Use it in your code:
1 defmodule MyApp.MyThing do
2 use MyApp.Dsl, extensions: [MyApp.Extensions.Logging]
3
4 dsl do
5 name "My Production Service"
6 env :prod
7
8 settings do
9 log_level :debug
10 structured_logging true # This option comes from the extension!
11 end
12 end
13 end
Spark merges everything beautifully.
17.7 A Real Ash Example
In Ash, resources use Spark under the hood. Here’s a simplified User resource:
1 defmodule MyApp.Accounts.User do
2 use Ash.Resource,
3 domain: MyApp.Accounts,
4 extensions: [AshAuthentication, AshJsonApi.Resource, AshGraphql.Resource]
5
6 attributes do
7 uuid_primary_key :id
8 attribute :email, :ci_string, allow_nil?: false
9 end
10
11 actions do
12 defaults [:read, :destroy]
13 create :register do
14 accept [:email, :password]
15 end
16 end
17
18 # Extension-added sections:
19 authentication do
20 api MyApp.Accounts
21 tokens do
22 enabled? true
23 store_all_tokens? true
24 end
25 add_ons do
26 confirmation :confirm
27 end
28 end
29
30 json_api do
31 type "user"
32 routes do
33 base "/users"
34 get :read
35 index :read
36 post :register
37 end
38 end
39
40 graphql do
41 type :user
42 queries do
43 get :get_user, :read
44 list :list_users, :read
45 end
46 end
47 end
Core Ash handles basics like attributes and actions. Extensions add the rest (e.g., authentication from AshAuthentication). This lets you pick and choose features like AshGraphql or AshAdmin without bloat.
17.8 Key Spark Concepts
Here’s a table summarizing the essentials:
| Concept | Description | Ash Example | Spark.Dsl |
|---|---|---|---|
| Dsl | The base module defining the core DSL. | Ash.Resource, Ash.Domain | Spark.Dsl |
| Extension | A module adding new sections/options using use Spark.Dsl.Extension. |
AshAuthentication, AshJsonApi.Resource | Spark.Dsl.Extension |
| Section | A named block in the DSL (e.g., authentication do ... end). |
authentication, json_api, postgres | Spark.Dsl.Section |
| Transformer | Code that runs after DSL parsing for validation, defaults, or side effects. | Used in Ash for complex checks and setups | Spark.Dsl.Transformer |
| Entities | Repeatable sub-items with their own DSL (e.g., multiple attributes). | attribute :name, :string; action :create | Spark.Dsl.Entity |
17.9 Why Spark’s Extension Model is Revolutionary
- No Global Changes: Extensions only apply where you list them.
- No Conflicts: Extensions can merge sections safely, avoiding conflicts issues.
- Discoverability: IDEs and docs can inspect the full DSL.
- Powerful Transformers: Move logic to compile time for efficiency.
This makes Ash feel built-in and polished, unlike many frameworks.
To recap:
- Elixir’s metaprogramming (via AST, quote, unquote) treats code as data.
- Spark builds on this to create extensible DSLs.
- It powers Ash’s modularity: Core is simple; extensions add features.
- Understanding Spark unlocks how Ash works and inspires building your own Ash Extensions.
Next, let’s build our own Ash Extension.