Preface: Why Specter?

Specter for selecting

Clojure has a a nice function, get-in, for looking deep into a data structure:

user=> (def x {:a [{:b 3}
                   {:b 4}
                   {:b 5 :c 3}]})
user=> (get-in x [:a 1 :b])
=> 4

Let’s say that the second argument of get-in is a path, which is composed of path elements. For get-in, every element must be a key into an associative structure. (Remember that, in Clojure, vectors are associative structures with integer keys, so 1 above is the same sort of element as :a and :b.)

get-in is a fine function, but it’s limited to traversal of associative structures to retrieve a single value (such as 4).

Specter is more powerful than get-in because it has more types of path elements. For example, ALL lets you select many leaves of the structure. Here’s how you select all values of :b from the structure above:

user=> (select [:a ALL :b] x)
=> [3 4 5]

You can combine ALL with arbitrary predicates to reduce the number of selected values to those that satisfy the predicate:

user=> (select [:a ALL :b odd?] x)
=> [3 5]

As a final teaser, walker is a path element for walking code. Here’s how you’d select every even integer in a nested expression:

user=> (select [(walker integer?) even?]
               '(+ 1 (- 2 3) (+ 4 (- 5 (+ 6 7)) 8)))
=> (2 4 6 8)

Specter for updating

transform is to select as update-in is to get-in: you provide a function that’s given each selected element, and the result is the original input “modified” with the results of the function. The following transforms even integers by multiplying them by 1000:

user=> (transform [(walker integer?) even?]    ; path
                  (partial * 1000)             ; transformer
                  '(+ 1 (- 2 3) (+ 4 (- 5 (+ 6 7)) 8)))
=> (+ 1 (- 2000 3) (+ 4000 (- 5 (+ 6000 7)) 8000))

As you can see from above, any of the variety of path elements you can use with select can also be used with transform. There are also elements especially useful in transform. For example, consider this structure:

user=> (def input [ {:a 1 :x {:y 2}}
                    {:a 3 :x {:y 4}} ])

We can change the :a values with the path elements you already know:

user=> (transform [ALL :a] (partial * 1000) input)
=> [{:a 1000, :x {:y 2}} {:a 3000, :x {:y 4}}]

… and do the same to the more-deeply-nested :y values:

user=> (transform [ALL :x :y] (partial * 1000) input)
=> [{:a 1, :x {:y 2000}} {:a 3, :x {:y 4000}}]

But how do we add 1000 times the :a values to the corresponding :y values? We can do that using the collect-one element, which stashes a value for later use. Before we get to its use with transform, consider collect-one in a select expression:

user=> (select [ALL (collect-one :a) :x :y] input)
=> [[1 2] [3 4]]

The result is not the :y values (2 and 4), but rather pairs of :a and :y values. (If you’re familiar with monads, you can think of this as something like the state monad.)

So we have a way to “grab” an intermediate value (:a) as well as the leaf value :y. Because of the use of collect-one, they will both be given to the transformation function, which allows us to combine the two:

user=> (transform [ALL (collect-one :a) :x :y]
                  (fn [a z] (+ (* 1000 a) z))
                  input)
=> [{:a 1, :x {:y 1002}} {:a 3, :x {:y 3004}}]

Specter is not as general-purpose as, say, zippers, because you can only visit a subtree once. (With zippers, you can navigate to a point in the tree and then, based on the value there, navigate backwards to revisit part of the tree.) However, many of the problems you’d otherwise use zippers for can be solved with Specter, often—I’d argue—more clearly. The above use of collect-one is one of many examples.

Specter for learning

In order to learn how Specter works, I decided to grow it myself, feature by feature, using the existing implementation as a guide. As I did, I kept having little shocks of recognition: “Ah, continuation-passing style. Clever!” or “Ah, recursively precomputing the call graph—I’ve seen that before!” I was recognizing design patterns¹, and it occurred to me that this book could also introduce those patterns to people who perhaps hadn’t seen them before.

Originally, I planned to discuss Specter’s design in a Part II of the book, about extending specter with your own path elements. You need to understand something of the design to do that well. But as I was writing the first chapter, about the most basic Specter paths, I realized that I could make that chapter a lot less dry by showing the design, and—in fact—by having you implement Specter yourself (or, if you prefer, follow along with a narrated implementation).

That was the moment when the title switched from Using and Extending Specter to Extending and Using Specter, and when the book became both a book about a useful tool and also a little bit of a book about software design.