Chapter 4: Caveman2

The web stack

Caveman2 is a rather simple but feature complete framework for web development. It includes most of what you’d expect such a tool to have, such as a template language, database access, configuration, url routing, ect.

Caveman is based on ningle, a micro-framework which handles only the most basic functionality, such as url routing. Ningle itself is based on the Clack HTTP library and server interface. Because common lisp apps can support many different web servers such as hunchentoot and woo, you need a common interface between them, clack fills that role. Other than abstracting the server backend it also provides request and response objects and handler middleware. We’ll take a deeper look into these features in due time. For now let’s take a closer look at caveman itself.

In this chapter we’ll take a look at what caveman provides for us and we’ll setup a simple project. Here’s what we’ve got:

• Ningle based url routing
• Database access using the datafly and sxql libraries
• The Djula templating library, a port of the Django template language.
• A configuration system called ENVY that allows us to configure our app using environment variables.

Setting up a simple project

First let’s install caveman with quicklisp

1  * (ql:quickload "caveman2") 
2  => ("caveman2")

Caveman comes with a built in project generator. Let’s say we want to write our own wiki, here is how we can generate a project skeleton for it:

 1  * (caveman2:make-project #P"~/.roswell/local-projects/fullstackwiki"
 2                           :author "Pavel" 
 3                           :license "MIT")
 4 
 5 writing ~/.roswell/local-projects/fullstackwiki/fullstackwiki.asd
 6 writing ~/.roswell/local-projects/fullstackwiki/fullstackwiki-test.asd
 7 writing ~/.roswell/local-projects/fullstackwiki/app.lisp
 8 writing ~/.roswell/local-projects/fullstackwiki/README.markdown
 9 writing ~/.roswell/local-projects/fullstackwiki/.gitignore
10 writing ~/.roswell/local-projects/fullstackwiki/db/schema.sql
11 writing ~/.roswell/local-projects/fullstackwiki/src/config.lisp
12 writing ~/.roswell/local-projects/fullstackwiki/src/db.lisp
13 writing ~/.roswell/local-projects/fullstackwiki/src/main.lisp
14 writing ~/.roswell/local-projects/fullstackwiki/src/view.lisp
15 writing ~/.roswell/local-projects/fullstackwiki/src/web.lisp
16 writing ~/.roswell/local-projects/fullstackwiki/static/css/main.css
17 writing ~/.roswell/local-projects/fullstackwiki/t/fullstackwiki.lisp
18 writing ~/.roswell/local-projects/fullstackwiki/templates/index.html
19 writing ~/.roswell/local-projects/fullstackwiki/templates/_errors/404.html
20 writing ~/.roswell/local-projects/fullstackwiki/templates/layouts/default.html
21 
22  => T

What we did was we called the function caveman2:make-project with the path to our project as an argument. We put our project in local-projects, so ASDF and quicklisp can find it. We called the project “fullstackwiki” and we gave the author name and license as keyword arguments. Note that we must pass it a pathname object rather than a string.

As you can see from the printed output of make-project it generated quite a bit of files. Let’s take closer look.

The root directory of our project contains asd files for our app and an auto generated test system, a README file and a lisp specific .gitignore file.

Let’s take a closer look at fullstackwiki.asd

 1 (in-package :cl-user)
 2 (defpackage fullstackwiki-asd
 3   (:use :cl :asdf))
 4 (in-package :fullstackwiki-asd)
 5 
 6 (defsystem fullstackwiki
 7   :version "0.1"
 8   :author "Pavel"
 9   :license "MIT"
10   :depends-on (:clack
11                :lack
12                :caveman2
13                :envy
14                :cl-ppcre
15                :uiop
16 
17                ;; for @route annotation
18                :cl-syntax-annot
19 
20                ;; HTML Template
21                :djula
22 
23                ;; for DB
24                :datafly
25                :sxql)
26   :components ((:module "src"
27                 :components
28                 ((:file "main" :depends-on ("config" "view" "db"))
29                  (:file "web" :depends-on ("view"))
30                  (:file "view" :depends-on ("config"))
31                  (:file "db" :depends-on ("config"))
32                  (:file "config"))))
33   :description ""
34   :in-order-to ((test-op (load-op fullstackwiki-test))))

Here we see that the code to our app is in the “src” directory(called a module by ASDF), and it has the following componets:

• config.lisp is the main configuration, it uses ENVY.
• db.lisp defines our database layer
• view.lisp handles html templates
• web.lisp defines our routes
• main.lisp handles how our app is started and stopped

We’ll take a look at all of these files in the next sections.

Configuration

Envy is a configuration utility which allows you to define several different configurations and then switch between them based on the value of an environment variable. For example you can have a separate configuration on a development machine and one for production deployment. It also allows you to define a common configuration and have the other ones just define the differences, rather than duplicate code. Let’s take a closer look at src/config.lisp:

 1 (in-package :cl-user)
 2 (defpackage fullstackwiki.config
 3   (:use :cl)
 4   (:import-from :envy
 5                 :config-env-var
 6                 :defconfig)
 7   (:export :config
 8            :*application-root*
 9            :*static-directory*
10            :*template-directory*
11            :appenv
12            :developmentp
13            :productionp))
14 (in-package :fullstackwiki.config)
15 
16 (setf (config-env-var) "APP_ENV")
17 
18 (defparameter *application-root*   (asdf:system-source-directory :fullstackwiki))
19 (defparameter *static-directory*   (merge-pathnames #P"static/" *application-roo\
20 t*))
21 (defparameter *template-directory* (merge-pathnames #P"templates/" *application-\
22 root*))
23 
24 (defconfig :common
25   `(:databases ((:maindb :sqlite3 :database-name ":memory:"))))
26 
27 (defconfig |development|
28   '())
29 
30 (defconfig |production|
31   '())
32 
33 (defconfig |test|
34   '())
35 
36 (defun config (&optional key)
37   (envy:config #.(package-name *package*) key))
38 
39 (defun appenv ()
40   (uiop:getenv (config-env-var #.(package-name *package*))))
41 
42 (defun developmentp ()
43   (string= (appenv) "development"))
44 
45 (defun productionp ()
46   (string= (appenv) "production"))

First the line (setf (config-env-var) "APP_ENV") tells ENVY to switch configuration based on the APP_ENV environment variable.

Next the *application-root*, *static-directory* and *temlate-directory* variables are defined which contain our app, our static resources and our html templates respectively.

The macro defconfig is used to define configurations as the name suggests. A config has a name and a body. As you can see above the body itself is a plist of values. The :common name is special, because it is included in every other configuration. In this case it only defines that the database used by default is sqlite3 and it’s to be stored in memory, rather than create a file on disk.

The other defined configurations are |development|, |production| and |test|, which are all empty. The reason they are named with || is because that’s how common lisp symbols are written in lower case. Remember that a configuration is chosen based on the value of an environment variable. You could name your config production, but since lisp symbols are upcased by default, you’ll have to set your environment variable in upcase: export APP_ENV=PRODUCTION.

Finally the file defines a few utility functions: * config gives returns back the configuration, or if an optional key argument is given, it looks it up and returns it’s value. * appenv returns the value of the configured environment variable * developmentp and productionp simply tell you if you’re running in dev or prod respectively.

That’s it. We’ll take a closer look again at configuration when we set up a database for our project other than sqlite3, for now you get the idea.

Views

By default caveman apps use Djula for html templating. Djula is an implementation of the Django template engine. If you haven’t used a template language before it’s basically an ordinary text file with special tags in between which insert or transform data. For example this is a very simple Djula template:

1 {% if username %}
2 Hello {{username}}!
3 {% else %}
4 Please login!
5 {% endif %}

This template defines a variable called username, when we render our template we’ll pass it an argument with the value of username and it will use it to transform the text. For example it’s value is Pavel, well get the string “Hello Pavel”, but if it’s nil we’ll get the string “Please login!”. Although template languages aren’t specific to HTML, they are extremely useful for just that purpose and that’s mostly what we’ll be using them for in this book. We’ll go through the template language itself in a separate chapter later.

In a caveman app, the Djula templates are kept in the directory designated by *template-directory*, which in our case is fullstackwiki/templates/. A fresh project will have a few such templates already defined in that directory. Like the Django engine, Djula supports template inheritance, which means that means we can put common code in separate files and reuse it in other templates. Our project already has such a shared template defined in the directory fullstackwiki/templates/layout/. Let’s take a look at the file named default.html in that directory:

 1 <!DOCTYPE html>
 2 <html>
 3 <head>
 4   <meta charset="utf-8">
 5   <title>{% block title %}{% endblock %}</title>
 6   <link rel="stylesheet" type="text/css" media="screen" href="/css/main.css">
 7 </head>
 8 <body>
 9   {% block content %}{% endblock %}
10 </body>
11 </html>

As you can see, this is just an ordinary HTML document, but with Djula tags added. In this case the tags are two pairs of {% block $}/{% endblock %}. What this tag does is it allows other templates to insert data in these blocks when they extend the default.html file. In our case we’ve defined two named blocks: title and content. Let’s take a look at index.html, which extends base.html:

1 {% extends "layouts/default.html" %}
2 {% block title %}Welcome to Caveman2{% endblock %}
3 {% block content %}
4 <div id="main">
5   Welcome to <a href="http://8arrow.org/caveman/">Caveman2</a>!
6 </div>
7 {% endblock %}

The first line introduces the extends tag, we give it the base template as an argument. Now any time we use the block tag with the name of a block, anything in it will be inserted into the base template.

Other than that caveman also defines a errors/404.html page for us.

That’s it for now. We’ll take a look at more tags as we need them.

Routing

The main purpose of a web framework is to match urls to code that returns a response to the client. Let’s see how caveman does that. The important code is in fullstackwiki/src/web.lisp. Let’s look at some of the code in that file.

First up, caveman2 stores url routing rules in an object called an app, which is an instance of the <app> class. Here we create a subclass called <web> and instantiate it, store it in a variable called fullstackwiki.web:*web* and clear it’s routing rules:

19 (defclass <web> (<app>) ())
20 (defvar *web* (make-instance '<web>))
21 (clear-routing-rules *web*)

Now we can begin defining routes. A route is a mapping of url template and additional conditions to Common Lisp functions. Caveman2 gives us two ways to define them. The first is annotation syntax, and the other is the defroute macro. Notice on line 14 of the file we have the following code: (syntax:use-syntax :annot). This allows us to annotate functions using the @annotation syntax, this is how it might look like:

1 @route GET "/"
2 (defun index ()
3   (render #P"index.html"))

The first line is an annotation, followed by a function. This isn’t standard Common Lisp, but a reader syntax extension. Annotations are essentially functions that take a form and transform it in some way, in this case it creates a routing rule for the url template "/" and associates the function index as it’s handler. We won’t be using annotation syntax in this book, since it is equivalent to the defroute macro in functionality and I have a strong aesthetic preference, at least for my projects, you are free to choose annotations. Now let’s look at the equivalent code as a defroute as it was defined in fullstackwiki/src/web.lisp:

26 (defroute "/" ()
27   (render #P"index.html"))

This is essentially equivalent to the annotation code. It defines a handler for the "/" url. The body of the route simply renders the index.html template and returns it to the client. We’ll take a look at much more complicated examples of routes later in the book.

Finally at the bottom of the file is this interesting method:

32 (defmethod on-exception ((app <web>) (code (eql 404)))
33   (declare (ignore app))
34   (merge-pathnames #P"_errors/404.html"
35                    *template-directory*))

In case no route matches the url of the request, this method will handle it by sending back a 404 Not Found response.

A few words about Clack/Lack

TBW

Running the app

The way clack based apps are run is with the clack:clackup function. Other than a function or a component it can also take an app file that configures our application as an argument:

1 (clack:clackup #P"/path/to/app.lisp" args)

This is useful since we might want to add middleware and other options to our app and we need a place where we can do that. Also the file can be used to start our application from the command line. We’ll see how that might be done in later chapters. In our case, caveman defined one such file in the root of our project called app.lisp, let’s take a look:

 1 (ql:quickload :fullstackwiki)
 2 
 3 (defpackage fullstackwiki.app
 4   (:use :cl)
 5   (:import-from :lack.builder
 6                 :builder)
 7   (:import-from :ppcre
 8                 :scan
 9                 :regex-replace)
10   (:import-from :fullstackwiki.web
11                 :*web*)
12   (:import-from :fullstackwiki.config
13                 :config
14                 :productionp
15                 :*static-directory*))
16 (in-package :fullstackwiki.app)
17 
18 (builder
19  (:static
20   :path (lambda (path)
21           (if (ppcre:scan "^(?:/images/|/css/|/js/|/robot\\.txt$|/favicon\\.ico$\
22 )" path)
23               path
24               nil))
25   :root *static-directory*)
26  (if (productionp)
27      nil
28      :accesslog)
29  (if (getf (config) :error-log)
30      `(:backtrace
31        :output ,(getf (config) :error-log))
32      nil)
33  :session
34  (if (productionp)
35      nil
36      (lambda (app)
37        (lambda (env)
38          (let ((datafly:*trace-sql* t))
39            (funcall app env)))))
40  *web*)

First it uses quicklisp to load our project, since the file itself isn’t part of it. It then imports our app and its configuration and uses lack:builder to build a lack component. It’s setting up static file handling, session and other middleware based on our configuration and applying them to our app. When you call clack:clackup with that file as an argument, it gets evaluated and the result of the lack:builder form is run as our app.

For actually running the app, caveman generated a few convenience functions in fullstackwiki/src/main.lisp let’s take a look:

 1 (in-package :cl-user)
 2 (defpackage fullstackwiki
 3   (:use :cl)
 4   (:import-from :fullstackwiki.config
 5                 :config)
 6   (:import-from :clack
 7                 :clackup)
 8   (:export :start
 9            :stop))
10 (in-package :fullstackwiki)
11 
12 (defvar *appfile-path*
13   (asdf:system-relative-pathname :fullstackwiki #P"app.lisp"))
14 
15 (defvar *handler* nil)
16 
17 (defun start (&rest args &key server port debug &allow-other-keys)
18   (declare (ignore server port debug))
19   (when *handler*
20     (restart-case (error "Server is already running.")
21       (restart-server ()
22         :report "Restart the server"
23         (stop))))
24   (setf *handler*
25         (apply #'clackup *appfile-path* args)))
26 
27 (defun stop ()
28   (prog1
29       (clack:stop *handler*)
30     (setf *handler* nil)))

This file defines our main package fullstackwiki. It also defines our app file and a pair of start/stop functions. As you can see, start checks if the app is running, and if it is, it offers to restart it, otherwise it calls clack:clackup with the file and sets the result to the *handler* variable. Stop is pretty self explanatory.

Conclusion

TBW