I The Basics
What are we trying to do and how do we do it? This should be enough to get you going.
What is literate programming?
A brief description of the philosophy and history of this mode of programming.
LitPro
A description of the tool, hopefully enough to get started with it.
Semi-literate programming
The first step towards literate programming, this is where comments become elevated to be at least as important, if not more important, than the code. Essentially, comments are the default text mode and code must be initiated.
Symphonic programming
Humans do not think in the same way computers process. Respecting the order of human thought is what symphonic programming is about. We cut up the code in many pieces, scatter it around and then the program acts as a conductor to put it all together in a harmonious symphony of beautiful code.
Artistic programming
This is the full power of LitPro released. The bits of code can be processed in arbitrary ways. We are not bound by the dictates of the language and can craft the code in whatever elegant manner we wish to craft it. With great power comes the possibilty of making a mess. This is painting code on a canvas with the blending of the bits and pieces in ways that are very powerful.
1. What is literate programming?
Literate programming is a way of organizing code developed in the late ’70s by Donald Knuth. It allows one to interweave documentation and code snippets laid out in a way agnostic to the underlying language needs. That is, it fits the human need, not the computer need, for how something is organized and explained.
As a first approximation, literate programming allows one to explain the purpose of the code, how it fits in with the larger program, what this specific bit is supposed to do, and some non-obvious maneuvers in the code. The details of how, of course, can be left to well-written code, but the hope is that the prose and section headings allow one to navigate around to the interesting bits.
This approximation is the elevation of comments out of code and into the primary view with code descending into a secondary viewpoint.
This is semi-literate programming and is the most often implemented view, being the easiest and fitting with the name “literate”.
At the next level of understanding, it allows one to sculpt the code to fit the human mind. The most interesting bits can be placed first. Or the overall structure can be laid out first. Whatever makes more sense, it can be done that way. Much like how a story may be laid out in non-chronological order for maximum impact of the human reading it, so too does literate programming enable the organization of code for maximal human understanding by shedding the dependency on what the computer requires for organization.
Just as humans may disagree on rather a story is told well so to does this increase the potential conflicts between reader and author of code. It becomes dependent on the author to try and think in the perspective of the reader as to how to present it well enough to the expected audience. This is not an easy task.
This is the level of arbitrary ordering of snippets. We can go even further and have arbitrary input and output files. That is, multiple literate program files can combine to make a single or multiple output files. This allows us to not only reorganize code within a single file, but organize the code across the project.
We could, for example, use a literate program in web development in which some html widget, with its css and javascript, all live in a single literate program file but upon compiling, those pieces go to there separate various destinations. We could also choose to do the opposite such as having javascript, say, in its own literate program, but then inject it (or a small, crucial subset of it) into the html file for performance reasons. We fit the organization of the project to the demands of the mental view of the project.
Related to this is the idea of a central project management file. Compile that literate program and it triggers the compilation of the entire project. But more to the point, it would be the entry point, a sort of table-of-contents with explanations, for the whole project. Instead of being limited to expressive directory and file names, one can use arbitrary comments to reveal a clear insight into the whole structure of the project.
We could call this literate projecting. Or perhaps symphonic programming, thinking of it as conducting all the disparate pieces of code threaded together in a harmonious fashion.
The third level is complete and arbitrary control of the code. At this stage, we can run snippets through any process we want. We can write a JSON description of an object and then translate that into a language specific object. We can have bits of markdown or pug and compile it into the same HTML document. We can create our own little mini-language for just that one bit of the project to reduce the amount of code written and have the literate program create the final rendered version.
We can also take almost common bits of code and programmatically change them as needed. This is a middle ground between copy-and-paste versus creating functions to implement common code.
This third level is where the bulk of the effort of litpro was spent. It is not hard to do the first two levels, particularly if syntax is not too worried about. But the third opens an entire world of asynchronous text processing with arbitrary tools. Here convenience is extremely important and, as anyone knows, the price of convenience is complexity.
This level is perhaps described as artistic programming. It allows for an unconstrained canvas to paint the code.
As we go up the levels, each one is increasingly more complex and potentially dangerous for the undisciplined. But it is also a big opportunity to learn discipline. There is nothing quite like making a mess to teach one discipline in coding.
2. LitPro
Here we briefly explain the tool that this book describes in detail.
The idea of litpro is to write literate programs in markdown. This is not a new idea. Markdown is a wonderfully simple format where paragraphs are normal text and indented blocks (or fenced off blocks) are treated as code.
Markdown has wide support and, in particular, GitHub supports viewing markdown quite nicely.
Litpro uses the CommonMark variant. This is a highly specified version with edge cases laid out in detail. Anything it considers a heading or a code block is what Litpro considers it to be. Some of the syntax choices have been constrained by following this format, but for the most part, these constraints have been most welcomed.
Litpro is written in JavaScript. The command-line tool runs using node.js and can be installed using npm install -g litpro. This is good for explorations and initializations. Use it by using the command litpro filename.
Any serious use of the tool for the project should have litpro, strictly versioned, as a
development dependency in the package.json file and refer to it that way. When
installed as a dependency, the command can be located from the top directory
as node_modules/.bin/litpro
To initiate a package.json file and put litpro as a dev dependency try
You can hook up scripts in the package.json file to make it easier to run. There are also configuration options and a file that we will get into eventually. We will also detail setting up a watch system to continuously compile and test the development.
Litpro is free and open source. Issue request are most welcome and you can find its repository at litpro
The command-line interface is a thin wrapper around a library which is platform agnostic (hopefully). The hope is to one day have a browser based version and the decoupling of the library from the command-line will be most useful to that end. The library can be found at literate-programming-lib
There are also some plugins available. The convention is to call them litpro-...
For example litpro-jshint implements a jshint command for applying that
library to JavaScript code in order to find likely errors.
But there is often very little that needs to be done to use an external node module. Thus, there is not a need to have very many plugins. Instead, we can write the code needed out in the configuration file.
There is one other version which is called literate-programming This was the original client, but it has been completely replaced with litpro powering it. It does differ from litpro in that it comes with “batteries included” for web development. While it is more than impossible to include much of what is used in web development, this does attempt to include some of those modules found most useful by the author. At the present time, they include a markdown compiler, jshint, some beautifiers and some minifiers, pug (formerly jade), and postcss.
Much of the inspiration for this tool comes from web development. The number of different languages and tools involved in web development coupled with best practices makes web development quite difficult. The hope is that this tool makes that easier.
While targeted at web development, it is by no means the only use. It is language agnostic and can be used for doing any type of text creation. It takes in markdown files and produces text files. Some languages need much more from an environment (such as code completion) which is orthogonal to this issue.
One possible solution to this is to use named language types for fenced code blocks and, if the editor is made aware of it, using that to do code completion, at least of the canonical language features.
3. Semi-literate programming
Here we explain the simplest use of litpro: elevating comments to readable paragraphs.
The idea here is quite simple. Write paragraphs for comments. Then use either indented blocks or code fences to write code blocks.
Here is a simple example:
With that text saved in a file, we can run litpro on it and it should produce
the file teens.js consisting of
console.log("hi");
setTimeout(bye, 1000);
function bye () {
console.log("bye");
}
This uses the save directive which is, at its simplest, [filename](# "save:")
where filename is where to save the file. The hash symbol says to use the current section; we’ll see later how to reference other sections.
4. Symphonic Programming
Here we want to explore reordering code. And to discuss it, let’s first talk about this.
So we have a conditional with some complicated bodies. And maybe we want to move those complications elsewhere to make it easier to understand the conditional flow. Typically, we would only have functions to do that role:
That works. But notice that we have introduced functions solely for the convenience of human reading. It adds in the complexity of new scopes and makes it harder to deal with error conditions effectively. It also means we never get to see the compiled code in its context.
In contrast, we can do the following with litpro.
This will compile to the first block. It allows us to see in the literate program the outline of that section and see the other bits separated. But if we want to see all the context together as the computer sees it, we can do that as well. We also do not lose the surrounding scope or placement in the flow. These are powerful tools at times.
4.1 Syntax
The syntax for different blocks, at its simplest, is a header block making a
new section and then referencing that section with _"section name" in a
code block. For example,
The above snippet will create
Note that for blank lines, if you want trailing or leading newline from a code block, you need to use the fenced blocks.
Some symbols are allowed in the heading. Given how it is used in the syntax,
pipes are not allowed and quotes can lead to conflicts (obviously ## "Quote"
has problems with _""quote"" but should be fine with _'"Quote"'
This syntax works pretty well, but there are a couple of more tricks to learn. In particular, colons are not allowed in headings either as they have been co-opted for minor blocks, discussed below.
The available markdown header syntaxes are those that convert into h1 to h4 headers, namely
#, ##, ###, #### or either of the underline heading syntax. Those headers become references to the code in their blocks. h5 and h6
headers are reserved for something else which is discussed later.
References should be unique. If you reuse the same reference name (the same h1 to h4 header twice in the same file), then it will concatenate the blocks. h5 and h6 headers can be repeated without them being combined, as will be discussed later.
One can use the same reference as many times as one likes.
One can use any of the quotes (", ', `) to start a substitution, just use the same quote to end.
Minor blocks
A minor block is designed for little snippets that seem too minor to create another section heading, but that one wants to move out of the way anyway. In a markdown toc, these will not appear nor do they need to be unique between different sections though they should be unique within one section.
A minor block is initiated with either a link whose title starts with a colon
or one whose target is completely empty. Let’s assume this is in section
bob.
[jane](# ":") would create reference bob:jane while
[jack 2]() would create reference bob:jack 2
Within section bob, we can reference it as ":jane" or ":jack 2" and
outside of bob, we need the full syntax of "bob:jane" and "bob:jack 2"
We can also do a save directive using section names as such
[whatever](#:jane "save:") if we are in the bob section or
[whatever](#bob:jane "save:") if not.
This works quite nicely (for headers, not minors) in terms of being an actual link when viewed on GitHub for on-minor references. For example, [whatever](#bob-is-cool "save:") will save the section “Bob is cool” and link to it.
Other literate program files
To use other literate program files, we use the load directive. We then use a
double colon syntax to refer to it. Let’s assume we have a file called
cool.md. Then we can load it in a litpro document using
[great](cool.md "load:") and then reference section bob:jane in it using
_"great::bob:jane" The load syntax has the alias as the linkname (bit in
square brackets) but it can be omitted. In either case, the filename itself
can be referenced. That is, in the above example we can also use
_"cool.md::jane"
Full Example
Here we have a full example with the different syntaxes being used. We will load from two files and save to three.
load.md is the main entry point. One would compile it with litpro -s . load.md The -s says to look for any other literate document to be loaded in the current directory; the default is src.
<html>
<head>
_"widget::files link"
</head>
<body>
<h2> Widgets for everybody!</h2>
_"sp-load2.md::html:top"
<p> snuck something in! </p>
_"widget::html:bottom"
</body>
</html>
[full.html](# "save:")
The above code loads up load2.md which has some nice bits in it.
When compiled, we get widget.js:
document.addEventListener("DOMContentLoaded", function () {
var button = document.querySelector(".widget button");
var h2 = document.querySelector(".widget h2").classList;
button.addEventListener("click", function () {
h2.add("big");
setTimeout(function () {
h2.remove("big");
}, 1000);
});
});
widget.css:
.widget h2 {
background-color : red;
}
.big {
font-size: 5em;
}
and full.html:
<html>
<head>
<link rel="stylesheet" href="widget.css" />
<script src="widget.js"></script>
</head>
<body>
<h2> Widgets for everybody!</h2>
<div class="widget">
<h2>Click a button</h2>
<p> snuck something in! </p>
<button>Awesome!</button>
</div>
</body>
</html>
4.2 Functions or reordering
A common point of debate is whether this reordering is significant. The argument is that in modern languages, the ability to define functions after they have been called allows one to shift the code elsewhere.
While true, this muddies the role of functions a bit. Functions are best used
to be repeated bits of code that are called over and over. In old JavaScript,
functions are also the only way to have scoped variables, but with the let
keyword, that has changed. Both are considerations for the computer and are
good reasons to use functions.
But to use functions solely for human readable purposes at the expense of increased complexity and cognitive load seems like a poor trade. Reordering using literate programming allows one complete freedom in how to cut up the code without any extra programming complexity. There is no wrestling with scope because of that reordering. There is no increased levels of redirection nor restrictions from redirecting from within the local code. It simply shifts the placement for the sole purpose of humans reading the code.
It is separation of concerns at its best. We have computer needs and we have human needs. We need to cater to both. Most paradigms try to balance the two needs simultaneously with various costs and trade-offs. This approach allows us to satisfy both constituents with the small price of an extra compile (or transpile) step.
5. Artistic programming
And so we reach the third and final level of our journey. The other two levels are pretty much complete in explaining them and how to use them. What follows for the rest of this tome is all about the third level.
The third level is the shedding of constraints. We take those snippets of code from the second level and we run them through code transformations. This enables us to do just about anything. Even more, we can call out to external programs, feeding the snippets in as standard input and receiving the output as the next form of the snippet.
This is the level of complete artistry. We can write little mini-languages to make efficient input. We can use a wide variety of text transformations of one language to the next all smoothly in the code. We can weave in user-end documentation, tests, sanity checks, compile contexts, and whatever else we can dream of, to make the job of coding, and all of its ancillary tasks, efficient and enjoyable.
The basic command syntax uses pipe syntax. Instead of just a reference,
such as _"name", we can pipe the incoming text into a command, or series of
commands, such as _"name | cat awesome, _"dude" | sub A, b". This would
take the text in the reference name and send it through the command cat which
concatenates stuff together. In particular it would join the incoming text
with awesome and the stuff in the dude section. Then that resulting text
goes through the sub command and all instances of A get replaced with b.
Note that section names can be referenced in the command arguments.
Sub
To start with, let’s say we want to create functions for the arithmetic
operators. That is, we want to be able to write add(4, 5) instead of 4+5.
In JavaScript, we could do
And that would be fine. We could then do that for subtraction, multiplication, and division. But that is a bit repetitive. So instead we can do
And that generates the following in a file called ops.js.
var add = function (a, b) {
return a + b;
}
var mul = function (a, b) {
return a * b;
}
var div = function (a, b) {
return a / b;
}
var sub = function (a, b) {
return a - b;
}
This is not that exciting of an example. And there are a number of ways we could have approached this with a simple copy and paste being one of them. Indeed, the four lines of litpro code is copy and pasted (it is also possible to run some custom code to do this more elegantly, but this is a simple introduction).
But you may wonder what is the advantage? The creative effort is about the same level, but now all four are dependent on each other. This may be good, it may not be good. It is good if you later decide to modify this. For example, you may want to check that the operators are numbers. Or you may want to round the result or cast it into some other form. By modifying the originating function, we get all of them modified at once. That is, the benefit is in the maintenance.
The downside is that they are coupled. So, for example if we want to inject a test for division by zero, we need to add something to facilitate that. One solution is to put in a comment line that gets replaced with what you need.
Turning into:
var add = function (a, b) {
if (Number.isNumber(a) && Number.isNumber(b) ) {
//rep
return a + b;
} else {
return NaN;
}
}
var mul = function (a, b) {
if (Number.isNumber(a) && Number.isNumber(b) ) {
//rep
return a * b;
} else {
return NaN;
}
}
var div = function (a, b) {
if (Number.isNumber(a) && Number.isNumber(b) ) {
if (b === 0) {
if (a > 0) {
return Infinity;
} else if (a < 0) {
return -Infinity;
} else {
return 1;
}
}
return a / b;
} else {
return NaN;
}
}
var sub = function (a, b) {
if (Number.isNumber(a) && Number.isNumber(b) ) {
//rep
return a - b;
} else {
return NaN;
}
}
This could easily become cumbersome. As is generally true with this style of programming, it is a matter of artistic choice as to which is the most useful and clear in any given situation. It is likely to evolve over time just as repeating similar code often evolves into pulling it out to functions. This is just one of the stops along the way.
5.1 Web
As another example, let’s say we are creating a web page. We want to have the
content written in markdown, but we want to create the structure in pug (formerly the jade language).
Let’s presume that the commands to convert them are md and pug,
respectively.
This yields
<nav>
<div class="left">Click</div>
<div class="right">Clack</div>
</nav>
<article><h1>Best Webs Ever</h1>
<p>We have the best webs <strong>ever</strong>.</p>
<h2>Silk</h2>
<p>Yeah, we got spider silk. <a href="kevlar.html">Kevlar strength</a>. Real coo\
l.</p>
</article>
<aside><p><em>Open Now!</em></p>
</aside>
<script>var a, b;
if (a = b) {
a = 2;
} else {
// never reached;
}</script>
The easiest way to run this is with literate-programming which has both markdown and pug included.
But to run this with litpro, one needs to use the lprc.js file that defines these commands.
There are other methods that we’ll discuss later, but this is perhaps the most useful method.
The lprc.js file requires the modules markdown-it and pug. We also include jshint which was done to check the script that we snuck in there.
module.exports = function (Folder, args) {
if (args.file.length === 0) {
args.file = ["web.md"];
}
if (!Folder.prototype.local) {
Folder.prototype.local = {};
}
require('litpro-jshint')(Folder, args);
var jade = require('pug');
Folder.sync("pug" , function (code, args) {
options = args.join(",").trim();
if (options) {
options = JSON.parse(options);
} else {
options = {'pretty':true};
}
return jade.render(code, options);
});
var md = require('markdown-it')({
html:true,
linkify:true
});
Folder.prototype.local.md = md;
Folder.sync( "md", function (code, args) {
return md.render(code);
});
};
We will discuss a lot of what is going on in this example as time goes on. But notice how we can cut up our assembly of the material, run them through commands, and get a nice compiled output.
The backslashes are escaping the substitution in the first round as mixing pug with compiled HTML content does not lead to good results. So we compile the pug and then when that is ready, we call the compile command to run that through our standard compiler. In this example, we need to include the structure named section as the overarching section.