Introduction

This is a book that specifically covers the author’s tool for doing literate programming. The tool is freely available via npm, with source code available on github.

The tool implements a version of Don Knuth’s idea of literate programming. The basic idea is to write code for being read and maintained rather than for the need of the computer. One can reorder code, cut up code into different snippets, promote comments to a top level, and do arbitrary transformations to the input text before outputting the code.

This book covers both the idea and the tool. Most of it is about what this specific tool does.

Part I, which is The Basics, covers the 80% of needs and uses. It is a fairly simple and straightforward concept. The freedom to break code up arbitrarily is quite nice and this part explains how to go about doing this.

Part II details more about what I term “artistic programming”. This is very much concerned with arbitrary actions and transformations of the compilation phase. At its core, it takes input text, does some stuff to it, and gives an output that is then arranged into the final code. The transformation is the artistic part, the assembling of small snippets is more of a symphonic part.

Transformations opens a wide world of possibilities and Part II delves into various scenarios and how to deal with them. This is a bit of a cookbook approach.

Part III focuses on this as a tool for the web environment. The web stack is a mess of different technologies, optimizations, and many files. The tool can handle all of this and this is the place to explain it in detail.

Part IV is a reference section. Every piece of syntax is documented, from the format to the commands, directives, and subcommands.

Part V is not included, but links are provided to PDFs of the various literate programs that make up this tool. Yes, the tool compiles itself. It lightly discusses the structure of the tools.

A note on terminology. I use the term compile because this transforms source code from one thing to another. Often it will be transpiling, that is, transforming from one language to another on the same level of abstraction, but there is nothing inherent about the process that limits us to transpiling. You could set it up to generate machine code, if you really want to. You can read a little more about compiling vs. transpiling if you like.

I The Basics

What are we trying to do and how do we do it? This should be enough to get you going.

What is literate programming?

A brief description of the philosophy and history of this mode of programming.

LitPro

A description of the tool, hopefully enough to get started with it.

Semi-literate programming

The first step towards literate programming, this is where comments become elevated to be at least as important, if not more important, than the code. Essentially, comments are the default text mode and code must be initiated.

Symphonic programming

Humans do not think in the same way computers process. Respecting the order of human thought is what symphonic programming is about. We cut up the code in many pieces, scatter it around and then the program acts as a conductor to put it all together in a harmonious symphony of beautiful code.

Artistic programming

This is the full power of LitPro released. The bits of code can be processed in arbitrary ways. We are not bound by the dictates of the language and can craft the code in whatever elegant manner we wish to craft it. With great power comes the possibilty of making a mess. This is painting code on a canvas with the blending of the bits and pieces in ways that are very powerful.

1. What is literate programming?

Literate programming is a way of organizing code developed in the late ’70s by Donald Knuth. It allows one to interweave documentation and code snippets laid out in a way agnostic to the underlying language needs. That is, it fits the human need, not the computer need, for how something is organized and explained.

As a first approximation, literate programming allows one to explain the purpose of the code, how it fits in with the larger program, what this specific bit is supposed to do, and some non-obvious maneuvers in the code. The details of how, of course, can be left to well-written code, but the hope is that the prose and section headings allow one to navigate around to the interesting bits.

This approximation is the elevation of comments out of code and into the primary view with code descending into a secondary viewpoint.

This is semi-literate programming and is the most often implemented view, being the easiest and fitting with the name “literate”.

At the next level of understanding, it allows one to sculpt the code to fit the human mind. The most interesting bits can be placed first. Or the overall structure can be laid out first. Whatever makes more sense, it can be done that way. Much like how a story may be laid out in non-chronological order for maximum impact of the human reading it, so too does literate programming enable the organization of code for maximal human understanding by shedding the dependency on what the computer requires for organization.

Just as humans may disagree on rather a story is told well so to does this increase the potential conflicts between reader and author of code. It becomes dependent on the author to try and think in the perspective of the reader as to how to present it well enough to the expected audience. This is not an easy task.

This is the level of arbitrary ordering of snippets. We can go even further and have arbitrary input and output files. That is, multiple literate program files can combine to make a single or multiple output files. This allows us to not only reorganize code within a single file, but organize the code across the project.

We could, for example, use a literate program in web development in which some html widget, with its css and javascript, all live in a single literate program file but upon compiling, those pieces go to there separate various destinations. We could also choose to do the opposite such as having javascript, say, in its own literate program, but then inject it (or a small, crucial subset of it) into the html file for performance reasons. We fit the organization of the project to the demands of the mental view of the project.

Related to this is the idea of a central project management file. Compile that literate program and it triggers the compilation of the entire project. But more to the point, it would be the entry point, a sort of table-of-contents with explanations, for the whole project. Instead of being limited to expressive directory and file names, one can use arbitrary comments to reveal a clear insight into the whole structure of the project.

We could call this literate projecting. Or perhaps symphonic programming, thinking of it as conducting all the disparate pieces of code threaded together in a harmonious fashion.

The third level is complete and arbitrary control of the code. At this stage, we can run snippets through any process we want. We can write a JSON description of an object and then translate that into a language specific object. We can have bits of markdown or pug and compile it into the same HTML document. We can create our own little mini-language for just that one bit of the project to reduce the amount of code written and have the literate program create the final rendered version.

We can also take almost common bits of code and programmatically change them as needed. This is a middle ground between copy-and-paste versus creating functions to implement common code.

This third level is where the bulk of the effort of litpro was spent. It is not hard to do the first two levels, particularly if syntax is not too worried about. But the third opens an entire world of asynchronous text processing with arbitrary tools. Here convenience is extremely important and, as anyone knows, the price of convenience is complexity.

This level is perhaps described as artistic programming. It allows for an unconstrained canvas to paint the code.

As we go up the levels, each one is increasingly more complex and potentially dangerous for the undisciplined. But it is also a big opportunity to learn discipline. There is nothing quite like making a mess to teach one discipline in coding.

2. LitPro

Here we briefly explain the tool that this book describes in detail.

The idea of litpro is to write literate programs in markdown. This is not a new idea. Markdown is a wonderfully simple format where paragraphs are normal text and indented blocks (or fenced off blocks) are treated as code.

Markdown has wide support and, in particular, GitHub supports viewing markdown quite nicely.

Litpro uses the CommonMark variant. This is a highly specified version with edge cases laid out in detail. Anything it considers a heading or a code block is what Litpro considers it to be. Some of the syntax choices have been constrained by following this format, but for the most part, these constraints have been most welcomed.

Litpro is written in JavaScript. The command-line tool runs using node.js and can be installed using npm install -g litpro. This is good for explorations and initializations. Use it by using the command litpro filename.

Any serious use of the tool for the project should have litpro, strictly versioned, as a development dependency in the package.json file and refer to it that way. When installed as a dependency, the command can be located from the top directory as node_modules/.bin/litpro

To initiate a package.json file and put litpro as a dev dependency try

npm init
npm install litpro --save-dev

You can hook up scripts in the package.json file to make it easier to run. There are also configuration options and a file that we will get into eventually. We will also detail setting up a watch system to continuously compile and test the development.

Litpro is free and open source. Issue request are most welcome and you can find its repository at litpro

The command-line interface is a thin wrapper around a library which is platform agnostic (hopefully). The hope is to one day have a browser based version and the decoupling of the library from the command-line will be most useful to that end. The library can be found at literate-programming-lib

There are also some plugins available. The convention is to call them litpro-... For example litpro-jshint implements a jshint command for applying that library to JavaScript code in order to find likely errors.

But there is often very little that needs to be done to use an external node module. Thus, there is not a need to have very many plugins. Instead, we can write the code needed out in the configuration file.

There is one other version which is called literate-programming This was the original client, but it has been completely replaced with litpro powering it. It does differ from litpro in that it comes with “batteries included” for web development. While it is more than impossible to include much of what is used in web development, this does attempt to include some of those modules found most useful by the author. At the present time, they include a markdown compiler, jshint, some beautifiers and some minifiers, pug (formerly jade), and postcss.

Much of the inspiration for this tool comes from web development. The number of different languages and tools involved in web development coupled with best practices makes web development quite difficult. The hope is that this tool makes that easier.

While targeted at web development, it is by no means the only use. It is language agnostic and can be used for doing any type of text creation. It takes in markdown files and produces text files. Some languages need much more from an environment (such as code completion) which is orthogonal to this issue.

One possible solution to this is to use named language types for fenced code blocks and, if the editor is made aware of it, using that to do code completion, at least of the canonical language features.

3. Semi-literate programming

Here we explain the simplest use of litpro: elevating comments to readable paragraphs.

The idea here is quite simple. Write paragraphs for comments. Then use either indented blocks or code fences to write code blocks.

Here is a simple example:

# Saying Hi

We want to write a little javascript command that says hi, waits one second,
and then says bye. We will save it in [teens.js](# "save:")

Greetings are great, right?

    console.log("hi");

Timer is in milliseconds, so 1000. We'll call the function `bye` defined
elsewhere

    setTimeout(bye, 1000);

And now let's define the bye function. For no apparent reason, we use a named
fence block. Hey, maybe we'll get some syntax highlighting!

```js
function bye () {
    console.log("bye");
}
```

Hey, we're all done!

With that text saved in a file, we can run litpro on it and it should produce the file teens.js consisting of

console.log("hi");
setTimeout(bye, 1000);
function bye () {
    console.log("bye");
}

This uses the save directive which is, at its simplest, [filename](# "save:") where filename is where to save the file. The hash symbol says to use the current section; we’ll see later how to reference other sections.

4. Symphonic Programming

Here we want to explore reordering code. And to discuss it, let’s first talk about this.

a, b ....
if (crazy) {
    // lots of lines
    c = a;
} else if (slightlyCrazy) {
    // fewer lines
    c = b;
} else {
    // very few lines
    c = d;
}

So we have a conditional with some complicated bodies. And maybe we want to move those complications elsewhere to make it easier to understand the conditional flow. Typically, we would only have functions to do that role:

a, b, ...
if (crazy) {
    c = lots(a, b);
} else if (slightlyCrazy) {
    c = fewer(a, b); 
} else {
    c = few(a, b);
}

function lots (a, b) {
    // lots of lines
    return a;
} 

function fewer(a,b) {
    // fewer lines
    return b;
} 

function few(a,b) {
    // very few lines
    return d;
}

That works. But notice that we have introduced functions solely for the convenience of human reading. It adds in the complexity of new scopes and makes it harder to deal with error conditions effectively. It also means we never get to see the compiled code in its context.

In contrast, we can do the following with litpro.

a, b ....
if (crazy) {
    _"lots"
    c = a;
} else if (slightlyCrazy) {
    _"fewer"
    c = b;
} else {
    _"few"
    c = d;
}

## Lots

Here we ...
    
    //lots of lines

## Fewer

And with less code we do 

    // fewer lines

## Few
   
    // very few lines

This will compile to the first block. It allows us to see in the literate program the outline of that section and see the other bits separated. But if we want to see all the context together as the computer sees it, we can do that as well. We also do not lose the surrounding scope or placement in the flow. These are powerful tools at times.

4.1 Syntax

The syntax for different blocks, at its simplest, is a header block making a new section and then referencing that section with _"section name" in a code block. For example,

Some text

## Awesome details-jack

We have some awesome code

    Yay      

 but what? 

 ```
 yeah
 
 ```

 ## Importing

 Here we put in the awesome

    So jack says

    _"awesome details-jack"

    And we get what we want

The above snippet will create

    So jack says
    
    Yay
    yeah
    

    And we get what we want

Note that for blank lines, if you want trailing or leading newline from a code block, you need to use the fenced blocks.

Some symbols are allowed in the heading. Given how it is used in the syntax, pipes are not allowed and quotes can lead to conflicts (obviously ## "Quote" has problems with _""quote"" but should be fine with _'"Quote"'

This syntax works pretty well, but there are a couple of more tricks to learn. In particular, colons are not allowed in headings either as they have been co-opted for minor blocks, discussed below.

The available markdown header syntaxes are those that convert into h1 to h4 headers, namely #, ##, ###, #### or either of the underline heading syntax. Those headers become references to the code in their blocks. h5 and h6 headers are reserved for something else which is discussed later.

References should be unique. If you reuse the same reference name (the same h1 to h4 header twice in the same file), then it will concatenate the blocks. h5 and h6 headers can be repeated without them being combined, as will be discussed later.

One can use the same reference as many times as one likes.

One can use any of the quotes (", ', `) to start a substitution, just use the same quote to end.

Minor blocks

A minor block is designed for little snippets that seem too minor to create another section heading, but that one wants to move out of the way anyway. In a markdown toc, these will not appear nor do they need to be unique between different sections though they should be unique within one section.

A minor block is initiated with either a link whose title starts with a colon or one whose target is completely empty. Let’s assume this is in section bob.

[jane](# ":") would create reference bob:jane while [jack 2]() would create reference bob:jack 2

Within section bob, we can reference it as ":jane" or ":jack 2" and outside of bob, we need the full syntax of "bob:jane" and "bob:jack 2"

We can also do a save directive using section names as such [whatever](#:jane "save:") if we are in the bob section or [whatever](#bob:jane "save:") if not.

This works quite nicely (for headers, not minors) in terms of being an actual link when viewed on GitHub for on-minor references. For example, [whatever](#bob-is-cool "save:") will save the section “Bob is cool” and link to it.

Other literate program files

To use other literate program files, we use the load directive. We then use a double colon syntax to refer to it. Let’s assume we have a file called cool.md. Then we can load it in a litpro document using [great](cool.md "load:") and then reference section bob:jane in it using _"great::bob:jane" The load syntax has the alias as the linkname (bit in square brackets) but it can be omitted. In either case, the filename itself can be referenced. That is, in the above example we can also use _"cool.md::jane"

Full Example

Here we have a full example with the different syntaxes being used. We will load from two files and save to three.

load.md is the main entry point. One would compile it with litpro -s . load.md The -s says to look for any other literate document to be loaded in the current directory; the default is src.

# Full HTML

Here we are building an HTML page. We will use a
[widget](load2.md "load:") that comes from another literate program. 

    <html>
        <head>
            _"widget::files link"
        </head>
        <body>
            <h2> Widgets for everybody!</h2>
            _"sp-load2.md::html:top"
            <p> snuck something in! </p>
            _"widget::html:bottom"
        </body>
    </html>

[full.html](# "save:")

The above code loads up load2.md which has some nice bits in it.

# Widget

Here we define a widget. This widget will be amazing!

We will save two files from here 

* [widget.js](#js "save:")
* [widget.css](#css-for-widget "save:")

## Files link

This is the loading tags for our pages

    <link rel="stylesheet" href="widget.css" />
    <script src="widget.js"></script>

## HTML

Here we put the html for our widget. We have a top and a bottom. The top
contains the div and a heading.  The minor block syntax can be either of the
below. 

[top]()

    <div class="widget">
        <h2>Click a button</h2>

[bottom](# ":")

    <button>Awesome!</button>
    </div>

## JS

When the button is clicked, we add the big class to the h2 element in the
widget 

    document.addEventListener("DOMContentLoaded", function () {
        _":add click"
    });
   
[add click]()

Here we add a click listener to the button. It should add a class to make the
text big and then one second later remove it.

    var button = document.querySelector(".widget button");
    var h2 = document.querySelector(".widget h2").classList;
    button.addEventListener("click", function () {
        h2.add("big");
        setTimeout(_":remove class", 1000);
    });

[remove class]()

This removes the big class

    function () {
        h2.remove("big");
    }

## CSS for widget

We want the h2 to have a red background!

    .widget h2 {
        background-color : red;
    }

    .big {
        font-size: 5em;
    }

When compiled, we get widget.js:

document.addEventListener("DOMContentLoaded", function () {
    var button = document.querySelector(".widget button");
    var h2 = document.querySelector(".widget h2").classList;
    button.addEventListener("click", function () {
        h2.add("big");
        setTimeout(function () {
            h2.remove("big");
        }, 1000);
    });
});

widget.css:

.widget h2 {
    background-color : red;
}

.big {
    font-size: 5em;
}

and full.html:

<html>
    <head>
        <link rel="stylesheet" href="widget.css" />
        <script src="widget.js"></script>
    </head>
    <body>
        <h2> Widgets for everybody!</h2>
        <div class="widget">
            <h2>Click a button</h2>
        <p> snuck something in! </p>
        <button>Awesome!</button>
        </div>
    </body>
</html>

4.2 Functions or reordering

A common point of debate is whether this reordering is significant. The argument is that in modern languages, the ability to define functions after they have been called allows one to shift the code elsewhere.

While true, this muddies the role of functions a bit. Functions are best used to be repeated bits of code that are called over and over. In old JavaScript, functions are also the only way to have scoped variables, but with the let keyword, that has changed. Both are considerations for the computer and are good reasons to use functions.

But to use functions solely for human readable purposes at the expense of increased complexity and cognitive load seems like a poor trade. Reordering using literate programming allows one complete freedom in how to cut up the code without any extra programming complexity. There is no wrestling with scope because of that reordering. There is no increased levels of redirection nor restrictions from redirecting from within the local code. It simply shifts the placement for the sole purpose of humans reading the code.

It is separation of concerns at its best. We have computer needs and we have human needs. We need to cater to both. Most paradigms try to balance the two needs simultaneously with various costs and trade-offs. This approach allows us to satisfy both constituents with the small price of an extra compile (or transpile) step.

5. Artistic programming

And so we reach the third and final level of our journey. The other two levels are pretty much complete in explaining them and how to use them. What follows for the rest of this tome is all about the third level.

The third level is the shedding of constraints. We take those snippets of code from the second level and we run them through code transformations. This enables us to do just about anything. Even more, we can call out to external programs, feeding the snippets in as standard input and receiving the output as the next form of the snippet.

This is the level of complete artistry. We can write little mini-languages to make efficient input. We can use a wide variety of text transformations of one language to the next all smoothly in the code. We can weave in user-end documentation, tests, sanity checks, compile contexts, and whatever else we can dream of, to make the job of coding, and all of its ancillary tasks, efficient and enjoyable.

The basic command syntax uses pipe syntax. Instead of just a reference, such as _"name", we can pipe the incoming text into a command, or series of commands, such as _"name | cat awesome, _"dude" | sub A, b". This would take the text in the reference name and send it through the command cat which concatenates stuff together. In particular it would join the incoming text with awesome and the stuff in the dude section. Then that resulting text goes through the sub command and all instances of A get replaced with b. Note that section names can be referenced in the command arguments.

Sub

To start with, let’s say we want to create functions for the arithmetic operators. That is, we want to be able to write add(4, 5) instead of 4+5. In JavaScript, we could do

var add = function (a, b) {
    return a + b;
}

And that would be fine. We could then do that for subtraction, multiplication, and division. But that is a bit repetitive. So instead we can do

# Operators

We want to convert the operators to functions.

We start with a basic form using addition

    var OP = function (a, b) {
        return a ?? b;
    }

 When done we will save it in [ops.js](#math-ops "save:")

## Math ops

Let's create the operators

    _"operators| sub OP, add, ??, +"
    _"operators| sub OP, mul, ??, *"
    _"operators| sub OP, div, ??, /"
    _"operators| sub OP, sub, ??, -"

And that generates the following in a file called ops.js.

var add = function (a, b) {
    return a + b;
}
var mul = function (a, b) {
    return a * b;
}
var div = function (a, b) {
    return a / b;
}
var sub = function (a, b) {
    return a - b;
}

This is not that exciting of an example. And there are a number of ways we could have approached this with a simple copy and paste being one of them. Indeed, the four lines of litpro code is copy and pasted (it is also possible to run some custom code to do this more elegantly, but this is a simple introduction).

But you may wonder what is the advantage? The creative effort is about the same level, but now all four are dependent on each other. This may be good, it may not be good. It is good if you later decide to modify this. For example, you may want to check that the operators are numbers. Or you may want to round the result or cast it into some other form. By modifying the originating function, we get all of them modified at once. That is, the benefit is in the maintenance.

The downside is that they are coupled. So, for example if we want to inject a test for division by zero, we need to add something to facilitate that. One solution is to put in a comment line that gets replaced with what you need.

# Operators

We want to convert the operators to functions.

The ?? will be replaced with operator symbols. We check for something to be a
number and then do the computation. This is not really necessary nor perhaps
the best to do it, but there you have it.

    var OP = function (a, b) {
        if (Number.isNumber(a) && Number.isNumber(b) ) { 
            //rep
            return a ?? b;
        } else {
            return NaN;
        }
    }

 When done we will save it in [ops2.js](#math-ops "save:")

## Math ops

Let's create the operators

    _"operators| sub OP, add, ??, +"
    _"operators| sub OP, mul, ??, *"
    _"operators| sub OP, div, ??, /, //rep, _"division by zero" "
    _"operators| sub OP, sub, ??, -"

## Division by zero

The division by zero is so bad so let's deal it. Let's be bad mathematical
people and give some values of oo, -oo, and 1

    if (b === 0) {
        if (a > 0) {
            return Infinity;
        } else if (a < 0) {
            return -Infinity;
        } else {
            return 1;
        }
    }

Turning into:

var add = function (a, b) {
    if (Number.isNumber(a) && Number.isNumber(b) ) { 
        //rep
        return a + b;
    } else {
        return NaN;
    }
}
var mul = function (a, b) {
    if (Number.isNumber(a) && Number.isNumber(b) ) { 
        //rep
        return a * b;
    } else {
        return NaN;
    }
}
var div = function (a, b) {
    if (Number.isNumber(a) && Number.isNumber(b) ) { 
        if (b === 0) {
            if (a > 0) {
                return Infinity;
            } else if (a < 0) {
                return -Infinity;
            } else {
                return 1;
            }
        }
        return a / b;
    } else {
        return NaN;
    }
}
var sub = function (a, b) {
    if (Number.isNumber(a) && Number.isNumber(b) ) { 
        //rep
        return a - b;
    } else {
        return NaN;
    }
}

This could easily become cumbersome. As is generally true with this style of programming, it is a matter of artistic choice as to which is the most useful and clear in any given situation. It is likely to evolve over time just as repeating similar code often evolves into pulling it out to functions. This is just one of the stops along the way.

5.1 Web

As another example, let’s say we are creating a web page. We want to have the content written in markdown, but we want to create the structure in pug (formerly the jade language). Let’s presume that the commands to convert them are md and pug, respectively.

# Webpage

We are going to make a web page. 

[web.html](#structure "save:|log | jade | compile structure ")

## Structure

We use the HTML5 elements of nav, article and aside in their appropriate
roles. The nav is very lame. 

    nav
        .left Click
        .right Clack

    article \_":content |md "

    aside \_"announcements |md "

    script \_"js stuff | jshint "

[content]() 

Here we have some great body content

    # Best Webs Ever

    We have the best webs **ever**. 

    ## Silk

    Yeah, we got spider silk. [Kevlar strength](kevlar.html). Real cool.


### Announcements

Whatever we want to say

    _Open Now!_ 

## js Stuff

Just some random javascript. The jshint will check it for errors and such. 

    var a, b; 
    if (a = b) {
        a = 2;
    } else {
        // never reached;
    }

Jshint just loves it when one uses assignment instead of equality testing in a
boolean expression. 

Note that jshint still allows it to be built. It will just give a warning on
the command line.

This yields

<nav>
  <div class="left">Click</div>
  <div class="right">Clack</div>
</nav>
<article><h1>Best Webs Ever</h1>
<p>We have the best webs <strong>ever</strong>.</p>
<h2>Silk</h2>
<p>Yeah, we got spider silk. <a href="kevlar.html">Kevlar strength</a>. Real coo\
l.</p>
</article>
<aside><p><em>Open Now!</em></p>
</aside>
<script>var a, b; 
if (a = b) {
    a = 2;
} else {
    // never reached;
}</script>

The easiest way to run this is with literate-programming which has both markdown and pug included.

But to run this with litpro, one needs to use the lprc.js file that defines these commands. There are other methods that we’ll discuss later, but this is perhaps the most useful method.

The lprc.js file requires the modules markdown-it and pug. We also include jshint which was done to check the script that we snuck in there.

module.exports = function (Folder, args) {

    if (args.file.length === 0) {
        args.file = ["web.md"];
    }


    if (!Folder.prototype.local) {
        Folder.prototype.local = {};
    }

    require('litpro-jshint')(Folder, args);

    var jade = require('pug');
    
    Folder.sync("pug" , function (code, args) {
        options = args.join(",").trim();
        if (options) {
            options = JSON.parse(options);
        } else {
            options = {'pretty':true};
        }
        return jade.render(code, options); 
    });
    
    var md = require('markdown-it')({
        html:true,
        linkify:true
    });
    
    
    Folder.prototype.local.md = md; 
    
    Folder.sync( "md", function (code, args) {
        return  md.render(code);
    });

};

We will discuss a lot of what is going on in this example as time goes on. But notice how we can cut up our assembly of the material, run them through commands, and get a nice compiled output.

The backslashes are escaping the substitution in the first round as mixing pug with compiled HTML content does not lead to good results. So we compile the pug and then when that is ready, we call the compile command to run that through our standard compiler. In this example, we need to include the structure named section as the overarching section.

II Cookbook

This is where we do a variety of specific examples of use. It mainly focuses on examples from the web stack. In particular, the programming is almost entirely JavaScript.

Constants

We can define constants once with explanations and then put them into whatever files need them.

Eval

How to run some JavaScript

Boilerplate

If you have need of boilerplate, then we can do that with some substitutions for templating it as well.

Making a Command

Commands are very useful. This is how to make them in a literate program.

Making a Directive

Inline custom directives are not that useful, but it can be insightful in going towards making plugins.

Project Files

Often there are other files needing managing for a project other than just code. For example, .gitignore, readme, package.json, … These can all live in a single literate programming document that generates these files. It could also be setup to have variables that you define once and then use when needed, such as the version number.

Making a lprc.js file

What constitutes a lprc.js file. This is where directives and commands are ideally added, either directly or from plugins.

Other languages

If you want to write in your own style or language or whatever, it can do that as well. Write in a way comfortable to you and have it compile it into any target language.

Making a Plugin

Making a plugin leads to good consistent behavior.

Data Entry

Dealing with little bits of data is a difficult problem. Here we look at inline data entry using a quick split style as well as reading in data from an external file. This pertains more to generating written output, then code itself, but it could be needed there as well such as if you want to pull in secrets from a file outside of the repo and put it in the compiled code that is also stored outside of the repo.

Conditionals

We may want to do one thing or another depending on command line options or based on some programmatic condition. We can do that.

Domain Specific Languages

We can write full blown domain specific languages and have them translated at build time. Or you can use standard languages that get transformed, such as markdown into html.

Making a Subcommand

These allow for doing complex actions in arguments. These are different from commands that expect to be a part of a pipe flow.

Linting

We can lint pieces of our code. It would even be possible to lint the pieces in isolation to see what shared variables might pop up.

h5 and h6 Headers

These are headers that generate path-escaped references using the most recent main block reference as the root path. These sections can not only be useful for commonly named parts, but can also have custom behavior automatically acting on the parts.

Testing

Testing should be easy. Here we give a strategy for testing little bits of code (unit testing) as well as testing the combined pieces (integrated testing).

Having fake data specified as well as expected output is also easy.

Documentation

While literate programming is designed to be the documentation for the maintenance programmer, there is still the little issue of user documentation. While that may call for a separate document, it could be included in the same literate program as well using the h5 and h6 mechanism, keeping it with the code.

Debugging

Debugging can be supported by conditional commands.

6. Constants

Let’s say we want to put constants at the top of a file. These could be snippets of texts, color variables for css, computed out numbers (for example, using a more powerful command line software to compute some arcane number and then putting it in various files when needed), etc.

6.1 Evaling

6.2 Command producing constants

7. Eval

Being able to run arbitrary code is a powerful tool. It helps in doing quick debugging as well as quick prototyping of an effect.

7.1 Command Eval

7.2 Command Async

7.3 Directive Eval

8. Boiler Plate

A few examples of doing boiler plate in a better way.

8.1 Repeated Use of Chunks

You can use the same chunk multiple times.

8.2 Evaling

8.3 Minors and Compiling

9. Making a Command

Commands are extremely useful in sculpting the code. You can make your own commands

9.1 Sync command

9.2 Async command

9.3 Plugins

10. Making a Directive

Directives are very powerful. They get invoked when they are encountered which means that defining directives in a document is less useful (and such definitions need to come first). Still, this is a good exercise in moving towards defining directives as part of a plugin.

10.1 Evaling to a Directive

10.2 lprc.js Directives

10.3 Plugins and Directives

11. Project Files

Many projects need a variety of files, not just code. Configurations are often in their own little files, even git has a file of gitignore that you might want to have around. It is possible to generate these files with litpro document and it may allow for easier creation and maintenance of such files.

11.1 Version

11.2 NPM

11.3 Readme

11.4 gitignore

12. lprc.js

This is the configuration file for a literate programming file. What can it do and what should it do?

12.1 Custom commands and directives

12.2 Loading plugins

12.3 Default arguments

13. Other Languages

Sometimes I just want to speak in another dialect. What can LitPro do for me?

13.1 Hyphenated var names

13.2 Space separated to JSON

13.3 CSV to JSON

13.4 Auto-prefixer

13.5 SASS to CSS

?? Example: using hyphens for spaces in javascript.

Example: math algorithm

14. Making a Plugin

Eventually, all good tools need a lot of plugins to be considered successful. So start writing them!

Most plugins are just a thin layer of wrapping over someone else’s carefully written plugin.

We’ll do a few case studies of plugins

14.1 Independent

Using nothing, creating whatever.

14.2 Jade

Very simple

14.3 JSHint

A little bit harder, also trying to shoehorn config details into the plugin

14.4 Markdown, PostCSS

Plugins of markdown-it and using them

15. Data Entry

Getting data to its right place.

15.1 Reading CSV into Template

15.2 Date/Time

both js and command line

15.3 Pulling from web

16. Conditionals

We can set flags for which environment is being run and this can cause different things to happen. One example might be production versus development, but that case might be better handled by simply compiling two versions every time with the output going to two separate places.

The example we will explore here, a real example, is that of a website which needs some information occasionally displayed. The website in our example is a school’s website that needs to display closing information in the event of inclement weather. As some pages have different setups, the exact placement of the announcement varies. Command line flags allow for convenient toggling of the announcements.

16.1 Flags

The two command line flags we will focus on here are the -f and -z flags. The -f flag is for setting the name that follows to true. So -f closed would set the boolean variable closed to false.

The other flag, -z, is a generic flag for passing in information. The term that follows should be of the form varname:value and the program will store the value into the variable name. In particular, one can extract that variable from the Folder.inputs.varname variable which is what is done in the :custom section piped into the evil command.

But there is a better way, implemented after this bit of code. Namely, the z command: z msg does the same.

16.2 Conditional Syntax

We have several conditional related syntax.

There is the if command which takes in a flag name, a command with arguments, and executes that command if the flag is true. This is what we use below.

There is also an if directive which basically does the same; the directive executes if the flag is present. We can also use a flag directive to set flags.

16.3 Full Example

Here is the actual relevant snippet from the school website generating literate program:

## Status

We have some command line flags for operating announcements or other options. 

Flag c  denotes being closed

Flag l means liberal leave

Flag o is for open (but questionable)

Flag z  is a custom flag and looks for the announcement in the
msg variable: `-z msg:...`. 

Example: `literate-programming -f l -f z -z "msg:Two hour delay; opening at 10AM\
";`


The flags are cumulative. 

    _"  |  if c, cat, _":closing | html-wrap p, danger" 
        |  if l, cat, _":liberal |  html-wrap p, warning "
        |  if o, cat, _":open | html-wrap p, okay "
        |  if z, cat, _":custom | evil | html-wrap p, custom" "
    
[closing]()

    WE ARE CLOSED.


[liberal]()

    We are on liberal leave today. Please notify us if you are not attending.

[open]()

    We are open today. 

[custom]()

    ret = doc.Folder.inputs.msg +'';

[css]()

    .danger, .warning, .okay, .custom {
        padding-top:5px;
        padding-bottom:3px;
        text-align:center;
    }
    .danger  { background-color: red; }
    .warning  { background-color: yellow; color:black; }
    .okay  { background-color: green;color:whitesmoke;}
    .custom { background-color:blue; color:whitesmoke;}


[big]()

    This for the homepage in which the announcement text is in h4 size.

    _"status | sub <p, <h4, </p, </h4"

This gets referenced in different ways. For the sidebar present in many of the pages, the block _status is inserted in the sidebar announcement html. If no flags are present, then _status will simply be empt.

The big announcement block is inserted into the home page’s content which happens to be in markdown, but that is not a problem as html gets ignored in the conversion.

For the calendar page, however, it gets generated in a way which is hard to put it in before compiling. So we add it afterwards as part of the save command: ` “save: | cheerio article, prepend, _‘status:big’ “) ` which says to put at the top of the article element the announcement.

16.4 Multiple Compiles

17. Domain Specific Languages

This is an interesting area of use. We could have some general purpose code created from very small little languages. They might be highly constrained and limited, but perhaps sufficient for a project.

17.1 Math Algorithms

17.2 Extra Markdownish

18. Making a Subcommand

This is about exploring the mysterious world of subcommands.

19. Linting

Linting little pieces of code.

20. h5 and h6 Headers

These headers create a path-like structure from the last major section. This allows for common headers that can have interesting stuff happening to them.

20.1 Paths

20.2 Collecting CSS

21. Testing

Here we detail a few different ways that we could use literate programming to help the first bugbear of programming: testing.

21.1 Generating Common Test Environments

21.2 h5 Test Headings

21.3 Inline Eval Testing of Code Snippets

The ultimate in unit testing, we test a variety of pieces of the code immediately.

22. Documentation

Strategies for dealing with the second bugbear of programming: documentation for end users. This is different than documenting for maintenance uses which is what literate programming does best. Here we try to use are tools to make writing end user documentation robust and, most importantly, maintainable. It will follow a similar flow to testing.

23. Debugging

Our third and final bugbear is debugging. How can we use these tools to pepper development code with debugger stuff easily while keeping code clean for production?

III Web Projects

The web is a particularly beautiful hot mess of tech stacks.

We first detail various useful parts related to particular pieces of the stack.

HTML

This is a language that covers both the structure of a webpage and the content. Combining the tasks lead to a language that is a bit painful to write out.

Litpro can help by offering boiler plate with subbing in of content. The boiler plate can be written in a structure focused (templating) language, such as pug, and the content can be written in markdown. Litpro can then transform and stitch together it all.

JS

Litpro is written in JavaScript and thus many of the common commands turn out to be quite useful in writing JavaScript.

There are a few very specific ways that can be very helpful.

The first is using JSHint. You can take any block of JS code and feed it into JSHint to get a log output of issues and problems. It is as easy as js code | jshintassuming you have installed litpro-jshint or using the full literate-programming.

Another handy command, in all versions, is js-string. This takes the text and breaks it by newline and outputs text suitable for creating a string in JS code that returns that text. This enables one to, say, write some HTML for templating and put it in there. In newer versions of JS, one can have multi-line strings, but this works for all versions of JS.

There is also the ife command which creates an immediate function execution around the incoming text. This is again less necessary in the newer variants, but it can still be handy for closures or if one does not want to use the new let version of var.

CSS

This is a language with great potential, but it is hampered by differing browser specs. While that has greatly improved, it is best to let a tool handle the differences. There can also be a lot of repetition at times. Both of these things Litpro can help with.

One can take any incoming text and run it through a preprocessor such as SASS or run it through PostCSS and the wonderful auto-prefixer. The latter is included in literate-programming.

One can also use some snippets or the caps command to help ease some of the more verbose parts of the syntax.

One can also do variables with the substitutions. This may help avoid using something like SASS entirely.

Tidy and Minify

Literate-programming includes tools for formatting the above three languages, either to make them pretty (tidy) or to reduce their size (minify).

One can have multiple directories generated say, one for development and one for production. Using the cd command, this is often quite convenient.

It is also possible to include some code in one, but not the other. Again, this comes to transforming the blocks of text.

Servers

In addition to the static content served up, one often needs dynamic interactions and thus writing a server and using a database comes up.

The same tools above for writing JavaScript are just as helpful with node. In particular, having development and production splits can be very useful.

There is also the database. For a database, one has the language of the server and the database language. Plus, one may have schemas to guide the development, perhaps even a specific language.

This tool may help with that by offering a mapping between the languages. Unlike the magic tools of ORM, this tool does the conversion and allows one to see the output in context.

AWS

AWS is an amazing land of tools. It is all very piecemeal and run by resource policies. There are a lot of configurations to handle and updating requires a process.

It seems like a perfect place for a text transformation tool to run.

writeweb

This is an attempt by the author to create automatic compiling of literate programs. Due to their insecure nature, this would be something that would need to be installed by an individual (or team).

static site

This is where we show how to setup a static site, one in which the content and individual page styling/js is done in a directory of files that are not literate programs, but a literate program processes them.

That is, we make a static site generator that runs as a literate program.

24. HTML

How can we handle HTML gracefully?

24.1 Boilerplate

24.2 Jade

24.3 Markdown

24.4 Template Runs

24.5 Cheerio

25. JavaScript

What can this do for our JavaScript woes?

25.1 A Little Code

If you have just a little bit of JS code, maybe you just want to put it in the page?

25.2 Linting

25.3 Testing

25.4 h5

25.5 Helpers

25.6 Transpilers

26. CSS

CSS is hard to organize correctly. However you decide to do it, this can work for you.

26.1 PostCSS

26.2 SASS

27. Servers

Web content needs to be served. Here are some ways this tool can help.

27.1 Making a Server

27.2 Pushing Content to Server

27.3 Pushing Content to S3

27.4 Database managing

28. AWS Lambda

AWS Lambda is awesome. But if anything needs an organizational strategy, it would be AWS Lambda. Here is how.

29. Writeweb

This is how to write up a literate program and have it compiled automatically.

This describes the setup of the writeweb server…

30. Static Sites

Putting it all together to show how to do great web development quickly and easily for whatever your needs are. Hah. Might as well talk about mobile development….

IV Reference

This part is the technical documentation of literate programming. While it does document itself, in some sense, this is a ready index of syntax, directives, commands, subcommands, and plugins.

Most of this applies to literate-programming core library, but there are directives and commands that may only be applicable in the command client. Those are marked in their own section, either with Litpro (thin and fat client applicable) or Literate-Programming (just the fat client)

Syntax

Here we describe the syntax as formally as possible.

Commonmark

This is the markdown specification and parser that we use to determine when something is a header, a code block or link. The rest of the syntax is not relevant for our purposes though it may be of relevance to readers.

Code blocks

Our main concern is to transform and string together code blocks. A code block is something that essentially is indented by 4 spaces or is fenced off (3 backticks before and after).

Minor Blocks

A minor block is a block that is being referenced by two parts: a header block name and, after a colon, the minor block’s name.

References and Substitutions

References are what we use to refer to the chunks of code, or rather, what we call the names of those chunks.

Substitutions are the syntax saying that something else should go there. Only substitutions and their escaped versions cause a change in the code chunks.

Substitutions can have null references. The point of these would be the pipes and commands.

References can occur before the first pipe or as part of a command or as part of a directive.

Pipes

These initiate commands that transform the incoming text.

Directive Syntax

This is a link syntax with the title text starting with a word followed by a colon. Each part of the link may or may not have relevance to the directive.

Command Syntax

No parentheses required for command arguments.

Subcommand Syntax

These are functions in the arguments of a command. These require parentheses.

Directives

Directives are generally intended for using text in a certain way, such as saving it to a file or defining a command with it. There is also the eval syntax which will act immediately upon being encountered.

Commands

Much of the heart of the transformations are commands and there are a great many commands that come by default.

Subcommands

These help in getting the right kind of arguments into a command. One can create arrays, objects, booleans, numbers, etc., as well as manipulate such structures.

Command-line

The command line programs have flags that we explain here.

Plugins

Plugins are a way to have commonly used function easily packaged up and used.

lprc.js

This is a file that gets executed upon startup. It is the place to load up plugins, define commands, directives, and defaults.

Compile Events

Here we detail some of the parts of the program that one may want to access for a variety of reasons. Further details can be read in the literate source code, mainly in that of literate-programming-lib.

31. A Complete and Full Syntax Specification

31.1 Document syntax

A literate program is a markdown document with some special conventions.

The basic idea is that each header line (regardless of level, either atx # or seText underline ) demarcates a full block. Code blocks within a full block are the bits that are woven together.

Code Block

Each code block can contain whatever kind of code, but there is a primary special syntax.

_"Block name" This tells the compiler to compile the block with “Block name” and then replace the _"Block name" with that code.

Note the the allowed quotes are double, single, and backtick. Matching types are expected. And yes, it is useful to have three different types.

The full syntax is something of the form _"scope name::block name:minor block name | cmd arg 1, arg 2 | cmd2 |cmd3 ..." where the scope name allows us to refer to other documents (or artificial common scopes) and the commands run the output of one to the input of the other, also taking in arguments which could they themselves be block substitutions.

Note that one can also backslash escape the underscore. To have multiple escapes (to allow for multiple compiling), one can use \#_" where the number gets decremented by one on each compile and, when it is compiled with a 0 there, the sub finally gets run.

A block of the form _":first" would look for a minor block, i.e., a block that has been created by a switch directive. See next section.

One can also visually hide parts of the document, without it being hidden to the compiler, by using html comments. If the start of a line is <!--+ then it will strip that and the next occurrence of --> before doing the markdown compiling.

Directive

A directive is a command that interacts with external input/output. Just about every literate program has at least one save directive that will save some compiled block to a file.

The syntax for the save directive is

[file.ext](#name-the-heading "save: encoding | pipe commands")  

where

• file.ext is the name of the file to save to
• name-the-heading is the heading of the block whose compiled version is being saved. Spaces in the heading get converted to dashes for id linking purposes. Colons can be used to reference other scopes and/or minor blocks. In particular, #:jack will refernce the jack minor in the current heading block where the save directive is located.
• save: is there to say this is the directive to save a file
• encoding is any valid encoding of iconv-lite. This is relevant more in the command line module, but is here as the save directive is here.
• pipe commands optional commands to process the text before saving. See next section.

For other directives, what the various parts mean depends, but it is always

[some](#stuff "dir: whatever")  

where the dir should be replaced with a directive name. If dir is absent, but the colon is there, then this demarcates a minor block start.

Pipes

One can also use pipes to pipe the compiled text through a command to do something to it. For example, _"Some JS code | jshint" will take the code in block some JS code and pipe it into the jshint command which can be a thin wrapper for the jshint module and report errors to the console. That command would then return the text in an untouched fashion. We can also use pipe commands to modify the text.

Commands can be used in block substitutions, minor block directive switches, and other directives that are setup to use them such as the save and out directive:
[code.js](#some-js-code "save: | jstidy) will tidy up the code before storing it in the file code.js.

If you want your own directive to process pipes, see the save directive in lp.md. Pay particular attention to the “process” and “deal with start” minor blocks. The functionality of pipe parsing is in the doc.pipeParsing command, but there are events that need to be respected in the setup.

Commands take arguments separated by commas and commands end with pipes or the block naming quote. One can also use a named code block as an argument, using any of the quote marks (same or different as surround block name). To escape commas, quotes, pipes, underscores, spaces (spaces get trimmed from the beginning and ending of an argument), newlines, one can use a backslash, which also escapes itself. Note that the commonmark parser will escape all backslash-punctuation combinations outside of code blocks. So you may need a double backslash in directive command pipings.

You can also use \n to pu ta newline in line or \u... where the … is a unicode codepoint per JavaScript spec implemented by string.fromcodepoint.

Minor Block

Finally, you can use distinct code blocks within a full block. If you simply have multiple code blocks with none of the switching syntax below, then they will get concatenated into a single code block.

You can also switch to have what I call minor blocks within a main heading. This is mainly used for small bits that are just pushed out of the way for convenience. A full heading change is more appropriate for something that merits separate attention.

To create a minor block, one can either use a link of the form [code name]() or [code name](#whatever ":|cmd ...") Note this is a bit of a break from earlier versions in which a link on its own line would create a minor block. Now it is purely on the form and not on placement.

Example: Let’s say in heading block ### Loopy we have [outer loop]() Then it will create a code block that can be referenced by _"Loopy:outer loop".

Note: If the switch syntax is [](#... ":|...") then this just transforms whatever is point to in href using the pipe commands. That is, it is not a switch, but fills in a gap for main blocks not having pipe switch syntax. The key is the empty link text.

Templating

One use of minor blocks is as a templating mechanism.

## Top

After the first compile, the numbers will be decremented, but the blocks
will not be evaluated.

    \1_":first"

    \2_":second"
    
    \1_":final"


This is now a template. We could use it as

[jack](# "store:| compile basic ")

[happy.txt](#jack "save:| compile great")
[sad.txt](# "save:| compile basic | compile grumpy")


# Basic

[first]()
    
    Greetings and Salutations

[final]()

    Sincerely,
    Jack

# Great

[second]()

    You are great.

# Grumpy

[second]()

    You are grumpy.

# Middle

[second]()

    You are okay.

## Another

    \_":first"

    \_"$2:second"
    
    \_":final"

[middle.txt](# "save:| sub $2, middle | compile basic")

This would produce the files:

happy.txt

Greetings and Salutations

You are great.

Sincerely,
Jack

sad.txt

Greetings and Salutations

You are grumpy.

Sincerely,
Jack

middle.txt

Greetings and Salutations

You are okay.

Sincerely,
Jack

Note that you need to be careful about feeding in the escaped commands into other parsers. For example, I was using Pugs to generate HTML structure and then using this templating to inject content (using markdown). Well, Pugs escapes quotes and this was causing troubles. So I used backticks to delimit the block name instead of quotes and it worked fine. Be flexible.

31.2 h5 and h6

So this design treats h5 and h6 headings differently. They become subheadings of h1-4 headings. So for example, if we have # top and then ##### doc and ###### something then the sections would be recorded as top, top/doc, top/doc/something and we have a path syntax such as ../ which would yield top/doc if placed in top/doc/something. Ideally, this should work as you imagine. See tests/h5.md for the test examples.

32. Directives

All the directives

…

32.1 Built in directives

There are a variety of directives that come built in.

• Save [filename](#start "save:options|commands") Save the text from start into file filename. The options can be used in different ways, but in the command client it is an encoding string for saving the file; the default encoding is utf8.
• Store [name|value](#start "store:value|...") If the value is present, then it is sent through the pipes. If there is no value, then the #start location is used for the value and that gets piped. The name is used to store the value. You can also use the pipe syntax in the linkname part for the value instead. This dominates over the start or option value. A little bit easer for the reader to see in rendered form.
• Transform [des|name](#start "transform:|...) or [des|name](#start ":|..."). This takes the value that start points to and transforms it using the pipe commands. Note one can store the transformed values by placing the variable name after a pipe in the link text. The description of link text has no role. For the syntax with no transform, it can be link text that starts with a pipe or it can be completely empty. Note that if it is empty, then it does not appear and is completely obscure to the reader.
• Load [alias](url "load:options") This loads the file, found at the url (file name probably) and stores it in the alias scope as well as under the url name. We recommend using a short alias and not relying on the filename path since the alias is what will be used repeatedly to reference the blocks in the loaded file. Options are open, but for the command line client it is the encoding string with default utf8. Note there are no pipes since there is no block to act on it.
• Cd [path](#ignore "cd: load/save") This creates the ability to change directories for either loading or saving. This is relative to the default directory. [](# "cd: load") (or save) will clear the path; it is always good to do that when done. Ideally, this would be a tightly grouped of files (listing like a directory) with the initial change right before the list and the changing back after the list.
• Define [commandName](#start "define: async/sync/raw/defaults|cmd") This allows one to define commands in a lit pro document. Very handy. Order is irrelevant; anything requiring a command will wait for it to be defined. This is convenient, but also a bit more of a bother for debugging. Anyway, the start is where we find the text for the body of the command. The post colon, pre pipe area expects one of three options which is explained below in plugins.You can also pipe your command definition through pipe commands before finally installing the function as a live function. Lots of power, lots of headaches :)
  • The basic signature of a command is function (input, args, name/callback) where the input is the text being piped in, the args are the arguments array of the command, and name is the name to be emitted when done. The this is the doc.
  • sync. This should return a value which will be used as the text being passed along. You can access name if you like, but it is not useful here.
  • async. Name is a callback function that should be called when done. Standard node signature of (err, data). So put the text in the second slot and null in the first if all is well.
  • raw. Nothing is setup for you. You have to get your hands dirty with the event emitter of the doc. You’ll need some good understanding of it. See the sub command definition for inspiration.
  • defaults. The idea is that this is mostly synchronous, but has some default arguments that come from the document and hence it is async for that reason. So we need to create a tag for the emitname, document variable names for default, and the function that is to be called when all is ready. To accomodate this, instead of a function, we should have an array that gets read in: [tag, arg0, arg1, ...., fun] where tag is either a string or a function that takes in the passed in arguments and generates a string, arg0 to arg…, are the default doc variables. It is fine to have some be empty. The final entry should be a function of the same type as the sync functions (return values pass along the input).

  This defines the command only for current doc. To do it across docs in the project, define it in the lprc.js. The commandName should be one word.

• Compose [cmdname](#useless "compose: cmd1, arg1, ..| cmd2, ...") This composes commands, even those not yet defined. The arguments specified here are passed onto the commands as they are executed. There are no subcommands used in these arguments, but subcommands can be used in the arguments differently. If an argi syntax has $i then that numbered argument when the command is invoked is subbed in. If the argi has @i, then it assumed the incoming argument is an array and uses the next available array element; if the @i appears at the end of the arg list, then it unloads the rest of its elements there. This may be a little klunky and the syntax may change. We also have as special commands in compose: `` which does nothing but handles two accidental pipes in a row smoothly, ->$i which stores the incoming into the ith variable to use later as a named dollar sign variable, $i-> which sends along the ith variable to the next pipe, ->@i which pushes the value onto the ith element, assuming it is an array (it creates an array if no array is found).
• Partial [cmdname](#block "partial: oldcmdname, argplace | pipes...") This takes a command, oldcmdname, and makes a new command, cmdname, by replacing an argument slot, argplace zero-based, with whatever the block and pipes result in.
• Subcommand [subcommandname](#cmdName "subcommand:") This defines subcommandname (one word) and attaches it to be active in the cmdName. If no cmdName, then it becomes available to all commands.
• Blocks on/off [off](# "block:") Stops recording code blocks. This is good when writing a bunch of explanatory code text that you do not want compiled. You can turn it back on with the [on](# "block:") directive. Directives and headings are still actively being run and used. These can be nested. Think “block comment” sections. Good for turning off troublesome sections.
• Eval [des|name](# "eval:) Whatever block the eval finds itself, it will eval. It will eval it only up to the point where it is placed. This is an immediate action and can be quite useful for interventions. The eval will have access to the doc object which gives one access to just about everything else. This is one of those things that make running a literate progamming insecure. The return value is nonexistent and the program will not usually wait for any async actions to complete. If you put a pipe in the link name text, then the anything after the pipe will become a name that the variable ret will be stored in.
• Ignore [language](# "ignore:") This ignores the language code blocks. For example, by convention, you could use code fence blocks with language js for compiled code and ignore those with javascript. So you can have example code that will not be seen and still get your syntax highlighting and convenience. Note that this only works with code fences, obviously. As soon as this is seen, it will be used and applied there after.
• Out [outname](#start "save:|commands") Sends the text from start to the console, using outname as a label.
• New scope [scope name](# "new scope:") This creates a new scope (stuff before a double colon). You can use it to store variables in a different scope. Not terribly needed, but it was easy to expose the underlying functionality.
• Push [var name |value](#start "push: |...") This takes the stuff in start, throws it through some pipes, and then stores it as an item in an array with the array stored under var name. These are stored in the order of appearance in the document. The optional pipe syntax after var name will yield the value that starts and we ignore #start in that case. This produces an augmented array.
• h5 [varname](#heading "h5: opt | cmd1, ...") This is a directive that makes h5 headings that match heading act like the push above where it is being pushed to an array that will eventually populate varname. It takes an optional argument which could be off to stop listening for the headings (this is useful to have scoped behavior) and full which will give the event name as well as the text; the default is just the text. This produces an augmented array.
• Link Scope [alias name](# "link scope:scopename") This creates an alias for an existing scope. This can be useful if you want to use one name and toggle between them. For example, you could use the alias v for dev or deploy and then have v::title be used with just switching what v points to depending on needs. A bit of a stretch, I admit.
• Log [match string](# "log:") This is a bit digging into the system. You can monitor the events being emitted by using what you want to match for. For example, you could put in a block name (all lower cased) and monitor all events for that. This gets sent to doc.log which by default prints to console.log. If you use \: in the match string, this becomes the triple colon separator that we use for techinical reasons for block:minor name syntax. This directive’s code gives a bit of insight as to how to get more out of the system.
• If [...](... "if: flag; directive:...") If flag holds true (think build flag), then the driective is executed with the arguments as given. A couple of great uses are conditional evaling which allows for a great deal of flexibility and conditional block on/off which may be useful if there is extensive debugging commands involved.
• Flag [flag name](# "flag:") This sets the named flag to true. Note there is no way to turn a flag off easily.
• Version [name](# "version: number ; tagline") This gives the name and version of the program. Note the semicolon separator. Saves g::docname, g::docversion, g::tagline.
• npminfo [author name](github/gituser "npminfo: author email; deps: ; dev: " ) This takes in a string for some basic author information and dependencies used. To add on or modify how it handles the deps, dev, etc., modify the types object on Folder.directives.npminfo. Saves g::authorname, g::gituser, g::authoremail, g::npm dependencies, g::npm dev dependencies.

32.2 Common Client Directives

• [name](# "exec:command line command") Executes command line as a directive. Not sure on usefulness.
• [var name](url "readfile:encoding|commands") Reads a file, pipes it in, stores it in var name.
• Save. Not new, but works to actually save the file on disk.

33. Commands

Commands …

33.1 Built in commands

Note commands need to be one word and are case-sensitive. They can be symbols as long as that does not conflict with anything (avoid pipes, commas, colons, quotes).

• eval code, arg1,... The first argument is the text of the code to eval. In its scope, it will have the incoming text as the text variable and the arguments, which could be objects, will be in the args array. The code is eval’d (first argument). The code text itself is available in the code variable. The variable text is what is passed along. This should make for quick hacking on text. The doc variable is also available for inspecting all sorts of stuff, like the current state of the blocks. If you want to evaluate the incoming text and use the result as text, then the line text = eval(text) as the first argument should work.
• async (async eval) code1, code2, ... Same deal as eval, except this code expects a callback function to be called. It is in the variable callback. So you can read a file and have its callback call the callback to send the text along its merry way.
• evil While the eval commands thinks of the first argument as code acting on the incoming text, its twin evil thinks of the incoming text as the code and the arguments as just environment variables. The value returned is the variable ret which defaults to the original code.
• funify This assumes the incoming text is a function-in-waiting and it evals it to become so. This is great if you want to do a .map or if you just want to mess with stuff. .call , args.. will call the function and return that result.
• sub key1, val1, key2, val2, ... This replaces key# in the text with val#. The replacement is sorted based on the length of the key value. This is to help with SUBTITLE being replaced before TITLE, for example, while allowing one to write it in an order that makes reading make sense. A little unorthodox. We’ll see if I regret it.
• store variable name This stores the incoming text into the variable name. This is good for stashing something in mid computation. For example, ...|store temp | sub THIS, that | store awe | _"temp" will stash the incoming text into temp, then substitute out THIS for that, then store that into awe, and finally restore back to the state of temp. Be careful that the variable temp could get overwritten if there are any async operations hanging about. Best to have unique names. See push and pop commands for a better way to do this.
• log This will output a concatenated string to doc.log (default console.log) with the incoming text and the arguments. This is a good way to see what is going on in the middle of a transformation.
• raw start, end This will look for start in the raw text of the file and end in the file and return everything in between. The start and end are considered stand-alone lines.
• trim This trims the incoming text, both leading and trailing whitespace. Useful in some tests of mine.
• join This will concatenate the incoming text and the arguments together using the first argument as the separator. Note one can use \n as arg1 and it should give you a newline (use \\n if in a directive due to parser escaping backslashes!). No separator can be as easy as |join ,1,2,....
• cat The arguments are concatenated with the incoming text as is. Useful for single arguments, often with no incoming text.
• echo echo This is output This terminates the input sequence and creates a new one with the first argument as the outgoing.
• get get blockname This is just like using _"blockname" but that fails to work in compositions. So get is its replacement. This ignores the input and starts its own chain of inputs.
• array array a1, a2, ... This creates an array out of the input and arguments. This is an augmented array.
• . . propname, arg1, arg2,... This is the dot command and it accesses property name which is the first argument; the object is the input (typically a string, but can be anything). If the property is a method, then the other arguments are passed in as arguments into the method. For the inspirational example, the push directive creates an array and to join them into text one could do | . join, \,. There is also an alias so that any .propname as a command works. For example, we could do | .join \, above. This avoids forgetting the comma after join in the prior example.
• push Simply pushes the current state of the incoming text on the stack for this pipe process.
• pop Replaces the incoming text with popping out the last unpopped pushed on text.
• if flag, cmd, arg1, arg2, .... If the flag is present (think build flag), then the command will execute with the given input text and arguments. Otherwise, the input text is passed on.
• when name1, name2, ... This takes in the event names and waits for them to be emitted by done or manually with a doc.parent.done.gcd.once(name, "done"). That would probably be used in directives. The idea of this setup is to wait to execute a cli command for when everything is setup. It passes through the incoming text.
• done name This is a command to emit the done event for name. It just passes through the incoming text. The idea is that it would be, say, a filename of something that got saved.
• arrayify This takes the incoming text and creates an array out of it. The first argument is an object with keys sep to know what to split on, esc to escape the separator and itself, trim a boolean that will trim the text for each entry. The defaults are newline, backslash, and true, respectively. You can also pass them as the first, second, and third argument, respectively. Note that this assumes that both sep and esc are single characters. You can have the usual block substitutions, of course, but it might be safer to escape the block and run it through compile, e.g., ` | arrayify | .mapc compile. This also allows nesting of objects. To get a string representation of the array, call | .toString`.
• objectify This takes the incoming text and creates an object out of it. The first argument is an object with keys key to know what to split on for the key, val to split on for the end of the value, esc to escape the separator and itself, trim a boolean that will trim the value for each entry; keys are automatically trimmed. The defaults are colon, newline, backslash, and true, respectively. Note that this assumes that all the characters are single characters. You can have the usual block substitutions, of course, but it might be safer to escape the block and run it through compile, e.g., ` | objectify | .mapc compile. This also allows nesting of objects. Call |.toString()` to get a string.
• ife This takes a snippet of code and creates an immediate function execution string for embedding in code. the arguments become the variable names in both the function call and the function definition. If an equals is present, then the right-hand side is in the function call and will not be hidden from access in the ife.
• caps This is a command that tries to match caps and replace them. The idea comes from wanting to write M W>900px and get @media (min-width:900px). This does that. By passing in a JSON object of possible matches as argument or setting the caps local object to an object of such matches, you can change what it matches. But it only will match a single character (though unicode is fine if you can input that).
• assert This asserts the equality of the input and first argument and if it fails, it reports both texts in a log with the second argument as a message. something | assert _"else", darn that else. This is a way to check that certain things are happening as they should.
• wrap This wraps the incoming text in the first and second argument: some text | wrap <, >" will result in <some text>`.
• js-string This breaks the incoming text of many lines into quoted lines with appropriate plus signs added.
• html-wrap This takes the incoming text and wraps it in a tag element, using the first argument as the element and the rest of the arguments as attributes. An equals sign creates an attribute with value, no equals implies a class. An attribute value will get wrapped in quotes. text-> | html-wrap p data, pretty, data-var=right will lead to <p class="data pretty" data-var="right">text</p>
• html-table This requires an array of arrays; augmented matrix is good. The first argument should either be an array of headers or nothing. It uses the same argument convention of html-wrap for the rest of the arguments, being attributes on the html table element. We could allow individual attributes and stuff on rows and columns, but that seems best left to css and js kind of stuff. Still thinking on if we could allow individual rows or entries to report something, but that seems complicated.
• html-escape This escapes <>& in html. It is mainly intended for needed uses, say in math writing. Very simple minded. One can modify the characters escaped by adding to Folder.plugins.html_escape. This is actually similar to caps and snippets.
• html-unescape The reverse of html-escape, depending on what the symbols are in plugins.html_unescape.
• snippets (alias s ). This is a function for things that are easily named, but long to write, such as a cdn download script tag for a common js library, say jquery. s jquery could then do that. Currently, there are no default snippets. To load them, the best bet is in the lprc.js file and store the object as Folder.plugins.snipets = obj or, if you are feeling generous, one could do Folder.merge(Folder.plugins.snippets, obj);. This is really a stand-alone command; incoming text is ignored.

  In writing a snippet, it can be a function which will take in the arguments. Alternatively, you can sprinkle ARG#||...| in your code for the Argument with numner # and the pipes give an optional default; if none, then ARG# is eliminated. So ARG0||1.9.0| yields a default of 1.9.0. Pipes cannot be in the default

  Be careful that the first argument is the snippet name.

• minidoc minidoc :title, :body This takes an array and converts into an object where they key value is either the args as keys and the values the relevant input items or the item in the array is a two-element array whose first is the key and second is the value. The named keys in the arguments skip over the two-element arrays. minidocs are augmented with some methods. See the augment section.
• augment augment type This augments the object with the methods contained in the type augment object. See the augment section.
• matrixify This takes in some text and splits into a two dimensional array using the passed in separators. The first separator divides the columns, the second divides the rows. The result is an array each of whose entries are the rows. There is also an escape character. The defaults are commas, newlines, and backslashes, respectively. The escpae character escapes the separators and itself, nothing else. There is also a boolean for whether to trim entries; that is true by default. Pass in f() in the fourth argument if not desired. All the characters should be just that, of length 1.

  This returns an augmented object, a matrix that has the properties: * transpose This returns a new matrix with flipped rows and columns. * trim This trims the entries in the matrix, returning the original. * num This converts every entry into a number, when possible. * clone This creates a copy. * traverse This runs through the matrix, applying a function to each entry, the arguments being element, inner index, outer index, the row object, the matrix.

Augment

We have . methods that we can invoke and the augment command adds in properties based on objects stored in doc.plugins.augment. Any key in that object is a valid type for the command. We currently have two pre-defined augment types: minidoc and arr. The ones with just a . only make sense as commands.

minidoc

• .store arg1 will take the object and store all of its properties with prefix arg1 if supplied. If the key has a colon in it, it will be escaped so that {":title" : "cool"} | .store pre can be accessed by cool:title.
• .clear arg1 Removes the variables stored in the scope (undoes store). Mostly to be used in the .compile command for tidying up. Best not to rely on the cleanup happening at any particular time so don’t use unless your sure.
• .mapc cmd, arg1, arg2, ... Applies the cmd and args to each of the values in the object, replacing the values with the new ones.
• .apply key, cmd, arg1, arg2, .. Applies the cmd and args with input being obj[key] value. This overwrites the obj[key] value with the new value.
• .clone Makes a new object with same properties and methods. This is a shallow clone. You can use this with push and pop to modify the object and then go back to the original state.
• .compile blockname This uses the blockname as a reference to a template whose sections will be filled with the corresponding keys of the minidoc. The assumption is that the keys stat with colons. Any that don’t will probably not match.
• .set key, val Sets the key to the value
• .get key Gets the value of that key
• .keys will give an array of the non-augmented keys. It takes one optional argument; a true boolean will cause it to be sorted with the default sort; a function will be presumed to be comparison function and it will be sorted according to that. Otherwise, the keys are returned as is.
• .toString will print a representation of the original object. By default the separators are colon and newlines, but the first two arguments can change that. It is also possible to pass in a function that acts on each key and value in third and fourth slots to wrap them.
• .strip This strips the object of its augmented properties.
• .forIn A foreach, map, or reduce rolled into one acting on the non-augment properties. It takes three arguments. The first is mandatory and is the function to be called on each pair. The second is an initial value or a container object. The third is a sort order, as in keys. The signature of the function is key, property value, intial value/last returned value, self. If the third stuff is undefined, then self becomes third. The final returned value of the function is what is returned, but if that is undefined, then the object itself is returned.

arr

These methods return new, augmented arrays.

• .trim Trims every entry in the array if it has that property. Undefined elements become empty strings. Other stuff becomes strings as well, trimmed of course.
• .splitsep sep This splits each entry into an array using the separator. The default separator is \n---\n.
• .mapc cmd, arg1, arg2, ... Maps each element through the commands as input with the given arguments being used.
• .pluck i This assumes the array is an array of arrays and takes the ith element of each array, returning a new array.
• .put i, full This takes an incoming array and places each of the elements in the ith position of the corresponding element in full. Reverse of pluck, in some ways.
• .get i Gets ith value. Negatives count down from last position, i.e., get -1 retrieves the last element of the array.
• .set i, val Sets ith value to val.

33.2 Common Client Commands

• exec exec cmd1, cmd2, ... This executes the commands on the commandline. The standard input is the incoming input and the standard output is what is passed along.
• execfresh execfresh Same as exec but no caching
• readfile readfile name Reads in file with filename. Starts at source directory. This terminates old input and replaces with file contents.
• readdir readdir name Generates a list of files in named directory. This generates an augmented array.
• savefile savefile name, encoding Saves the input into the named file using the encoding if specified.
• z z msg will return the value in msg from the command line flag -z msg:value. If the value contains multiple colons, it is interpreted as an array and an array will be returned.

33.3 Full Client Commands

• jshint This takes the input and runs it through JSHint. The command is of the form js stuff | jshint options, globals, shortname, print clean.
  • The options is an object that corresponds to the options that JShint accepts; you can use a subcommand to create the options object if you like. Default is unused:true, else is their defaults.
  • Globals is an array of global names; if they can be written over, pass in name:true instead of name.
  • Shortname is the shortname to present in understanding what is being jshinted. Otherwise, it does its best to give you a cryptic but informative name.
  • If the fourth argument is a boolean, t() or f() will do it, then that toggles whether to print the message that it all went smoothly or not, respectively. The default is to not print it.
  • You can override the defaults repeatedly by modifying the Folder.plugins.jshint object with the names: options, globals, and clean.
• md This takes the input as markdown and puts out html. The first argument is an optional string naming the renderer to use. The other arguments should be booleans, namely, f(), if one does not want preprocessing/post to occur. The default preprocessors, in order, are literate programming subs and math subs rendering to katex.

  To create a renderer, you can use Folder.plugins.md.req as the markdoan object and then render it per the instructions (an options object req(options).use(...). This is all best done in the lprc.js file. Store the rendered under the preferred name in plugins.md.

  See the logs test directory and its lprc.js.

• cheerio This gives access to the cheerio module, a lightweight node version of jQuery-esque without the overhead of jsdom. It can’t do everything, but it does most things: cheeriojs. To use, the incoming text is the html doc to modify, the first argument is the selector, the second the method, and then the arguments to the method, e.g., somehtml | cheerio h2.title, .text, Hello there!
• ch-replace This is a convenience method for cheerio. This will use the first argument as a selector and the second argument as a html replacement.
• postcss This takes incoming text and runs it through postcss. To do something useful, you need to have the arguments be the commands to use. At the moment, the only one shipping with this is autoprefixer, but others are likely to be added (minimizers and fixers, in particular). You can add them yourself by, in lprcs.js, saying (installing cssnano as example) Folder.plugins.postcss[cssnano] = require('cssnano'); and ensuring that the cssnano module is installed in npm.
• tidy This uses js-beautify The first argument is the type: js, css, or html. The second argument are options that get merged with the defaults. The js has a default of indent_size of 4 and jslint_happy true. An unrecognized first argument (or none) will default to js.
• minify The first argument says the type of minifier: js, css, and html. js is the default if the first argument is not realized. The second argument is an object of options that get passed in. This uses uglify-js, clean-css, and html-minifier, respectively. For css, the second argument can be a boolean indicating whether to pass on the results object (if true, t() ) or just the css output text (default).

34. Subcommands

Existing in a weird argument world…

34.1 Built-in Subcommands

With command arguments, one can run commands on arguments to get them in some appropriate form or use, including passing in objects or arrays. You can use them as cmd a, subcmd(arg1, arg2, arg3) would have subcmd acting on the args and the result of that would be the argument place The a would be passed into cmd as the first argument, but anything might get passed into cmd by subcmd’s return value. It could also store an object into a state for configuration.

There are several built-in subcommands. Note that these are case insensitive.

• e or echo This expects a quote-delimited string to be passed in and will strip the quotes. This is useful as the appearance of a quote will mask all other mechanics. So e("a, b and _this") will produce a literal argument of a, b, and _this. Multiple arguments will be stripped and passed on as multipel arguments.
• j or join The first entry is the joiner separator and it joins the rest of the arguments. For arrays, they are flattened with the separator as well (just one level – then it gets messy and wrong, probably).
• a or arr or array This creates an array of the arguments.
• arguments or args Inverse of array. This expects an array and each element becomes a separate argument that the command will see. E.g., cmd arguments(arr(3, 4)) is equivalent to cmd 3, 4. This is useful for constructing the args elsewhere. In particular, args(obj(_"returns json of an array")) will result in the array from the subsitution becoming the arguments to pass in.
• o or obj or object This presumes that a JSON stringed object is ready to be made into an object.
• merge Merge arrays or objects, depending on what is there.
• kv or key-value This produces an object based on the assumption that a key, value pairing are the arguments. The key should be text. Multiple pairs welcome.
• act This allows one to do obj, method, args to apply a method to an object with the slot 2 and above being arguments. For example, one could do act( arr(3, 4, 5), slice, 2, 3) to slice the array to [5].
• prop or property. This will take the arguments as a property chain to extract the value being pointed to.
• json This will convert an object to JSON representation.
• set The presumption is that an object is passed in whose key:values should be added to the command state. gSet does this in a way that other commands in the pipe chain can see it. set(kv(name, val, ...)) would probably be the typical way.
• get This retrieves the value for the given key argument. gGet does the same for the pipe chain. Multiple keys can be given and each associated value will be returned as distinct arguments.
• n or # or number This converts the argument(s) to numbers, using js Number function. n(1, 2, 3) will create three arguments of integers. To get an array, use arr(n(1, 2, 3)
• eval will evaluate the argument and use the magic ret variable as the value to return. This can also see doc (and doc.cmdName) and args has the arguments post code. Recommend using backticks for quoting the eval; it will check for that automatically (just backticks, can do echo for the others if needed).
• log This logs the argument and passes them along as arguments.
• t or true. This returns the true value.
• f or false. This returns the false value.
• null. This returns the null value.
• doc. This returns the doc variable. This could be useful in connection to the property method and the log subcommand.
• skip. This returns no arguments.

To build one’s own command, you can attach a function whose arguments will be the arguments passed in. The this is the doc object. The current name (say for scope storing) is in doc.cmdName. This will point to within a whole pipe chunk. Pop off the last part (delimited by triple colon) to get to the whole command scope. The return value will be used as in an argument into the command or another subcommand. If it is an array and the flag args is set to true, then each entry in the array will be expanded into a set of arguments. So instead of 1 argument, several could be returned. If nothing is returned, then no arguments are passed on and it is as if it wasn’t there.

35. Command Line

All you need to know about LitPro’s command line.

Save, load, out defined already, but here linked up to do something.

35.1 Flags

The various flags are

• -b, –build The build directory. Defaults to build. Will create it if it does not exist. Specifying . will use the current directory.
• –checksum This gives an alternate name for the file that lists the hash for the generate files. If the compiled text matches, then it is not written. Default is .checksum stored in the build directory.
• -d, –diff This computes the difference between each files from their existing versions. There is no saving of files.
• -e, –encoding Specify the default encoding. It defaults to utf8, but any encoding supported by node works. To have more encodings, use the plugin litpro-iconv-lite To override the command lined behavior per loaded file from a document, one can put the encoding between the colon and pipe in the directive title. This applies to both reading and writing.
• –file A specified file to process. It is possible to have multiple files, each proceeded by an option. Also any unclaimed arguments will be assumed to be a file that gets added to the list.
• -f, –flag This passes in flags that can be used for conditional branching within the literate programming. For example, one could have a production flag that minimizes the code before saving.
• -i, –in This takes in standard input as another litpro doc to read from.
• -l, –lprc This specifies the lprc.js file to use. None need not be provided. The lprc file should export a function that takes in as arguments the Folder constructor and an args object (what is processed from the command line). This allows for quite a bit of sculpting. See more in lprc.
• -o, –out This directs all saved files to standard out; no saving of compiled texts will happen. Other saving of files could happen; this just prevents those files being saved by the save directive from being saved.
• -s, –src The source directory to look for files from load directives. The files specified on the command line are used as is while those loaded from those files are prefixed. Shell tab completion is a reason for this difference.
• -z, –other This is a place that takes in an array of options for plugins. Since plugins are loaded after initial parsing, this allows one to sneak in options. The format is key:value. So -z cache:cool would set the value cache to cool.
• –scopes This shows at the end of the run all the variables and values that the document thinks is there. Might be useful for debugging purposes.

36. Plugins

These are some of the plugins available for LitPro.

37. lprc.js

A full specification of how to use this file. It includes a variety of information about the structure of the folder and doc objects.

37.1 Plugins

This is a big topic which I will only touch on here. You can define commands in the text of a literate program, and we will discuss this a bit here, but mostly, both commands and directives get defined in module plugins or the lprc.js file if need be.

Defining Commands

The define directive allows one to create commands within a document. This is a good place to start getting used to how things work.

A command has the function signature function (input, args, name)-> void where the input is the incoming text (we are piping along when evaluating commands), args are the arguments that are comma separated after the command name, and the name is the name of the event that needs to be emitted with the outgoing text. The function context is the doc example.

A minimal example is

function ( input, args, name) {
    this.gcd.emit(name, input);
}

We simply emit the name with the incoming text as data. We usually use doc for the this variable. This is the raw option in the define directive.

The default is sync and is very easy.

function ( input, args, name) {
    return input;
}

That is, we just return the text we want to return. In general, the name is not needed though it may provide context clues.

The third option is an async command. For those familiar with node conventions, this is easy and natural.

function (input, args, callback, name) {
    callback(null, input);
}

The callback takes in an error as first argument and, if no error, the text to output. One should be able to use this as a function callback to pass into other callback async setups in node.

So that’s the flow. Obviously, you are free to do what you like with the text inside. You can access the document as this and from there get to the event emitter gcd and the parent, folder, leading to other docs. The scopes are available as well. Synchronous is the easiest, but asynchronous control flow is just as good and is needed for reading files, network requests, external process executions, etc.

Plugin convention.

I recommend the following npm module conventions for plugins for literate-programming.

1. litpro-… is the name. So all plugins would be namespaced to litpro. Clear, but short.
2. Set module.exports = function(Folder, other) The first argument is the Folder object which construts folders which constructs documents. By accessing Folder, one can add a lot of functionality. This access is granted in the command line client before any folder is created.

   The other argument depends on context, but for the command line client it is the parsed in arguments object. It can be useful for a number of purposes, but one should limit its use as it narrows the context of the use.

3. Define commands and, less, directives. Commands are for transforming text, directives are for doing document flow maipulations. Other hacks on Folder should be even less rare than adding directives.
4. Commands and directives are globally defined.
5. Folder.commands[cmd name] = function (input, args, name)... is how to add a command function. You can use Folder.sync(cmdname, cmdfun) and Folder.async to install sync and async functions directly in the same fashion as used by the define directive.
6. Folder.directives[directive name] = function (args) is how to install a directive. There are no helper functions for directives. These are more for controlling the flow of the compiling in the large. The arg keys are read off from [link](href "directive:input"). Also provided is the current block name which is given by the key cur.
7. If you want to do stuff after folder and its event emitter, gcd, is created, then you can modify Folder.postInit to be a function that does whatever you want on a folder instance. Think of it as a secondary constructor function.
8. The Folder has a plugins object where one can stash whatever under the plugin’s name. This is largely for options and alternatives. The folder and doc object have prototyped objects on this as well which allows one to choose the scope of applicability of objects. But beware that subobjects are not prototyped (unless setup in that way; you may want to implement that by Object.creating what is there, if anything). Think of it as deciding where options should live when creating them.

Structure of Doc and Folder

To really hack the doc compiling, one should inspect the structure of Folder, folder, and doc. The Folder is a constructor and it has a variety of properties on it that are global to all folders. But it also has several prototype properties that get inherited by the folder instances. Some of those get inherited by the docs as well. For each folder, there is also a gcd object which is the event emitter, which comes from the, ahem, magnificient event-when library (I wrote it with this use in mind). In many ways, hacking on gcd will manipulate the flow of the compiling.

I wrote the folder instance to maintain flexibility, but typically (so far at least), one folder instance per run is typical. Still, there might be a use for it in say have a development and production compile separate but running simultaneously?

Folder

These are the properties of Folder that may be of interest.

• commands. This is an object that is prototyped onto the instance of a folder. Changing this adds commands to all created folder instances.
• directives. This holds the directives. Otherwise same as commands.
• reporter. This holds the functions that report out problems. See reporters below. This is not prototyped and is shared across instances.
• postInit. This does modification of the instance. Default is a noop.
• sync, async. These install sync and async commands, respectively.
• defSubCommand. Installs a subcommand.
• plugins. This is a space to stash stuff for plugins. Use the plugin sans litpr as the key. Then put there whatever is of use. The idea is if you require something like jshint and then want default options, you can put that there. Then in a lprc file, someone can override those options it will be applied across the project.

folder

Each instance of folder comes with its own instances of:

• docs. Holds all the documents.
• scopes. Holds all the scopes which are the stuff before the double colon. It includes the blocks from the compiled docs but also any created scopes.
• reports. This holds all the reports of stuff waiting. As stuff stops waiting, the reports go away. Ideally, this should be empty when all is done.
• stack. This is for the push and pop of text piping.
• gcd. This is the event-emitter shared between docs, but not folders. Default actions are added during the instantiation, largely related to the parsing which sets up later. If you want to log what goes on, you may want to look at the event-when docs (makeLog is a good place to start).
• flags. This holds what flags are present.

and shares via the prototype

• parse. This parses the text of docs using commonmark spec
• newdoc. This creates a new document. Kind of a constructor, but simply called as a function. it calls the constructor Doc.
• colon. We replace colons with a unicode triple colon for emitting purposes of block names (event-when uses colon as separators too). This contains the escape (does replacement), restore (undoes it), and v which is the unicode tripe colon. If the v is replaced entirely, everything should hopefully work just fine with a new separator.
• createScope. Creating a scope.
• join. What is used to concatenate code blocks under same block heading. Default is “\n”
• log. What to do with logging. Defaults to console.log.
• indicator. An internal use to allow escaping of whitespace in command arguments that would otherwisebe trimmed.
• wrapSync, wrapAsync. These wrap functions up for command sync, async, but do not install them. Not sure why not install them.
• subnameTransform. A function that deals with shorthand minor substitutions that avoid using the main block heading. This can be overwritten if you want some custom behavior.
• reportwaits. This is a function that produces the reports of what is still waiting. Very useful for debugging. This returns an array.
• simpleReport. This reports on the substitutions that did not get resolved. This returns an array. It also includes any commands that were called but not defined. Subcommands throw errors when not defined, but since commands can be defined later, they will not. Hence this mechanism.
• Doc. This is the constructor for documents.

and uses Object.create to kind of share

• commands
• directives
• plugins

and direct copying from

• reporters

doc

Each file leads to a doc which is stored in the folder. Each doc has a variety of stuff going on.

Unique to each instance

• file. The actual path to the file. It is treated as unique and there is a scope dedicated to it. Don’t mess with it. It is also how docs are keyed in the folder.docs object.
• text. The actual text of the file.
• blockOff. This tracks whether to take in code blocks as usual. See blocks directive. If 0, code blocks are queued up. If greater than 1, code blocks are ignored.
• levels. This tracks the level of the heading that is currently being used. See h5/h6 description
• blocks. Each heading gets its own key in the blocks and the raw code blocks are put here.
• heading, curname. These are part of the block parsing. curname is the full name while heading excludes minor block names.
• vars. This is where the variables live. As each code block is compiled, its result gets stored here. But one can also add any bit of var name and text to this.
• parent. This is the folder that contains this doc.

Inherited from folder

• commands, modifications affect all
• directives, modifications affect all
• scopes, modifications affect all
• gcd, modifications affect all. Be careful to scope added events to files, etc.
• plugins, modifications affect all
• colon, Object.created
• join, overwriting will only affect doc
• log, overwriting will only affect doc
• subnameTransform, overwriting will only affect doc
• indicator, overwriting will only affect doc
• wrapSync, wrapAsync, overwriting will only affect doc
• augment, this augments the object with the type.
• cmdworker, this will call the command. needed as with the dot command, it can get tricky. Used in .apply, .mapc, compose.
• compose, this creates a function from composing multiple commands

Prototyped on Doc. Almost all are internal and are of little to no interest.

• pipeParsing. This parses the pipes. This may be useful if you want to do something like in the save or define directives. Check them out in the source if you want to see how to use it.
• blockCompiling. This is what the compile command taps into. See how it is done there.
• getscope. Looks up a scope and does appropriate async waiting for an existing scope if need be.
• retrieve. retrieves variable.
• createLinkedScope. Creates a link to a scope and notifies all.
• indent. This is the default indenting function for subbing in multi-line blocks. The default idea is to indent up to the indent of the line that contains the block sub; further existing indentation in sublines is respected on top of that.
• getIndent. Figuring out the indent
• substituteParsing
• regexs. Some regular expressions that are used in the parsing of the code blocks.
• backslash. The backslash function applied to command arguments.
• whitespaceEscape. Handling whitespace escaping in conjunction with backslash. Putting the whitespace back.
• store. stores a variable.

38. Compile Events

This details ways to get information out about what is going on in the compiling of a litpro document: events, reports, evaling, ..

Reporting

A key feature of any programming environment is debugging. It is my hope that this version has some better debugging information. The key to this is the reporting function of what is waiting around.

The way it works is that when an event of the form waiting for:type:... is emitted with data [evt, reportname, ...] then reporters gets a key of the event string wthout the waiting for:, and when the evt is emitted, it is removed.

If it is still waiting around when all is done, then it gets reported. The reportname is used to look up which reporter is used. Then that reporter takes in the remaining arguments and produces a string that will be part of the final report that gets printed out.

Some of the waiting is not done by the emitting, but rather by presence in .when and .onces.

V The Literate Programs

Here we give descriptions and links to all the various pieces of the literate program family. They are all written in a literate fashion and the source code is freely available at their repositories, listed below.

All of this relies on node.js.

To understand these tools, the most interesting repositories are those for the underlying library, literate-programming-lib, and the event library, event-when.

litpro

This is our main tool. It is a command-line tool. It can be installed globally or locally into a repository. Use npm install litpro -g to install it globally.

• github is where the code lives
• npm is the package.

literate-programming

This is the fat command-line client. It can do everything that litpro can do, but it also comes bundled with pug, markdown-it, postcss, tidy, minifiers. This is a “batteries loaded” tool for web development.

• github is where the code lives
• npm is the package.

This can also be installed globally with npm install literate-programming -g. The command is then literate-programming.

litpro-jshint

If you all you need is jshint, you might want to install the module litpro-jshint instead of the full one.

• github is where the code lives
• npm is the package.

literate-programming-cli

This is an intermediate project between the library and litpro. This is where most of the command line client code is written.

• github is where the code lives
• npm is the package.

literate-programming-cli-test

This is our test framework for the command line client. Basically, we have a directory with the files to process, a canonical directory showing what we expect to be in the build directory.

• github is where the code lives
• npm is the package.

literate-progamming-lib

This is the place to read the literate program construction in detail. Start with project.md and migrate to the other files in the structure. Parsing and stitching are the two main algorithmic pages.

• github is where the code lives
• npm is the package.

event-when

This is an event library that allows one to wait for multiple events with ease. We use it, for example, to assemble a variety of substitutions into a single code block. The return of that code block must wait for when the events of the other blocks being subbed in have fired.

• github is where the code lives
• npm is the package.