Quizzes and Exercises

The final two block elements that Markua provides are quizzes and exercises. These two block elements are very special, however, in that they enable a single Markua document to construct everything from traditional textbooks and paper-based quizzes to entire online courses (or MOOCs). So, they’re discussed in their own chapter.

Quizzes and exercises are essentially the same. The only difference is that quizzes are intended to be marked, and exercises are not. Because of their similarities, they are discussed here together.

Quizzes or exercises in a textbook consist of two things:

Questions, typically in the chapter itself.
Answers, typically at the back of the book.

The questions in the chapter essentially are placed there like any other block element, such as an aside or blurb. The answers are positioned at the back of the book, along with other elements like the index and appendices.

There is only one syntax to create a quiz or exercise. For a quiz, it’s by wrapping the quiz in {quiz} … {/quiz}; for an exercise, it’s by wrapping the exercise in {exercise} … {/exercise}.

Here is a brief example of a quiz:

{quiz, id: quiz1}
? How many letters are in the word Markua?

a) 5
B) 6
c) 7

? How many unique letters are in the word Markua?

! 5
{/quiz}

This quiz contains two questions: a multiple-choice question where the correct answer is b, and a fill-in-the-blank question where the correct answer is 5. Quizzes and exercises have the same question types, discussed later.

With a quiz, the id attribute is required. This is so the identity of a quiz can be preserved across generations of a course.

Here is the same example, but as an exercise:

{exercise, id: exercise1}
? How many letters are in the word Markua?

a) 5
B) 6
c) 7

? How many unique letters are in the word Markua?

! 5
{/exercise}

Just like with quiz, with an exercise the id attribute is required. This is so the identity of an exercise can be preserved across generations of a course.

Quiz and Exercise Headings and Other Content

A quiz or exercise can contain any Markua content, not just questions and answers. This is true regardless of whether the quiz or exercise is in a MOOC, an ebook or on paper. Note that video and audio resources don’t work so well on paper, however.

If a quiz or exercise starts with any type of heading immediately after the {quiz} or {exercise} line, this heading’s content should be considered the name of the quiz or exercise. This can be used in a list of quizzes or exercises produced by Leanpub. Typically the heading will be a chapter heading (# ), but section headings (## ) and lower headings also are supported. (The reason for this is that quizzes are sometimes top-level things, and other times are nested inside chapters, sections or sub-sections. Some course authors would correctly feel that the quiz should have the appropriate level of heading given their position in the document.

Example:

{quiz, id: quiz2}
# Markua Quiz

Watch this [video](https://www.youtube.com/watch?time_continue=1&v=VOCYL-FNbr0) of Peter explaining Markua.

? What year was that video from?

What year? Really? Did it really take that long? What was going on???

a) 2012
b) 2013
C) 2014
d) 2015

{words: 500}
? Why do you think the first version of the Markua spec took so long?

Look at the Leanpub website and read the [pricing essay](https://leanpub.com/pricing).

! Answers could include "bootstrapped startup", the spec evolving, removing HTML mapping, etc.

That's it for this quiz, and this MOOC!

**Thanks for taking my course!**
{/quiz}

An Empty Quiz or Exercise is Not an Error

A quiz or exercise which contains no questions is not an error. Instead, a if Leanpub encounters a quiz or exercise with no questions it must filter the quiz or exercise from the output, optionally providing a warning to the author.

This lets authors create placeholders for quizzes or exercises in their courses before the quizzes or exercises are ready, which is potentially very useful in an in-progress course.

A Malformed Quiz or Exercise is an Error

If Leanpub encounters a malformed quiz or exercise it must treat this as an error and not generate the output from the Markua document. Quizzes and exercises are not something that should ever be produced in a broken state.

However, it is also an error to parse quiz syntax outside a quiz or exercise block. Leanpub must not parse lines starting with ? or ! as representing questions or answers unless those are contained in a quiz or exercise block.

Supported Attributes on Quizzes and/or Exercises

attempts: The number of allowed attempts on a quiz. The default is defined by the value of default-quiz-attempts on the containing course, or 1 if this is not present. A value of 0 means the quiz cannot be taken (yet). A value of -1 means the quiz has an unlimited number of attempts. Since an exercise does not count toward the mark on a course, an exercise always has an unlimited number of attempts.
case-sensitive: true or false. The default is true. This sets the default behaviour of fill in the blank questions. If true, the fill in the blank question answers are case-sensitive. If false, they are not.
id: All Markua elements support an id attribute. The reason the id attribute is explicitly listed here is to emphasize that Leanpub may require an id attribute on a quiz or exercise. For example, Leanpub requires the id attribute on all quizzes, in order to determine the identity of quizzes when a course is being published in-progress. (As a student, you’d be pretty unhappy if you had to re-take an unchanged quiz simply because a professor published a new course version.)
points: If present, this is the total number of points the quiz or exercise is worth. (This really only matters for quizzes, but is supported for exercises as well, in case Leanpub wishes to display the points on exercises to make them feel more real.) If points is not present, the worth of the quiz is determined by summing the points of the questions. (Questions are worth 1 point each if they have no points attribute.) If the quiz has a points attribute and its questions also have points attributes, the worth of each question in a larger course context is determined as follows: its points are the numerator, and the total points in the quiz or exercise is the denominator.
random-choice-order: true or false. The default is false. This sets the default behaviour of multiple choice questions. If true, the choices in the multiple choice question are randomly arranged; if false, they are presented in the order written.
random-question-order: true or false. The default is false. This sets the default behaviour of the quiz or exercise. If true, the questions are randomly arranged; if false, they are presented in the order written.
start-at: The start-at is the number of the first question. The default is 1. Any integer is permitted. Subsequent questions will have a number which is 1 higher than the previous question.
version: The version of the quiz. This does not replace the function of the id; it’s more for use in analytics by the instructor. The default is 1.

As discussed above, there is no title or caption attribute for a quiz–you can just add a heading inside the quiz or exercise itself, using the normal Markua formatting for a chapter heading.

Question Types: Multiple Choice, Fill In The Blank

There are two types of questions supported by Leanpub:

Multiple Choice
Fill In The Blank

These types are not specified by a {type} attribute. Instead, they are inferred from properties of the answers or from other attributes of the question.

Multiple Choice Questions

A multiple choice question has 2 or more answer choices, and 1 correct answer choice.

The correct answer choice is in capital letters before the parentheses; incorrect answer choices have lowercase letters before the parentheses.

Example:

? How many letters are in the word Markua?

a) 5
B) 6
c) 7

Obviously, when generating the question in the actual quiz or exercise, Leanpub must make all answer choices have the same type of letter. This is usually a lowercase letter, although either all lowercase or all uppercase letters would be fine.

Unless a choose-answers attribute is used, the multiple choice answers all must start from a or A, and must use a right-parenthesis after the a or A. Any line starting with a) ar A) in a quiz is considered a set of multiple choice quiz answers, not a numbered list using a) or A) as a delimiter. If you want to put a numbered list in a quiz body, use periods for the delimiter.

A multiple choice question may also have a dynamic number of answer choices, including for the correct answer. This done with the special choose-answers attribute, shown and explained below.

{choose-answers: 4}
? How many grams are in a pound?

C) 454
C) 453
m) 451
m) 1000
o) 100
o) 150
o) 200
o) 250
o) 300
o) 500

The choose-answers attribute specifies how many answer choices should be shown. This includes exactly one of the correct answers (indicated with C), all of the mandatory incorrect answers (indicated with m) and as many of the optional incorrect answers (indicated with o) as are needed for the question to have the total number of answers as indicated by the choose-answers attribute.

So, in the above example, either 453 or 454 will be shown, along with the mandatory incorrect answer choices 451 (a literary joke) and 1000 (a kilogram, not a pound) and one of the optional incorrect answers (100, 150, 200, 250, 300 or 500).

When a choose-answers attribute is used, the question will always have random-choice-order.

The following are errors in a question where a choose-answers attribute is used:

0 correct (C) answers
not enough mandatory (m) incorrect or optional (o) incorrect answers for the question to have the choose-answers number of answers
if choose-answers is n, a number of mandatory (m) incorrect answers >= n (since there needs to be one correct answer shown)
if choose-answers is n and the number of mandatory (m) answers is n - 1, then any optional (o) incorrect answers existing
answers starting with something other than C, m or o

Supported Attributes on Multiple Choice Questions

choose-answers: This is described above. If choose-answers is used, random-choice-order is forced to true.
points: The number of points the question is worth. This number can be 0 or higher. The default is 1.
random-choice-order: true or false. The default is false, unless choose-answers is used. This sets the behaviour of the specific multiple choice question. If true, the choices in the multiple choice question are randomly arranged; if false, they are presented in the order written. If this attribute is omitted, its value is determined by the random-choice-order attribute on the quiz itself, which defaults to false if absent.

Fill In The Blank Questions

A fill in the blank question consists of a question and a set of answers. Each answer is specified by !, an optional points value, a space, and then a semicolon-separated list of the acceptable values of that answer. Each answer value can be a text string (quoted or not) or a regular expression (regex). If a points value is not specified for an answer, the answer is worth full points.

Support for regular expression answer values is optional. However, Leanpub which supports regular expression marking must interpret any answer which starts with a forward slash (/) and ends with a forward slash followed by some word characters (e.g. i) as being a regular expression. Note that Leanpub uses Ruby regular expressions.

Finally, note that you can separate regular expressions with semicolons, just like any other answer value. There’s no reason not to support this, and it may lead to simpler regular expressions. However, if you’re good at regular expressions, you can also combine them into one regular expression, of course.

Note that since a semicolon is used to separate answer values, to provide an actual semicolon as part of an answer value you must either put the answer value in quotes, use a backslash-escape \; or make the semicolon part of a regular expression.

Examples:

? How many unique letters are in the word Markua?

! 5

? What's the global capital of investment banking?

! New York ; London

? What's the global capital of investment banking?

! "New York" ; "London"

? What's the global capital of investment banking?

! New York
! London

? What's the global capital of investment banking?

! "New York"
! "London"

{case-sensitive: false}
? What's pi?

! "The ratio of a circle's circumference to its diameter" ; 3.14 ... 3.1416 ; an irrational number

{case-sensitive: false}
? What's pi?

! "The ratio of a circle's circumference to its diameter"
! 3.14 ... 3.1416
! an irrational number

? Where's the Eiffel Tower?

! /(Paris|France)/i

? Where's the Eiffel Tower?

! /Paris/i ; /France/i

{points: 2}
? Where's the Eiffel Tower?

! /Paris/i
! /France/i

{points: 2}
? Where's the Eiffel Tower?

!2 /Paris/
!1 /paris/i
!.5 /France/i

{points: 2}
? Where's the Eiffel Tower?

! /Paris/
!1 /paris/i
!.5 /France/i

{points: 2}
? Where's the global capital of investment banking?

!2 New York ; London
!1 USA ; UK

As shown by the answer ("The ratio of a circle's circumference to its diameter" ; 3.14 ... 3.1416; an irrational number), acceptable answer values in a fill in the blank question can be of completely different types, and numeric answer values can be expressed as ranges (min <= x <= max), expressed as min ... max. Also, this answer shows that quotes are optional around text strings. The reason to use quotes is for clarity, or to ensure that any semicolons used are treated as semicolons instead of as answer choice delimiters. Semicolons inside quotes are just semicolons and do not need to be backslash-escaped. You do, however, need to backslash-escape a quote if you want it to be treated as a literal quote, instead of the start or end of a string.

Supported Attributes on Fill In The Blank Questions

points: The number of points the question is worth. This number can be 0 or higher. The default is the 1. The answers must either not specify points (in which case they are worth the full value of points that the question is worth), or they must specify points between 0 and the points value.
case-sensitive: true or false. The default is true. This sets the behaviour of the specific fill in the blank question. If true, the fill in the blank question answer is case-sensitive. If false, it is not. In the case of multiple acceptable answer values, this attribute applies to all of them. Note that this only applies to text string answers, not to regular expressions. For a regular expression to be case-insensitive, you must end it with an i after the closing backtick.

Creating a Course or MOOC from a Markua Document

Over the past decade, there has been a steady growth of interest in courses delivered over the internet at massive scale. These Massive Open Online Courses, or MOOCs, consist of essentially four things:

Reading material
Video or audio lectures
Exercises, with answers provided to the student
Quizzes, with answers used to automatically mark the quiz

It turns out the four things in this list all work perfectly in a Markua document. So, not only can Markua be used to easily create a textbook which includes video, audio, images and quizzes, it is also an amazingly simple and flexible way of creating a MOOC. A MOOC is essentially just a textbook which is executable, plus discussion forums and credentials. For example, Leanpub authors can click one button to create a massive open online course (MOOC), complete with automated marking for all the quizzes in the course, entirely from one Markua source document.

The fact that a Markua document can be used to create an online course or MOOC means that certain aspects of the syntax for quizzes and exercises are more robust than they would otherwise. One example of this is question alternates.

Question Alternates

In an online course or MOOC, some professors might not want every question the same, despite the fact that question order and answer order can be randomized. So, Markua supports question alternates, using a simple (if slightly ugly) syntax. Question alternates are only supported in quizzes, since they make no sense to include in exercises.

To create question alternates, every question in the quiz (not just those with alternates) must be numbered sequentially, starting from 1, using a ?# syntax. This is a question mark followed by the number of the question, e.g. ?1, ?2, ?3. The questions in a quiz are numbered using sequential positive integers starting from 1: 1, 2, 3, etc.

The alternates are specified by providing the same number for multiple questions, e.g. ?1, ?1, ?1, ?2, ?3, ?4, ?4, ?5. When the actual quiz is given, only one of the questions for the given question number is used.

Note that only the first question with a given number may have a points attribute–since all other alternates must use the same points value, specifying it would be pointless.

The following is an example of a quiz which uses question alternates. This ensures that to ensure that students get randomly selected versions of questions 1 and 4. Also, since random-question-order: true is used, the actual position of the questions is randomized after the specific questions are selected from the alternates.

{quiz, id: "midterm", random-question-order: true}
?1 What's 2 + 2?

! 4

?1 What's 2.2 + 2.2?

! 4.4

?2 what's 3 + 3?

! 6

?3 What's 4 + 4?

! 8

{points: 2}
?4 What's 5 + 5?

! 10

?4 What's 6 + 6?

! 12

?5 What's 7 + 7?

! 14
{/quiz}

Note that the syntax for question alternates is very strict. Every question must have a number, and these numbers must be in ascending order (except for the alternates, which have the same number as each other).

Question alternates can also be grouped by a choose-questions attribute attached to the first question alternate. In this case, Leanpub must choose the number of questions m specified from the given alternates with that number n, or n choose m. Note that in this scenario, the numbering after the alternates increases by m: for example, if a quiz starts with a choose: 3, the next question is numbered 4, not 2. This ensures that the person constructing the quiz knows what they are doing, and saves them from having to keep track in a scenario where there are multiple questions with a choose-questions attribute.

{quiz, id: "midterm", random-question-order: true}

{choose-questions: 3}
?1 What's 2 + 2?

! 4

?1 What's 2.2 + 2.2?

! 4.4

?1 what's 3 + 3?

! 6

?1 What's 4 + 4?

! 8

?1 What's 5 + 5?

! 10

{points: 2}
?4 What's 6 + 6?

! 12

?5 What's 7 + 7?

! 14
{/quiz}

Leanpub must treat any error in the numbering of question alternates (and the questions which follow) as an error, and not generate the quiz if there is any error. This is preferable to Leanpub of trying to guess at what the author meant, and trying to do the right thing. Fixing a syntax error takes a couple minutes of editing and a few minutes to publish the book or course again. However, fixing the consequences of a quiz being administered to hundreds–or thousands, or tens of thousands–of people with an incorrect number of questions, or with questions incorrectly used as alternates for each other, would be much more difficult.