Leanpub: Publish Early, Publish Often

Working with materials

The power of RACHEL and related projects comes from the materials made available for people to use.

Some practical context (a personal perspective)

In the last few years and during my travels I’ve come to the conclusion that the packaging and distribution of materials is one of the biggest challenges for RACHEL and similar projects that aim to provide contents to the vast majority of people and locations that don’t have reliable, high speed Internet access.

Here are my heuristics I’m working towards:

the maximum chunk should be designed to fit on a USB stick one could buy at any tourist resort or medium sized city globally e.g. climbing the Andes, at a beach resort in India, in a shopping mall in Nairobi, etc. That’s somewhere either 8GB or 16GB currently based on my admittedly very limited research (I check wherever I go in the world). I also consider ways to reduce the chunk size to one that’d fit (and be useful[1]) on a DVD-ROM and even a CD-ROM. The aim is to be able to deliver whatever we want on one or more practical, available pieces of hardware (and by that I include USB sticks, etc).
Modularity helps reduce the update size dramatically. Until recently folk had to redownload the entire image if they want the latest updates. <dev.worldpossible.org/mods> now allows people to download individual modules (using rsync, which is available in cygwin for Windows users). I’d like to get even more modular.
Compression also helps. Emmanuel of Kiwix is helping me with our PhET experiment to see if we can script the packaging of PhET. The principles and the commands we end up using may be generally applicable for rich HTML5 apps.

Money to pay good developers / devops / system administrator people may help us make better progress. Also, the industry is solving these problems for commercial apps, content delivery, etc. so let’s investigate the practices that seem to work and try to apply them as a starting point.

[1] having smaller usable chunks of content and software can also help people who simply don’t have the luxury of a device with 32GB available e.g. where a useful subset of RACHEL contents are being deployed by a project in Kenya who only had a few GB available; and I have various cheap Android tablets with roughly either 3GB or 6GB available from new (depending on whether they were equipped with 4GB or 8GB storage). Kiwix and zim files are one approach to package and distribute much of the materials.

Finding stuff

Making ‘offline copies’ of web sites

In principle it’s possible to create offline copies of sites. Technically, the key proviso is: provided the contents can work locally (without needing access to that organisation’s servers, databases, etc) then it should be possible. However before we start trying to download someone’s web site, we need to ensure we have their permission.

TBC more on ways to copy/download online materials

PhET github example
How Kiwix does it
Web content crawlers
Downloaders e.g. for YouTube videos. Picking the most suitable format for videos, documents etc.

Permission to use existing stuff

Writing new stuff

Blogging software, such as WordPress, can be added to computers running RACHEL so that people can create and share their work locally. Potentially they could also make it available on the Internet if they have a working connection.
Free software called Pachyderm can be used to create presentations in HTML5 https://sourceforge.net/p/pachyderm/wiki/P3_Whats_New/. We can then package and use the recorded presentations as part of our projects.

Collaborating vs. Single author

Enhancing public materials

Preparing and packaging materials

‘Local contents’ is a good place to share ad-hoc materials, for instance individual PDF files. however it’s not well suited to packaging a collection of content such as a set of videos, lessons, exercises, etc.

Packaging materials enables them to become a distinct entity, where they can be distributed, incorporated, and used - hopefully as a cohesive body of stuff. There are various ways the material can be packaged, there are some inevitable variations needed so they are acceptable to the different programs that will read and present them.

RACHEL Modules

RACHEL consists of modules that contain various materials, such as videos, web contents, etc. Each package has a consistent structure: each is a separate folder, with a file called index.htmlf.

index.htmlf contains HTML, part of a web page, a fragment. The fragment starts with a <div> tag and ends with the closing </div> tag. The div tag has a class of indexmodule which helps keep the styling consistent.

Here’s a very simple example, from the current music theory module:

1 <div class="indexmodule">
2     <a href="<?php echo $dir ?>/index.html"><img src="<?php echo $dir ?>/musicth\
3 eorylogo.jpg" alt="Music Theory"></a>
4     <h2><a href="<?php echo $dir ?>/index.html">Music Theory</a></h2>
5     <p>Comprehensive music theory lessons and exercises to begin your journey in\
6 to the realm of music.</p>
7 </div>

you may notice it references an index.html page. Most (but not all) modules have an index.html page. This index.html makes the contents easier to navigate from a File Explorer program such as Microsoft Explorer or Nautilus on Ubuntu.

The modules are sub-folders of the following folder /RACHEL/bin/www/modules/ (in Windows the / are replaced by ).

Here’s the current set of fragments for the packages provided with USB RACHEL 32GB.

 1 ./powertyping/index.htmlf
 2 ./medline_plus/index.htmlf
 3 ./hesperian_health/index.htmlf
 4 ./math_expression/index.htmlf
 5 ./khan_academy/index.htmlf
 6 ./ebooks/index.htmlf
 7 ./understanding_algebra/index.htmlf
 8 ./khan_health/index.htmlf
 9 ./olpc/index.htmlf
10 ./ck12/index.htmlf
11 ./practical_action/index.htmlf
12 ./infonet/index.htmlf
13 ./iicba/index.htmlf
14 ./musictheory/index.htmlf
15 ./wikipedia_for_schools/index.htmlf
16 ./scratch/index.htmlf

Packaging for Kiwix & kiwix-serve

Kiwix and kiwix-serve use a common file format called zim, or zimfiles. These have an extension of .zim. The materials should be usable in a modern web browser, either served by a web server (we test with nginx), or using the file:// protocol (which is what happens if you open an html file in Windows Explorer, and the equivalent for other operating systems. A program called zimwriterfs is used to package the files.

Using Contents management software

We could try a simple contents management program such as Statamic http://statamic.com/learn (Thank you Damien for the suggestion).

Packaging for RACHEL php Server

RACHEL Web Server currently runs a small php script (index.php) that scans sub-folders for files with an extension of .htmlf. If you create one of these files (called index.htmlf) in the root of the package, then your contents can be integrated in the homepage of RACHEL. The file needs to contain an HTML fragment, where everything is wrapped in a <div> tag.

TBC add an example skeleton here.

Preparing materials

While we can simply package the files, preparating the structure, removing unnecessary files, and making the materials more consistent in terms of layout, naming, etc. can help make the materials easier to maintain and manage. Also, consider that some users will have relatively slow, small computers, to read or interact with your materials, so you may want to check the materials are usable on low-cost underpowered tablet devices, etc.

Shrinking the materials

Image compression - image resolution that’s still usable and attractive for the target devices.
Removing bulky stuff - ask is it necessary, can we device a low-bandwidth alternative, etc.