Raspberry Pi Computing: Temperature Measurement
Raspberry Pi Computing: Temperature Measurement
Buy on Leanpub
Raspberry Pi Computing: Temperature Measurement

Table of Contents

Introduction

Welcome!

Hi there. Congratulations on being interested enough in the process of learning about measuring temperature with the Raspberry Pi to have gotten your hands on this book.

Hopefully this will be a journey of discovery for both of us. The goal is to experiment with computers and use them to know a bit more about what is happening in the physical environment. I know that this sort of thing has been done already by others, but I have an ulterior motive. I write books to learn and document what I’ve done. The hope is that by sharing the journey perhaps others can learn something from my efforts :-).

Ambitious? Perhaps :-). But I’d like to think that if you’re reading this, I managed to make some headway. I dare say that like other books I have written (or are in the process of writing) it will remain a work in progress. They are living documents, open to feedback, comment, expansion, change and improvement. Please feel free to provide your thoughts on ways that I can improve things. Your input would be much appreciated.

You will find that I have typically eschewed a simple “Do this approach” for more of a story telling exercise. This means that some explanations are longer and more flowery than might be to everyone’s liking, but there you go, try to be brave :-).

There’s a lot of information in the book. There’s a vast amount of ‘stuff’ that people with a faint idea about computing will find excessive. Sorry about that. I’ve deliberately gathered together a lot of the content from other books I’ve written to create a guide that is as full of usable information as possible to help people who could conceivably be using the Pi and coding for the first time. This is ONE fairly simple project that might otherwise be described in 5 pages stretched out into about 170. But I’m working on the principle that if I need to recreate what I have here from scratch, this guide will leave nothing out. My plan is that it will also form the basis of other similarly explained books. Evolving the description as the OS versions and Raspberry Pi’s are updated.

I’m sure most authors try to be as accessible as possible. I’d like to do the same, but be warned… There’s a good chance that if you ask me a technical question I may not know the answer. So please be gentle with your emails :-).

Email: d3noobmail+temp@gmail.com

Cover photo via Good Free Photos.

What are we trying to do?

Put simply, we are going to examine the wonder that is the Raspberry Pi computer and use it to accomplish something.

In this specific case we will be connecting several temperature probes to the Pi, measuring the values that they produce, recording them in a database and then making a graph out of that information that we can view on a web page!

Along the way we’ll;

  • Look at the Raspberry Pi and its history.
  • Work out how to get software loaded onto the Pi.
  • Learn about networking and configure the Pi accordingly.
  • Install and configure a web server, a database and a bunch of other software to get stuff working.
  • Write some code to interface with our temperature probes.
  • Make a web page that will make a pretty picture of our temperature readings.

Who is this book for?

You!

Just by virtue of taking an interest and getting hold of a copy of this book you have demonstrated a desire to learn, to explore and to challenge yourself. That’s the most important criteria you will want to have when trying something new. Your experience level will come second place to a desire to learn.

Having said that, it may be useful to be comfortable using the Windows operating system (I’ll be using Windows 7 for the set-up of the devices), you should be aware of Linux as an alternative operating system, but you needn’t have tried it before. The best thing to remember is that before you learn anything new, it pretty much always appears indistinguishable from magic, but once you start having a play, the mystery quickly falls away.’

What will we need?

Well, you could just read the book and learn a bit. By itself that’s not a bad thing, but trust me when I say that actually experimenting with physical computers is fun and rewarding.

The list below is pretty flexible in most cases and will depend on how you want to measure the temperatures.

  • A Raspberry Pi (I’m using a Raspberry Pi Model B 2 / 3)
  • Probably a case for the Pi
  • A MicroSD card
  • A power supply for the Pi
  • A keyboard and monitor that you can plug into the Pi (there are a few options here, read on for details)
  • A remote computer (like your normal desktop PC that you can use to talk to connect to the Pi). This isn’t strictly necessary, but it makes the experience way cooler.
  • Some DS18B20 temperature sensors (the waterproof kind). They are available from lots of places. Google is your friend.
  • A 10k Ohm resister
  • Some soldering equipment
  • Some dupont connectors (that’s what I used, but you could connect to the Pi in different ways)
  • An Internet connection for getting and updating the software.

As we work through the book we will be covering off the different parts required and you should get a good overview of what your options are in different circumstances.

Why on earth did I write this rambling tome?

That’s a really good question. This is a project that I use in the real world (to measure water temperatures) and I originally wrote it up as a project in the book “Raspberry Pi: Measure, Record, Explore”. However, that was some time ago and after a particularly embarrassing incident where I disconnected the power to the Pi without shutting it down correctly I found that it refused to read from the MicroSD card any more. That meant that I needed to reinstall and configure the software. When I had originally installed the project the most current version of the Raspbian operating system was ‘Wheezy’ (more on this later in the book) and the latest version (‘Stretch’) had some configuration changes that meant the instructions in Raspberry Pi: Measure, Record, Explore were difficult to follow. I could have updated the entire book, but that would have meant updating a very large number of projects that I have dotted about the house. I realised that even if I did this, eventually the project would become outdated again and the process would repeat itself. Therefore I thought that if I made a book that went from whoa (or woe in my case) to go for this specific moment in time I could get everything up and running and I could adapt the book for subsequent projects by simply adapting whatever the proceeding one was.

Will this work?

Who knows.

What I can tell you is that I now have a new benchmark for building my temperature measurement project and you have a book that tells you how I did it :-).

Included is a bunch of information from my books on the Raspberry Pi, Linux and d3.js. I hope you find it useful.

Where can I get more information?

The Raspberry Pi as a concept has provided an extensible and practical framework for introducing people to the wonders of computing in the real world. At the same time there has been a boom of information available for people to use them. The following is a far from exhaustive list of sources, but from my own experience it represents a useful subset of knowledge.

raspberrypi.org

Google+

reddit

Raspberry Pi Stack Exchange

The History of the Raspberry Pi

The story of how the Raspberry Pi came to be started in 2006. Eben Upton, Rob Mullins, Jack Lang and Alan Mycroft, who were based at the University of Cambridge’s Computer Laboratory, became concerned at the decline in the volume and skills of students applying to study Computer Science. They saw part of the problem being that a typical student applicant was not arriving with a history of hobby programming and tinkering with hardware. Instead they were coming on board with some web design experience, but little else.

They established that the way that kids were interacting with computers had changed. A school curriculum that was pre-loaded to emphasise working with Word and Excel and building web pages meant that the focus was more on working with applications. Games consoles were replacing the traditional hobbyist computer platforms that had existed in the era when the Amiga, Apple II, ZX Spectrum and the ‘build your own’ approach were king.

So, in 2006, Eben and the team began to design and prototype a platform that was cheap, simple, booted into a programming environment and most of all, one which would inspire the next generation of computer enthusiasts to recover the joy of experimenting with computers.

Between 2006 and 2008, they developed a number of prototypes based on the Atmel ATmega644 microcontroller. By 2008 processors designed for mobile devices were becoming affordable and powerful enough to support a range of multimedia features. This allowed the boards to support an graphical environment which they believed would make the board more attractive to children who were looking for an alternative to a purely programming-oriented device.

Eben, Rob, Jack and Alan, then teamed up with Pete Lomas, and David Braben to form the Raspberry Pi Foundation to bring the project to the world. The Foundation’s goal was to offer two versions of the board, priced at US$25 and US$35.

In August 2011 50 alpha boards are manufactured. These boards were functionally identical to the model B. Based on these, twenty-five model B Beta boards were assembled in December 2011 with the same component layout as what would be the production boards.

Early Alpha Board (Credit: Paul Downey)
Early Alpha Board (Credit: Paul Downey)

Interest in the project escalated as they were demonstrated booting Linux, playing a 1080p movie trailer and running benchmarking programs. During the first week of 2012, the first 10 boards were put up for auction on eBay. One was bought anonymously and donated to the museum at The Centre for Computing History in Suffolk, England. While the ten boards together raised over 16,000 Pounds (about $25,000 USD) the last to be auctioned (serial number No. 01) raised 3,500 Pounds by itself.

The Raspberry Pi Model B entered mass production through licensed manufacturing deals with element 14/Premier Farnell and RS Electronics. They started accepting orders for the model B on the 29th of February 2012. It was quickly apparent that they had identified a need in the marketplace as their servers struggled to cope with the load placed by watchers repeatedly refreshing their browsers. The official Raspberry Pi Twitter account reported that Premier Farnell sold out within few minutes of the initial launch, while RS Components took over 100,000 pre orders on the first day of sales.

raspberrypi.org blog lights the fuse.
raspberrypi.org blog lights the fuse.

Within two years they had sold over two million units.

The the lower cost model A went on sale for $25 on 4 February 2013. By that stage the Raspberry Pi was already a hit with the model B being manufactured at a rate of 4000 units per day and the amount of on-board ram had been increased to 512MB.

The official Raspberry Pi blog reported that the three millionth Pi shipped in early May 2014 and in July 2014 announced the Raspberry Pi Model B+, “the final evolution of the original Raspberry Pi. For the same price as the original Raspberry Pi model B, but incorporating numerous small improvements”. In November of the same year the even lower cost (US$20) A+ was announced. Like the A, it would have no Ethernet port, and just one USB port, but like the B+, it would have lower power requirements, a micro-SD-card slot and 40-pin HAT compatible GPIO.

On 2 February 2015 the official Raspberry Pi blog announced that the Raspberry Pi 2 was available. With the same form factor and connector layout as the Model B+, it has a 900 MHz quad-core ARMv7 Cortex-A7 CPU, twice the memory (for a total of 1 GB) and complete compatibility with the original generation of Raspberry Pis.

Raspberry Pi B+ and Raspberry Pi 2 B
Raspberry Pi B+ and Raspberry Pi 2 B

Following a meeting with Eric Schmidt (of Google fame) in 2013, Eben embarked on the design of a new form factor for the Pi and in on the 26th of November 2015 the Pi Zero was released. The Pi Zero is a significantly smaller version of a Pi with similar functionality but with a retail cost of $5. On release it sold out (20,000 units) World wide in 24 hours and a free copy was affixed to the cover of the MagPi magazine.

The Raspberry Pi 3 was released in February 2016 and was notable for the inclusion of on-board WiFi and Bluetooth.

In February 2017 the Raspberry Pi Zero W was announced. This device had the same small form factor of the Pi Zero, but included the WiFi and Bluetooth functionality of the Raspberry Pi 3.

While it is easy to consider that the measurement of the success of the Raspberry Pi would be in the number of computer boards sold, This would most likely not be the opinion of those visionaries who began the journey to develop the boards. To them the stated aim was to re-invigorate the desire of young people to experiment with computers and to have fun doing it. Their success can be measured as the numerous projects, blogs, updated school curriculum’s and all around enjoyment that their efforts have produced.

Raspberry Pi Versions

In the words of the totally awesome Raspberry Pi foundation;

The Raspberry Pi is a low cost, credit-card sized computer that plugs into a computer monitor or TV, and uses a standard keyboard and mouse. It’s capable of doing everything you’d expect a desktop computer to do, from browsing the internet and playing high-definition video, to making spreadsheets, word-processing, playing games and learning how to program in languages like Scratch and Python.

The Raspberry Pi B+ Board
The Raspberry Pi B+ Board

There are (at time of writing) seven different models on the market. The A, B, A+, B+, ‘2 model B’, ‘3 model B’ (which I’m just going to call the 2 and 3 respectively), the Zero and Zero W. A lot of projects will typically use either the the 2 or the 3 for no reason other than they offer a good range of USB ports (4), 1024 MB of RAM, an HMDI video connection and an Ethernet connection. For all intents and purposes either the 2 or 3 can be used interchangeably for the projects depending on connectivity requirements as the 3 has WiFi and Bluetooth built in. For size limited situations or where lower power is an advantage, the Zero or Zero W is useful, although there is a need to cope with reduced connectivity options (a single micro USB connection) although the Zero W has WiFi and Bluetooth built in. Always aim to use the latest version of the Raspbian operating system (or at least one released on or after the 31st of January 2015). For best results browse the ‘Downloads’ page of raspberrypi.org.

Raspberry Pi B+, 2 B and 3 B

Raspberry Pi B+
Raspberry Pi B+

The model B+, 2 B and 3 B all share the same form factor and have been a consistent standard for the layout of connectors since the release of the B+ in July 2014. They 85 x 56 x 17mm, weighs 45g and are powered by Broadcom chipsets of varying speeds, numbers of cores and architectures.

USB Ports

They include 4 x USB Ports (with a maximum output of 1.2A)

Raspberry Pi B+ USB Ports
Raspberry Pi B+ USB Ports

Video Out

Integrated Videocore 4 graphics GPU capable of playing full 1080p HD video via a HDMI video output connector. HDMI standards rev 1.3 & 1.4 are supported with 14 HDMI resolutions from 640×350 to 1920×1200 plus various PAL and NTSC standards.

Raspberry Pi B+ HDMI Video Output
Raspberry Pi B+ HDMI Video Output

Ethernet Network Connection

There is an integrated 10/100Mb Ethernet Port for network access.

Raspberry Pi B+ Ethernet Connector
Raspberry Pi B+ Ethernet Connector

USB Power Input Jack

The boards include a 5V 2A Micro USB Power Input Jack.

Raspberry Pi B+ USB Power Input
Raspberry Pi B+ USB Power Input

MicroSD Flash Memory Card Slot

There is a microSD card socket on the ‘underside ‘of the board. On the Model 2 this is a ‘push-push’ socket. On the 3 this is a simple friction fit.

Raspberry Pi B+ MicroSD Card Socket
Raspberry Pi B+ MicroSD Card Socket

Stereo and Composite Video Output

The B+, 2 B and 3 B includes a 4-pole (TRRS) type connector that can provide stereo sound if you plug in a standard headphone jack and composite video Output with stereo audio if you use a TRRS adapter.

Raspberry Pi B+ A/V Connector
Raspberry Pi B+ A/V Connector

40 Pin Header

The Raspberry Pi B+, 2 B and 3 B includes a 40-pin, 2.54mm header expansion slot (Which allows for peripheral connection and expansion boards).

Raspberry Pi B+ GPIO Connector
Raspberry Pi B+ GPIO Connector

Raspberry Pi Peripherals

To make a start using the Raspberry Pi we will need to have some additional hardware to allow us to configure it.

SD Card

The Raspberry Pi needs to store the Operating System and working files on a MicroSD card (actually a MicroSD card for the A+, B+ B2, B3 and Zero models and a full size SD card if you’re using an older A or B model).

MicroSD Card
MicroSD Card

The MicroSD card receptacle is on the rear of the board and on the Model 2 it is a ‘push-push’ type which means that you push the card in to insert it and then to remove it, give it a small push and it will spring out.

MicroSD Card Positioning
MicroSD Card Positioning

This is the equivalent of a hard drive for a regular computer, but we’re going for a minimal effect. We will want to use a minimum of an 8GB card (smaller is possible, but 8 is recommended). Also try to select a higher speed card if possible (class 10 or similar) as this should speed things up a bit.

Keyboard / Mouse

While we will be making the effort to access our system via a remote computer, we will need a keyboard and a mouse for the initial set-up. Because the B+, B2 and B3 models of the Pi have 4 x USB ports, there is plenty of space for us to connect wired USB devices.

Wired Keyboard and Mouse
Wired Keyboard and Mouse

A wireless combination would most likely be recognised without any problem and would only take up a single USB port, but if we will build towards a remote capacity for using the Pi (using it headless, without a keyboard / mouse / display), the nicety of a wireless connection is not strictly required.

Wireless Keyboard and Mouse
Wireless Keyboard and Mouse

Video

The Raspberry Pi comes with an HDMI port ready to go which means that any monitor or TV with an HDMI connection should be able to connect easily.

HDMI Connected Monitor
HDMI Connected Monitor

Because this is kind of a hobby thing you might want to consider utilising an older computer monitor with a DVI or 15 pin D connector. If you want to go this way you will need an adapter to convert the connection.

VGA to HDMI Adapter
VGA to HDMI Adapter

Network

The B+, B2 and B3 models of the Raspberry Pi have a standard RJ45 network connector on the board ready to go. In a domestic installation this is most likely easiest to connect into a home ADSL modem or router.

HDMI Connected Monitor
HDMI Connected Monitor

This ‘hard-wired’ connection is great for a simple start, but we will work through using a wireless solution later in the book.

Power supply

The Pi can be powered up in a few ways. The simplest is to use the micro USB port to connect from a standard USB charging cable. You probably have a few around the house already for phones or tablets.

Power Supply Connection
Power Supply Connection

It is worth knowing that depending on what use we wish to put our Raspberry Pi to we might want to pay a certain amount of attention to the amount of current that our power supply can supply. The A+, B+ and Zero models will function adequately with a 700mA supply, but with the version 2 and 3 models of the Pi, or if we want to look towards using multiple wireless devices or supplying sensors that demand increased power, we should consider a supply that is capable of an output up to 2.5A.

Cases

We should get ourselves a simple case to sit the Pi out of the dust and detritus that’s floating about. There are a wide range of options to select from. These range from cheap but effective to more costly than the Pi itself (not hard) and looking fancy.

You could use a simple plastic case that can be brought for a few dollars;

Simple ABS plastic case
Simple ABS plastic case

At the high end of the market is a high quality aviation grade anodized aluminium case from ebay seller sauliakasas This will cost you more than the Pi itself, but it is a beautiful case;

High quality aviation grade anodized aluminium case
High quality aviation grade anodized aluminium case

For a sense of style, a very practical design and a warm glow from knowing that you’re supporting a worthy cause, you could go no further than the official Raspberry Pi case that includes removable side-plates and loads of different types of access. All for the paltry sum of about $9.

Official Raspberry Pi case
Official Raspberry Pi case

Operating Systems

An operating system is software that manages computer hardware and software resources for computer applications. For example Microsoft Windows could be the operating system that will allow the browser application Firefox to run on our desktop computer.

Variations on the Linux operating system are the most popular on our Raspberry Pi. Often they are designed to work in different ways depending on the function of the computer.

Linux is a computer operating system that is can be distributed as free and open-source software. The defining component of Linux is the Linux kernel, an operating system kernel first released on 5 October 1991 by Linus Torvalds.

Linux was originally developed as a free operating system for Intel x86-based personal computers. It has since been made available to a huge range of computer hardware platforms and is one of the most popular operating system on servers, mainframe computers and supercomputers. Linux also runs on embedded systems, which are devices whose operating system is typically built into the firmware and is highly tailored to the system; this includes mobile phones, tablet computers, network routers, facility automation controls, televisions and video game consoles. Android, the most widely used operating system for tablets and smart-phones, is built on top of the Linux kernel. In our case we will be using a version of Linux that is assembled to run on the ARM CPU architecture used in the Raspberry Pi.

The development of Linux is one of the most prominent examples of free and open-source software collaboration. Typically, Linux is packaged in a form known as a Linux distribution, for both desktop and server use. Popular mainstream Linux distributions include Debian, Ubuntu and the commercial Red Hat Enterprise Linux. Linux distributions include the Linux kernel, supporting utilities and libraries and usually a large amount of application software to carry out the distribution’s intended use.

A distribution intended to run as a server may omit all graphical desktop environments from the standard install, and instead include other software to set up and operate a solution stack such as LAMP (Linux, Apache, MySQL and PHP). Because Linux is freely re-distributable, anyone may create a distribution for any intended use.

Welcome to Raspbian

The Raspbian Linux distribution is based on Debian Linux. At the time of writing there have been three different editions published. ‘Wheezy’, ‘Jessie’ and ‘Stretch’. Debian is a widely used Linux distribution that allows Raspbian users to leverage a huge quantity of community based experience in using and configuring software. The Wheezy edition is the earlier of the three and was the stock edition from the inception of the Raspberry Pi till the end of 2015. From that point Jessie was the default distribution until mid 2017 when Stretch took over.

Downloading

The best place to source the latest version of the Raspbian Operating System is to go to the raspberrypi.org page; http://www.raspberrypi.org/downloads/. We will download the ‘Lite’ version (which doesn’t use a desktop GUI). If you’ve never used a command line environment, then good news! You’re about to enter the World of ‘real’ computer users :-).

Raspbian Download
Raspbian Download

You can download via bit torrent or directly as a zip file, but whatever the method you should eventually be left with an ‘img’ file for Raspbian.

To ensure that the projects we work on can be used with either the B+, 2 or 3 models we need to make sure that the version of Raspbian we download is from 2015-01-13 or later. Earlier downloads will not support the more modern CPU of the 2 or 3.

Image File
Image File

We should always try to download our image files from the authoritative source and we can normally do so in a couple of different ways. We can download via bit torrent or directly as a zip file.

To ensure that the projects we work on can be used with the full range of Raspberry Pi models we need to make sure that the versions of the image files we download are from 2015-01-13 or later. Earlier downloads will not support the more modern CPU of models from the B2 onwards.

Writing the Operating System image to the SD Card

Once we have an image file we need to get it onto our SD card.

We will work through an example using Windows 7 but the process should be very similar for other operating systems as we will be using the excellent open source software Etcher which is available for Windows, Linux and MacOS.

Download and install Etcher and start it up.

Etcher Start
Etcher Start

Select the img file that you want to install.

Etcher SD Card Selection
Etcher SD Card Selection

You will need an SD card reader capable of accepting your MicroSD card (you may require an adapter or have a reader built into your desktop or laptop). Place the card in the reader and you should see Etcher automatically select it for writing (Etcher is very good at presenting options for installing that are only SD cards).

Flash the drive
Flash the drive

Then click on ‘Flash!’ to burn the card.

Etcher in progress
Etcher in progress

Etcher will write the image to the SD card. The time taken can vary a little, but it should only take about 3-4 minutes with a class 10 SD card.

Once written, Etcher will validate the write process (this can be disabled if desired).

Flash Complete!
Flash Complete!

When the process is finished exit Etcher and eject the card from the computer and we’re done.

Powering On

Insert the card into the slot on the Raspberry Pi and turn on the power.

You will see a range of information scrolling up the screen before eventually being presented with a login prompt.

The Command Line interface

Because we have installed the ‘Lite’ version of Raspbian, when we first boot up, the process should automatically re-size the root file system to make full use of the space available on your SD card. If this isn’t the case, no need to worry as the facility to do it can be accessed from the Raspberry Pi configuration tool.

Once the reboot is complete (if it occurs) you will be presented with the console prompt to log on;

Raspbian GNU/Linux 7 raspberrypi tty1

raspberrypi login:

The default username and password is:

Username: pi

Password: raspberry

Enter the username and password.

Congratulations, you have a working Raspberry Pi and are ready to start getting into the thick of things!

Firstly we’ll do a bit of house keeping.

Raspberry Pi Software Configuration Tool

As mentioned earlier, if we weren’t prompted to use the Raspberry Pi Software Configuration Tool we should run it now to enable full use of the storage on the SD card and changes in the locale and keyboard configuration as well as enabling ssh (more on that later). This can be done by running the following command;

Raspberry Pi Software Configuration Tool
Raspberry Pi Software Configuration Tool

Use the up and down arrow keys to move the highlighted section to the selection you want to make then press tab to highlight the <Select> option (or <Finish> if you’ve finished).

If you didn’t see the file system expanded on the SD card on first boot, select ‘7 Advanced Options’;

Advanced Options
Advanced Options

Then ‘A2 Expand Filesystem’;

Expand Filesystem
Expand Filesystem

Once selected there will be a dialogue box that will tell us that the changes will be enabled on the next boot.

Reboot Time
Reboot Time

Once this has been completed we can continue to configure other options safe in the knowledge that when we reboot the Pi we will have the use of the full capacity of the SD card.

While we are here it would probably be a good idea to change the settings for our operating system to reflect our location for the purposes of having the correct time, language and WiFi regulations. These can all be located via selection ‘4 Localisation Options’ on the main menu.

Select Localisation Options
Select Localisation Options

Select this and work through any changes that are required for your installation based on geography.

Localisation Options
Localisation Options

The last main menu item that is worth considering is to enable remote access via ssh. This will allow us to access the Raspberry Pi on our local network via a desktop computer or laptop, removing the need to have a keyboard and monitor connected directly to the Pi. More on the options available here are in the Remote Access section. To enable this select ‘5 Interfacing Options’ from the main menu.

Interfacing Options
Interfacing Options

From here we select ‘P2 SSH’

Enabling ssh
Enabling ssh

ssh used to be enabled by default, but doing so presents a potential security concern, so it has been disabled by default as of the end of 2016.

Once you exit out of the raspi-config menu system, if you have made a few changes, there is a probability that you will be asked if you want to re-boot the Pi. That’s a pretty good idea.

Once the reboot is complete you will be presented with the console prompt to log on again;

Software Updates

After configuring our Pi we’ll want to make sure that we have the latest software for our system. This is a useful thing to do as it allows any additional improvements to the software we will be using to be enhanced or security of the operating system to be improved. This is probably a good time to mention that we will need to have an Internet connection available.

Type in the following line which will find the latest lists of available software;

You should see a list of text scroll up while the Pi is downloading the latest information.

Then we want to upgrade our software to latest versions from those lists using;

The Pi should tell you the lists of packages that it has identified as suitable for an upgrade along with the amount of data that will be downloaded and the space that will be used on the system. It will then ask you to confirm that you want to go ahead. Tell it ‘Y’ and we will see another list of details as it heads off downloading software and installing it.

Power Up the Pi

To configure the Raspberry Pi for our purpose we will extend our Pi a little. This makes configuring and using the device easier and to be perfectly honest, making life hard for ourselves is so exhausting! Let’s not do that.

Static IP Address

Enabling remote access is a really useful thing. This will allow us to configure and operate our raspberry Pi from a separate computer. To do so we will want to assign our Raspberry Pi a static IP address.

An Internet Protocol address (IP address) is a numerical label assigned to each device (e.g., computer, printer) participating in a computer network that uses the Internet Protocol for communication.

There is a strong likelihood that our Raspberry Pi already has an IP address and it should appear a few lines above the ‘login’ prompt when you first boot up;

My IP address is 10.1.1.25

Raspbian GNU/Linux 7 raspberrypi tty1

raspberrypi login:

The My IP address... part should appear just above or around 15 lines above the login line, depending on the version of Raspbian we’re using. In this example the IP address 10.1.1.25 belongs to the Raspberry Pi.

This address will probably be a ‘dynamic’ IP address and could change each time the Pi is booted. For the purposes of using the Raspberry Pi with a degree of certainty when logging in to it remotely it’s easier to set a fixed IP address.

This description of setting up a static IP address makes the assumption that we have a device running on the network that is assigning IP addresses as required. This sounds like kind of a big deal, but in fact it is a very common service to be running on even a small home network and it will be running on the ADSL modem or similar. This function is run as a service called DHCP (Dynamic Host Configuration Protocol). You will need to have access to this device for the purposes of knowing what the allowable ranges are for a static IP address. The most likely place to find a DHCP service running in a normal domestic situation would be an an ADSL modem or router.

The Netmask

A common feature for home modems and routers that run DHCP devices is to allow the user to set up the range of allowable network addresses that can exist on the network. At a higher level we should be able to set a ‘netmask’ which will do the job for us. A netmask looks similar to an IP address, but it allows you to specify the range of addresses for ‘hosts’ (in our case computers) that can be connected to the network.

A very common netmask is 255.255.255.0 which means that the network in question can have any one of the combinations where the final number in the IP address varies. In other words with a netmask of 255.255.255.0 the IP addresses available for devices on the network 10.1.1.x range from 10.1.1.0 to 10.1.1.255 or in other words any one of 256 unique addresses.

CIDR Notation

An alternative to specifying a netmask in the format of ‘255.255.255.0’ is to use a system called Classless Inter-Domain Routing, or CIDR. The concept is that you can add a specification in the IP address itself that indicates the number of significant bits that make up the netmask.

For example, we could designate the IP address 10.1.1.17 as associated with the netmask 255.255.255.0 by using the CIDR notation of 10.1.1.17/24. This means that the first 24 bits of the IP address given are considered significant for the network routing.

Using CIDR notation allows us to do some very clever things to organise our network, but at the same time it can have the effect of freaking people out by introducing a pretty complex topic when all they want to do is get their network going :-). So for the sake of this explanation we can assume that if we wanted to specify an IP address and a netmask, it could be accomplished by either specifying each seperatly (IP address = 10.1.1.17 and netmask = 255.255.255.0) or in CIDR format (10.1.1.1/24)

Distinguish Dynamic from Static

The other service that our DHCP server will allow is the setting of a range of addresses that can be assigned dynamically. In other words we will be able to declare that the range from 10.1.1.20 to 10.1.1.255 can be dynamically assigned which leaves 10.1.1.0 to 10.1.1.19 which can be set as static addresses.

You might also be able to reserve an IP address on your modem / router. To do this you will need to know what the MAC (or hardware address) of the Raspberry Pi is. To find the hardware address on the Raspberry Pi type;

(For more information on the ifconfig command check out the Linux commands section)

This will produce an output which will look a little like the following;

eth0 Link encap:Ethernet HWaddr 00:08:C7:1B:8C:02
     inet addr:10.1.1.26 Bcast:10.1.1.255 Mask:255.255.255.0
     UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
     RX packets:53 errors:0 dropped:0 overruns:0 frame:0
     TX packets:44 errors:0 dropped:0 overruns:0 carrier:0
     collisions:0 txqueuelen:1000
     RX bytes:4911 (4.7 KiB)  TX bytes:4792 (4.6 KiB)

The figures 00:08:C7:1B:8C:02 are the Hardware or MAC address.

Because there are a huge range of different DHCP servers being run on different home networks, I will have to leave you with those descriptions and the advice to consult your devices manual to help you find an IP address that can be assigned as a static address. Make sure that the assigned number has not already been taken by another device. In a perfect World we would hold a list of any devices which have static addresses so that our Pi’s address does not clash with any other device.

For the sake of the upcoming projects we will assume that the address 10.1.1.17 is available.

Default Gateway

Before we start configuring we will need to find out what the default gateway is for our network. A default gateway is an IP address that a device (typically a router) will use when it is asked to go to an address that it doesn’t immediately recognise. This would most commonly occur when a computer on a home network wants to contact a computer on the Internet. The default gateway is therefore typically the address of the modem / router on your home network.

We can check to find out what our default gateway is from Windows by going to the command prompt (Start > Accessories > Command Prompt) and typing;

This should present a range of information including a section that looks a little like the following;

Ethernet adapter Local Area Connection:

  IPv4 Address. . . . . . . . . . . : 10.1.1.15
  Subnet Mask . . . . . . . . . . . : 255.255.255.0
  Default Gateway . . . . . . . . . : 10.1.1.1

The default router gateway is therefore ‘10.1.1.1’.

Lets edit the dhcpcd.conf file

On the Raspberry Pi at the command line we are going to start up a text editor and edit the file that holds the configuration details for the network connections.

The file is /etc/dhcpcd.conf. That is to say it’s the dhcpcd.conf file which is in the etc directory which is in the root (/) directory.

To edit this file we are going to type in the following command;

The nano file editor will start and show the contents of the dhcpcd.conf file which should look a little like the following;

 A sample configuration for dhcpcd.
# See dhcpcd.conf(5) for details.

# Allow users of this group to interact with dhcpcd via the control socket.
#controlgroup wheel

# Inform the DHCP server of our hostname for DDNS.
hostname

# Use the hardware address of the interface for the Client ID.
clientid
# or
# Use the same DUID + IAID as set in DHCPv6 for DHCPv4 ClientID per RFC4361.
#duid

# Persist interface configuration when dhcpcd exits.
persistent

# Rapid commit support.
# Safe to enable by default because it requires the equivalent option set
# on the server to actually work.
option rapid_commit

# A list of options to request from the DHCP server.
option domain_name_servers, domain_name, domain_search, host_name
option classless_static_routes
# Most distributions have NTP support.
option ntp_servers
# Respect the network MTU.
# Some interface drivers reset when changing the MTU so disabled by default.
#option interface_mtu

# A ServerID is required by RFC2131.
require dhcp_server_identifier

# Generate Stable Private IPv6 Addresses instead of hardware based ones
slaac private

# A hook script is provided to lookup the hostname if not set by the DHCP
# server, but it should not be run by default.
nohook lookup-hostname

We are going to add the information that tells the network interface to use eth0 at our static address that we decided on earlier (10.1.1.17) along with information on the netmask to use (in CIDR format) and the default gateway of our router. To do this we will add the following lines to the end of the information in the dhcpcd.conf file;

# Custom static IP address for eth0.
interface eth0
static ip_address=10.1.1.17/24
static routers=10.1.1.1
static domain_name_servers=10.1.1.1

Here we can see the IP address and netmask (static ip_address=10.1.1.17/24), the gateway address for our router (static routers=10.1.1.1) and the address where the computer can also find DNS information (static domain_name_servers=10.1.1.1).

Once you have finished press ctrl-x to tell nano you’re finished and it will prompt you to confirm saving the file. Check your changes over and then press ‘y’ to save the file (if it’s correct). It will then prompt you for the file-name to save the file as. Press return to accept the default of the current name and you’re done!

To allow the changes to become operative we can type in;

This will reboot the Raspberry Pi and we should see the (by now familiar) scroll of text and when it finishes rebooting you should see;

My IP address is 10.1.1.17

Raspbian GNU/Linux 7 raspberrypi tty1

raspberrypi login:

Which tells us that the changes have been successful (bearing in mind that the IP address above should be the one you have chosen, not necessarily the one we have been using as an example).

Remote access

To allow us to work on our Raspberry Pi from our normal desktop we can give ourselves the ability to connect to the Pi from another computer. The will mean that we don’t need to have the keyboard / mouse or video connected to the Raspberry Pi and we can physically place it somewhere else and still work on it without problem. This process is called ‘remotely accessing’ our computer .

To do this we need to install an application on our windows desktop which will act as a ‘client’ in the process and have software on our Raspberry Pi to act as the ‘server’. There are a couple of different ways that we can accomplish this task, but because we will be working at the command line (where all we do is type in our commands (like when we first log into the Pi)) we will use what’s called SSH access in a ‘shell’.

Remote access via SSH

Secure Shell (SSH) is a network protocol that allows secure data communication, remote command-line login, remote command execution, and other secure network services between two networked computers. It connects, via a secure channel over an insecure network, a server and a client running SSH server and SSH client programs, respectively (there’s the client-server model again).

In our case the SSH program on the server is running sshd and on the Windows machine we will use a program called ‘PuTTY’.

Setting up the Server (Raspberry Pi)

SSH is already installed on Raspbian, but it needs to be enabled before it can be used. We worked through this earlier, but to check that it is there and working type the following from the command line;

The Pi should respond with the message that the program sshd is active (running).

pi@raspberrypi:~ $ /etc/init.d/ssh status
● ssh.service - OpenBSD Secure Shell server
   Loaded: loaded (/lib/systemd/system/ssh.service; enabled)
   Active: active (running) since Tue 2017-04-25 03:30:16 UTC; 1h 28min ago
 Main PID: 2135 (sshd)
   CGroup: /system.slice/ssh.service
           └─2135 /usr/sbin/sshd -D

If it isn’t, run the following command;

Raspberry Pi Software Configuration Tool
Raspberry Pi Software Configuration Tool

Use the up and down arrow keys to move the highlighted section to the selection you want to make then press tab to highlight the <Select> option (or <Finish> if you’ve finished).

To enable SSH select ‘5 Interfacing Options’ from the main menu.

Interfacing Options
Interfacing Options

From here we select ‘P2 SSH’

Enabling ssh
Enabling ssh

And we should be done!

Setting up the Client (Windows)

The client software we will use is called ‘Putty’. It is open source and available for download from here.

On the download page there are a range of options available for use. The best option for us is most likely under the ‘For Windows on Intel x86’ heading and we should just download the ‘putty.exe’ program.

Save the file somewhere logical as it is a stand-alone program that will run when you double click on it (you can make life easier by placing a short-cut on the desktop).

Once we have the file saved, run the program by double clicking on it and it will start without problem.

The first thing we will set-up for our connection is the way that the program recognises how the mouse works. In the ‘Window’ Category on the left of the PuTTY Configuration box, click on the ‘Selection’ option. On this page we want to change the ‘Action of mouse’ option from the default of ‘Compromise (Middle extends, Right paste)’ to ‘Windows (Middle extends, Right brings up menu)’. This keeps the standard Windows mouse actions the same when you use PuTTY.

PuTTY Selection Set-up
PuTTY Selection Set-up

Now select the ‘Session’ Category on the left hand menu. Here we want to enter our static IP address that we set up earlier (10.1.1.17 in the example that we have been following, but use your one) and because we would like to access this connection on a frequent basis we can enter a name for it as a saved session (In the screen-shot below it is imaginatively called ‘Raspberry Pi’). Then click on ‘Save’.

PuTTY Session Set-up
PuTTY Session Set-up

Now we can select our raspberry Pi Session (per the screen-shot above) and click on the ‘Open’ button.

The first thing you will be greeted with is a window asking if you trust the host that you’re trying to connect to.

PuTTY Session Connection
PuTTY Session Connection

In this case it is a pretty safe bet to click on the ‘Yes’ button to confirm that we know and trust the connection.

Once this is done, a new terminal window will be shown with a prompt to login as: . Here we can enter our user name (‘pi’) and then our password (if it’s still the default, the password is ‘raspberry’).

PuTTY Session Connected
PuTTY Session Connected

There you have it. A command line connection via SSH. Well done.

If this is the first time that you’ve done something like this it can be a very liberating feeling. To complete the feeling of freedom let’s set up a wireless network connection.

Setting up a WiFi Network Connection

Our set-up of the Raspberry Pi will allow us to carry out all the (computer interface) interactions via a remote connection. However, the Raspberry Pi is currently making that remote connection via a fixed network cable. It could be argued that the lower number of connections that we need to run to our machine the better. The most obvious solution to this conundrum is to enable a wireless connection.

It should be noted that enabling a wireless network will not be a requirement for everyone, and as such, I would only recommend it if you need to. If you’re using a model 3 or Zero W you have WiFi built in, otherwise you will need to purchase a USB WiFi dongle and correctly configure it.

Built in WiFi Enabling

We need to edit the file wpa_supplicant.conf at /etc/wpa_supplicant/wpa_supplicant.conf. This looks like the following;

country=NZ
ctrl_interface=DIR=/var/run/wpa_supplicant GROUP=netdev
update_config=1

Use the nano command as follows;

We need to add the ssid (the wireless network name) and the password for the WiFi network here so that the file looks as follows (using your ssid and password of course);

country=NZ
ctrl_interface=DIR=/var/run/wpa_supplicant GROUP=netdev
update_config=1
network={
    ssid="homenetwork"
    psk="h0mepassw0rd"
    key_mgmt=WPA-PSK
}
Make the changes operative

To allow the changes to become operative we can type in;

Once we have rebooted, we can check the status of our network interfaces by typing in;

This will display the configuration for our wired Ethernet port, our ‘Local Loopback’ (which is a fancy way of saying a network connection for the machine that you’re using, that doesn’t require an actual network (ignore it in the mean time)) and the wlan0 connection which should look a little like this;

wlan0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 10.1.1.99  netmask 255.255.255.0  broadcast 10.1.1.255
        inet6 fe80::8b9f:3e4f:dcf0:12a9  prefixlen 64  scopeid 0x20<link>
        ether b8:27:eb:e3:b7:f2  txqueuelen 1000  (Ethernet)
        RX packets 51  bytes 9384 (9.1 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 35  bytes 6078 (5.9 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

This would indicate that our wireless connection has been assigned the dynamic IP address 10.1.1.99.

We should be able to test our connection by connecting to the Pi via SSH and ‘PuTTY’ on the Windows desktop using the address 10.1.1.99.

In theory you are now the proud owner of a computer that can be operated entirely separate from all connections except power!

Make the built in WiFi IP address static

In the same way that we would edit the /etc/dhcpcd.conf file to set up a static IP address for our physical connection (eth0) we will now edit it with the command…

This time we will add the details for the wlan0 connection to the end of the file. Those details (assuming we will use the 10.1.1.17 IP address) should look like the following;

# Custom static IP address for wlan0.
interface wlan0
static ip_address=10.1.1.17/24
static routers=10.1.1.1
static domain_name_servers=10.1.1.1

Our wireless lan (wlan0) is now designated to be a static IP address (with the details that we had previously assigned to our wired connection) and we have added the ‘ssid’ (the network name) of the network that we are going to connect to and the password for the network.

Make the changes operative

To allow the changes to become operative we can type in;

We’re done!

WiFi Via USB Dongle

Using an external USB WiFi dongle can be something of an exercise if not done right. In my own experience, I found that choosing the right wireless adapter was the key to making the job simple enough to be able to recommend it to new users. Not all WiFi adapters are well supported and if you are unfamiliar with the process of installing drivers or compiling code, then I would recommend that you opt for an adapter that is supported and will work ‘out of the box’. There is an excellent page on elinux.org which lists different adapters and their requirements. I eventually opted for the Edimax EW-7811Un which literally ‘just worked’ and I would recommend it to others for it’s ease of use and relatively low cost (approximately $15 US).

Edimax WiFi USB Adapter
Edimax WiFi USB Adapter

To install the wireless adapter we should start with the Pi powered off and install it into a convenient USB connection. When we turn the power on we will see the normal range of messages scroll by, but if we’re observant we will note that there are a few additional lines concerning a USB device. These lines will most likely scroll past, but once the device has finished powering up and we have logged in we can type in…

… which will show us a range of messages about drivers that are loaded to support discovered hardware.

Somewhere in that list (hopefully towards the end) will be a series of messages that describe the USB connectors and what is connected to them. In particular we could see a group that looks a little like the following;

[3.382731] usb 1-1.2: new high-speed USB device number 4 using dwc_otg
[3.494250] usb 1-1.2: New USB device found, idVendor=7392, idProduct=7811
[3.507749] usb 1-1.2: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[3.520230] usb 1-1.2: Product: 802.11n WLAN Adapter
[3.542690] usb 1-1.2: Manufacturer: Realtek
[3.560641] usb 1-1.2: SerialNumber: 00345767831a5e

That is our USB adapter which is plugged into USB slot 2 (which is the ‘2’ in usb 1-1.2:). The manufacturer is listed as ‘Realtek’ as this is the manufacturer of the chip-set in the adapter that Edimax uses.

Editing files

We need to edit two files. The first is the file wpa_supplicant.conf at /etc/wpa_supplicant/wpa_supplicant.conf. This looks like the following;

country=NZ
ctrl_interface=DIR=/var/run/wpa_supplicant GROUP=netdev
update_config=1

Use the nano command as follows;

We need to add the ssid (the wireless network name) and the password for the WiFi network here so that the file looks as follows (using your ssid and password of course);

country=NZ
ctrl_interface=DIR=/var/run/wpa_supplicant GROUP=netdev
update_config=1
network={
    ssid="homenetwork"
    psk="h0mepassw0rd"
    key_mgmt=WPA-PSK
}
Make the changes operative

To allow the changes to become operative we can type in;

Once we have rebooted, we can check the status of our network interfaces by typing in;

This will display the configuration for our wired Ethernet port, our ‘Local Loopback’ (which is a fancy way of saying a network connection for the machine that you’re using, that doesn’t require an actual network (ignore it in the mean time)) and the wlan1 connection which should look a little like this;

wlan1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 10.1.1.97  netmask 255.255.255.0  broadcast 10.1.1.255
        inet6 fe80::c4e4:a6e5:9788:d2c2  prefixlen 64  scopeid 0x20<link>
        ether 00:ec:0b:4c:6b:99  txqueuelen 1000  (Ethernet)
        RX packets 106  bytes 18616 (18.1 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 34  bytes 5681 (5.5 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

This would indicate that our wireless connection has been assigned the dynamic IP address 10.1.1.97.

We should be able to test our connection by connecting to the Pi via SSH and ‘PuTTY’ on the Windows desktop using the address 10.1.1.97.

Make the built in WiFi IP address static

In the same way that we would edit the /etc/dhcpcd.conf file to set up a static IP address for our physical connection (eth0) we will now edit it with the command…

This time we will add the details for the wlan1 connection to the end of the file. Those details (assuming we will use the 10.1.1.17 IP address) should look like the following;

# Custom static IP address for wlan1.
interface wlan1
static ip_address=10.1.1.17/24
static routers=10.1.1.1
static domain_name_servers=10.1.1.1
Make the changes operative

To allow the changes to become operative we can type in;

We’re done!

Reconnecting to the wireless network automatically

I have found with experience that in spite of my best intentions, sometimes when setting up a Raspberry Pi to maintain a WiFi connection, if it disconnects for whatever reason it may not reconnect automatically.

To solve this problem we’re going to write a short script that automatically reconnects our Pi to a WiFi network. The script will check to see if the Pi is connected to our local network and, if it’s off-line, will restart the wireless network interface. We’ll use a cron job to schedule the execution of this script at a regular interval.

Let’s write a script

First, we’ll need to check if the Pi is connected to the network. This is where we’ll try to ping an IP address on our local network (perhaps our gateway address?). If the ping command succeeds in getting a response from the IP address, we have network connectivity. If the command fails, we’ll turn off our wireless interface (wlan1) and then turn it back on (yes, the timeless solution of turning it off and on).

The script looks a little like this;

#!/bin/bash

# The IP address of our gateway on our local router
GATEWAY=10.1.1.1

# Send two pings, with the output going to /dev/null
ping -c2 ${GATEWAY} > /dev/null

# Check to see if the returned value from ping ($?)
# is not 0 and then act to restart wlan1 if necessary
if [ $? == 0 ]
then
    # Restart wlan1 (the wireless interface)
    ifconfig wlan1 down
    ifconfig wlan1 up
fi

Use nano to create the script, name it something like wifistart.sh, and save it in /usr/local/bin.

We also need to make sure it’s executable by running chmod (using sudo) as follows;

Lets run our script on a regular schedule

To make our WiFi checking script run automatically, we’ll schedule a cron job using crontab;

… and add this line to the bottom:

*/5 * * * * /usr/bin/sudo -H /usr/local/bin/wifistart.sh >> /dev/null 2>&1

This runs the script every 5 minutes with sudo permissions, writing its output to /dev/null so it doesn’t spam syslog.

Let’s test it

To test that the script works as expected, we will want to take down the wlan1 interface and wait for the script to bring it back up. Before taking down wlan1, we might want to adjust the interval in crontab to 1 minute. And fair warning, when we disconnect wlan1, we will lose that network interface, so we will need to either have a local keyboard / monitor connected, have another network interface set up or be really confident that we’ve got everything set up right first time.

To take down wlan1 to confirm the script works, run:

After waiting for 5 (or 1) minutes, we could try ssh-ing back into the Raspberry Pi or if we’re keen we could have a ping command running on another server checking the interface to show when it stops and when it (hopefully) starts again. Assuming everything works, our Pi should reconnect seamlessly.

Setting up the Raspberry Pi Software

While the Raspberry Pi is a capable computer, we still need to install software on it to allow us to gather our data, store it and display it.

The software we will be using is based on the Linux Operating System. Because this is potentially unfamiliar territory (for those who haven’t used Linux or had some practical computing experience), we will take our time and explain things as we go.

Web Server, PHP and Database

Because we want to be able to explore the data we will be collecting, we need to set up a web server that will present web pages to other computers that will be browsing within the network (remembering that this is not intended to be connected to the Internet, just inside your home network). At the same time as setting up a web server on the Pi we will install PHP. PHP is a scripting language that is widely used in developing web pages. And because we will want to store our data somewhere we will add in the MariaDB database. MariaDB is a ‘fork’ of a database called ‘MySQL’ (a ‘fork’ is referring to a copied branch of software that developers want to take in a particular direction). You will see in the instructions below plenty of mention of MySQL since to maintain compatibility, MariaDB maintains a common naming convention for commands and file locations.

Install NGINX and PHP

The web server that we will use is called NGINX (pronounced “Engine X”). NGINX is an open-source web server that is often recommended for its performance and low resource consumption. This obviously makes it ideal for hardware such as the Raspberry Pi. In spite of being targeted as something of a ‘light’ application, it is extremely powerful, capable and it is widely used in large scale applications.

We can start the install process using the following;

We’re familiar with apt-get already, but this time we’re including more than one package in the installation process. Specifically we’re including nginx and php-fpm.

‘nginx’ is obviously the name of the NGINX web server and php-fpm is for PHP.

The Raspberry Pi will advise you of the range of additional packages that will be installed at the same time (to support those we’re installing (these additional packages are ‘dependencies’)). Agree to continue and the installation will proceed. This should take a few minutes or more (depending on the speed of your Internet connection).

Configuration

Firstly we should edit the default file that will get displayed as a web page if none is specified. What we want to do is to include the option to redirect to an index.php file.:

Replace the line:

index index.html index.htm index.nginx-debian.html;

with

index index.html index.htm index.php;

Now we want to edit the portion of the file that will handle all requests for resources that end in .php.

Replace the section of the file that looks like this;

        #location ~ \.php$ {
        #       include snippets/fastcgi-php.conf;
        
        #       # With php-fpm (or other unix sockets):
        #       fastcgi_pass unix:/var/run/php/php7.0-fpm.sock;
        #       # With php-cgi (or other tcp sockets):
        #       fastcgi_pass 127.0.0.1:9000;
        #}

With this;

        location ~ \.php$ {
                include snippets/fastcgi-php.conf;

                # With php-fpm (or other unix sockets):
                fastcgi_pass unix:/var/run/php/php7.0-fpm.sock;
        #       # With php-cgi (or other tcp sockets):
        #       fastcgi_pass 127.0.0.1:9000;
        }

NGINX has its default web page location at /var/www/html on Raspbian. We are going to change the permissions / ownership of that folder by running following two commands:

Now lets create a suitable index.php test file with the following command:

Now lets restart the NGINX service so that the changes we have made can take effect:

Now we can go to to the IP address of our Pi in a browser (in the example we are using it’s 10.1.1.17) and type it in to the URL area to test. Something like the following should be displayed;

Testing the web server
Testing the web server

Marvellous.

Database

As mentioned earlier, we will use a MariaDB database to store the information that we collect.

We will install MariaDB in a couple of steps and then we will install the database administration tool phpMyAdmin to make our lives easier accessing the database.

Then secure the installation by running

Follow the instructions to add a root password and to restrict access in whatever way you think appropriate to your situation (don’t be afraid to lock things down). The following is the listing of the messages that are shown and the questions that are asked.

NOTE: RUNNING ALL PARTS OF THIS SCRIPT IS RECOMMENDED FOR ALL MariaDB
      SERVERS IN PRODUCTION USE!  PLEASE READ EACH STEP CAREFULLY!

In order to log into MariaDB to secure it, we'll need the current
password for the root user.  If you've just installed MariaDB, and
you haven't set the root password yet, the password will be blank,
so you should just press enter here.

Enter current password for root (enter for none):
OK, successfully used password, moving on...

Setting the root password ensures that nobody can log into the MariaDB
root user without the proper authorisation.

Set root password? [Y/n] y
New password:
Re-enter new password:
Password updated successfully!
Reloading privilege tables..
 ... Success!


By default, a MariaDB installation has an anonymous user, allowing anyone
to log into MariaDB without having to have a user account created for
them.  This is intended only for testing, and to make the installation
go a bit smoother.  You should remove them before moving into a
production environment.

Remove anonymous users? [Y/n] y
 ... Success!

Normally, root should only be allowed to connect from 'localhost'.  This
ensures that someone cannot guess at the root password from the network.

Disallow root login remotely? [Y/n] y
 ... Success!

By default, MariaDB comes with a database named 'test' that anyone can
access.  This is also intended only for testing, and should be removed
before moving into a production environment.

Remove test database and access to it? [Y/n] y
 - Dropping test database...
 ... Success!
 - Removing privileges on test database...
 ... Success!

Reloading the privilege tables will ensure that all changes made so far
will take effect immediately.

Reload privilege tables now? [Y/n] y
 ... Success!

Cleaning up...

All done!  If you've completed all of the above steps, your MariaDB
installation should now be secure.

Thanks for using MariaDB!

Installing phpMyAdmin

phpMyAdmin is free software written in PHP to carry out administration of a MySQL or MariaDB database. While administration can be carried out at the command line, the web interface of phpMyAdmin is easy to use and well supported.

To begin installation run the following from the command line;

During the installation, you will be prompted for which web server you would like the software to configure. Since NGINX is not one of the available options, you can just hit TAB to bypass this prompt.

Selecting the web server for phpMyAdmin
Selecting the web server for phpMyAdmin

The next prompt will ask if you would like dbconfig-common to configure a database for phpMyAdmin to use. Select ‘Yes’ to continue.

Selecting the web server for PHPMyAdmin
Selecting the web server for PHPMyAdmin

You will need to enter a password for phpMyadmin to register with the database server;

enter the password for phpMyadmin to register with the database server
enter the password for phpMyadmin to register with the database server

And confirm it.

For the Nginx web server to find and serve the phpMyAdmin files correctly, we need to create a symbolic link from the phpMyAdmin installation files (in /usr/share/) to our NGINX web root directory (var/www/html/) by running following command:

Now we will install and enable the mcrypt PHP module, which phpMyAdmin relies on for encryption.

We should also disable the database from trying to authenticate root using plugin. Instead it will use the password that we set up earlier.

Run;

And then at the mysql prompt

Now we can go to our browser on a local machine and enter the IP address followed by /phpmyadmin (in the case of our example 10.1.1.17/phpmyadmin) it should start up phpMyAdmin in the browser.

phpMyAdmin Web
phpMyAdmin Web

If we enter the username as ‘root’ and the MySQL root password that we set earlier, it will open up the phpMyAdmin interface.

phpMyAdmin Web Interface
phpMyAdmin Web Interface

Create users for the database

Our database has an administrative (root) user, but for our future purposes, we will want to add a couple more users to manage the security of the database in a responsible way (using the root account to access the database from our python and PHP scripts would be possible, but irresponsible).

If we click on the ‘User accounts’ tab in phpMyAdmin we can see the range of users that are already set up.

phpMyAdmin Users
phpMyAdmin Users

We will create an additional two users. One that can only read (select) data from our database and another that can put data into (insert) the database. Keeping these functions separate gives us some flexibility in how we share the data in the future. For example we could be reasonably comfortable providing the username and password to allow reading the data to a wide range of people, but we should really only allow our future scripts the ability to insert data into the database.

From the ‘User accounts’ tab, select ‘Add a new user’;

phpMyAdmin Login Information
phpMyAdmin Login Information

Enter a user name and password.

Then we scroll down a little and select the type of access we want the pi_select user to have in the ‘Global privileges’ section. For the pi_select user under ‘Data’ we want to tick ‘SELECT’.

phpMyAdmin SELECT user
phpMyAdmin SELECT user

Then press the ‘Go’ button and we’ve created a user.

For the second user with the ability to insert data (let’s call the user ‘pi_insert’), go through the same process, but tick the ‘SELECT’ and the ‘INSERT’ options for the data.

When complete we should be able to see that both of our users look similar to the following;

phpMyAdmin INSERT user
phpMyAdmin INSERT user

Create a database

When we read data from our sensors, we will record them in a database. MariaDB is a database program, but we still need to set up a database inside that program. In fact while we will set up a database in this step, when we come to record and explore our data we will be dealing with a ‘table’ of data that will exist inside a database.

For the purposes of getting ourselves set up we need to create a database. If we go to the ‘Databases’ tab we can enter a name for a new database (here I’ve called one ‘measurements’) and click on ‘Create’.

phpMyAdmin New Database
phpMyAdmin New Database

Database preparation

Create the Database Table
Create the Database Table

Enter in the name of the table and the number of columns that we are going to use for our measured values. In the screen-shot above we can see that the name of the table is ‘temperature’ and the number of columns is ‘3’.

We will use three columns so that we can store a temperature reading, the time it was recorded and the unique ID of the sensor that recorded it.

Once we click on ‘Go’ we are presented with a list of options to configure our table’s columns. Don’t be intimidated by the number of options that are presented, we are going to keep the process as simple as practical.

For the first column we can enter the name of the ‘Column’ as ‘dtg’ (short for date time group) the ‘Type’ as ‘TIMESTAMP’ and the ‘Default’ value as ‘CURRENT_TIMESTAMP’. For the second column we will enter the name ‘temperature’ and the ‘Type’ is ‘FLOAT’ (we won’t use a default value). For the third column we will enter the name ‘sensor_id’ and the type is ‘VARCHAR’ with a ‘Length/Values’ of 15.

Configure the Database Table Columns
Configure the Database Table Columns

Scroll down a little and click on the ‘Save’ button and we’re done.

Save the Database Table Columns
Save the Database Table Columns

Multiple Temperature Measurements

This project will measure the temperature at multiple points using DS18B20 sensors. We will use the waterproof version of the sensors since they are more practical for external applications.

The DS18B20 Sensor

The DS18B20 is a ‘1-Wire’ digital temperature sensor manufactured by Maxim Integrated Products Inc. It provides a 9-bit to 12-bit precision, Celsius temperature measurement and incorporates an alarm function with user-programmable upper and lower trigger points.

Its temperature range is between -55C to 125C and they are accurate to +/- 0.5C between -10C and +85C.

It is called a ‘1-Wire’ device as it can operate over a single wire bus thanks to each sensor having a unique 64-bit serial code that can identify each device.

While the DS18B20 comes in a TO-92 package, it is also available in a waterproof, stainless steel package that is pre-wired and therefore slightly easier to use in conditions that require a degree of protection. The measurement project that we will undertake will use the waterproof version.

Single DS18B20 Sensor
Single DS18B20 Sensor

The sensors can come with a couple of different wire colour combinations. They will typically have a black wire that needs to be connected to ground. A red wire that should be connected to a voltage source (in our case a 3.3V pin from the Pi) and a blue or yellow wire that carries the signal.

The DS18B20 can be powered from the signal line, but in our project we will use an external voltage supply (from the Pi).

Measure

Hardware required

  • 3 x DS18B20 sensors (the waterproof version)
  • 10k Ohm resister
  • Jumper cables with Dupont connectors on the end
  • Solder
  • Heat-shrink

Connect

The DS18B20 sensors needs to be connected with the black wires to ground, the red wires to the 3V3 pin and the blue or yellow (some sensors have blue and some have yellow) wires to the GPIO4 pin. A resistor between the value of 4.7k Ohms to 10k Ohms needs to be connected between the 3V3 and GPIO4 pins to act as a ‘pull-up’ resistor.

The Raspbian Operating System image that we are using only supports GPIO4 as a 1-Wire pin, so we need to ensure that this is the pin that we use for connecting our temperature sensor.

The following diagram is a simplified view of the connection.

Triple DS18B20 Connection
Triple DS18B20 Connection

Connecting the sensor practically can be achieved in a number of ways. You could use a Pi Cobbler break out connector mounted on a bread board connected to the GPIO pins. But because the connection is relatively simple we could build a minimal configuration that will plug directly onto the appropriate GPIO pins using Dupont connectors. The resister is concealed under the heat-shrink and indicated with the arrow.

Minimal Triple DS18B20 Connection
Minimal Triple DS18B20 Connection

This version uses a recovered header connector from a computers internal USB cable.

Enable

We’re going to go back to the Raspberry Pi Software Configuration Tool as we need to enable the 1-wire option. This can be done by running the following command;

Select ‘Interfacing options`;

Interfacing Options
Interfacing Options

Then select the 1-wire option and enable it.

1-wire enabling
1-wire enabling

When you back out of the menu you will be asked to reboot the device. Do this and then log in again.

Test

From the terminal as the ‘pi’ user run the command;

modprobe w1-gpio registers the new sensors connected to GPIO4 so that now the Raspberry Pi knows that there is a 1-Wire device connected to the GPIO connector (For more information on the modprobe command check out the details here).

Then run the command;

modprobe w1-therm tells the Raspberry Pi to add the ability to measure temperature on the 1-Wire system.

To allow the w1_gpio and w1_therm modules to load automatically at boot we can edit the the /etc/modules file and include both modules there where they will be started when the Pi boots up. To do this edit the /etc/modules file;

Add in the w1_gpio and w1_therm modules so that the file looks like the following;

# /etc/modules: kernel modules to load at boot time.
#
# This file contains the names of kernel modules that should be loaded
# at boot time, one per line. Lines beginning with "#" are ignored.

w1-gpio
w1-therm

Save the file.

Then we change into the /sys/bus/w1/devices directory and list the contents using the following commands;

(For more information on the cd command check out the reference here. Or to find out more about the ls command go here)

This should list out the contents of the /sys/bus/w1/devices which should include a number of directories starting 28-. The number of directories should match the number of connected sensors. The portion of the name following the 28- is the unique serial number of each of the sensors.

We then change into one of those directories;

We are then going to view the ‘w1_slave’ file with the cat command using;

The output should look something like the following;

73 01 4b 46 7f ff 0d 10 41 : crc=41 YES
73 01 4b 46 7f ff 0d 10 41 t=23187

At the end of the first line we see a YES for a successful CRC check (CRC stands for Cyclic Redundancy Check, a good sign that things are going well). If we get a response like NO or FALSE or ERROR, it will be an indication that there is some kind of problem that needs addressing. Check the circuit connections and start troubleshooting.

At the end of the second line we can now find the current temperature. The t=23187 is an indication that the temperature is 23.187 degrees Celsius (we need to divide the reported value by 1000).

cd into each of the 28-xxxx directories in turn and run the cat w1_slave command to check that each is operating correctly. It may be useful at this stage to label the individual sensors with their unique serial numbers to make it easy to identify them correctly later.

Record

To record this data we will use a Python program that checks all the sensors and writes the temperature and the sensor name into our database. At the same time a time stamp will be added automatically.

Our Python program will execute the program at a regular interval using cron which we used earlier to automatically reconnect to the network if required.

Record the temperature values

The following Python code (which is based on the code that is part of the great temperature sensing tutorial on iot-project) is a script which allows us to check the temperature reading from multiple sensors and write them to our database with a separate entry for each sensor.

The full code can be found in the code samples bundled with this book (m_temp.py).

#!/usr/bin/python
# -*- coding: utf-8 -*-

import os
import fnmatch
import time
import MySQLdb as mdb
import logging

logging.basicConfig(filename='/home/pi/DS18B20_error.log',
  level=logging.DEBUG,
  format='%(asctime)s %(levelname)s %(name)s %(message)s')
logger=logging.getLogger(__name__)

# Load the modules (not required if they are loaded at boot) 
# os.system('modprobe w1-gpio')
# os.system('modprobe w1-therm')

# Function for storing readings into MySQL
def insertDB(IDs, temperature):

  try:

    con = mdb.connect('localhost',
                      'pi_insert',
                      'xxxxxxxxxx',
                      'measurements');
    cursor = con.cursor()

    for i in range(0,len(temperature)):
      sql = "INSERT INTO temperature(temperature, sensor_id) \
      VALUES ('%s', '%s')" % \
      ( temperature[i], IDs[i])
      cursor.execute(sql)
      sql = []
      con.commit()

    con.close()

  except mdb.Error, e:
    logger.error(e)

# Get readings from sensors and store them in MySQL

temperature = []
IDs = []

for filename in os.listdir("/sys/bus/w1/devices"):
  if fnmatch.fnmatch(filename, '28-*'):
    with open("/sys/bus/w1/devices/" + filename + "/w1_slave") as f_obj:
      lines = f_obj.readlines()
      if lines[0].find("YES"):
        pok = lines[1].find('=')
        temperature.append(float(lines[1][pok+1:pok+6])/1000)
        IDs.append(filename)
      else:
        logger.error("Error reading sensor with ID: %s" % (filename))

if (len(temperature)>0):
  insertDB(IDs, temperature)

This script can be saved in our home directory (/home/pi) and can be run by typing;

While we won’t see much happening at the command line, if we use our web browser to go to the phpMyAdmin interface and select the ‘measurements’ database and then the ‘temperature’ table we will see a range of temperature measurements for the different sensors and their associated time of reading.

Now you can be forgiven for thinking that this is not going to collect the sort of range of data that will let us ‘Explore’ very much, but let’s do a quick explanation of the Python code first and then we’ll work out how to record a lot more data :-).

Save the MySQL Table Columns
Save the MySQL Table Columns
Code Explanation

The script starts by importing the modules that it’s going to use for the process of reading and recording the temperature measurements;

import os
import fnmatch
import time
import MySQLdb as mdb
import logging

Then the code sets up the logging module. We are going to use the basicConfig() function to set up the default handler so that any debug messages are written to the file /home/pi/DS18B20_error.log.

logging.basicConfig(filename='/home/pi/DS18B20_error.log',
  level=logging.DEBUG,
  format='%(asctime)s %(levelname)s %(name)s %(message)s')
logger=logging.getLogger(__name__)

Later in the section where we are getting our readings or writing to the database we write to the log file if there is an error.

The program can then issue the modprobe commands that start the interface to the sensor;

# os.system('modprobe w1-gpio')
# os.system('modprobe w1-therm'))

Our code has them commented out since we have already configured the /etc/modules file using raspi-config, but if you didn’t want to start the modules at start-up (for whatever reasons), you can un-comment these.

We then declare the function that will insert the readings into the database;

def insertDB(IDs, temperature):

  try:

    con = mdb.connect('localhost',
                      'pi_insert',
                      'xxxxxxxxxx',
                      'measurements');
    cursor = con.cursor()

    for i in range(0,len(temperature)):
      sql = "INSERT INTO temperature(temperature, sensor_id) \
      VALUES ('%s', '%s')" % \
      ( temperature[i], IDs[i])
      cursor.execute(sql)
      sql = []
      con.commit()

    con.close()

  except mdb.Error, e:
    logger.error(e)

This is a neat piece of script that uses arrays to recall all the possible temperature and ID values.

Then we have the main body of the script that finds all our possible sensors and reads the IDs and the temperatures;

temperature = []
IDs = []

for filename in os.listdir("/sys/bus/w1/devices"):
  if fnmatch.fnmatch(filename, '28-*'):
    with open("/sys/bus/w1/devices/" + filename + "/w1_slave") as f_obj:
      lines = f_obj.readlines()
      if lines[0].find("YES"):
        pok = lines[1].find('=')
        temperature.append(float(lines[1][pok+1:pok+6])/1000)
        IDs.append(filename)
      else:
        logger.error("Error reading sensor with ID: %s" % (filename))

if (len(temperature)>0):
  insertDB(IDs, temperature)

After declaring our two arrays temperature and IDs we start the for loop that checks all the file names in /sys/bus/w1/devices;

for filename in os.listdir("/sys/bus/w1/devices"):

If it finds a filename that starts with 28- then it processes it;

  if fnmatch.fnmatch(filename, '28-*'):

First it opens the w1_slave file in the 28-* directory…

    with open("/sys/bus/w1/devices/" + filename + "/w1_slave") as f_obj:

… then it pulls out the lines in the file;

      lines = f_obj.readlines()

If it finds the word “YES” in the first line (line 0) of the file…

      if lines[0].find("YES"):

…then it uses the position of the equals (=) sign in the second line (line 1)…

        pok = lines[1].find('=')

… to pull out the characters following the ‘=’, and manipulate them to form the temperature in
degrees Centigrade;

        temperature.append(float(lines[1][pok+1:pok+6])/1000)

We then add the filename to the IDs array;

        IDs.append(filename)

If we didn’t find a “yes” in the first line we log the error in the log file

        logger.error("Error reading sensor with ID: %s" % (filename))

Then finally if we have been successful in reading at least one temperature value, we push the IDs and temperature array to the insertDB function;

if (len(temperature)>0):
  insertDB(IDs, temperature)

Recording data on a regular basis with cron

As mentioned earlier, while our code is a thing of beauty, it only records a single entry for each sensor every time it is run.

What we need to implement is a schedule so that at a regular time, the program is run. This is achieved using cron via the crontab.

To set up our schedule we need to edit the crontab file. This is is done using the following command;

Once run it will open the crontab in the nano editor. We want to add in an entry at the end of the file that looks like the following;

*/1 * * * * /usr/bin/python /home/pi/m_temp.py

This instructs the computer that every minute of every hour of every day of every month we run the command /usr/bin/python /home/pi/m_temp.py (which, if we were at the command line in the pi home directory, we would run as python m_temp.py, but since we can’t guarantee where we will be when running the script, we are supplying the full path to the python command and the m_temp.py script.

Save the file and when the next minute rolls over our program will run on its designated schedule and we will have sensor entries written to our database every minute. Go ahead, check it out by refreshing the temperature table.

Explore

This section has a working solution for presenting multiple streams of temperature data. This is a slightly complex use of JavaScript and specifically the JavaScript library d3.js.

The final form of the graph should look something like the following (depending on the number of sensors you are using, and the amount of data you have collected)

Multiple Line Graphs of Temperature
Multiple Line Graphs of Temperature

One of the neat things about this graph is that it ‘builds itself’ in the sense that aside from us deciding what we want to label the specific temperature streams as, the code will organise all the colours and labels for us. Likewise, if the display is getting a bit messy we can click on the legend labels to show / hide the corresponding line.

The Code

The following code is a PHP file that we can place on our Raspberry Pi’s web server (in the /var/www/html directory) that will allow us to view all of the results that have been recorded in the temperature directory on a graph. When we want to see the graph we simply go (in our browser’s URL bar) to the IP address of the Pi followed by the name of our php file (‘m_temp.php’ is used below). This would look a little like ‘10.1.1.17/m_temp.php’.

The full code can be found in the code samples bundled with this book (m_temp.php).

<?php

$hostname = 'localhost';
$username = 'pi_select';
$password = 'xxxxxxxxxx';

try {
    $dbh = new PDO("mysql:host=$hostname;dbname=measurements",
                               $username, $password);

    /*** The SQL SELECT statement ***/

    $sth = $dbh->prepare("
       SELECT `temperature`, LEFT(`dtg`,19) AS dtg, sensor_id
       FROM `temperature` 
       WHERE `dtg` > (NOW() - INTERVAL 36 HOUR)
       GROUP BY `sensor_id`,`dtg`
       ORDER BY `temperature`.`dtg` DESC
    ");
    $sth->execute();

    /* Fetch all of the remaining rows in the result set */
    $result = $sth->fetchAll(PDO::FETCH_ASSOC);

    /*** close the database connection ***/
    $dbh = null;
    
}
catch(PDOException $e)
    {
        echo $e->getMessage();
    }

$json_data = json_encode($result);     

//echo $json_data;

?>

<!DOCTYPE html>
<meta charset="utf-8">
<style> /* set the CSS */

body { font: 12px Arial;}

path { 
    stroke: steelblue;
    stroke-width: 2;
    fill: none;
}

.axis path,
.axis line {
    fill: none;
    stroke: grey;
    stroke-width: 1;
    shape-rendering: crispEdges;
}

.legend {
    font-size: 16px;
    font-weight: bold;
    text-anchor: middle;
}

</style>
<body>

<!-- load the d3.js library -->    
<script src="http://d3js.org/d3.v3.min.js"></script>

<script>

// Set the dimensions of the canvas / graph
var margin = {top: 30, right: 20, bottom: 70, left: 50},
    width = 900 - margin.left - margin.right,
    height = 300 - margin.top - margin.bottom;

// Parse the date / time
var parseDate = d3.time.format("%Y-%m-%d %H:%M:%S").parse;

// Set the ranges
var x = d3.time.scale().range([0, width]);
var y = d3.scale.linear().range([height, 0]);

// Define the axes
var xAxis = d3.svg.axis().scale(x)
    .orient("bottom");

var yAxis = d3.svg.axis().scale(y)
    .orient("left").ticks(5);

// Define the line
var temperatureline = d3.svg.line()	
    .x(function(d) { return x(d.date); })
    .y(function(d) { return y(d.temperature); });
    
// Adds the svg canvas
var svg = d3.select("body")
    .append("svg")
        .attr("width", width + margin.left + margin.right)
        .attr("height", height + margin.top + margin.bottom)
    .append("g")
        .attr("transform", 
              "translate(" + margin.left + "," + margin.top + ")");

// Get the data
<?php echo "data=".$json_data.";" ?>

data.forEach(function(d) {
	d.date = parseDate(d.dtg);
	d.temperature = +d.temperature;
});

// Scale the range of the data
x.domain(d3.extent(data, function(d) { return d.date; }));
y.domain([0, d3.max(data, function(d) { return d.temperature; })]);

// Nest the entries by sensor_id
var dataNest = d3.nest()
	.key(function(d) {return d.sensor_id;})
	.entries(data);

var color = d3.scale.category10();   // set the colour scale

legendSpace = width/dataNest.length; // spacing for the legend

// Loop through each sensor_id / key
dataNest.forEach(function(d,i) { 

	svg.append("path")
		.attr("class", "line")
		.style("stroke", function() { // Add the colours dynamically
			return d.color = color(d.key); })
		.attr("id", 'tag'+d.key.replace(/\s+/g, '')) // assign ID
		.attr("d", temperatureline(d.values));

	// Add the Legend
	svg.append("text")
		.attr("x", (legendSpace/2)+i*legendSpace)  // space legend
		.attr("y", height + (margin.bottom/2)+ 5)
		.attr("class", "legend")    // style the legend
		.style("fill", function() { // Add the colours dynamically
			return d.color = color(d.key); })
		.on("click", function(){
			// Determine if current line is visible 
			var active   = d.active ? false : true,
			newOpacity = active ? 0 : 1; 
			// Hide or show the elements based on the ID
			d3.select("#tag"+d.key.replace(/\s+/g, ''))
				.transition().duration(100) 
				.style("opacity", newOpacity); 
			// Update whether or not the elements are active
			d.active = active;
			})  
		.text(
		    function() {
		        if (d.key == '28-00043b6ef8ff') {return "Inlet";}
		        if (d.key == '28-00043e9049ff') {return "Ambient";}
		        if (d.key == '28-00043e8defff') {return "Outlet";}
		        else {return d.key;}
		        }); 
});

// Add the X Axis
svg.append("g")
	.attr("class", "x axis")
	.attr("transform", "translate(0," + height + ")")
	.call(xAxis);

// Add the Y Axis
svg.append("g")
	.attr("class", "y axis")
	.call(yAxis);

</script>
</body>

The graph that will look a little like this (except the data will be different of course).

Multiple Temperature Line Graph
Multiple Temperature Line Graph

This is a fairly basic graph (i.e, there is no title or labelling of axis). It will automatically try to collect 36 hours of measurements. So if (as is the case here) we have three sensors, this will result in three lines, each of which will displayed a day and a half’s data.

It does include some cool things though.

  • It will automatically include as many lines as we have data for. So if we have 7 sensors, there will be 7 lines.
  • Currently the graph is showing the three lines in the legend as ‘Outlet’, ‘Inlet’ and ‘Ambient’. This is because our code specifically assigns a name to a sensor ID. But, if we do not assign a specific label it will automagically use the sensor ID as the label.
  • The colours for the lines and the legend will automatically be set as nicely distinct colours.
  • We can click on a legend label and it will turn on / off the corresponding line to make it easier to read.
'Inlet' and 'Ambient' De-selected
‘Inlet’ and ‘Ambient’ De-selected
PHP

The <?php line at the start and the ?> line at the end form the wrappers that allow the requesting page to recognise the contents as PHP and to execute the code rather than downloading it for display.

The following lines set up a range of important variables;

$hostname = 'localhost';
$username = 'pi_select';
$password = 'xxxxxxxxxx';

Hopefully you will recognise that these are the configuration details for the database that we set up. There’s the user and the password (the script isn’t returned to the browser, so the browser doesn’t get to see the password and in this case our user has a very limited set of privileges). There’s the host address that contains our database (in this case it’s local (on the same server), so we use localhost, but if the database was on a remote server, we would just include its address (10.1.1.17 in our example)) and there’s the database we’re going to access.

Then we use those variables to define how we will connect to the server and the database…

    $dbh = new PDO("mysql:host=$hostname;dbname=measurements", 
                               $username, $password);

… then we prepare our query that will request the data from the database table…

    $sth = $dbh->prepare("
       SELECT `temperature`, LEFT(`dtg`,19) AS dtg, sensor_id
       FROM `temperature` 
       WHERE `dtg` > (NOW() - INTERVAL 36 HOUR)
       GROUP BY `sensor_id`,`dtg`
       ORDER BY `temperature`.`dtg` DESC
    ");

(This particular query is telling the database to SELECT the temperature column and only the left most 19 characters from our date/time data (from the dtg column) and the sensor ID names (from the sensor_id column) FROM the table temperature. All thatis is done WHERE the date / time (dtg) is more recent than 36 hours ago.)

… and execute the query.

    $sth->execute();

We fetch all the returned data and place it in an associative array.

    $result = $sth->fetchAll(PDO::FETCH_ASSOC);

Then we close the connection to the database.

    $dbh = null;

…and then we check to see if it was successful. If it wasn’t, we output the exception message with our catch section;

catch(PDOException $e)
    {
        echo $e->getMessage();
    }

We then encode the contents of our array in a the JSON format.

$json_data = json_encode($result); 

Whew!

HTML

This stands for HyperText Markup Language and is the stuff that web pages are made of. Check out the definition and other information on Wikipedia for a great overview. Just remember that all we’re going to use HTML for is to hold the code that we will use to present our information.

The HTML code we are going to examine also includes Cascading Style Sheets (everyone appears to call them ‘Style Sheets’ or ‘CSS’) which are a language used to describe the formatting (or “look and feel”) of a document written in a markup language (HTML in this case). The job of CSS is to make the presentation of the components we will draw with D3 simpler by assigning specific styles to specific objects. One of the cool things about CSS is that it is an enormously flexible and efficient method for making everything on the screen look more consistent and when you want to change the format of something you can just change the CSS component and the whole look and feel of your graphics will change.

Here’s the HTML portions of the code;

<!DOCTYPE html>
<meta charset="utf-8">
<style>

    The CSS is in here

</style>
<body>
<script src="http://d3js.org/d3.v3.min.js"></script>

<script>

    The D3 JavaScript code is here

</script>
</body>

Compare it with the full code. It kind of looks like a wrapper for the CSS and JavaScript. You can see that it really doesn’t boil down to much at all (that doesn’t mean it’s not important).

There are plenty of good options for adding additional HTML stuff into this very basic part for the file, but for what we’re going to be doing, we really don’t need to bother too much.

One thing probably worth mentioning is the line;

<script src="http://d3js.org/d3.v3.min.js"></script>

That’s the line that identifies the file that needs to be loaded to get D3 up and running. In this case the file is sourced from the official d3.js repository on the internet (that way we are using the most up to date version). The D3 file is actually called d3.v3.min.js which may come as a bit of a surprise. That tells us that this is version 3 of the d3.js file (the v3 part) which is an indication that it is separate from the v2 release, which was superseded in late 2012. The other point to note is that this version of d3.js is the minimised version (hence min). This means that any extraneous information has been removed from the file to make it quicker to load.

The two parts that we left out are the CSS and the D3 JavaScript.

CSS

The CSS is as follows;

body { font: 12px Arial;}

path {
    stroke: steelblue;
    stroke-width: 2;
    fill: none;
}

.axis path,
.axis line {
    fill: none;
    stroke: grey;
    stroke-width: 1;
    shape-rendering: crispEdges;
}

.legend {
    font-size: 16px;
    font-weight: bold;
    text-anchor: middle;
}

Cascading Style Sheets give you control over the look / feel / presentation of web content. The idea is to define a set of properties to objects in the web page.

They are made up of ‘rules’. Each rule has a ‘selector’ and a ‘declaration’ and each declaration has a property and a value (or a group of properties and values).

For instance in the example code for this web page we have the following rule;

body { font: 12px Arial;}

body is the selector. This tells you that on the web page, this rule will apply to the ‘body’ of the page. This actually applies to all the portions of the web page that are contained in the ‘body’ portion of the HTML code (everything between <body> and </body> in the HTML bit). { font: 12px Arial;} is the declaration portion of the rule. It only has the one declaration which is the bit that is in between the curly braces. So font: 12px Arial; is the declaration. The property is font: and the value is 12px Arial;. This tells the web page that the font that appears in the body of the web page will be in 12 px Arial.

What about the bit that’s like;

path {
    stroke: steelblue;
    stroke-width: 2;
    fill: none;
}

Well, the whole thing is one rule, ‘path’ is the selector. In this case, ‘path’ is referring to a line in the D3 drawing nomenclature.

For that selector there are three declarations. They give values for the properties of ‘stroke’ (in this case colour), ‘stroke-width’ (the width of the line) and ‘fill’ (we can fill a path with a block of colour).

JavaScript and d3.js

JavaScript is what’s called a ‘scripting language’. It is the code that will be contained inside the HTML file that will make D3 do all its fanciness. In fact, D3 is a JavaScript Library. JavaScript is the language D3 is written in.

Knowing a little bit about this would be really good, but to be perfectly honest, I didn’t know anything about it before I started writing a book. With a bit of trial and error, you can figure out what’s going on.

There’s quite a bit of detail in the code, but it’s not so long that we can’t work out what’s doing what.

Let’s examine the blocks bit by bit to get a feel for it.

Setting up the margins and the graph area.

The part of the code responsible for defining the image area (or the area where the graph and associated bits and pieces is placed ) is this part.

var margin = {top: 30, right: 20, bottom: 70, left: 50},
    width = 900 - margin.left - margin.right,
    height = 300 - margin.top - margin.bottom;

This is really (really) well explained on Mike Bostock’s page on margin conventions here http://bl.ocks.org/3019563, but at the risk of confusing you here’s my crude take on it.

The first line defines the four margins which surround the block where the graph (as an object) is positioned.

var margin = {top: 30, right: 20, bottom: 70, left: 50},

So there will be a border of 30 pixels at the top, 20 at the right and 70 and 50 at the bottom and left respectively. Now the cool thing about how these are set up is that they use an array to define everything. That means if you want to do calculations in the JavaScript later, you don’t need to put the numbers in, you just use the variable that has been set up. In this case margin.right = 20!

So when we go to the next line;

    width = 900 - margin.left - margin.right,

the width of the inner block of the canvas where the graph will be drawn is 900 pixels – margin.left – margin.right or 900-50-20 or 830 pixels wide. Of course now you have another variable ‘width’ that we can use later in the code.

Obviously the same treatment is given to height.

Another cool thing about all of this is that just because you appear to have defined separate areas for the graph and the margins, the whole area in there is available for use. It just makes it really useful to have areas designated for the axis labels and graph labels without having to juggle them and the graph proper at the same time.

That is the really cool part of this whole business. D3 is running in the background looking after the drawing of the objects, while you get to concentrate on how the data looks without too much maths!

Getting the Data

We’re going to jump forward a little bit here to the bit of the JavaScript code that loads the data for the graph.

I’m going to go out of the sequence of the code, because if you know what the data is that you’re using, it will make explaining some of the other functions that are coming up much easier.

The section that grabs the data is this bit.

<?php echo "data=".$json_data.";" ?>

data.forEach(function(d) {
	d.date = parseDate(d.dtg);
	d.temperature = +d.temperature;
});

There’s lots of different ways that we can get data into our web page to turn into graphics. And the method that you’ll want to use will probably depend more on the format that the data is in than the mechanism you want to use for importing.

In our case we actually use our old friend PHP to declare it. The line…

<?php echo "data=".$json_data.";" ?>

…uses PHP to inject a line of code into our JavaScript that declares the data variable and its values. This is actually a fairly pivotal moment in the understanding of how PHP on a web page works. On the server (our Raspberry Pi) the script contains the line;

<?php echo "data=".$json_data.";" ?>

But, when a client (the web browser) accesses the script, that line is executed as code and the information that is pushed to the client is

data=[{"dtg":"2014-12-13 18:08:08","temperature":"22"},...}];

The data= portion is ‘echo’ed to the script directly and the dtg and temperature information is from the variable that we declared earlier in our initial large PHP block when we queried our database.

After declaring what our data is, we need to do a little housekeeping to make sure that the values are suitable for the script.

data.forEach(function(d) {
	d.date = parseDate(d.dtg);
	d.temperature = +d.temperature;
});

This block of code simply ensures that all the numeric values that are pulled out of the data are set and formatted correctly. The first line sets the data variable that is being dealt with (called slightly confusingly ‘data’) and tells the block of code that, for each group within the ‘data’ array it should carry out a function on it. That function is designated ‘d’.

data.forEach(function(d) {

The information in the array can be considered as being stored in rows. Each row consists of three values: one value for ‘dtg’ another value for ‘temperature’ and the final one for the sensor.

The function is pulling out values of ‘dtg’ and ‘temperature’ one row at a time.

Each time it gets a value of ‘dtg’ and ‘temperature’ it carries out the following operations;

	d.date = parseDate(d.dtg);

For this specific value of date/time being looked at (d.date), d3.js changes it into a date/time format that is processed via a separate function ‘parseDate’. (The ‘parseDate’ function is defined in a separate part of the script, and we will examine that later.) For the moment, be satisfied that it takes the raw date information from the data in a specific row and converts it into a format that D3 can then process. That value is then re-saved in the same variable space.

The next line then sets the ‘temperature’ variable to a numeric value (if it isn’t already) using the ‘+’ operator.

	d.temperature = +d.temperature;

So, at the end of that section of code, we have gone out and picked up our data and ensured that it is formatted in a way that the rest of the script can use correctly.

But anyway, let’s get back to figuring what the code is doing by jumping back to the end of the margins block.

Formatting the Date / Time.

One of the glorious things about the World is that we all do things a bit differently. One of those things is how we refer to dates and time.

In my neck of the woods, it’s customary to write the date as day - month – year. E.g 23-12-2012. But in the United States the more common format would be 12-23-2012. Likewise, the data may be in formats that name the months or weekdays (E.g. January, Tuesday) or combine dates and time together (E.g. 2012-12-23 15:45:32). So, if we were to attempt to try to load in some data and to try and get D3 to recognise it as date / time information, we really need to tell it what format the date / time is in.

The line in the JavaScript that parses the time is the following;

var parseDate = d3.time.format("%Y-%m-%d %H:%M:%S").parse;

This line is used when the data.forEach(function(d) portion of the code (that we looked at a couple of pages back) used d.dtg = parseDate(d.dtg); as a way to take a date/time in a specific format and to get it recognised by D3. In effect it said “take this value that is supposedly a date and make it into a value I can work with”.

The function used is the d3.time.format(specifier) function where the specifier in this case is the mysterious combination of characters %Y-%m-%d %H:%M:%S. The good news is that these are just a combination of directives specific for the type of date we are presenting.

The % signs are used as prefixes to each separate format type and the ‘-’ (minus) signs are literals for the actual ‘-’ (minus) signs that appear in the date to be parsed.

The Y refers to the year with century as a decimal number.

The m refers to the month as a decimal number [01,12].

The d refers to a zero-padded day of the month as a decimal number [01,31].

The H refers to the hour (24-hour clock) as a decimal number [00,23].

The M refers to the number of minutes as a decimal number [00,59].

The S refers to seconds as a decimal number [00,61].

And the y refers to the year (without the centuries) as a decimal number.

If we look at a subset of the data from our database we see that indeed, the dates therein are formatted in this way.

That’s all well and good, but what if your data isn’t formatted exactly like that?

Good news. There are multiple different formatters for different ways of telling time and you get to pick and choose which one you want. Check out the Time Formatting page on the D3 Wiki for a the authoritative list and some great detail, but the following is the list of currently available formatters (from the d3 wiki);

  • %a - abbreviated weekday name.
  • %A - full weekday name.
  • %b - abbreviated month name.
  • %B - full month name.
  • %c - date and time, as “%a %b %e %H:%M:%S %Y”.
  • %d - zero-padded day of the month as a decimal number [01,31].
  • %e - space-padded day of the month as a decimal number [ 1,31].
  • %H - hour (24-hour clock) as a decimal number [00,23].
  • %I - hour (12-hour clock) as a decimal number [01,12].
  • %j - day of the year as a decimal number [001,366].
  • %m - month as a decimal number [01,12].
  • %M - minute as a decimal number [00,59].
  • %p - either AM or PM.
  • %S - second as a decimal number [00,61].
  • %U - week number of the year (Sunday as the first day of the week) as a decimal number [00,53].
  • %w - weekday as a decimal number [0(Sunday),6].
  • %W - week number of the year (Monday as the first day of the week) as a decimal number [00,53].
  • %x - date, as “%m/%d/%y”.
  • %X - time, as “%H:%M:%S”.
  • %y - year without century as a decimal number [00,99].
  • %Y - year with century as a decimal number.
  • %Z - time zone offset, such as “-0700”.
  • There is also a a literal “%” character that can be presented by using double % signs.
Setting Scales Domains and Ranges

This is another example where, if you set it up right, D3 will look after you forever.

From our basic web page we have now moved to the section that includes the following lines;

var x = d3.time.scale().range([0, width]);
var y = d3.scale.linear().range([height, 0]);

The purpose of these portions of the script is to ensure that the data we ingest fits onto our graph correctly. Since we have two different types of data (date/time and numeric values) they need to be treated separately (but they do essentially the same job). To examine this whole concept of scales, domains and ranges properly, we will also move slightly out of sequence and (in conjunction with the earlier scale statements) take a look at the lines of script that occur later and set the domain. They are as follows;

x.domain(d3.extent(data, function(d) { return d.date; }));
y.domain([0, d3.max(data, function(d) { return d.temperature; })]);

The idea of scaling is to take the values of data that we have and to fit them into the space we have available.

First we make sure that any quantity we specify on the x axis fits onto our graph.

var x = d3.time.scale().range([0, width]);

Here we set our variable that will tell D3 where to draw something on the x axis. By using the d3.time.scale() function we make sure that D3 knows to treat the values as date / time entities (with all their ingrained peculiarities). Then we specify the range that those values will cover (.range) and we specify the range as being from 0 to the width of our graphing area (See! Setting those variables for margins and widths are starting to pay off now!).

Then we do the same for the Y axis.

var y = d3.scale.linear().range([height, 0]);

There’s a different function call (d3.scale.linear()) but the .range setting is still there.

Now hang on, what’s going on with the [height, 0] part in y axis scale statement? The astute amongst you will note that for the time scale we set the range as [0, width] but for this one ([height, 0]) the values look backwards.

Well spotted.

This is all to do with how the screen is laid out and referenced. Take a look at the following diagram showing how the coordinates for drawing on our screen work;

Coordinates that the browser expects
Coordinates that the browser expects

The top left hand of the screen is the origin or 0,0 point and as we go left or down the corresponding x and y values increase to the full values defined by height and width.

That’s good enough for the time values on the x axis that will start at lower values and increase, but for the values on the y axis we’re trying to go against the flow. We want the low values to be at the bottom and the high values to be at the top.

No problem. We just tell D3 via the statement y = d3.scale.linear().range([height, 0]); that the larger values (height) are at the low end of the screen (at the top) and the low values are at the bottom (as you most probably will have guessed by this stage, the .range statement uses the format .range([closer_to_the_origin, further_from_the_origin]). So when we put the height variable first, that is now associated at the top of the screen.

Coordinates with adjusted ranges
Coordinates with adjusted ranges

We’ve scaled our data to the graph size and ensured that the range of values is set appropriately. What’s with the domain part that was in this section’s title?

Come on, you remember this little piece of script don’t you?

x.domain(d3.extent(data, function(d) { return d.date; }));
y.domain([0, d3.max(data, function(d) { return d.temperature; })]);

While it exists in a separate part of the file from the scale / range part, it is certainly linked.

That’s because there’s something missing from what we have been describing so far with the set up of the data ranges for the graphs. We haven’t actually told D3 what the range of the data is.

So, the .domain function is designed to let D3 know what the scope of the data will be. This is what is then passed to the scale function.

Looking at the first part that is setting up the x axis values, it is saying that the domain for the x axis values will be determined by the d3.extent function which in turn is acting on a separate function which looks through all the ‘date’ values that occur in the ‘data’ array. In this case the .extent function returns the minimum and maximum value in the given array.

  • function(d) { return d.date; } returns all the ‘date’ values in ‘data’. This is then passed to…
  • The .extent function that finds the maximum and minimum values in the array and then…
  • The .domain function which returns those maximum and minimum values to D3 as the range for the x axis.

Pretty neat really. At first you might think it was overly complex, but breaking the function down into these components allows additional functionality with differing scales, values and quantities. In short, don’t sweat it. It’s a good thing.

Setting up the Axes

Now we come to our next piece of code;

var xAxis = d3.svg.axis().scale(x)
    .orient("bottom");

var yAxis = d3.svg.axis().scale(y)
    .orient("left").ticks(5);

I’ve included both the x and y axes because they carry out the formatting in very similar ways. It’s worth noting that this is not the point where the axes get drawn. That occurs later in the script.

D3 has it’s own axis component that aims to take the fuss out of setting up and displaying the axes. So it includes a number of configurable options.

Looking first at the x axis;

var xAxis = d3.svg.axis().scale(x)
    .orient("bottom");

The axis function is called with d3.svg.axis(). Then the scale is set using the x values that we set up in the scales, ranges and domains section using .scale(x). Then we tell the graph to orientate itself to the bottom of the graph .orient("bottom").

The code that formats the y axis is pretty similar;

var yAxis = d3.svg.axis().scale(y)
    .orient("left").ticks(5);

The only significant difference is that we define the number of ticks we would like to have on that axis (for the x axis we just let D3 pick an appropriate number)

Adding data to the line function

We’re getting towards the end of our journey through the script now. The next step is to get the information from the array ‘data’ and to place it in a new array that consists of a set of coordinates that we are going to plot.

var temperatureline = d3.svg.line()	
    .x(function(d) { return x(d.date); })
    .y(function(d) { return y(d.temperature); });

I’m aware that the statement above may be somewhat ambiguous. You would be justified in thinking that we already had the data stored and ready to go. But that’s not strictly correct.

What we have is data in a raw format, we have added pieces of code that will allow the data to be adjusted for scale and range to fit in the area that we want to draw, but we haven’t actually taken our raw data and adjusted it for our desired coordinates. That’s what the code above does.

The main function that gets used here is the d3.svg.line() function. This function uses accessor functions to store the appropriate information in the right area and in the case above they use the x and y accessors (that would be the bits that are .x and .y). The d3.svg.line() function is called a ‘path generator’ and this is an indication that it can carry out some pretty clever things on its own accord. But in essence its job is to assign a set of coordinates in a form that can be used to draw a line.

Each time this line function is called on, it will go through the data and will assign coordinates to ‘dtg’ and ‘temperature’ pairs using the ‘x’ and ‘y’ functions that we set up earlier (which of course are responsible for scaling and setting the correct range / domain).

Of course, it doesn’t get the data all by itself, we still need to actually call the temperatureline function with ‘data’ as the source to act on. But never fear, that’s coming up soon.

Adding the SVG area.

As the title states, the next piece of script forms and adds the space that D3 will then use to draw on.

var svg = d3.select("body")
    .append("svg")
        .attr("width", width + margin.left + margin.right)
        .attr("height", height + margin.top + margin.bottom)
    .append("g")
        .attr("transform",
              "translate(" + margin.left + "," + margin.top + ")");

So what exactly does that all mean?

Well D3 needs to be able to have a space defined for it to draw things. When you define the space it’s going to use, you can also give it an identifying name and attributes.

In the example we’re using here, we are ‘appending’ an SVG element (an element that we are going to draw things on) to the <body> element of the HTML page.

We also add an element ‘g’ that is referenced to the top left corner of the actual graph area on the canvas. ‘g’ is actually a grouping element in the sense that it is normally used for grouping together several related elements. So in this case those grouped elements will have a common reference.

Interesting things to note about the code. The .attr("stuff in here") parts are attributes of the appended elements they are part of.

For instance;

    .append("svg")
        .attr("width", width + margin.left + margin.right)
        .attr("height", height + margin.top + margin.bottom)

tells us that the ‘svg’ element has a “width” of width + margin.left + margin.right and the “height” of height + margin.top + margin.bottom.

Likewise…

    .append("g")
        .attr("transform",
              "translate(" + margin.left + "," + margin.top + ")");

tells us that the element “g” has been transformed by moving(translating) to the point margin.left, margin.top. Or to the top left of the graph space proper. This way when we tell something to be drawn on our canvas, we can use the reference point “g” to make sure everything is in the right place.

Data Wrangling!

The PHP block at the start of the code selects three columns of information. The temperature, the date-time-group and the sensor ID;

temperature date and sensor_id
temperature date and sensor_id

In this project, we are going to need to ‘pivot’ the data that we are retrieving from our database so that it is produced in a multi-column format that the script can deal with easily. This is not always easy in programming, but it can be achieved using the d3 nest function which we will examine. Ultimately we want to be able to use the data in a format that looks a little like this;

Pivoted sensor, temperature readings
Pivoted sensor, temperature readings

We can see that the information is still the same, but there has been a degree of redundancy removed.

Nesting the data

The following code nest’s the data

var dataNest = d3.nest()
	.key(function(d) {return d.sensor_id;})
	.entries(data);

We declare our new array’s name as dataNest and we initiate the nest function;

var dataNest = d3.nest()

We assign the key for our new array as sensor_id. A ‘key’ is like a way of saying “This is the thing we will be grouping on”. In other words our resultant array will have a single entry for each unique sensor_id which will itself be an array of dates and values.

	.key(function(d) {return d.sensor_id;})

Then we tell the nest function which data array we will be using for our source of data.

		}).entries(data);

Then we use the nested data to loop through our sensor IDs and draw the lines and the legend labels;

dataNest.forEach(function(d,i) { 

	svg.append("path")
		.attr("class", "line")
		.style("stroke", function() { // Add the colours dynamically
			return d.color = color(d.key); })
		.attr("id", 'tag'+d.key.replace(/\s+/g, '')) // assign ID
		.attr("d", temperatureline(d.values));

	// Add the Legend
	svg.append("text")
		.attr("x", (legendSpace/2)+i*legendSpace)  // space legend
		.attr("y", height + (margin.bottom/2)+ 5)
		.attr("class", "legend")    // style the legend
		.style("fill", function() { // Add the colours dynamically
			return d.color = color(d.key); })
		.on("click", function(){
			// Determine if current line is visible 
			var active   = d.active ? false : true,
			newOpacity = active ? 0 : 1; 
			// Hide or show the elements based on the ID
			d3.select("#tag"+d.key.replace(/\s+/g, ''))
				.transition().duration(100) 
				.style("opacity", newOpacity); 
			// Update whether or not the elements are active
			d.active = active;
			})  
		.text(
		    function() {
		        if (d.key == '28-00043b6ef8ff') {return "Inlet";}
		        if (d.key == '28-00043e9049ff') {return "Ambient";}
		        if (d.key == '28-00043e8defff') {return "Outlet";}
		        else {return d.key;}
		        }); 
});  

The forEach function being applied to dataNest means that it will take each of the keys that we have just declared with the d3.nest (each sensor ID) and use the values for each sensor ID to append a line using its values.

There is a small and subtle change that might other wise go unnoticed, but is nonetheless significant. We include an i in the forEach function;

    dataNest.forEach(function(d,i) {

This might not seem like much of a big deal, but declaring i allows us to access the index of the returned data. This means that each unique key (sensor ID) has a unique number.

Then the code can get on with the task of drawing our lines;

	svg.append("path")
		.attr("class", "line")
		.style("stroke", function() { // Add the colours dynamically
			return d.color = color(d.key); })
		.attr("id", 'tag'+d.key.replace(/\s+/g, '')) // assign ID
		.attr("d", temperatureline(d.values));  

Applying the colours

Making sure that the colours that are applied to our lines (and ultimately our legend text) is unique from line to line is actually pretty easy.

The set-up for this is captured in an earlier code snippet.

var color = d3.scale.category10();   // set the colour scale

This declares an ordinal scale for our colours. This is a set of categorical colours (10 of them in this case) that can be invoked which are a nice mix of difference from each other and pleasant on the eye.

We then use the colour scale to assign a unique stroke (line colour) for each unique key (sensor ID) in our dataset.

    .style("stroke", function() {
        return d.color = color(d.key); })

It seems easy when it’s implemented, but in all reality, it is the product of some very clever thinking behind the scenes when designing d3.js and even picking the colours that are used.

Then we need to make sure that we can have a good reference between our lines and our legend labels. To do this we need to add assign an id to each legend text label.

        .attr("id", 'tag'+d.key.replace(/\s+/g, ''))

Being able to use our key value as the id means that each label will have a unique identifier. “What’s with adding the 'tag' piece of text to the id?” I hear you ask. Good question. If our key starts with a number we could strike trouble (in fact I’m sure there are plenty of other ways we could strike trouble too, but this was one I came across). As well as that we include a little regular expression goodness to strip any spaces out of the key with .replace(/\s+/g, '').

Adding the legend

If we think about the process of adding a legend to our graph, what we’re trying to achieve is to take every unique data series we have (sensor ID) and add a relevant label showing which colour relates to which sensor. At the same time, we need to arrange the labels in such a way that they are presented in a manner that is not offensive to the eye. In the example I will go through I have chosen to arrange them neatly spaced along the bottom of the graph. so that the final result looks like the following;

Multi-line graph with legend
Multi-line graph with legend

Bear in mind that the end result will align the legend completely automatically. If there are three sensors it will be equally spaced, if it is six sensors they will be equally spaced. The following is a reasonable mechanism to facilitate this, but if the labels for the data values are of radically different lengths, the final result will looks ‘odd’ likewise, if there are a LOT of data values, the legend will start to get crowded.

There are three broad categories we will want to enable to make this possible;

  1. Declare a style for the legend font
  2. Change the area and margins for the graph to accommodate the additional text
  3. Add the text

Declaring the style for the legend text is as easy as making an appropriate entry in the <style> section of the code. For this example we have the following;

.legend {
    font-size: 16px;
    font-weight: bold;
    text-anchor: middle;
} 

To change the area and margins of the graph we can make the following small changes to the code.

var margin = {top: 30, right: 20, bottom: 70, left: 50}, 
    width = 900 - margin.left - margin.right,
    height = 300 - margin.top - margin.bottom;  

The bottom margin is now 70 pixels high and the overall space for the area that the graph (including the margins) covers is 300 pixels.

To add the legend text is slightly more work, but only slightly more.

One of the ‘structural’ pieces of code that we needed to put in was the piece that understood the physical layout of what we are trying to achieve;

legendSpace = width/dataNest.length; // spacing for the legend

This finds the spacing between each legend label by dividing the width of the graph area by the number of sensor IDs (key’s).

The following code can then go ahead and add the legend;

	// Add the Legend
	svg.append("text")
		.attr("x", (legendSpace/2)+i*legendSpace)  // space legend
		.attr("y", height + (margin.bottom/2)+ 5)
		.attr("class", "legend")    // style the legend
		.style("fill", function() { // Add the colours dynamically
			return d.color = color(d.key); })
		.on("click", function(){
			// Determine if current line is visible 
			var active   = d.active ? false : true,
			newOpacity = active ? 0 : 1; 
			// Hide or show the elements based on the ID
			d3.select("#tag"+d.key.replace(/\s+/g, ''))
				.transition().duration(100) 
				.style("opacity", newOpacity); 
			// Update whether or not the elements are active
			d.active = active;
			})  
		.text(
		    function() {
		        if (d.key == '28-00043b6ef8ff') {return "Inlet";}
		        if (d.key == '28-00043e9049ff') {return "Ambient";}
		        if (d.key == '28-00043e8defff') {return "Outlet";}
		        else {return d.key;}
		        }); 

There are some slightly complicated things going on in here, so we’ll make sure that they get explained.

Firstly we get all our positioning attributes so that our legend will go into the right place;

		.attr("x", (legendSpace/2)+i*legendSpace)  // space legend
		.attr("y", height + (margin.bottom/2)+ 5)
		.attr("class", "legend")    // style the legend

The horizontal spacing for the labels is achieved by setting each label to the position set by the index associated with the label and the space available on the graph. To make it work out nicely we add half a legendSpace at the start (legendSpace/2) and then add the product of the index (i) and legendSpace (i*legendSpace).

We position the legend vertically so that it is in the middle of the bottom margin (height + (margin.bottom/2)+ 5).

And we apply the same colour function to the text as we did to the lines earlier;

		.style("fill", function() { // Add the colours dynamically
			return d.color = color(d.key); })

Making it interactive

The last significant step we’ll take in this example is to provide ourselves with a bit of control over how the graph looks. Even with the multiple colours, the graph could still be said to be ‘busy’. To clean it up or at least to provide the ability to more clearly display the data that a user wants to see we will add code that will allow us to click on a legend label and this will toggle the corresponding graph line on or off.

		.on("click", function(){
			// Determine if current line is visible 
			var active   = d.active ? false : true,
			newOpacity = active ? 0 : 1; 
			// Hide or show the elements based on the ID
			d3.select("#tag"+d.key.replace(/\s+/g, ''))
				.transition().duration(100) 
				.style("opacity", newOpacity); 
			// Update whether or not the elements are active
			d.active = active;
			})  

We use the .on("click", function(){ call to carry out some actions on the label if it is clicked on. We toggle the .active descriptor for our element with var active = d.active ? false : true,. Then we set the value of newOpacity to either 0 or 1 depending on whether active is false or true.

From here we can select our label using its unique id and adjust it’s opacity to either 0 (transparent) or 1 (opaque);

			d3.select("#tag"+d.key.replace(/\s+/g, ''))
				.transition().duration(100) 
				.style("opacity", newOpacity); 

Just because we can, we also add in a transition statement so that the change in transparency doesn’t occur in a flash (100 milli seconds in fact (.duration(100))).

Lastly we update our d.active variable to whatever the active state is so that it can toggle correctly the next time it is clicked on.

Since it’s kind of difficult to represent interactivity in a book, head on over to the live example on bl.ocks.org to see the toggling awesomeness that could be yours!

Printing out custom labels

The only thing left to do is to decide what to print for our labels. If we wanted to simply show each sensor ID we could have the following;

		.text(d.key); 

This would produce the following at the bottom of the graph;

Multi-line graph with legend
Multi-line graph with legend

But it makes more sense to put a real-world label in place so the user has a good idea about what they’re looking at. To do this we can use an if statement to match up our sensors with a nice human readable representation of what is going on;

		.text(
		    function() {
		        if (d.key == '28-00043b6ef8ff') {return "Inlet";}
		        if (d.key == '28-00043e9049ff') {return "Ambient";}
		        if (d.key == '28-00043e8defff') {return "Outlet";}
		        else {return d.key;}
		        }); 

The final result is a neat and tidy legend at the bottom of the graph;

Multi-line graph with legend
Multi-line graph with legend
Add the axes

And finally we get to draw in the axes;

svg.append("g")
	.attr("class", "x axis")
	.attr("transform", "translate(0," + height + ")")
	.call(xAxis);

svg.append("g")
	.attr("class", "y axis")
	.call(yAxis);

We have covered the formatting of the axis components earlier. So this part is actually just about getting those components drawn onto our graph.

Both axes start by being appended to the g group. Then each has its own classes applied for styling via CSS.

Feel free to mess about with these CSS styles to change the appearance of your axes.

On the x axis, we have a transform statement (.attr("transform", "translate(0," + height + ")")). If you recall, our point of origin for drawing is in the top left hand corner. Therefore if we want our x axis to be on the bottom of the graph, we need to move (transform) it to the bottom by a set amount. The set amount in this case is the height of the graph proper (height).

The last part of the two sections of script ( .call(xAxis); and .call(yAxis); ) call the x and y axis functions and initiate the drawing action.

Wrap Up

Well that’s it. In theory, you should now be a complete D3 ninja.

OK, perhaps a slight exaggeration. In fact there is a strong possibility that the information I have laid out here is at best borderline useful and at worst laden with evil practices and gross inaccuracies.

But look on the bright side. Irrespective of the nastiness of the way that any of it was accomplished or the inelegance of the code, if the picture drawn on the screen is relatively pretty, you can walk away with a smile. :-)

This section concludes a very basic description of one type of a graphic that can be built with D3.

Those with a smattering of knowledge of any of the topics I have butchered above (or below) are fully justified in feeling a large degree of righteous indignation. To those I say, please feel free to amend where practical and possible, but please bear in mind this was written from the point of view of someone with only a little experience in the topic and therefore try to keep any instructions at a level where a new entrant can step in.

Linux Concepts

What is Linux?

In it’s simplest form, the answer to the question “What is Linux?” is that it’s a computer operating system. As such it is the software that forms a base that allows applications that run on that operating system to run.

In the strictest way of speaking, the term ‘Linux’ refers to the Linux kernel. That is to say the central core of the operating system, but the term is often used to describe the set of programs, tools, and services that are bundled together with the Linux kernel to provide a fully functional operating system.

An operating system is software that manages computer hardware and software resources for computer applications. For example Microsoft Windows could be the operating system that will allow the browser application Firefox to run on our desktop computer.

Linux is a computer operating system that is can be distributed as free and open-source software. The defining component of Linux is the Linux kernel, an operating system kernel first released on 5 October 1991 by Linus Torvalds.

Linux was originally developed as a free operating system for Intel x86-based personal computers. It has since been made available to a huge range of computer hardware platforms and is a leading operating system on servers, mainframe computers and supercomputers. Linux also runs on embedded systems, which are devices whose operating system is typically built into the firmware and is highly tailored to the system; this includes mobile phones, tablet computers, network routers, facility automation controls, televisions and video game consoles. Android, the most widely used operating system for tablets and smart-phones, is built on top of the Linux kernel.

The Linux mascot 'Tux'
The Linux mascot ‘Tux’

The development of Linux is one of the most prominent examples of free and open-source software collaboration. Typically, Linux is packaged in a form known as a Linux distribution, for both desktop and server use. Popular mainstream Linux distributions include Debian, Ubuntu and the commercial Red Hat Enterprise Linux. Linux distributions include the Linux kernel, supporting utilities and libraries and usually a large amount of application software to carry out the distribution’s intended use.

A distribution intended to run as a server may omit all graphical desktop environments from the standard install, and instead include other software to set up and operate a solution stack such as LAMP (Linux, Apache, MySQL and PHP). Because Linux is freely re-distributable, anyone may create a distribution for any intended use.

Linux is not an operating system that people will typically use on their desktop computers at home and as such, regular computer users can find the barrier to entry for using Linux high. This is made easier through the use of Graphical User Interfaces that are included with many Linux distributions, but these graphical overlays are something of a shim to the underlying workings of the computer. There is a greater degree of control and flexibility to be gained by working with Linux at what is called the ‘Command Line’ (or CLI), and the booming field of educational computer elements such as the Raspberry Pi have provided access to a new world of learning opportunities at this more fundamental level.

Linux Directory Structure

To a new user of Linux, the file structure may feel like something at best arcane and in some cases arbitrary. Of course this isn’t entirely the case and in spite of some distribution specific differences, there is a fairly well laid out hierarchy of directories and files with a good reason for being where they are.

We are frequently comfortable with the concept of navigating this structure using a graphical interface similar to that shown below, but to operate effectively at the command line we need to have a working knowledge of what goes where.

Linux Directories
Linux Directories

The directories we are going to describe form a hierarchy similar to the following;

Directory Hierarchy
Directory Hierarchy

For a concise description of the directory functions check out the cheat sheet. Alternatively their function and descriptions are as follows;

/

The / or ‘root’ directory contains all other files and directories. It is important to note that this is not the root users home directory (although it used to be many years ago). The root user’s home directory is /root. Only the root user has write privileges for this directory.

/bin

The /bin directory contains common essential binary executables / commands for use by all users. For example: the commands cd, cp, ls and ping. These are commands that may be used by both the system administrator and by users, but which are required when no other filesystems are mounted.

/boot

The /boot directory contains the files needed to successfully start the computer during the boot process. As such the /boot directory contains information that is accessed before the Linux kernel begins running the programs and process that allow the operating system to function.

/dev

The /dev directory holds device files that represent physical devices attached to the computer such as hard drives, sound devices and communication ports as well as ‘logical’ devices such as a random number generator and /dev/null which will essentially discard any information sent to it. This directory holds a range of files that strongly reinforces the Linux precept that Everything is a file.

/etc

The /etc directory contains configuration files that control the operation of programs. It also contains scripts used to startup and shutdown individual programs.

/etc/cron.d

The /etc/cron.d, /etc/cron.hourly, /etc/cron.daily, /etc/cron.weekly, /etc/cron.monthly directories contain scripts which are executed on a regular schedule by the crontab process.

/etc/rc?.d

The /rc0.d, /rc1.d, /rc2.d, /rc3.d, /rc4.d, /rc5.d, /rc6.d, /rcS.d directories contain the files required to control system services and configure the mode of operation (runlevel) for the computer.

/home

Because Linux is an operating system that is a ‘multi-user’ environment, each user requires a space to store information specific to them. This is done via the /home directory. For example, the user ‘pi’ would have /home/pi as their home directory.

/lib

The /lib directory contains shared library files that supports the executable files located under /bin and /sbin. It also holds the kernel modules (drivers) responsible for giving Linux a great deal of versatility to add or remove functionality as needs dictate.

/lost+found

The /lost+found directory will contain potentially recoverable data that might be produced if the file system undergoes an improper shut-down due to a crash or power failure. The data recovered is unlikely to be complete or undamaged, but in some circumstances it may hold useful information or pointers to the reason for the improper shut-down.

/media

The /media directory is used as a directory to temporarily mount removable devices (for example, /media/cdrom or /media/cdrecorder). This is a relatively new development for Linux and comes as a result of a degree of historical confusion over where was best to mount these types of devices (/cdrom, /mnt or /mnt/cdrom for example).

/mnt

The /mnt directory is used as a generic mount point for filesystems or devices. Recent use of the directory is directing it towards it being used as a temporary mount point for system administrators, but there is a degree of historical variation that has resulted in different distributions doing things different ways (for example, Debian allocates /floppy and /cdrom as mount points while Redhat places them in /mnt/floppy and /mnt/cdrom respectively).

/opt

The /opt directory is used for the installation of third party or additional optional software that is not part of the default installation. Any applications installed in this area should be installed in such a way that it conforms to a reasonable structure and should not install files outside the /opt directory.

/proc

The /proc directory holds files that contain information about running processes and system resources. It can be described as a pseudo filesystem in the sense that it contains runtime system information, but not ‘real’ files in the normal sense of the word. For example the /proc/cpuinfo file which contains information about the computers cpus is listed as 0 bytes in length and yet if it is listed it will produce a description of the cpus in use.

/root

The /root directory is the home directory of the System Administrator, or the ‘root’ user. This could be viewed as slightly confusing as all other users home directories are in the /home directory and there is already a directory referred to as the ‘root’ directory (/). However, rest assured that there is good reason for doing this (sometimes the /home directory could be mounted on a separate file system that has to be accessed as a remote share).

/sbin

The /sbin directory is similar to the /bin directory in the sense that it holds binary executables / commands, but the ones in /sbin are essential to the working of the operating system and are identified as being those that the system administrator would use in maintaining the system. Examples of these commands are fdisk, shutdown, ifconfig and modprobe.

/srv

The /srv directory is set aside to provide a location for storing data for specific services. The rationale behind using this directory is that processes or services which require a single location and directory hierarchy for data and scripts can have a consistent placement across systems.

/tmp

The /tmp directory is set aside as a location where programs or users that require a temporary location for storing files or data can do so on the understanding that when a system is rebooted or shut down, this location is cleared and the contents deleted.

/usr

The /usr directory serves as a directory where user programs and data are stored and shared. This potential wide range of files and information can make the /usr directory fairly large and complex, so it contains several subdirectories that mirror those in the root (/) directory to make organisation more consistent.

/usr/bin

The /usr/bin directory contains binary executable files for users. The distinction between /bin and /usr/bin is that /bin contains the essential commands required to operate the system even if no other file system is mounted and /usr/bin contains the programs that users will require to do normal tasks. For example; awk, curl, php, python. If you can’t find a user binary under /bin, look under /usr/bin.

/usr/lib

The /usr/lib directory is the equivalent of the /lib directory in that it contains shared library files that supports the executable files for users located under /usr/bin and /usr/sbin.

/usr/local

The /usr/local directory contains users programs that are installed locally from source code. It is placed here specifically to avoid being inadvertently overwritten if the system software is upgraded.

/usr/sbin

The /usr/sbin directory contains non-essential binary executables which are used by the system administrator. For example cron and useradd. If you can’t locate a system binary in /usr/sbin, try /sbin.

/var

The /var directory contains variable data files. These are files that are expected to grow under normal circumstances For example, log files or spool directories for printer queues.

/var/lib

The /var/lib directory holds dynamic state information that programs typically modify while they run. This can be used to preserve the state of an application between reboots or even to share state information between different instances of the same application.

/var/log

The /var/log directory holds log files from a range of programs and services. Files in /var/log can often grow quite large and care should be taken to ensure that the size of the directory is managed appropriately. This can be done with the logrotate program.

/var/spool

The /var/spool directory contains what are called ‘spool’ files that contain data stored for later processing. For example, printers which will queue print jobs in a spool file for eventual printing and then deletion when the resource (the printer) becomes available.

/var/tmp

The /var/tmp directory is a temporary store for data that needs to be held between reboots (unlike /tmp).

Everything is a file in Linux

A phrase that will often come up in Linux conversation is that;

Everything is a file

For someone new to Linux this sounds like some sort of ‘in joke’ that is designed to scare off the unwary and it can sometimes act as a barrier to a deeper understanding of the philosophy behind the approach taken in developing Linux.

The explanation behind the statement is that Linux is designed to be a system built of a group of interacting parts and the way that those parts can work together is to communicate using a common method. That method is to use a file as a common building block and the data in a file as the communications mechanism.

The trick to understanding what ‘Everything is a file’ means, is to broaden our understanding of what a file can be.

Traditional Files

The traditional concept of a file is an object with a specific name in a specific location with a particular content. For example, we might have a file named foo.txt which is in the directory /home/pi/ and it could contain a couple of lines of text similar to the following;

This is the first line
This is the second line

Directories

As unusual as it sounds a directory is also a file. The special aspect of a directory is that is is a file which contains a list of information about which files (and / or subdirectories) it contains. So when we want to list the contents of a directory using the ls command what is actually happening is that the operating system is getting the appropriate information from the file that represents the directory.

System Information

However, files can also be conduits of information. The /proc/ directory contains files that represent system and process information. If we want to determine information about the type of CPU that the computer is using, the file cpuinfo in the /proc/ directory can list it. By running the command `cat /proc/cpuinfo’ we can list a wealth of information about our CPU (the following is a subset of that information by the way);

pi@raspberrypi ~ $ cat /proc/cpuinfo
processor       : 0
model name      : ARMv7 Processor rev 5 (v7l)
BogoMIPS        : 57.60
Features        : half thumb fastmult vfp edsp neon vfpv3 tls vfpv4 idiva idi\
vt vfpd32 lpae evtstrm
CPU implementer : 0x41
CPU architecture: 7
CPU variant     : 0x0
CPU part        : 0xc07
CPU revision    : 5

Hardware        : BCM2709
Revision        : a01041
Serial          : 000000002a4ea712

Now that might not mean a lot to us at this stage, but if we were writing a program that needed a particular type of CPU in order to run successfully it could check this file to ensure that it could operate successfully. There are a wide range of files in the /proc/ directory that represent a great deal of information about how our system is operating.

Devices

When we use different devices in a Linux operating system these are also represented as a file. In the /dev/ directory we have files that represent a range of physical devices that are part of our computer. In larger computer systems with multiple disks they could be represented as /dev/sda1 and /dev/sda2, so that when we wanted to perform an action such as formatting a drive we would use the command mkfs on the /dev/sda1 file.

The /dev/ directory also holds some curious files that are used as tools for generating or managing data. For example /dev/random is an interface to the kernels random number device. /dev/zero represents a file that will constantly stream zeros (while this might sound weird, imagine a situation where you want to write over an area of disk with data to erase it). The most well known of these unusual files is probably /dev/null. This will act as a ‘null device’ that will essentially discard any information sent to it.

File Editing

Working in Linux is an exercise in understanding the concepts that Linux uses as its foundations such as ‘Everything is a file’ and the use of wildcards, pipes and the directory structure.

While working at the command line there will very quickly come the realisation that there is a need to know how to edit a file. Linux being what it is, there are many ways that files can be edited.

An outstanding illustration of this is via the excellent cartoon work of the xkcd comic strip (Buy his stuff, it’s awesome!).

Real Programmers
Real Programmers

For a taste of the possible options available Wikipedia has got our back. Inevitably where there is choice there are preferences and where there are preferences there is bias. Everyone will have a preference towards a particular editor and don’t let a particular bias influence you to go down a particular direction without considering your options. Speaking from personal experience I was encouraged to use ‘vi’ as it represented the preference of the group I was in, but because I was a late starter to the command line I struggled for the longest time to try and become familiar with it. I know I should have tried harder, but I failed. For a while I wandered in the editor wilderness trying desperately to cling to the GUI where I could use ‘gedit’ or ‘geany’ and then one day I was introduced to ‘nano’.

This has become my preference and I am therefore biased towards it. Don’t take my word for it. Try alternatives. I’ll describe ‘nano’ below, but take that as a possible path and realise that whatever editor works for you will be the right one. The trick is simply to find one that works for you.

The nano Editor

The nano editor can be started from the command line using just the command and the /path/name of the file.

If the file requires administrator permissions it can be executed with ‘sudo`.

When it opens it presents us with a working space and part of the file and some common shortcuts for use at the bottom of the console;

nano Interface
nano Interface

It includes some simple syntax highlighting for common file formats;

nano Syntax Highlighting
nano Syntax Highlighting

This can be improved if desired (cue Google).

There is a swag of shortcuts to make editing easier, but the simple ones are as follows;

  • CTRL-x - Exit the editor. If we are in the middle of editing a file we will be asked if we want to save our work
  • CTRL-r - Read a file into our current working file. This enables us to add text from another file while working from within a new file.
  • CTRL-k - Cut text.
  • CTRL-u - Uncut (or Paste) text.
  • CTRL-o - Save file name and continue working.
  • CTRL-t - Check the spelling of our text.
  • CTRL-w - Search the text.
  • CTRL-a - Go to the beginning of the current working line.
  • CTRL-e - Go to the end of the current working line.
  • CTRL-g - Get help with nano.

Linux Commands

Executing Commands in Linux

A command is an instruction given by a user telling the computer to carry out an action. This could be to run a single program or a group of linked programs. Commands are typically initiated by typing them in at the command line (in a terminal) and then pressing the ENTER key, which passes them to the shell.

The Terminal
The Terminal

A terminal refers to a wrapper program which runs a shell. This used to mean a physical device consisting of little more than a monitor and keyboard. As Unix/Linux systems advanced the terminal concept was abstracted into software. Now we have programs such as LXTerminal (on the Raspberry Pi) which will launch a window in a Graphical User Interface (GUI) which will run a shell into which you can enter commands. Alternatively we can dispense with the GUI all together and simply start at the command line when we boot up.

The shell is a program which actually processes commands and returns output. Every Linux operating system has at least one shell, and most have several. The default shell on most Linux systems is bash.

The Commands

Commands on Linux operating systems are either built-in or external commands. Built-in commands are part of the shell. External commands are either executables (programs written in a programming language and then compiled into an executable binary) or shell scripts.

A command consists of a command name usually followed by one or more sequences of characters that include options and/or arguments. Each of these strings is separated by white space. The general syntax for commands is;

commandname [options] [arguments]

The square brackets indicate that the enclosed items are optional. Commands typically have a few options and utilise arguments. However, there are some commands that do not accept arguments, and a few with no options. As an example we can run the ls command with no options or arguments as follows;

The ls command will list the contents of a directory and in this case the command and the output would be expected to look something like the following;

pi@raspberrypi ~ $ ls
Desktop                   python_games
Options

An option (also referred to as a switch or a flag) is a single-letter code, or sometimes a single word or set of words, that modifies the behaviour of a command. When multiple single-letter options are used, all the letters are placed adjacent to each other (not separated by spaces) and can be in any order. The set of options must usually be preceded by a single hyphen, again with no intervening space.

So again using ls if we introduce the option -l we can show the total files in the directory and subdirectories, the names of the files in the current directory, their permissions, the number of subdirectories in directories listed, the size of the file, and the date of last modification.

The command we execute therefore looks like this;

And so the command (with the -l option) and the output would look like the following;

pi@raspberrypi ~ $ ls -l
total 26
drwxr-xr-x 2 pi pi 4096 Feb 20 08:07 Desktop
drwxrwxr-x 2 pi pi 4096 Jan 27 08:34 python_games

Here we can see quite a radical change in the formatting and content of the returned information.

Arguments

An argument (also called a command line argument) is a file name or other data that is provided to a command in order for the command to use it as an input.

Using ls again we can specify that we wish to list the contents of the python_games directory (which we could see when we ran ls) by using the name of the directory as the argument as follows;

The command (with the python_games argument) and the output would look like the following (actually I removed quite a few files to make it a bit more readable);

pi@raspberrypi ~ $ ls  python_games
4row_arrow.png           gem4.png                    pentomino.py
4row_black.png           gem5.png                    pinkgirl.png
4row_board.png           gem6.png                    Plain_Block.png
4row_computerwinner.png  gem7.png                    princess.png
4row_humanwinner.png     gemgem.py                   RedSelector.png
gem1.png                 match5.wav                  Wall_Block_Tall.png
gem2.png                 memorypuzzle_obfuscated.py  Wood_Block_Tall.png
gem3.png                 memorypuzzle.py             wormy.py
Putting it all together

And as our final example we can combine our command (ls) with both an option (-l) and an argument (python_games) as follows;

Hopefully by this stage, the output shouldn’t come as too much surprise, although again I have pruned some of the files for readabilities sake;

pi@raspberrypi ~ $ ls -l  python_games
total 1800
-rw-rw-r-- 1 pi pi   9731 Jan 27 08:34 4row_arrow.png
-rw-rw-r-- 1 pi pi   7463 Jan 27 08:34 4row_black.png
-rw-rw-r-- 1 pi pi   8666 Jan 27 08:34 4row_board.png
-rw-rw-r-- 1 pi pi  18933 Jan 27 08:34 4row_computerwinner.png
-rw-rw-r-- 1 pi pi  25412 Jan 27 08:34 4row_humanwinner.png
-rw-rw-r-- 1 pi pi   8562 Jan 27 08:34 4row_red.png
-rw-rw-r-- 1 pi pi  14661 Jan 27 08:34 tetrisc.mid
-rw-rw-r-- 1 pi pi  15759 Jan 27 08:34 tetrominoforidiots.py
-rw-rw-r-- 1 pi pi  18679 Jan 27 08:34 tetromino.py
-rw-rw-r-- 1 pi pi   9771 Jan 27 08:34 Tree_Short.png
-rw-rw-r-- 1 pi pi  11546 Jan 27 08:34 Tree_Tall.png
-rw-rw-r-- 1 pi pi  10378 Jan 27 08:34 Tree_Ugly.png
-rw-rw-r-- 1 pi pi   8443 Jan 27 08:34 Wall_Block_Tall.png
-rw-rw-r-- 1 pi pi   6011 Jan 27 08:34 Wood_Block_Tall.png
-rw-rw-r-- 1 pi pi   8118 Jan 27 08:34 wormy.py

apt-get

The apt-get command is a program, that is used with Debian based Linux distributions to install, remove or upgrade software packages. It’s a vital tool for installing and managing software and should be used on a regular basis to ensure that software is up to date and security patching requirements are met.

There are a plethora of uses for apt-get, but we will consider the basics that will allow us to get by. These will include;

  • Updating the database of available applications (apt-get update)
  • Upgrading the applications on the system (apt-get upgrade)
  • Installing an application (apt-get install *package-name*)
  • Un-installing an application (apt-get remove *package-name*)
The apt-get command

The apt part of apt-get stands for ‘advanced packaging tool’. The program is a process for managing software packages installed on Linux machines, or more specifically Debian based Linux machines (Since those based on ‘redhat’ typically use their rpm (red hat package management (or more lately the recursively named ‘rpm package management’) system). As Raspbian is based on Debian, so the examples we will be using are based on apt-get.

APT simplifies the process of managing software on Unix-like computer systems by automating the retrieval, configuration and installation of software packages. This was historically a process best described as ‘dependency hell’ where the requirements for different packages could mean a manual installation of a simple software application could lead a user into a sink-hole of despair.

In common apt-get usage we will be prefixing the command with sudo to give ourselves the appropriate permissions;

apt-get update

This will resynchronize our local list of packages files, updating information about new and recently changed packages. If an apt-get upgrade (see below) is planned, an apt-get update should always be performed first.

Once the command is executed, the computer will delve into the internet to source the lists of current packages and download them so that we will see a list of software sources similar to the following appear;

pi@raspberrypi ~ $ sudo apt-get update
Hit http://raspberrypi.collabora.com wheezy Release.gpg
Get:1 http://mirrordirector.raspbian.org wheezy Release.gpg [490 B]
Get:2 http://archive.raspberrypi.org wheezy Release.gpg [473 B]
Hit http://raspberrypi.collabora.com wheezy Release
Get:3 http://mirrordirector.raspbian.org wheezy Release [14.4 kB]
Get:4 http://archive.raspberrypi.org wheezy Release [17.6 kB]
Hit http://raspberrypi.collabora.com wheezy/rpi armhf Packages
Get:5 http://mirrordirector.raspbian.org wheezy/main armhf Packages [6,904 kB]
Get:6 http://archive.raspberrypi.org wheezy/main armhf Packages [130 kB]
Ign http://raspberrypi.collabora.com wheezy/rpi Translation-en
Ign http://mirrordirector.raspbian.org wheezy/contrib Translation-en
Ign http://mirrordirector.raspbian.org wheezy/main Translation-en
Ign http://mirrordirector.raspbian.org wheezy/non-free Translation-en
Ign http://mirrordirector.raspbian.org wheezy/rpi Translation-en
Fetched 7,140 kB in 35s (200 kB/s)
Reading package lists... Done
apt-get upgrade

The apt-get upgrade command will install the newest versions of all packages currently installed on the system. If a package is currently installed and a new version is available, it will be retrieved and upgraded. Any new versions of current packages that cannot be upgraded without changing the install status of another package will be left as they are.

As mentioned above, an apt-get update should always be performed first so that apt-get upgrade knows which new versions of packages are available.

Once the command is executed, the computer will consider its installed applications against the databases list of the most up to date packages and it will prompt us with a message that will let us know how many packages are available for upgrade, how much data will need to be downloaded and what impact this will have on our local storage. At this point we get to decide whether or not we want to continue;

pi@raspberrypi ~ $ sudo apt-get upgrade
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages will be upgraded:
  bind9-host cups-bsd cups-client cups-common libapache2-mod-php5 libbind9-80 
  libisccc80 libisccfg82 liblwres80 libsdl1.2debian libsqlite3-0 libssl1.0.0 
  php5-mcrypt php5-mysql raspi-config
6 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Need to get 10.7 MB of archives.
After this operation, 556 kB disk space will be freed.
Do you want to continue [Y/n]?

Once we say yes (‘Y’) the upgrade kicks off and we will see a list of the packages as they are downloaded unpacked and installed (what follows is an edited example);

Do you want to continue [Y/n]? y
Get:1 http://archive.raspberrypi.org/debian/wheezy/main libsdl1.2debian
armhf 1.2.15-5+rpi1 [205 kB]
Get:2 http://archive.raspberrypi.org/debian/wheezy/main raspi-config all
20150131-5 [13.3 kB]
Get:3 http://mirrordirector.raspbian.org/raspbian/ wheezy/main libsqlite3-0
armhf 3.7.13-1+deb7u2 [414 kB]
Fetched 10.7 MB in 31s (343 kB/s)
Preconfiguring packages ...
(Reading database ... 80703 files and directories currently installed.)
Preparing to replace cups-common 1.5.3-5+deb7u5 
(using .../cups-common_1.5.3-5+deb7u6_all.deb) ...
Unpacking replacement cups-common ...
Preparing to replace cups-bsd 1.5.3-5+deb7u5 
(using .../cups-bsd_1.5.3-5+deb7u6_armhf.deb) ...
Unpacking replacement cups-bsd ...
Preparing to replace php5-gd 5.4.39-0+deb7u2 
(using .../php5-gd_5.4.41-0+deb7u1_armhf.deb) ...
Unpacking replacement php5-gd ...
Processing triggers for man-db ...
Setting up libssl1.0.0:armhf (1.0.1e-2+rvt+deb7u17) ...
Setting up libsqlite3-0:armhf (3.7.13-1+deb7u2) ...
Setting up cups-common (1.5.3-5+deb7u6) ...
Setting up cups-client (1.5.3-5+deb7u6) ...

There can often be alerts as the process identifies different issues that it thinks the system might strike (different aliases, runtime levels or missing fully qualified domain names). This is not necessarily a sign of problems so much as an indication that the process had to take certain configurations into account when upgrading and these are worth noting. Whenever there is any doubt about what has occurred, Google will be your friend :-).

apt-get install

The apt-get install command installs or upgrades one (or more) packages. All additional (dependency) packages required will also be retrieved and installed.

If we want to install multiple packages we can simply list each package separated by a space after the command as follows;

apt-get remove

The apt-get remove command removes one (or more) packages.

cat

The cat command is a really versatile command that is typically used to carry out three different functions. It can display a file on the screen, combine different files together (concatenate them) or create new files. This is another core command that is immensely useful to learn in when working with Linux from the command line. It’s simple, flexible and versatile.

  • cat [options] filename filename : Display, combine or create new files.

For example: To display the file foo.txt on the screen we would use;

To display the files foo.txt and bar.txt on the screen one after the other we would use;

Or to combine the files foo.txt and bar.txt into a new file called foobar.txt using the redirection symbol >;

The cat command

The cat command is a vital tool to use on the Linux command line because ultimately Linux is an operating system that is file driven. Without a graphical user interface there needs to be a mechanism whereby creating and manipulating text files can be accomplished easily. The cat command is one of the commands that makes this possible. The name ‘cat’ is short for ‘catenate’ or ‘concatenate’ (either appears to be acceptable, but ‘concatenate’ appears to be more widely used), which is to say to connect things in a series. This is certainly one of it’s common uses, but a better overview would be to say that the cat command is used to;

  • Display text files at the command line
  • Join one text file to the end of another text file, combining them
  • Copy text files into a new document
Options

The only option that gets any degree of use with cat is the -n option that numbers the output lines.

Arguments and Examples

To display text

For example, to display a text file (foo.txt) on the screen we can use the following command;

The output would be;

pi@raspberrypi ~ $ cat foo.txt
This line is the contents of foo.txt

As we can see, the contents of the file ‘foo.txt’ is sent to the screen (be aware, if the file is sufficiently large, it will simple dump the contents in a long scrolling waterfall of text).

To join more than one file together

We could just as easily display two files one after the other (concatenated) as follows;

The output would be;

pi@raspberrypi ~ $ cat foo.txt bar.txt
This line is the contents of foo.txt
This line is the contents of bar.txt

To create a new file

Instead of having the file sent to the screen, we can specify that cat send our file to a new (renamed) file as follows;

This could be thought of an a equivalent to a file copy action and uses the redirection symbol >.

Taking the process one step further we can take our original two files and combine them into a single file with;

We can then check the results of our combination using cat on the new file as follows;

And the output would be;

pi@raspberrypi ~ $ cat foobar.txt
This line is the contents of foo.txt
This line is the contents of bar.txt

Then we could use cat to append a file to an already existing file by using the redirection operator >>;

Here we use the redirection operator >> to add the contents of the file newfoo.txt to the already existing file foobar.txt.

The resulting file content would be;

pi@raspberrypi ~ $ cat foobar.txt
This line is the contents of foo.txt
This line is the contents of bar.txt
And this is newfoo.txt

Finally, we can use cat to create a file from scratch. In this scenario if we use cat without a source file and redirect to a new file (here called newfile.txt. It will take the input from the command line to add to the file until CONTROL-d is pressed.

The resulting file content would be;

pi@raspberrypi ~ $ cat newfile.txt
Just keep typing
and entering information until...
we press ctrl-d to finish and the file gets written!
Test yourself
  1. Which is the safest redirector to use?
  2. Create a new file using cat and enter a few lines of text to give it some content
  3. Copy that file using cat to a new file.
  4. Combine the original file and the copy into a new file.
  5. Display that new file on the screen.

cd

The cd command is used to move around in the directory structure of the file system (change directory). It is one of the fundamental commands for navigating the Linux directory structure.

cd [options] directory : Used to change the current directory.

For example, when we first log into the Raspberry Pi as the ‘pi’ user we will find ourselves in the /home/pi directory. If we wanted to change into the /home directory (go up a level) we could use the command;

Take some time to get familiar with the concept of moving around the directory structure from the command line as it is an important skill to establish early in Linux.

The cd command

The cd command will be one of the first commands that someone starting with Linux will use. It is used to move around in the directory structure of the file system (hence cd = change directory). It only has two options and these are seldom used. The arguments consist of pointing to the directory that we want to go to and these can be absolute or relative paths.

The cd command can be used without options or arguments. In this case it returns us to our home directory as specified in the /etc/passwd file.

If we cd into any random directory (try cd /var) we can then run cd by itself;

… and in the case of a vanilla installation of Raspbian, we will change to the /home/pi directory;

pi@raspberrypi ~ $ cd /var
pi@raspberrypi /var $ cd
pi@raspberrypi ~ $ pwd
/home/pi

In the example above, we changed to /var and then ran the cd command by itself and then we ran the pwd command which showed us that the present working directory is /home/pi. This is the Raspbian default home directory for the pi user.

Options

As mentioned, there are only two options available to use with the cd command. This is -P which instructs cd to use the physical directory structure instead of following symbolic links and the -L option which forces symbolic links to be followed.

For those beginning Linux, there is little likelihood of using either of these two options in the immediate future and I suggest that you use your valuable memory to remember other Linux stuff.

Arguments

As mentioned earlier, the default argument (if none is included) is to return to the users home directory as specified in the /etc/passwd file.

When specifying a directory we can do this by absolute or relative addressing. So if we started in the /home/pi directory, we could go the /home directory by executing;

… or using relative addressing and we can use the .. symbols to designate the parent directory;

Once in the /home directory, we can change into the /home/pi/Desktop directory using relative addressing as follows;

We can also use the - argument to navigate to the previous directory we were in.

Examples

Change into the root (/) directory;

Test yourself
  1. Having just changed from the /home/pi directory to the /home directory, what are the five variations of using the cd command that will take the pi user to the /home/pi directory
  2. Starting in the /home/pi directory and using only relative addressing, use cd to change into the /var directory.

chmod

The chmod command allows us to set or modify a file’s permissions. Because Linux is built as a multi-user system there are typically multiple different users with differing permissions for which files they can read / write or execute. chmod allows us to limit access to authorised users to do things like editing web files while general users can only read the files.

  • chmod [options] mode files : Change access permissions of one or more files & directories

For example, the following command (which would most likely be prefixed with sudo) sets the permissions for the /var/www directory so that the user can read from, write to and change into the directory. Group owners can also read from, write to and change into the directory. All others can read from and change into the directory, but they cannot create or delete a file within it;

This might allow normal users to browse web pages on a server, but prevent them from editing those pages (which is probably a good thing).

The chmod command

The chmod command allows us to change the permissions for which user is allowed to do what (read, write or execute) to files and directories. It does this by changing the ‘mode’ (hence chmod = change file mode) of the file where we can make the assumption that ‘mode’ = permissions.

Every file on the computer has an associated set of permissions. Permissions tell the operating system what can be done with that file and by whom. There are three things you can (or can’t) do with a given file:

  • read it,
  • write (modify) it and
  • execute it.

Linux permissions specify what the owning user can do, what the members of the owning group can do and what other users can do with the file. For any given user, we need three bits to specify access permissions: the first to denote read (r) access, the second to denote (w) access and the third to denote execute (x) access.

We also have three levels of ownership: ‘user’, ‘group’ and ‘others’ so we need a triplet (three sets of three) for each, resulting in nine bits.

The following diagram shows how this grouping of permissions can be represented on a Linux system where the user, group and others had full read, write and execute permissions;

Linux permissions as rwx
Linux permissions as rwx

If we had a file with more complex permissions where the user could read, write and execute, the group could read and write, but all other users could only read it would look as follows;

Slightly more complex Linux permissions
Slightly more complex Linux permissions

This description of permissions is workable, but we will need to be aware that the permissions are also represented as 3 bit values (where each bit is a ‘1’ or a ‘0’ (where a ‘1’ is yes you can, or ‘0’ is no you can’t)) or as the equivalent octal value.

Linux permissions as symbolic, 3 bit and octal
Linux permissions as symbolic, 3 bit and octal

The full range of possible values for these permission combinations is as follows;

Permission              Symbolic 3-bit Octal
read, write and execute rwx      111   7
read and write          rw-      110   6
read and execute        r-w      101   5
read only               r--      100   4
write and execute       -wx      011   3
write only              -w-      010   2
execute only            --x      001   1
none                    ---      000   0

Another interesting thing to note is that permissions take a different slant for directories.

  • read determines if a user can view the directory’s contents, i.e. execute ls in it.
  • write determines if a user can create new files or delete file in the directory. (Note here that this essentially means that a user with write access to a directory can delete files in the directory even if he/she doesn’t have write permissions for the file! So be careful.)
  • execute determines if the user can cd into the directory.

We can check the check the permissions of files using the ls -l command which will list files in a long format as follows;

This command will list the details of the file foo.txt that is in the /tmp directory as follows

pi@raspberrypi ~ $ ls -l /tmp
-rwxrw-r-- 1 pi pi-group 20 Jul 10 13:14 foo.txt

The permissions on the file, the user and the group owner can be found as follows;

File details
File details

From this information we can see that the file’s user (‘pi’) has permissions to read, write and execute the file. The group owner (‘pi-group’) can read and write to the file and all other users can read the file.

Options

The main option that is worth remembering is the -R option that will Recursively apply permissions on the files in the specified directory and its sub-directories.

The following command will change the permissions for all the files in the /srv/foo directory and in all the directories that are under it;

Arguments

Simplistically (in other words it can be more complicated, but we’re simplifying it) there are two main ways that chmod is used. In either symbolic mode where the permissions are changed using symbols associated with read, write and execute as well as symbols for the user (u), the group owner (g), others (o) and all users (a). Or in numeric mode where we use the octal values for permission combinations.

Symbolic Mode

In symbolic mode we can change the permissions of a file with the following syntax:

  • chmod [who][op][permissions] filename

Where who can be the user (u), the group owner (g) and / or others (o). The operator (op) is either + to add a permission, - to remove a permission or = to explicitly set permissions. The permissions themselves are either readable (r), writeable (w), or executable (x).

For example the following command adds executable permissions (x) to the user (u) for the file /tmp/foo.txt;

This command removes writing (w) and executing (x) permissions from the group owner (g) and all others (o) for the same file;

Note that removing the execute permission from a directory will prevent you from being able to list its contents (although root will override this). If you accidentally remove the execute permission from a directory, you can use the +X argument to instruct chmod to only apply the execute permission to directories.

Numeric Mode

In numeric mode we can explicitly state the permissions using the octal values, so this form of the command is fairly common.

For example, the following command will change the permissions on the file foo.txt so that the user can read, write and execute it, the group owner can read and write it and all others can read it;

Examples

To change the permissions in your home directory to remove reading and executing permissions from the group owner and all other users;

To make a script executable by the user;

Windows marks all files as executable by default. If you copy a file or directory from a Windows system (or even a Windows-formatted disk) to your Linux system, you should ideally strip the unnecessary execute permissions from all copied files unless you specifically need to retain it. Note of course we still need it on all //directories// so that we can access their contents! Here’s how we can achieve this in one command:

This instructs chmod to remove the execute permission for each file and directory, and then immediately set execute again if working on a directory.

crontab

The crontab command give the user the ability to schedule tasks to be run at a specific time or with a specific interval. If you want to move beyond using Linux from a graphical user interface, you will most likely want to schedule a task to run at a particular time or interval. Even just learning about it might give you ideas of what you might do.

  • crontab [-u user] [-l | -r | -e] : Schedule a task to run at a particular time or interval

For example, you could schedule a script to run every day to carry our a backup process in the middle of the night. or capture some data every hour to store in a database.

The crontab command

The command crontab is a concatenation of ‘cron table’ because it uses the job scheduler cron to execute tasks which are stored in a ‘table’ of sorts in the users crontab file. cron is named after ‘Khronos’, the Greek personification of time.

While each user who sets up a job to run using the crontab creates a crontab file, the file is not intended to be edited by hand. It is in different locations in different flavour of Linux distributions and the most reliable mechanism for editing it is by running the crontab -e command. Each user has their own crontab file and the root user can edit another users crontab file. This would be the situation where we would use the -u option, but honestly once we get to that stage it can probably be assumed that we know a fair bit about Linux.

There are only three main options that are used with crontab.

Options

The first option that we should examine is the -l option which allows us to list the crontab file;

Once run it will list the contents of the crontab file directly to the screen. The output will look something like;

# Edit this file to introduce tasks to be run by cron.
#
# Each task to run has to be defined through a single line
# indicating with different fields when the task will be run
# and what command to run for the task
#
# To define the time you can provide concrete values for
# minute (m), hour (h), day of month (dom), month (mon),
# and day of week (dow) or use '*' in these fields (for 'any').#
# Notice that tasks will be started based on the cron's system
# daemon's notion of time and timezones.
#
# Output of the crontab jobs (including errors) is sent through
# email to the user the crontab file belongs to (unless redirected).
#
# For example, you can run a backup of all your user accounts
# at 5 a.m every week with:
# 0 5 * * 1 tar -zcf /var/backups/home.tgz /home/
#
# For more information see the manual pages of crontab(5) and cron(8)
#
# m h  dom mon dow   command

*/10 * * * * /usr/bin/php /home/pi/scrape-books.php

Here we can see that the main part of the file (in fact everything except the final line) is comments that explain how to include an entry into the crontab file.

The entry in this case is specified to run every 10 minutes and when it does, it will run the PHP script scrape-books.php (we’ll explain how this is encoded later in the examples section).

If we want to remove the current crontab we can use the -r option. Probably not something that we would do an a regular basis, as it would be more likely to be editing the content rather than just removing it wholesale.

Lastly there is the option to edit the crontab file which is initiated using -e. This is the main option that would be used and the one we will cover in detail in the examples below.

Examples

As an example, consider that we wish to run a Python script every day at 6am. The following command will let us edit the crontab;

Once run it will open the crontab in the default editor on your system (most likely ‘vi’, ‘vim’ or ‘nano’). The file will look as follows;

# Edit this file to introduce tasks to be run by cron.
#
# Each task to run has to be defined through a single line
# indicating with different fields when the task will be run
# and what command to run for the task
#
# To define the time you can provide concrete values for
# minute (m), hour (h), day of month (dom), month (mon),
# and day of week (dow) or use '*' in these fields (for 'any').#
# Notice that tasks will be started based on the cron's system
# daemon's notion of time and timezones.
#
# Output of the crontab jobs (including errors) is sent through
# email to the user the crontab file belongs to (unless redirected).
#
# For example, you can run a backup of all your user accounts
# at 5 a.m every week with:
# 0 5 * * 1 tar -zcf /var/backups/home.tgz /home/
#
# For more information see the manual pages of crontab(5) and cron(8)
#
# m h  dom mon dow   command

As stated earlier, the default file obviously includes some explanation of how to format an entry in the crontab. In our case we wish to add in an entry that tells the script to start at 6 hours and 0 minutes each day. The crontab accepts six pieces of information that will allow that action to be performed. each of those pieces is separated by a space.

  1. A number (or range of numbers), m, that represents the minute of the hour (valid values 0-59);
  2. A number (or range of numbers), h, that represents the hour of the day (valid values 0-23);
  3. A number (or range of numbers), dom, that represents the day of the month (valid values 0-31);
  4. A number (or list, or range), or name (or list of names), mon, that represents the month of the year (valid values 1-12 or Jan-Dec);
  5. A number (or list, or range), or name (or list of names), dow, that represents the day of the week (valid values 0-6 or Sun-Sat); and
  6. command, which is the command to be run, exactly as it would appear on the command line.

The layout is therefore as follows;

Linux permissions as rwx
Linux permissions as rwx

Assuming that we want to run a Python script called ‘m_temp.py` which was in the ‘pi’ home directory the line that we would want to add would be as follows;

0 6 * * * /usr/bin/python /home/pi/m_temp.py

So at minute 0, hour 6, every day of the month (where the asterisk denotes ‘everything’), every month, every day of the week we run the command /usr/bin/python /home/pi/m_temp.py (which, if we were at the command line in the pi home directory we would run as python m_temp.py, but since we can’t guarantee where we will be when running the script, we are supplying the full path to the python command and the m_temp.py script.

If we wanted to run the command twice a day (6am and 6pm (1800hrs)) we can supply a comma separated value in the hours (h) field as follows;

0 6,18 * * * /usr/bin/python /home/pi/m_temp.py

If we wanted to run the command at 6am but only on weekdays (Monday through Friday) we can supply a range in the dow field as follows (remembering that 0 = Sunday);

0 6 * * 1-5 /usr/bin/python /home/pi/m_temp.py

If we want to run the same command every 2 hours we can use the */2 notation, so that our line in the crontab would look like the following;

0 */2 * * * /usr/bin/python /home/pi/m_temp.py

It’s important to note that we need to include the 0 at the start (instead of the *) so that it doesn’t run every minute every 2 hours (every minute in other words)

Test yourself
  1. How could you set up a schedule job in crontab that ran every second?
  2. Create a crontab line to run a command on the 20th of July every year at 2 minutes past midnight.

ifconfig

The ifconfig command can be used to view the configuration of, or to configure a network interface. Networking is a fundamental function of modern computers. ifconfig allows us to configure the network interfaces to allow that connection.

  • ifconfig [arguments] [interface]

or

  • ifconfig [arguments] interface [options]

Used with no ‘interface’ declared ifconfig will display information about all the operational network interfaces. For example running;

… produces something similar to the following on a simple Raspberry Pi.

eth0      Link encap:Ethernet  HWaddr 76:12:45:56:47:53
          UP BROADCAST MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

wlan0     Link encap:Ethernet  HWaddr 09:87:65:54:43:32
          inet addr:10.1.1.8  Bcast:10.1.1.255  Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:3978 errors:0 dropped:898 overruns:0 frame:0
          TX packets:347 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:859773 (839.6 KiB)  TX bytes:39625 (38.6 KiB)

The output above is broken into three sections; eth0, lo and wlan0.

  • eth0 is the first Ethernet interface and in our case represents the RJ45 network port on the Raspberry Pi (in this specific case on a B+ model). If we had more than one Ethernet interface, they would be named eth1, eth2, etc.
  • lo is the loopback interface. This is a special network interface that the system uses to communicate with itself. You can notice that it has the IP address 127.0.0.1 assigned to it. This is described as designating the ‘localhost’.
  • wlan0 is the name of the first wireless network interface on the computer. This reflects a wireless USB adapter (if installed). Any additional wireless interfaces would be named wlan1, wlan2, etc.
The ifconfig command

The ifconfig command is used to read and manage a servers network interface configuration (hence ifconfig = interface configuration).

We can use the ifconfig command to display the current network configuration information, set up an ip address, netmask or broadcast address on an network interface, create an alias for network interface, set up hardware addresses and enable or disable network interfaces.

To view the details of a specific interface we can specify that interface as an argument;

Which will produce something similar to the following;

eth0      Link encap:Ethernet  HWaddr b8:27:eb:2c:bc:62
          inet addr:10.1.1.8  Bcast:10.1.1.255  Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:119833 errors:0 dropped:0 overruns:0 frame:0
          TX packets:8279 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:8895891 (8.4 MiB)  TX bytes:879127 (858.5 KiB)

The configuration details being displayed above can be interpreted as follows;

  • Link encap:Ethernet - This tells us that the interface is an Ethernet related device.
  • HWaddr b8:27:eb:2c:bc:62 - This is the hardware address or Media Access Control (MAC) address which is unique to each Ethernet card. Kind of like a serial number.
  • inet addr:10.1.1.8 - indicates the interfaces IP address.
  • Bcast:10.1.1.255 - denotes the interfaces broadcast address
  • Mask:255.255.255.0 - is the network mask for that interface.
  • UP - Indicates that the kernel modules for the Ethernet interface have been loaded.
  • BROADCAST - Tells us that the Ethernet device supports broadcasting (used to obtain IP address via DHCP).
  • RUNNING - Lets us know that the interface is ready to accept data.
  • MULTICAST - Indicates that the Ethernet interface supports multicasting.
  • MTU:1500 - Short for for Maximum Transmission Unit is the size of each packet received by the Ethernet card.
  • Metric:1 - The value for the Metric of an interface decides the priority of the device (to designate which of more than one devices should be used for routing packets).
  • RX packets:119833 errors:0 dropped:0 overruns:0 frame:0 and TX packets:8279 errors:0 dropped:0 overruns:0 carrier:0 - Show the total number of packets received and transmitted with their respective errors, number of dropped packets and overruns respectively.
  • collisions:0 - Shows the number of packets which are colliding while traversing the network.
  • txqueuelen:1000 - Tells us the length of the transmit queue of the device.
  • RX bytes:8895891 (8.4 MiB) and TX bytes:879127 (858.5 KiB) - Indicates the total amount of data that has passed through the Ethernet interface in transmit and receive.
Options

The main option that would be used with ifconfig is -a which will will display all of the interfaces on the interfaces available (ones that are ‘up’ (active) and ‘down’ (shut down). The default use of the ifconfig command without any arguments or options will display only the active interfaces.

Arguments

We can disable an interface (turn it down) by specifying the interface name and using the suffix ‘down’ as follows;

Or we can make it active (bring it up) by specifying the interface name and using the suffix ‘up’ as follows;

To assign a IP address to a specific interface we can specify the interface name and use the IP address as the suffix;

To add a netmask to a a specific interface we can specify the interface name and use the netmask argument followed by the netmask value;

To assign an IP address and a netmask at the same time we can combine the arguments into the same command;

Test yourself
  1. List all the network interfaces on your server.
  2. Why might it be a bad idea to turn down a network interface while working on a server remotely?
  3. Display the information about a specific interface, turn it down, display the information about it again then turn it up. What differences do you see?

ls

The ls command lists the contents of a directory and can show the properties of those objects it lists. It is one of the fundamental commands for knowing what files are where and the properties of those files.

  • ls [options] directory : List the files in a particular directory

For example: If we execute the ls command with the -l option to show the properties of the listings in long format and with the argument /var so that it lists the content of the /var directory…

… we should see the following;

pi@raspberrypi ~ $ ls -l /var
total 102440
drwxr-xr-x  2 root     root          4096 Mar  7 06:25 backups
drwxr-xr-x 12 root     root          4096 Feb 20 08:33 cache
drwxr-xr-x 43 root     root          4096 Feb 20 08:33 lib
drwxrwsr-x  2 root     uucp          4096 Jan 11 00:02 local
lrwxrwxrwx  1 root     root             9 Feb 15 11:23 lock -> /run/lock
drwxr-xr-x 11 root     root          4096 Jul  7 06:25 log
drwxrwsr-x  2 root     mail          4096 Feb 15 11:23 mail
drwxr-xr-x  2 root     root          4096 Feb 15 11:23 opt
lrwxrwxrwx  1 root     root             4 Feb 15 11:23 run -> /run
drwxr-xr-x  4 root     root          4096 Feb 15 11:26 spool
-rw-------  1 root     root     104857600 Feb 16 14:03 swap
drwxrwxrwt  2 root     root          4096 Jan 11 00:02 tmp
drwxrwxr-x  2 www-data www-data      4096 Feb 20 08:21 www
The ls command

The ls command will be one of the first commands that someone starting with Linux will use. It is used to list the contents of a directory (hence ls = list). It has a large number of options for displaying listings and their properties in different ways. The arguments used are normally the name of the directory or file that we want to show the contents of.

By default the ls command will show the contents of the current directory that the user is in and just the names of the files that it sees in the directory. So if we execute the ls command on its own from the pi users home directory (where we would be after booting up the Raspberry Pi), this is the command we would use;

… and we should see the following;

pi@raspberrypi ~ $ ls
Desktop  python_games

This shows two directories (Desktop and python_games) that are in pi’s home directory, but there are no details about the directories themselves. To get more information we need to include some options.

Options

There are a very large number of options available to use with the ls command. For a full listing type man ls on the command line. Some of the most commonly used are;

  • -l gives us a long listing (as explained above)
  • -a shows us aLL the files in the directory, including hidden files
  • -s shows us the size of the files (in blocks, not bytes)
  • -h shows the size in “human readable format” (ie: 4K, 16M, 1G etc). (must be used in conjunction with the -s option).
  • -S sorts by file Size
  • -t sorts by modification time
  • -r reverses order while sorting

A useful combination of options could be a long listing (-l) that shows all (-a) the files with the file size being reported in human readable (-h) block size (-s).

… will produce something like the following;

pi@raspberrypi ~ $ ls -lash
total 84K
4.0K drwxr-xr-x 13 pi   pi   4.0K May  7 11:46 .
4.0K drwxr-xr-x  3 root root 4.0K May  7 10:20 ..
4.0K -rw-r--r--  1 pi   pi     69 May  7 11:46 .asoundrc
4.0K -rw-------  1 pi   pi    854 Jul  8 12:55 .bash_history
4.0K -rw-r--r--  1 pi   pi   3.2K May  7 10:20 .bashrc
4.0K drwxr-xr-x  4 pi   pi   4.0K May  7 11:46 .cache
4.0K drwxr-xr-x  7 pi   pi   4.0K May  7 11:46 .config
4.0K drwxr-xr-x  2 pi   pi   4.0K May  7 11:46 Desktop
4.0K drwxr-xr-x  2 pi   pi   4.0K May  7 11:46 .fontconfig
4.0K drwxr-xr-x  2 pi   pi   4.0K May  7 11:46 .gstreamer-0.10
4.0K drwx------  3 pi   pi   4.0K May  7 11:46 .local
4.0K -rw-r--r--  1 pi   pi    675 May  7 10:20 .profile
4.0K drwxrwxr-x  2 pi   pi   4.0K Jan 27 21:34 python_games
4.0K drwxr-xr-x  3 pi   pi   4.0K May  7 11:46 .themes
Arguments

The default argument (if none is included) is to list the contents of the directory that the user is currently in. Otherwise we can specify the directory to list. This might seem like a simple task, but there are a few tricks that can make using ls really versatile.

The simplest example of using a specific directory for an argument is to specify the location with the full address. For example, if we wanted to list the contents of the /var directory (and it doesn’t matter which directory we run this command from) we simply type;

… will produce the following;

pi@raspberrypi ~ $ ls /var
backups  cache  lib  local  lock  log  mail  opt  run  spool  swap  tmp  www

We can also use some of the relative addressing characters to shortcut our listing. We can list the home directory by using the tilde (ls ~) and the parent directory by using two full stops (ls ..).

The asterisk (*) can be used as a wildcard to list files with similar names. E.g. to list all the png file in a directory we can use ls *.png.

If we just want to know the details of a specific file we can use its name explicitly. For example if we wanted to know the details of the swap file in /var we would use the following command;

… which will produce the following;

pi@raspberrypi ~ $ ls -l /var/swap
-rw------- 1 root root 104857600 May  7 11:29 /var/swap
Examples

List all the configuration (.conf) files in the /etc directory;

… which will produce the following;

pi@raspberrypi ~ $ ls /etc/*.conf
/etc/adduser.conf          /etc/host.conf       /etc/ntp.conf
/etc/ca-certificates.conf  /etc/idmapd.conf     /etc/pam.conf
/etc/debconf.conf          /etc/insserv.conf    /etc/resolv.conf
/etc/deluser.conf          /etc/ld.so.conf      /etc/resolvconf.conf
/etc/dhcpcd.conf           /etc/libaudit.conf   /etc/rsyslog.conf
/etc/fuse.conf             /etc/logrotate.conf  /etc/sysctl.conf
/etc/gai.conf              /etc/mke2fs.conf     /etc/ts.conf
/etc/gssapi_mech.conf      /etc/nsswitch.conf   /etc/ucf.conf

modprobe

The modprobe command allows us to add (or remove) modules to the Linux Kernel. The Linux kernel is the code that forms the core of the Linux operating system, so it’s kind of important. When changing hardware, the modprobe command allows us to import or remove the equivalent of windows device drivers to / from the kernel to enable / disable additional functionality.

  • modprobe [options] [modulename] : Load or remove a Linux kernel module

For example to add the module w1-therm to support measurement of temperature via the 1-Wire bus we would execute the following command;

The modprobe command

The Linux kernel is designed with a monolithic structure, but with the ability to be able to change kernel modules while running. (Windows 7 and OS X use hybrid kernels which offer the advantage of being smaller in size, but they require more management of the drivers by the user and manufacturer).

To work around the disadvantage of having a large footprint, Linux kernel developers have incorporated the facility to add or remove kernel modules on the fly. This can be taken to the extreme where the the entire kernel module can be replaced without needing to reboot.

Kernel modules are typically located in the lib/modules directory and can be listed using the following command;

An output might look something like the following;

pi@raspberrypi /lib/modules $ ls /lib/modules/$(uname -r)
kernel             modules.builtin      modules.dep.bin  modules.softdep
modules.alias      modules.builtin.bin  modules.devname  modules.symbols
modules.alias.bin  modules.dep          modules.order    modules.symbols.bin

We can also list all the loaded modules using the command lsmod as follows;

A sample output might looks similar to the following;

pi@raspberrypi /lib/modules $ lsmod
Module                  Size  Used by
w1_therm                2559  0
wire                   25680  1 w1_therm
cn                      4636  1 wire
rfkill                 16651  1 cfg80211
i2c_dev                 6027  0
snd_bcm2835            18649  0
snd_pcm                73475  1 snd_bcm2835
snd_seq                53078  0
snd_seq_device          5628  1 snd_seq
snd_timer              17784  2 snd_pcm,snd_seq
snd                    51038  5 snd_bcm2835,snd_timer,snd_pcm,snd_seq,snd_seq\
_device
i2c_bcm2708             4990  0

We can also see the range of drivers that are available via the command;

Which would provide an output similar to the following;

pi@raspberrypi /lib/modules $ ls /lib/modules/$(uname -r)/kernel/drivers/
base       cdrom      extcon  input  mfd   nfc    rtc   staging  w1
bcma       char       gpio    leds   misc  of     scsi  uio      watchdog
block      connector  hid     md     mmc   power  spi   usb
bluetooth  cpufreq    i2c     media  net   pps    ssb   video
Options

The modprobe command has several options, but the vast majority of users will simply need to install a module which is done using the sudo modprobe [modulename] command (no options needed).

The only realistic command option that a novice user might use would be -r which would remove a module.

Arguments

The module name is the main argument used when executing the modprobe command. Multiple modules can be loaded by simply putting a space between the module names.

Test yourself
  1. What type of module is used in the Linux kernel vs the Windows kernel and what is the advantage of the Linux kernel approach
  2. Show how the modules mod1, mod2 and mod3 can all be loaded using one use of the modprobe command.

ping

The ping command allows us to check the network connection between the local computer and a remote server. It does this by sending a request to the remote server to reply to a message (kind of like a read-request in email). This allows us to test network connectivity to the remote server and to see if the server is operating. The ping command is a simple and commonly used network troubleshooting tool.

  • ping [options] remote server : checks the connection to a remote server.

To check the connection to the server at CNN for example we can simple execute the following command (assuming that we have a connection to the internet);

Which will return something like the following;

PING cnn.com (157.166.226.25) 56(84) bytes of data.
64 bytes from www.cnn.com (157.166.226.25): icmp_seq=1 ttl=111 time=265 ms
64 bytes from www.cnn.com (157.166.226.25): icmp_seq=2 ttl=111 time=257 ms
64 bytes from www.cnn.com (157.166.226.25): icmp_seq=3 ttl=111 time=257 ms
64 bytes from www.cnn.com (157.166.226.25): icmp_seq=4 ttl=111 time=258 ms
64 bytes from www.cnn.com (157.166.226.25): icmp_seq=5 ttl=111 time=257 ms
64 bytes from www.cnn.com (157.166.226.25): icmp_seq=6 ttl=111 time=257 ms
^C
--- cnn.com ping statistics ---
14 packets transmitted, 13 received, 7% packet loss, time 13016ms
rtt min/avg/max/mdev = 257.199/267.169/351.320/24.502 ms

The first thing to note is that by default the ping command will just keep running. When we want to stop it we need to press CTRL-c to get it to stop.

The information presented is extremely useful and tells us that www.cnn.com’s IP address is 157.166.226.25 and that the time taken for a ping send and return message took about 250 milliseconds.

The ping command

The ping command is a very simple network / connectivity checking tool that is one of the default ‘go-to’ commands for system administrators. You might be wondering about how the name has come about. It is reminiscent of the echo-location technique used by dolphins, whales and bats to send out a sound and to judge their surroundings by the returned echo. In the dramatised world of the submariner, a ping is the sound emitted by a submarine in the same way to judge the distance and direction to an object. It was illustrated to best effect in the book by Tom Clancy and the subsequent movie “The Hunt for Red October” where the submarine commander makes the request for “One Ping Only”.

One Ping Only
One Ping Only

It works by sending message called an ‘Echo Request’ to a specific network location (which we specify as part of the command). When (or if) the server receives the request it sends an ‘Echo Reply’ to the originator that includes the exact payload received in the request. The command will continue to send and (hopefully) receive these echoes until the command completes its requisit number of attempts or the command is stopped by the user (with a CTRL-c). Once complete, the command summarises the effort.

From the example used above we can see the output as follows;

PING cnn.com (157.166.226.25) 56(84) bytes of data.
64 bytes from www.cnn.com (157.166.226.25): icmp_seq=1 ttl=111 time=265 ms
64 bytes from www.cnn.com (157.166.226.25): icmp_seq=2 ttl=111 time=257 ms
64 bytes from www.cnn.com (157.166.226.25): icmp_seq=3 ttl=111 time=257 ms
64 bytes from www.cnn.com (157.166.226.25): icmp_seq=4 ttl=111 time=258 ms
64 bytes from www.cnn.com (157.166.226.25): icmp_seq=5 ttl=111 time=257 ms
64 bytes from www.cnn.com (157.166.226.25): icmp_seq=6 ttl=111 time=257 ms
^C
--- cnn.com ping statistics ---
14 packets transmitted, 13 received, 7% packet loss, time 13016ms
rtt min/avg/max/mdev = 257.199/267.169/351.320/24.502 ms

We can see from the returned pings that the IP address of the server that is designated as ‘www.cnn.com’ is ‘157.166.226.25’ The resolution of the IP address would be made possible by DNS, but using a straight IP address is perfectly fine). The icmp_seq= column tells us the sequence of the returned replies and ttl indicates how many IP routers the packet can go through before being thrown away. The time provides the measured return trip of the request and reply.

The summary at completion tells us how many packets were sent and how many received back. This forms a percentage of lost packets which is established over the specified time. The final line provides a minimum, average maximum and standard deviation from the mean.

Options

There are a few different options for use, but the more useful are as follows;

  • -c only ping the connection a certain number (count) of times
  • -i change the time interval between pings

It’s really useful to have ping running continuously so that we can make changes to networking while watching the results, but it’s also useful to run the command for a limited amount of time. This is where the -c option comes in. This will simply restrict the number of pings that are sent out and will then cease and summarise the effort. This can be used as follows;

Which will return something like the following;

PING cnn.com (157.166.226.25) 56(84) bytes of data.
64 bytes from www.cnn.com (157.166.226.25): icmp_seq=1 ttl=111 time=266 ms
64 bytes from www.cnn.com (157.166.226.25): icmp_seq=2 ttl=111 time=263 ms
64 bytes from www.cnn.com (157.166.226.25): icmp_seq=3 ttl=111 time=288 ms
64 bytes from www.cnn.com (157.166.226.25): icmp_seq=4 ttl=111 time=280 ms

--- cnn.com ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3003ms
rtt min/avg/max/mdev = 263.283/274.586/288.637/10.386 ms

Sometimes it can be convenient to set our own time interval between pings. This can be accomplished with the -i option which will let us vary the repeat time. The default is 1 second, however the value cannot be set below 0.2 seconds without doing so as the superuser. Interestingly there is an option to flood the network with pings (flood mode) to test the network infrastructure. However, this would be something typically left to research carefully when you really need it.

Test yourself
  1. How does the ping command to a server name know how to return an IP address?
  2. What does ‘ttl’ stand for?

sudo

The sudo command allows a user to execute a command as the ‘superuser’ (or as another user). It is a vital tool for system administration and management.

  • sudo [options] [command] : Execute a command as the superuser

For example, if we want to update and upgrade our software packages, we will need to do so as the super user. All we need to do is prefix the command apt-get with sudo as follows;

One of the best illustrations of this is via the excellent cartoon work of the xkcd comic strip (Buy his stuff, it’s awesome!).

Sudo courtesy xkcd
Sudo courtesy xkcd
The sudo command

The sudo command is shorthand for ‘superuser do’.

When we use sudo an authorised user is determined by the contents of the file /etc/sudoers.

As an example of usage we should check out the file /etc/sudoers. If we use the cat command to list the file like so;

We get the following response;

pi@raspberrypi ~ $ cat /etc/sudoers
cat: /etc/sudoers: Permission denied

That’s correct, the ‘pi’ user does not have permissions to view the file

Let’s confirm that with ls;

Which will result in the following;

pi@raspberrypi ~ $ ls -l /etc/sudoers
-r--r----- 1 root root 696 May  7 10:39 /etc/sudoers

It would appear that only the root user can read the file!

So let’s use sudo to cat](#cat) the file as follows;

That will result in the following output;

pi@raspberrypi ~ $ sudo cat /etc/sudoers
#
# This file MUST be edited with the 'visudo' command as root.
#
# Please consider adding local content in /etc/sudoers.d/ instead of
# directly modifying this file.
#
# See the man page for details on how to write a sudoers file.
#
Defaults        env_reset
Defaults        mail_badpass
Defaults        secure_path="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bi\
n:/sbin:/bin"

# Host alias specification

# User alias specification

# Cmnd alias specification

# User privilege specification
root    ALL=(ALL:ALL) ALL

# Allow members of group sudo to execute any command
%sudo   ALL=(ALL:ALL) ALL

# See sudoers(5) for more information on "#include" directives:

#includedir /etc/sudoers.d
pi ALL=(ALL) NOPASSWD: ALL

There’s a lot of information in the file, but there, right at the bottom is the line that determines the privileges for the ‘pi’ user;

pi ALL=(ALL) NOPASSWD: ALL

We can break down what each section means;

pi

pi ALL=(ALL) NOPASSWD: ALL

The pi portion is the user that this particular rule will apply to.

ALL

pi ALL=(ALL) NOPASSWD: ALL

The first ALL portion tells us that the rule applies to all hosts.

ALL

pi ALL=(ALL) NOPASSWD: ALL

The second ALL tells us that the user ‘pi’ can run commands as all users and all groups.

NOPASSWD

pi ALL=(ALL) NOPASSWD: ALL

The NOPASSWD tells us that the user ‘pi’ won’t be asked for their password when executing a command with sudo.

All

pi ALL=(ALL) NOPASSWD: ALL`

The last ALL tells us that the rules on the line apply to all commands.

Under normal situations the use of sudo would require a user to be authorised and then enter their password. By default the Raspbian operating system has the ‘pi’ user configured in the /etc/sudoers file to avoid entering the password every time.

If your curious about what privileges (if any) a user has, we can execute sudo with the -l option to list them;

This will result in output that looks similar to the following;

pi@raspberrypi ~ $ sudo -l
Matching Defaults entries for pi on this host:
    env_reset, mail_badpass,
    secure_path=/usr/local/sbin\:/usr/local/bin\:/usr/sbin\:/usr/bin\:/sbin\:\
/bin

User pi may run the following commands on this host:
    (ALL : ALL) ALL
    (ALL) NOPASSWD: ALL
The ‘sudoers’ file

As mentioned above, the file that determines permissions for users is /etc/sudoers. DO NOT EDIT THIS BY HAND. Use the visudo command to edit. Of course you will be required to run the command using sudo;

sudo vs su

There is a degree of confusion about the roles of the sudo command vs the su command. While both can be used to gain root privileges, the su command actually switches the user to another user, while sudo only runs the specified command with different privileges. While there will be a degree of debate about their use, it is widely agreed that for simple on-off elevation, sudo is ideal.

Test yourself
  1. Write an entry for the sudoers file that provides sudo priviledges to a user for only the cat command.
  2. Under what circumstances can you edit the sudoers file with a standard text editor.

Directory Structure Cheat Sheet

  • / : The ‘root’ directory which contains all other files and directories
  • /bin : Common commands / programs, shared by all users
  • /boot : Contains the files needed to successfully start the computer during the boot process
  • /dev : Holds device files that represent physical and ‘logical’ devices
  • /etc : Contains configuration files that control the operation of programs
  • /etc/cron.d: One of the directories that allow programs to be run on a regular schedule
  • /etc/rc?.d : Directories containing files that control the mode of operation of a computer
  • /home : A directory that holds subdirectories for each user to store user specific files
  • /lib : Contains shared library files and kernel modules
  • /lost+found : Will hold recoverable data in the event of an an improper shut-down
  • /media : Used to temporarily mount removable devices
  • /mnt : A mount point for filesystems or temporary mount point for system administrators
  • /opt : Contains third party or additional software that is not part of the default installation
  • /proc : Holds files that contain information about running processes and system resources
  • /root : The home directory of the System Administrator, or the ‘root’ user
  • /sbin : Contains binary executables / commands used by the system administrator
  • /srv : Provides a consistent location for storing data for specific services
  • /tmp : A temporary location for storing files or data
  • /usr : Is the directory where user programs and data are stored and shared
  • /usr/bin : Contains binary executable files for users
  • /usr/lib : Holds shared library files to support executables in /usr/bin and /usr/sbin
  • /usr/local : Contains users programs that are installed locally from source code
  • /usr/sbin : The directory for non-essential system administration binary executables
  • /var : Holds variable data files which are expected to grow under normal circumstances
  • /var/lib : Contains dynamic state information that programs modify while they run
  • /var/log : Stores log files from a range of programs and services
  • /var/spool : Contains files that are held (spooled) for later processing
  • /var/tmp : A temporary store for data that needs to be held between reboots (unlike /tmp)