Table of Contents

• Introduction
  • About Me
  • Conventions
  • Terminology
  • Prerequisites
  • The Scenario
• 1 Configuration and Vagrant
  • 1.1 Go Headless
  • 1.2 Making configuration easier
  • 1.3 Using a pre-made Vagrantfile
• 2 File Sharing and SSHFS
  • 2.1 Installing SSHFS
  • 2.2 Mounting a guest directory to the host with SSHFS
• 3 Integration and Apollo
  • 3.1 Building And Installing Apollo
  • 3.2 Using Apollo
• 4 Where to Go From Here

Introduction

You may be a Web developer working on a Macbook. You may work for a company whose back-end depends upon scripts that are written only to work on Debian Linux. You may find out that the practical way for you to work with your company’s tools is by installing Linux on a VM (Virtual Machine), and running the company code inside that VM.

You may be reluctant to spend two whole days setting up the VM and figuring out how to comfortably communicate with it.

This book is meant to help you set up and interact with a new VM running Linux in a matter of hours rather than days. The goal is to make the experience of working with, and working within, your VM as close as possible to the experience of working from your native shell.

A working knowledge of the command-line is essential to understanding the examples in this book. If you are a full-stack or a back-end developer you will have no problem. If you are a front-end developer who sweats at the mere mention of the command-line, I promise you, you’re going to be okay :). Learning to use the command-line is not only essential to your work as a Web developer, but will expand your abilities to succeed in front-end development immensely. There are plenty of tutorials that can be found on Google to teach basic command-line usage. Here’s one from Tuts+

I’ve distilled the frustrations I’ve experienced when working with VMs in to 3 basic problems:

• Configuration
• File Sharing
• Integration

The next three chapters of this book will explain each of these problems, and then present a solution to the problem with an example of that solution applied.

About Me

I work as a full-stack Web developer at Space Monkey, Inc. Our back-end private API is written mostly in Python, and it has many dependencies which we’ve prepared for distribution to the servers as Debian packages. The most straightforward way for me to get our API (Application Programming Interface) up and running locally for development was to do it on a Virtual Machine; I work on a Macbook Pro running OS X which isn’t compatible with Debian packages. (Debian is a Linux operating system, and Debian packages are a standardized method of installing, configuring, and managing software). I got so frustrated with the large number of steps necessary not only to set up a VM, but to work within one that I dedicated a number of days to simplifying the process. Now when I need to make a new VM it’s a matter of minutes rather than hours.

Conventions

Most of the figures in this book will be examples of commands that should be issued into your terminal. All terminal commands will appear in code blocks, and will be preceded by the ‘>’ character at the beginning of the line like so:

> echo "Hello, World!"

Lines that end with a backslash: \ will be joined by the shell with the line that comes after. This happens when a single command is too long to fit on one line:

> this command is so very long that I can\
barely keep it in my head!

I’ll accompany code with comments that are meant solely for clarification and should not be run in the terminal. These comments will be preceded by a ‘#’ character like so:

# This is a comment.
# Don't enter it into the command line!

If a line is not begun with the characters >, or #, and is not preceded by a line ending in \, then it shows the shell’s response to the command entered:

> print "I'm printing a command!"
I'm printing a command!

Terminology

I’ll use lots of abbreviations and acronyms in this book. The first time I use an acronym, I’ll write its expanded form in parenthesis, like this: AAG (Acronyms Are Great). For your convenience, here’s a short list of acronyms, abbreviations, and potentially confusing terms that I use in this book:

CLI

Command-line interface. A textual interface to a tool that is meant to run on the command-line.

guest

An operating system running in a VM.

GUI

A graphical user interface.

host

An operating system running VM software.

OS

Operating System.

provider

Software that runs a VM on a host.

repo

A source-code repository. A location on a local computer or on a server where source code changes are tracked and stored.

VM

Virtual Machine. An emulated environment to run one OS inside of another.

Prerequisites

I’ll be making regular use of the Git revision control system in this book. If you want to follow along with the examples I suggest visiting the Git Web site. to learn how to install and use Git.

Homebrew is a package manager built for OS X that I will also be using in the examples to install the dependencies that Apollo has. To learn about installation and basic usage of Homebrew I refer you to the Homebrew Web site.

As I mentioned above in the introduction, the tools in this book are all used from the command-line. If you aren’t comfortable in the command-line, I suggest reading a few tutorials and practicing a little before diving into this book.

In order to run a VM, software is necessary that emulate the hardware that a certain OS (Operating System) expects to encounter. There are a variety of VM software packages (providers) available for Mac. This guide will use Virtual Box, a free provider that’s actively developed and supported by Oracle. However, this book should work with any providers that are supported by Vagrant. More information about Vagrant can be found in Chapter 1.

The Scenario

The scenario for the examples in this book is a simplified version of a Web development task. Explaining the complex back-end we use at Space Monkey wouldn’t be appropriate for this short book. Instead, I’ll show how to use a VM to run a new tool called Docker, which will help me set up a fresh instance of the Ghost blogging platform in a Docker container.

This book is not a Docker tutorial. It only aims to make working with VMs easier. I’m using Docker in the examples, however, because it’s a new tool that’s rapidly growing in popularity. Since it only runs on Linux, developers who use a Mac need run Docker within a VM.

Docker containers are similar to VMs in that a container is a self-contained copy of a Linux OS. But containers can only be used to run a Linux OS within another Linux OS.

In the end, we’ll have a Docker container running inside of a VM with Ubuntu installed, which will be managed by Vagrant, running on OS X. See Figure A for a graphic that shows the final machine configuration that the example will lead to.

Figure A: The example will end with a Docker container running inside a Ubuntu VM, managed by Vagrant, and running on OS X

1 Configuration and Vagrant

1.1 Go Headless

Before I get into the meat of the configuration problem, there’s an important point that I need to address. When developing, VMs are best used from the command-line only.

When I began to work on VMs, I tried to use them through the Virtualbox GUI (Graphical User Interface). I’d install Linux with Virtual Box, and then I’d boot up the VM and work within a fully graphical virtual environment, as if a window were cut through OS X, looking in on Linux. There are problems with this arrangement. By running a window manager (a graphically heavy user interface like Gnome) in the VM, I was wasting a lot of computing power on driving the graphics for that guest interface. Also, in order for the guest OS to look good, I had to install the Virtualbox Guest Additions, which usually took anywhere between fifteen minutes and three hours to figure out depending on how good the community written guides were that I could find. Also, using a guest OS GUI just feels clunky. It wasn’t engineered to work with the Mac hardware like OS X is. And furthermore, I’d have to spend time installing all of the shell and command-line customizations that I’m used to in the new VM before I could feel comfortable working there.

There’s a better way.

Virtualbox can be run in what’s called headless mode. That means the guest OS runs in the background without needing any GUI attached to it. That way, we can get a version of the OS that doesn’t even include a window manager, which means fewer megabytes to download. Yay! And then we can set it up to run in the background listening for SSH (Secure Shell login for remote control) connections. This allows us to log in and use the guest OS through just the command-line. Much speedier!

1.2 Making configuration easier

My first hour or two setting up a VM at work was spent browsing the Web to find the download link for an installation disk image of the Linux distribution I wanted, downloading it (that kind of file is typically between 500 and 800 megabytes), and wading through the menus, forms, and fields of the installation process. The next few hours were spent trying to figure out how to correctly install the Virtual Box add-ons (necessary to make the VM get along nicely with the host.) and configure network interfaces so that the VM had Internet access, and so that I could SSH in to it.

Vagrant is a piece of software written by Mitchell Hashimoto that automates all of the chores mentioned above, and more. A user can write a small script, called a Vagrantfile that specifies what guest OS to install on a new VM, what the network configuration should be, what actions to take on first launch, and much more.

Instructions for installation and basic use of Vagrant can be found on the Vagrant Web site.

1.3 Using a pre-made Vagrantfile

Here’s where the example starts, and it will stretch across the next two chapters.

Getting the Docker source code

I’m going to clone the Docker source code from Github (Github is a service which hosts public Git repositories, also knows as a ‘repo’ for free).

The Docker source has a Vagrantfile that the Docker developers have prepared which sets up a new Ubuntu VM, installs Docker, configures the VM for SSH access, and forwards a range of ports from the host to the guest. The port forwarding means that when a service is run on the VM that listens on one of those open ports, 49000 for example, a web request to localhost:49000 on the host will be forwarded to the service listening from the virtual machine. That way, I can run the server for the website that I’m working on inside the VM, but view it from the browser in OS X.

Here I create a new directory for the Docker source to be cloned to. I usually put the projects that I get from Github into a folder called github in my home folder.

> cd ~/Github

Then I clone Docker from the source repository on Github:

> git clone https://github.com/dotcloud/docker.git

That will clone the source for Docker into a new folder called docker. I’ll make it a little bit easier to remember where the Docker source lives on disk by creating an environment variable that points to that directory. I’ll edit the file ~/.zshrc and add the line:

export DOCKERSRC=“~/Github/docker"

That way my alias will get restored every time a new shell starts.

ZSH or BASH I’m running zsh and I highly recommend it, as do others. If you’re using BASH (Bourne-again Shell) which is the default on OS X, you’ll need to change ~/.zshrc to ~/.bashrc.)

Now, I can get back to the Docker source directory any time I want to just by typing:

> cd $DOCKERSRC

Setting up the VM for the first time

Vagrant knows to start up (or set up) a VM when it receives the command vagrant up. It will first check to see if the environment variable VAGRANT_CWD is set. If it is, then Vagrant will look in the directory pointed to by that variable for a file called Vagrantfile. If the variable is not set, it will look for the Vagrantfile in the current directory. I will only be using this single VM for development, and I don’t want to have to be in the $DOCKERSRC directory every time I use vagrant to manage my VM, so I’ll add another line to my ~/.zshrc: export VAGRANT_CWD=$DOCKERSRC.

Make sure to add the line that sets VAGRANT_CWD after the line that defines DOCKERSRC.

Essential Step Later in the book, Apollo will rely on the ability to call Vagrant from any directory. If the VAGRANT_CWD environment variable isn’t set, Apollo won’t work.

Now, there’s one last line to add to the zshrc before starting up the VM for the first time. The environment variable FORWARD_DOCKER_PORTS indicates to Vagrant that we want to forward a block of ports as described in the introduction to this chapter. By default, it is set to 0, which means that Vagrant won’t forward those ports. I’ll add the following line to my ~/.zshrc: export FORWARD_DOCKER_PORTS=1.

Essential Step If this environment variable isn’t set, I won’t be able to communicate with any of the software that I run inside the VM from OS X.

Okay, to recap, my ~/.zshrc now has a section that looks like the following code block.

# For easy access to the Docker source
export DOCKERSRC=~/Github/docker

# Make Vagrant always look in the Docker source
# directory for the Vagrantfile
export VAGRANT_CWD=$DOCKERSRC

# Instruct Vagrant to forward a block of ports
export FORWARD_DOCKER_PORTS=1

i> FORWARD_DOCKER_PORTS is specific only to the Docker Vagrantfile. There’s custom code in the Docker Vagrantfile that checks for the value of that environment variable and instructs Vagrant to forward all of those ports if it’s enabled. Vagrantfiles are interpreted with Ruby, so you can write Ruby code when writing your Vagrantfiles. Nifty!

Now I’m ready. I just use one command, and then sit back and let Vagrant download Ubuntu, install it, configure it, and install Docker:

vagrant up

2 File Sharing and SSHFS

The next big frustration I faced when working with VMs came when I tried to work with my source code. I had to make it available for the VM to run it. If I put the code inside the VM, I could run it without an issue, but that meant I either had to edit it inside the VM, or copy one file at a time over to OS X, edit it, and then copy it back. I like using Sublime Text for a lot of my coding. I work on a Macbook Pro Retina, and Sublime Text has accelerated performance on the retina screen, which makes for an enjoyable editing experience. It also has some excellent features for fast file switching that I use constantly. Moving the source code into the VM meant that I had to either give up Sublime Text, or give up the fast file switching functionality. Neither of these options was acceptable to me.

If the code can’t be in the VM, the next logical place to put it on the host (OS X). In fact, Vagrant can automatically set up file sharing between the host and the VM, but there’s a catch.

Mac OS X and Linux speak different languages when it comes to understanding how to store files on a hard-drive. OS X uses a file system called HFS+, and my Ubuntu VM uses a file system called Ext4. HFS+ is a case-insensitive file system, while Ext4 is case-sensitive. With Vagrant’s default file sharing, the files are shared from OS X to the VM. That means that the VM is reading files through a case-insensitive translation. Much of the time, this won’t be an issue. But for our back-end code at Space Monkey, it was an issue. I encountered all kinds of runtime errors when I tried to use them through Vagrant’s default file sharing.

Enter SSHFS.

SSHFS (Secure Shell File System) is an open-source tool that builds a virtual file system and keeps it in sync across the same kind of connection that I’m using to control the VM: SSH. SSHFS is just one of many solutions that have been invented to solve this problem, but I found SSHFS to be the easiest to set up, and so fast that I barely even notice that I’m using it.

I’m going to use SSHFS to mount the home directory from my VM into a folder on OS X. That way, I can write to, and read from any location under my VM’s home folder from within OS X. Case sensitivity is satisfied, because the code lives on the VM. And I’m satisfied because I can easily edit the code with Sublime Text.

2.1 Installing SSHFS

Building software can be a big pain in the neck, but Homebrew on the Mac makes it easy. If you don’t know what Homebrew is, I gave a short explanation in the Introduction to this book, and I invite you to give it a read.

Installing SSHFS with Homebrew is simple:

> install sshfs

Homebrew will download the source for SSHFS, build it, install it, and make it runnable from the command-line.

During the installation, Homebrew printed out a section called Caveats that looks like this:

==> Caveats
Make sure to follow the directions given by
	`brew info fuse4x-kext`
before trying to use a FUSE-based filesystem.

Watch out for those Caveats from Homebrew. It usually means that you need to take extra steps to finish the installation process.

So I issue the command:

> brew info fuse4x-kext

And I’m informed that I need to run the following commands in order to complete the installation.

sudo /bin/cp -rfX /usr/local/Cellar/\
fuse4x-kext/0.9.2/Library/Extensions/\
fuse4x.kext /Library/Extensions
sudo chmod +s /Library/Extensions/fuse4x.kext/\
Support/load_fuse4x

I copy, paste, and execute those commands, and then my installation is done. I can check to make sure I have access to the command by typing:

> sshfs

If I see output like this:

> sshfs
missing host
see `sshfs -h' for usage

That’s good! It means I have access to the sshfs command now.

2.2 Mounting a guest directory to the host with SSHFS

Now, I’m going to connect a folder on OS X with a folder on my VM. This process is called mounting. In other words, I’ll mount the folder from my VM onto OS X. In order to do that, I’ll make a folder on OS X to hold the files from my VM, and then I’ll issue a command to set up the connection. Once I’m done with the connection I can unmount it, and the folder that I made will once again appear empty.

# Make a folder to mount the VM into.
> mkdir ~/vmfiles

# Mount with the sshfs command:
> sshfs vagrant@localhost: ~/vmfiles -p 2222 -o follow_symlinks
# I'll have to enter the password for the 
# VM now, which in this case is "vagrant".

Now I’ll break that command down. Here are the parts:

sshfs: The executable

vagrant@localhost:: Means login to the host localhost as the user vagrant and the colon at the end means that I’m going to be mounting the home folder of the user named vagrant.

~/vmfiles: Is the folder I created to mount the VM files into.

-p 2222: The VM is hosted on localhost and it’s listening for incoming SSH connections on port 2222. If I don’t pass in the port here, I won’t get a connection.

-o follow_symlinks: Means that if there’s a symbolic link that exists in the directory we’re mounting from the VM, we should treat it as a link just like Linux would, instead of viewing it as a useless file with no contents.

And now, if I change to my ~/vmfiles directory, I should see all of the files from my VM’s home folder.

Using ssh-copy-id

Right now when I connect to the VM over SSH, I’m prompted to enter the vagrant user’s password, which is “vagrant”. But I don’t want to type this over and over again. ssh-copy-id is a little script that will tell the VM to remember who I am after the first login, and let me in without a password forever after.

I just have to install it and connect to the VM with it once. Then I’ll be remembered when I next try to log in.

# Install ssh-copy-id with Homebrew
> brew install ssh-copy-id

# Login once with it.
> ssh-copy-id -p 2222 vagrant@localhost

3 Integration and Apollo

The last big hurdle I faced when learning to work with VMs was that I didn’t want to feel like I was working on a VM. I have enough to think about when I’m programming that I don’t want to have to remember which custom tools I have installed on my VM and which I don’t. I just want to work as if I’m working on the host, which I’ve already spent a large amount of time customizing to my development style.

Mounting the VM files on OS X, which is explained in chapter 2, is a huge step in the right direction because I can browse those files from my OS X shell, and I can use my favorite programs (like ag) to operate on those files.

The last step for me was to be able to issue commands on the VM without leaving the comfort of my home shell. So I wrote an open-source tool called Apollo which allows me to do it.

Apollo wraps up Vagrant and SSH into one bundle. Once Apollo, Vagrant, and SSHFS are installed, and Vagrant is configured to run a VM, Apollo can:

• Start up the VM and mount its files
• Shut down the VM and unmount its files
• Issue a single command to the VM
• Instantly connect the user to an interactive SSH session on the VM

3.1 Building And Installing Apollo

Apollo is written in Golang, Google’s popular new language. It’s a compiled language, so I’ll show in this section how to get the Apollo source code, build it, and install it.

Dependencies

The previous two chapters of this book address Apollo’s dependencies:

• Vagrant
• SSHFS
• ssh-copy-id

All three of these tools need to be installed and set up in order for Apollo to work.

Golang

Golang, also known as “Go” is built around the idea of sharing open-source code. Before getting the source for Apollo, It’d be best to install and set up Golang. After following the instructions on Golang’s Web site I should have the following things:

• A folder in my home directory called go
• An environment variable $GOPATH which points to ~/go
• An environment variable $GOROOT which points to the directory where Go itself is installed
• An executable called go that I can call from the command line.
• The directory $GOPATH/bin should be added to my path. That’s where programs compiled with go install get put by default.

Getting the Source and compiling

Apollo is hosted publicly on Github under the MIT license. It can be obtained by using the command go get. The go command knows how to use a number of version control systems. When a url is given to it that starts with github.com, it knows to use git to clone that repo. And it will automatically place it under $GOPATH/src/github.com under a folder named after the author of the project.

# Clone the source code for Apollo with the `go` tool.
> go get github.com/murphyrandle/Apollo

# Change to the directory where `go` organizes it.
> cd $GOPATH/src/github.com/murphyrandle/Apollo

Now, if I use the command go install, Go will compile Apollo, and place the executable into the directory $GOPATH/bin. If that directory is added to my path, I can use Apollo immediately after by typing apollo.

# Build and install Apollo.
> go install

# Make sure I can run it.
> apollo
Apollo version: 0.0
# Great! I can now use the apollo command.

3.2 Using Apollo

At this point I’ll recap the example scenario. I’ll be using Docker inside of my VM to create a container with a fresh copy of the Ghost blogging platform inside, ready for me to work with.

This section will use Apollo to: - Start up the VM and mount its files so that I can clone the container source into the VM. - Issue single commands to the VM to build the container, and save it. - Connect to an interactive session, launch the container, and save its container ID into an environment variable.

Managing a VM with Apollo

The apollo up command will use Vagrant to launch the VM, authorize the user with ssh-copy-id, and mount the VM home directory over SSHFS.

# Launch the VM and mount the files
# Note, there's no need to create a folder to mount 
# to, Apollo does that if ~/apollo doesn't exist.
> apollo up

# Change to the mounted folder.
> cd ~/apollo

I need to clone the source for my Ghost blog container into my VM now. I’ll use Git to clone it into the home directory of my VM.

# Clone the source for the Ghost container.
> git clone git@github.com:murphyrandle/docker-ghost.git

Now a folder exists that is accessible on OS X under ~/apollo/docker-ghost AND on the VM under ~/docker-ghost.

Issuing single commands to the VM

I tried to make Apollo as transparent as possible. Because of that, any text that is typed after apollo on the command-line that is not one of Apollo’s flags is treated as a command to be run on the remote machine.

t> Apollo makes sure to run any remote commands in the directory on the VM that corresponds with the current directory on OS X. t> If I type apollo pwd in the directory ~/apollo/foo on OS X, pwd will be run in the directory ~/foo on the VM.

w> Since the shell does substitutions before Apollo can see the command being typed in, special shell substitution characters like $(foo) won’t be passed to the VM unless they are first escaped like so: '$(foo)'.

To build the docker-ghost container, I just navigate to that directory, and issue the command for building, prefixed by “apollo”.

i> To find out more about using Docker, you can visit the Docker Web site

# Go to the directory where the Dockerfile is located
> cd ~/apollo/docker-ghost

# Build the Docker image
> apollo sudo docker build -t ghost .

# Make sure it was built successfully
> apollo sudo docker images
Running in spaaace!
REPOSITORY          TAG                 IMAGE ID
ghost               latest              ae0650bbb06e
~ Fin. ~

Great! Now I have a docker image with ghost inside of it, all configured to run.

# Run the ghost image, configured to listen on localhost:49000
> apollo sudo docker run -p :49000:2368 -i -t ghost

Now, on OS X, if I open a browser and visit localhost:49000, I’ll find a new Ghost blog waiting to greet me (see Figure B).

Figure B: A Ghost blog in a Docker container in a VM, displayed on a browser in OS X

Using Apollo to connect to an interactive session

Occasionally, the commands that I want to issue on the VM are complicated enough that I don’t want to worry about escaping shell substitutions, and I’ll log in to a full interactive SSH session. This can be done through Apollo with the following command:

# Log in to a full interactive SSH session
> apollo -i

That will drop me into the default bash command-prompt on the VM until I terminate the session with CTRL-C, when I’ll be returned to this shell.

All of the same steps from the docker-ghost example can be repeated by starting an interactive session and issuing the same commands with the “apollo” removed from the beginning.

> apollo -i
> sudo docker build -t ghost .
> sudo docker run -p :49000:2368 -i -t ghost

4 Where to Go From Here

In this book, we covered how to make VM setup a breeze, make our files easy to access, and make remote commands easy to run. However, the defaults that Apollo assumes may not work for you. For example, Apollo currently won’t accomodate:

• More than one Virtual Machine
• A different mount point than ~/apollo
• A shell on the VM other than BASH

And it won’t auto-install any of its dependencies.

The source code for Apollo is fewer than 200 lines long. I invite you to dive in and start changing code to fit your own workflow. If you add a feature that you think would be useful to others, I’d love to recieve a pull-request on the Github repo.

Thank you for reading this book.

Any suggestions or corrections should be emailed to murphyspublic@gmail.com.