Read Backend Handbook

Backend Handbook

Francisco Quintero

Introduction

My very first intensive Ubuntu servers experience was back then in year 2014. The company had hosted 20+ Drupal websites in a dedicated server in hosting provider. As you should know, dedicated machines are expensive. One day, my boss in that company told me:

Frank, we need to start saving money. I think we can save a lot if we move all sites to basic Linode VPS. If you’re able to move all those websites to Linode before [GIVEN DATE], the amount of dollars in the hosting service bill will be yours.

Then, to buy me in, he said:

I rather give you that money than to the hosting provider.

I was in.

After the first two migrated websites, everything went very good and real quick, though it was a real pain in the ass. The whole migration, in general terms, consisted of:

In the hosting provider:

Backing up the MySQL database for the Drupal website
Zipping the website files in a tar format
Downloading the zip

In my local environment:

Create a new db and import the exported data
Unzip the zipped file in the adequate folder
Check everything was in order

Doing the process local helped a lot because I used Vagrant to replicate an Ubuntu server. Every step done locally was replicated, almost exactly, in the server.

When everything local was completed, I’d move to the next steps.

In Linode:

Launch a new basic Ubuntu server VPS
Create all necessary server records(A, TXT, CNAME)

In the Linode server:

Setup a new system user
Setup the new user SSH login
Harden the server
- Setup firewall rules
- Deactivate root login
- Set an SSH port greater than 10000
Install required software
Configure software
Create folder structure
Upload zip file
Upload database backup
Unzip file and place files in given folder
Import database data
Create Apache virtual host

After everything was loading correctly in a subdomain, the original domain would point to the Linode nameservers and I’d checked it was loading as it should. Then moved on to the next website.

Yes, they were lots of steps for 20+ websites. None of those steps were automatized and back then I didn’t know how to automat them.

Fortunately, I finished on time and got USD 500 in my bank account. However, I told myself I wouldn’t do that kind of manual work again. I set to learn Bash scripting and ways to automate everything I could as much as it would be possible.

From that day on I got into the Ops world. I created scripts for doing a kind of deploy of Drupal websites and kept trying and doing more scripts for several things I saw were very repetitive and could be automated. One thing took me to another and learnt even more about Bash, also learnt Ansible, Terraform, Docker, Jenkins.

I took every chance I had to try and learn new tools to make my life as a developer a bit easier and now all that knowledge is in this handbook.

Objective

This handbook main objective is to help you get started in the process of setting up and configure Ubuntu servers(from 14.04+) to host Ruby on Rails apps. Born from doing several configurations for previous Ruby on Rails applications, many sections described in this handbook can be used for many other type of web apps.

The Backend Handbook contains information to configure development tools such as Vagrant or Jenkins. Also covers how to setup servers like the ones you get in services like in Digital Ocean or Linode, from scratch, creating system users, and more.

List of software installed in the process include:

Development libraries(curl, htop, gcc, python)
PostgreSQL
RVM
Ruby
NodeJS
Redis
WKHTMLTOPDF
etc

About the Author

Francisco Quintero is a software engineer from Colombia. Although his first experiences were with Visual Basic .NET, he fell in love with web development when discovered free books(HTML, CSS, and JavaScript) published by recognized Spanish instructor Javier Eguiluz. Since then, web development is what he does and became passionated about.

Setup

This section will help you to get started setting up your brand new server. This process, though easy to complete, can get a wrong turn when commands are not issued carefully.

One of the things I’ve learnt about setting up servers before writing this handbook is that computers are better than humans when automating things, so, it’s way better to have scripts and then use them as many times as we need rather than manually issuing commands in the console one by one.

Linode

Configurig an Ubuntu machine in Linode requires you to set it up from scratch(they give you the OS and root user). To have a working Ubuntu server you’d need to:

Create a non-root user
Set up user SSH access
Harden SSH service configuration(change default port, etc)
Activate firewall(iptables or UFW)
More…

Create New non-root user

A non-root user is a system user that can log in to the server but in order to run superuser commands needs to precede them with sudo and inputing her password.

Add a user

NOTE: the # symbol denotes the commands are run as root user, not as a system user. NOTE: the $ symbol denotes the commands are run as a system user.

1 # adduser [USERNAME]

Example:

1 # adduser deployer

Give that user permissions to run sudo commands

1 # usermod -a -G sudo [USERNAME]

Example:

1 # usermod -a -G sudo deployer

References

You can always see the man pages for previous commands to learn more about them or reading their docs online.

1 $ man adduser
2 $ man usermod

Online documentation:

Set up User SSH Access

In order to access a Linode machine from your development computer(or any other machine) you need an SSH key that identifies that user on the server. Using SSH authentication is recomended, and certainly, more secure than password authentication.

Generate a New SSH Key

Using a computer with Linux or macOS:

1 $ ssh-keygen -b 4096

When generating ssh keys, the prompt will ask you to enter a passphrase. That is a secret word to further protect your key. I, normally, don’t assign any to my ssh keys for server access.

Follow the steps in the console prompt and take note of where the generated keys(they come in pair) are located in your computer.

Now, login into the server (via password access) and create a .ssh folder in the user’s home folder and then give all permissions only to the owner:

1 $ ssh [USERNAME]@[SERVER-IP-ADDRESS]
2 $ mkdir -p ~/.ssh && chmod -R 700 ~/.ssh

Back in your computer, proceed to upload the generated ssh key. In the folder you generated the keys should be placed two files: id_rsa and id_rsa.pub. We are going to upload the public key (id_rsa.pub).

In the ssh key generation prompt you can give any other name to the keys. I usually name them according to the project I’m working on.

Upload the file using the scp command. It has the syntax scp ORIGIN DESTINATION

1 $ scp /path/to/file [USERNAME]@[IP-ADDRESS]:~/

Example:

1 $ scp /home/cesc/id_rsa.pub ubuntu@167.34.56.18:~/

Finally, login once again into the server and move the id_rsa.pub key into the authorized_keys file that is read by the SSH service.

1 $ ssh [USERNAME]@[SERVER-IP-ADDRESS]
2 $ mv id_rsa.pub .ssh/authorized_keys

References

See man pages for more details in the previously used commands.

1 $ man ssh
2 $ man ssh-keygen
3 $ man scp

Online docs:

Harden SSH Service from Configuration File

Leaving the default configuration for the ssh service is not recommendable because it leaves your server open to attacks from intruders or anyone with the intention of breaking your machine.

By modifying some settings in the sshd_config file your server will be more secured though not impenetrable, but something is better than nothing.

The sshd_config is located (in Ubuntu 14.04) at /etc/ssh/sshd_config. The next commands suppose you are editing that file.

Open the Configuration File

1 $ sudo nano /etc/ssh/sshd_config

Following settings is how the file settings should look. If you find them this way, it is all ok.

If you are using nano editor you can find lines in a file using the CTRL + W key combination and typing the desire word.

Change the Default Port

1 Port [PORT-NUMBER]

22 is the default. Change to a 5 digit number, like 12781, to make it more secure.

Deactive Root Access

1 PermitRootLogin no

Deactivate Password Access

1 PasswordAuthentication no

Restart SSH Service Configuration

Exit the nano editor with CTRL + O (save changes) & CTRL + X.

1 $ sudo service ssh restart

References

In the man pages, as always.

1 $ man ssh

Online docs:

sshd_config

Dependencies

This section includes all software a basic Ruby on Rails application would require in order to run: databases, datastores, web servers, app servers, libraries, and more.

The list of packages or software projects described in this section was gathered from my experience working in different RoR projects. It might be very useful for several scenarios or might lack other packages for other, but regardless, it works as a good starting point.

Remember # and $ symbols

The # symbol denotes the commands are run as root user, not as a system user. In the other hand, the $ symbol denotes the commands are run as a system user.

Examples

Command run as root

1 # adduser peter-griffin

Command run as system user

1 $ sudo service nginx status

Library Essentials

This installation makes sure the machine has the basic software the rest of the programs you’re going to install need. They provide libraries, commands or modules to those software.

What are they for?

Some of them are standard libraries the software that you install in your machine is going to need to work correctly.

Others are just useful software that should be installed because some guides tells you to use them or because sooner or later you’d be needing them.

How to install them?

Create a file $ nano install_basic_dependencies.sh and add the following content:

 1 #!/bin/bash
 2 
 3 apt-get update
 4 apt-get -y install \
 5   build-essential \
 6   curl \
 7   git-core \
 8   python-software-properties \
 9   htop \
10   vim \
11   libfontconfig1 \
12   libgmp-dev \
13   zlib1g-dev \
14   libssl-dev \
15   libreadline6-dev \
16   libyaml-dev \
17   libncurses5-dev \
18   libxml2-dev \
19   libxslt-dev \
20   libsqlite3-dev

To save the file press CTRL + O and then to exit press CTRL + X.

and run it

1 $ sudo bash install_basic_dependencies.sh

NOTE The list of packages to install is splited with continuation lines for readability.

-y flag is for accepting the prompt without needing the user to type y/N

Docs

See man pages for apt-get command $ man apt-get or online

Database Systems and Datastores

A database is:

A database management system (DBMS) is a computer software application that interacts with the user, other applications, and the database itself to capture and analyze data.

Wikipedia

A datastore is:

A data store is a repository for persistently storing and managing collections of data which include not just repositories like databases, but also simpler store types such as simple files, emails etc.

Wikipedia

PostgreSQL

PostgreSQL is a ORDBMS(Object-Relational Database Management System).

What is this for?

Storing and accessing data through the Structured Query Language(SQL).

How to install it?

Create a file $ nano install_postgresql.sh and add the following content:

1 #!/bin/bash
2 
3 apt-get update
4 apt-get -y install libpq-dev postgresql

To save the file press CTRL + O and then to exit press CTRL + X.

Then run it

1 $ sudo bash install_postgresql.sh

Verify

You can check PostgreSQL is up and running by issuing $ sudo service postgresql status command in your terminal. It should return the version and port the database server is working.

Docs

A full list of PostgreSQL manuals can be found their official site.

MySQL

MySQL is a RDBMS(Relation Database Management System).

What is this for?

Storing and accessing data through the Structured Query Language(SQL).

How to install it?

Create a file $ nano install_mysql.sh and add the following content:

1 #!/bin/bash
2 
3 apt-get update
4 apt-get -y install libmysqlclient-dev mysql-server-5.5
5 apt-get -f install mysql-server

To save the file press CTRL + O and then to exit press CTRL + X.

Run the script

1 $ sudo bash install_mysql.sh

Docs

All MySQL documentation and manuals can be found in their official page.

Redis

Redis is an in-memory data structure store that can be used as a database, cache and/or message broker.

What is this for?

Regarding Ruby on Rails applications, Redis can be used as a database for storing background jobs data. Sidekiq is a popular gem that leverages Redis to work.

You can also use a Redis as a cache or message broker, however, solutions like RabbitMQ are better suited for this job as there have been issues about Redis losing data.

How to Install Redis

There packages available for many operating systems that make it easy to install Redis with just a couple of commands.

Ubuntu 14.04+

Run the following commands:

1 $ apt-get update
2 $ apt-get install -y redis-server

MacOS

If you Homebrew installed just run:

1 $ brew install redi

From Source

Create a file $ nano install_redis.hs and add the following content:

 1 #!/bin/bash
 2 
 3 apt-get update
 4 apt-get -y install tcl8.5
 5 curl -sSL http://download.redis.io/releases/redis-stable.tar.gz -o /tmp/redis.tar.gz
 6 mkdir -p /tmp/redis
 7 tar -xzf /tmp/redis.tar.gz -C /tmp/redis --strip-components=1
 8 make -C /tmp/redis
 9 make -C /tmp/redis install
10 echo -n | /tmp/redis/utils/install_server.sh
11 rm -rf /tmp/redis*
12 sysctl vm.overcommit_memory=1
13 sed -ie 's/# bind 127.0.0.1/bind 127.0.0.1/g' /etc/redis/6379.conf
14 service redis_6379 restart

To save the file press CTRL + O and then to exit press CTRL + X, then run the instructions in the file

1 $ sudo bash install_redis.sh

and wait for redis to be installed on your machine.

NOTE this way of install is called from source and the main difference between this and installing from apt-get is getting up to date packages. Sometimes the apt packages are really outdated.

Verify Installation

Check redis is running with sudo service redis_6379 status. It should output something like Redis is running (PORT NUMBER).

Docs

You can find all redis documentation in its official page.

Web Servers and App Servers

A web server is:

A web server is a computer system that processes requests via HTTP, the basic network protocol used to distribute information on the World Wide Web.

Wikipedia

An application server is:

An application server is a software framework that provides both facilities to create web applications and a server environment to run them.

Wikipedia

Nginx Web Server

Nginx is a web server that happens to be as well a reverse proxy, load balancer, and HTTP cache.

What is this for?

Using nginx you can have many things that otherwise would require you to install individual pieces of software to have a great architecture for your app:

You can use nginx as a web server in a client-server architecture
Also works as a reverse proxy, meaning that nginx can work as first layer and send requests, according to predefined rules, to one or more backend services(Jenkins, Apache, etc)
Similar to what a reverse proxy would do, you can use nginx to work as a load balancer and distribute request to other web servers throught redirects
Nginx can also be used as a cache and you would not need to install things like Varnish, Redis or Memcache

How to install it?

Create a file $ nano install_nginx.sh and add the following content:

1 #!/bin/bash
2 
3 apt-get update
4 apt-get -y install nginx

To save the file press CTRL + O and then to exit press CTRL + X.

Finally, run the script to install it:

1 $ sudo bash install_nginx.sh

Verify Installation

If you need to verify Nginx is up, you can do it with $ sudo service nginx status.

Docs

Nginx documentation can be found here.

Phusion Passenger

Passenger is an application server that allows you to securely operate web apps, microservices & APIs with outstanding reliability, performance and control. By acting as a process manager, reverse proxy and by providing operations tools, Passenger enables you to quickly launch and easily maintain Ruby, Node.js, Python and Meteor apps.

Phussion Passenger

What is this for?

Let’s try to define first what is an application server.

In one side we have web servers which are normally used for serving static content via the HTTP protocol.

In the other side, we got applications servers which can also do the job of a web server and do more things. The main difference(when both use no extra plugins) is that app servers can server dynamic content via the HTTP protocol or others.

In the case of Passenger, its job is to handle business logic in applications coded in the supported languages.

How to install it?

Create a file $ nano install_passenger.sh and add the following content:

 1 #!/bin/bash
 2 
 3 apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys 561F9B9CAC40B2F7
 4 apt-get install -y apt-transport-https ca-certificates
 5 
 6 sh -c 'echo deb https://oss-binaries.phusionpassenger.com/apt/passenger trusty main \
 7 > /etc/apt/sources.list.d/passenger.list'
 8 apt-get update
 9 
10 apt-get install -y nginx-extras passenger

save the file by pressing CTRL + O and then press CTRL + X to exit it.

Now, run the script:

1 $ sudo bash install_passenger.sh

Depending on the version of passenger you installed you either need to:

A) uncomment lines passenger_root and passenger_ruby in /etc/nginx/nginx.conf file and then restart nginx with sudo service nginx restart.

B) Edit /etc/nginx/nginx.conf and uncomment include /etc/nginx/passenger.conf; by removing the # character in the beginning of the line and then restart nginx with sudo service nginx restart.

Verify Installation

One of the ways to verify Passenger is working is by taking a look to the processes list and check one called watchdog is available.

Docs

See Phussion Passenger extensive documentation for more help.

Libraries

This section covers installation of some specific dependencies, libraries, software or stuff that some application might need.

WKHTMLTOPDF for generating PDF files from HTML pages
Image Magick for manipulating image files
Jenkins for continuous integration
RVM(Ruby Version Manager) for managing Ruby versions
NVM(Node Version Manager) for managing Node versions

NodeJS

NodeJS is an open source and cross platform JavaScript runtime environment for developing server tools and/or web applications.

Although Node.js is not a JavaScript framework, many of its basic modules are written in JavaScript, and developers can write new modules in JavaScript.

Wikipedia

What is this for?

Because of it non-blocking and event-driven architecture real time communication applications, browser games and web applications that demand high throughput, scalability and concurrence can be designed and develop using nodejs.

How to install it?

Create a file $ nano install_nodejs.sh and add the following content:

1 #!/bin/bash
2 
3 apt-add-repository ppa:chris-lea/node.js
4 apt-get update
5 apt-get -y install nodejs

To save the file press CTRL + O and then to exit press CTRL + X.

Run its instructions:

1 $ sudo bash install_nodejs.sh

Verify Installation

You can verify Node is installed with either $ node -v or $ node --version. The output should return a number which is the current Node version in your system.

Additionally, you can check the installation path with $ which node or $ command -v node.

Docs

NodeJS documentation can be found in the official site

NVM - Node Version Manager

By using NVM you can manage(install, uninstall, use) different NodeJS versions just a few commands away.

What is this for?

Imagine you work in three projects where NodeJS is being used. Now, in your system you have installed Node version 8.11.1. Good, you can work all of your projects with no problem with that version… Until a dependency/colleague updates something and your current version is no longer supported.

Now, of course you could uninstall it and install a newer one but, What if one of those projects can’t work with an updated version(greater than 8.11, in this example)?

You could either use virtual environments with VirtualBox or Vagrant or you can use NVM to easily install and handle different NodeJS versions in your computer with not much trouble.

How to install it?

Create a file $ nano install_nvm.sh and add the following content:

1 #!/bin/bash
2 
3 curl -o- https://raw.githubusercontent.com/creationix/nvm/v0.33.11/install.sh | bash

To save the file press CTRL + O and then to exit press CTRL + X, then run the file:

1 $ bash install_nvm.sh

Verify Install

You can verify NVM is installed with command -v nvm or nvm or nvm --version.

Docs

See NVM GitHub project’s page for more details.

RVM

Easy, manage(install, uninstall, use) different ruby versions just a few commands away.

What is this for?

With RVM you can manage several Ruby versions easily. Without a solution like RVM(rbenv, for example) you might need to uninstall a previous Ruby version to install a differente one. Forget about it with RVM.

Besides, you can handle gemsets to have customized and context specific ruby gems. This context can be different projects or environments.

Most important thing about RVM is that rubies and gemsets are isolated from each other.

How to install it?

Create a file $ nano install_rvm.sh and add the following content:

NOTE: do not run as sudo

1 #!/bin/bash
2 
3 gpg --keyserver hkp://keys.gnupg.net --recv-keys 409B6B1796C275462A1703113804BB82D39\
4 DC0E3
5 \curl -sSL https://get.rvm.io | bash -s stable

To save the file press CTRL + O and then to exit press CTRL + X nano editor.

Now you can run the instructions:

1 $ bash install_rvm.sh

and wait for RVM to be installed.

Bonus: install ruby

After installing RVM, you can proceed to install ruby:

NOTE: you should change the version of ruby for the desired one

You can run these commands one by one or place them in a file. Before create a file $ nano install_ruby.sh and add the following

 1 #!/bin/bash
 2 
 3 echo "Sourcing RVM and reloading shell"
 4 . /home/$(whoami)/.rvm/scripts/rvm
 5 
 6 echo "RVM requirements"
 7 rvm requirements
 8 
 9 echo "Installing ruby-2.4.1"
10 rvm install ruby-2.4.1
11 
12 echo "Setting ruby-2.4.1 as default"
13 rvm --default use ruby-2.4.1
14 
15 echo "Install bundler"
16 gem install bundler --no-document

run the instructions in the file

1 $ bash install_ruby.sh

Verify Installation

Check RVM is installed with $ rvm -v or $ rvm --version and the install path with $ which rvm.

Docs

Find everything about RVM in its official site

WKHTMLTOPDF

WKHTMLTOPDF is a command line tool for rendering HTML docs into PDF. This utility runs headless, meaning it needs no GUI to be operated or used.

What is this for?

Many web applications display useful information to their users. That info can be in the form of tables or well-crafted UIs, and though most of the data is not so important, there’s important info user might required in a portable format.

A good example of important info is invoices for payments or subscriptions. The customer might want to download them to enter the info in a system they own/use or for backup purposes. Whatever the case, they’d like to export that HTML view into something they can grasp. This is when tools such as WKHTMLTOPDF can prove themselves useful.

Be aware WKHTMLTOPDF makes sense to be used when the info is, mainly, generated from a backend because of its multiple implementations in programming languages such as Ruby.

How to install it?

Create a file $ nano install_wkhtmltopdf.sh and add the following content:

1 #!/bin/bash
2 
3 apt-get install -y openssl build-essential xorg libssl-dev xfonts-75dpi
4 
5 wget http://download.gna.org/wkhtmltopdf/0.12/0.12.2/wkhtmltox-0.12.2_linux-trusty-a\
6 md64.deb
7 dpkg -i wkhtmltox-0.12.2_linux-trusty-amd64.deb

NOTE: this script installs an specific version for this library.

You can find more versions in the downloads page

Save the file press CTRL + O and then to exit press CTRL + X. Run the file:

1 $ sudo bash install_wkhtmltopdf.sh

Verify Installation

Test it is working

1 wkhtmltopdf http://www.google.com google.pdf

You should see a generated PDF file with the Google front page as content.

Docs

See complete WKHTMLTOPDF docs in the project page

Image Magick

Image magick is both a command line tool and a library to work and manipulate images.

What is this for?

When working with images in rails applications, you’re mostly going to work using carrierwave or paperclip gems. Whenever you need to crop or process images, you’re going to need to install image magick so those gems can do the processing.

How to install it?

Create a file $ nano install_image_magick.sh and add the following content:

 1 #!/bin/bash
 2 
 3 apt-get update
 4 apt-get -y install libpng12-dev libglib2.0-dev zlib1g-dev libbz2-dev libtiff4-dev li\
 5 bjpeg8-dev
 6 mkdir -p ~/image_magick_download
 7 cd ~/image_magick_download
 8 wget http://www.imagemagick.org/download/ImageMagick.tar.gz
 9 tar -xzf ImageMagick.tar.gz
10 cd ImageMagick-*
11 sudo ./configure
12 sudo make
13 sudo make install
14 sudo ldconfig /usr/local/lib

To save the file press CTRL + O and then to exit press CTRL + X.

Run the installation script

1 $ sudo bash install_image_magick.sh

NOTE: this is takes a lot of time to be completed. Be patient.

Docs

More documentation about image magick can be found the official site

Jenkins

Jenkins helps to automate the non-human part of the whole software development process, with now common things like continuous integration, but by further empowering teams to implement the technical part of a Continuous Delivery.

Wikipedia

What is this for?

As an automation tool, Jenkins is widely(but not only) for continuous integration. However, with a Jenkins basic install you can also run test suites, generate documents, and even deploy software to servers.

Also, Jenkins is not limited to any specific software or programming language.

Regarding RoR apps(which are the reason of this handbook), using Jenkins you could:

run test suite when commits are sent to repository(i.e, GitHub, Bitbucket)
email specified developers in case tests break
package source code to make it deployable
with plugins or its bash integration, define deployment rules or commands
run other tasks, email developers about deployment, and more.

How to install it?

Create a file nano install_jenkins.sh and add the following content:

1 #!/bin/bash
2 
3 wget -q -O - https://pkg.jenkins.io/debian/jenkins-ci.org.key | sudo apt-key add -
4 sh -c 'echo deb http://pkg.jenkins.io/debian-stable binary/ > /etc/apt/sources.list.\
5 d/jenkins.list'
6 apt-get update
7 apt-get -y install jenkins

To save the file press CTRL + O and then to exit press CTRL + X nano. Now you can run it:

1 $ sudo bash install_jenkins.sh

See more installation options in Jenkins docs.

Docs

See Jenkins documentation in the official web site.

Configure Tools

In this chapter we’ll see how to configure tools for doing backend tasks.

Let’s see how to do it with Vagrant and Jenkins. They’re both independent tools but once we learn how they work will find out they can work together.

Jenkins

Jenkins is an automation server that supports plugins to help in the process of building, deploying and/or automating software projects.

A Jenkins installation can be as complex as we want it to be. Has the main advantage of being free and open source and the disadvantage is that once we install it on a server, we have to take care of everything.

Initial Configuration

Once Jenkins is installed in your system(local, Vagrant, or web server) we need to follow some steps to complete the installation and then proceed to configure it.

In your browser go to http://localhost:8080 and wait for a Jenkins page to appear.

The page will indicate where to find the Jenkins initial admin password.

The file is located in /var/jenkins/secrets, so the command is:

1 $ cat /var/jenkins/secrets/initialAdminPassword

Paste the output in the input and click the Continue button at the end of the form.

Now your Jenkins is ready to be used!

Setup for the Cloud

To access Jenkins instance installed in a EC2 machine, for example, probably you’re going to need to configure a virtual host for the web server your EC2 server uses.

As I’m a fan of Nginx, here’s a config for nginx:

 1 upstream app_server {
 2   server 127.0.0.1:8080 fail_timeout=0;
 3 }
 4  
 5 server {
 6   listen 80;
 7   listen [::]:80 default ipv6only=on;
 8   server_name CI.YOURCOMPANY.COM;
 9  
10   location / {
11     proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;        
12     proxy_set_header Host $http_host;
13     proxy_redirect off;
14  
15     if (!-f $request_filename) {
16       proxy_pass http://app_server;
17       break;
18     }
19   }
20 }

Replace CI.YOURCOMPANY.COM for your server IP address or a valid subdomain.

Checkout other configs in this gist and this other.

Plugins

One of the most popular plugins in Jenkins is the GitHub one. Jenkins alone can’t do much so installing plugins is a most to take the best out of Jenkins.

For projects hosted on GitHub, installed the respective plugin. The docs provide instructions on how to install and configure.

One of the most important benefits of using this plugin is being able to trigger actions in Jenkins whenever a new commit is made to a given branch.

So one could setup a job to do a deployment whenever master branch gets new commits. For this kind of setup you’d be better using a GitHub Personal Token.

Be mindful the token should have the scope admin:org_hook

For Ruby on Rails

When using Jenkins or any other CI tool to run tests for Rails apps, there’s one thing we need to be aware:

The Jenkins server need everything your Rails app needs to run.

Let’s say your app in development needs the following list of dependencies:

RVM
Ruby
Bundler
A database: PostgreSQL or MySQL
NodeJS

Then you’d have to install them all in the Jenkins server. Once they’re available you can run commands for the Rails app in the Jenkins instance:

1 bundle install
2 rails db:migrate
3 rspec spec --fail-fast

These commands would go in a build step. If any of those commands fail, Jenkins will notify you and you should take actions to fix them.

This article shows how to do a setup that run tests and deploys using Capistrano. It’s a bit complex but can serve as an example.

Tips and Tricks

Restart your Jenkins server from the a web browser with JENKINSURL/safeRestart
Force restart making running jobs to be lost with JENKINSURL/restart
In the server you can restart it with sudo service jenkins restart
Jenkins execution logs normally should live in /var/log/jenkins

Docs

Vagrant

Vagrant is a tool for creating development environments with ease. Leveraging VirtualBox, you can use Vagrant to create unique and isolated development environments for almost everyting(I’ve tried PHP and RoR development environments).

What’s best about Vagrant is that you can make these environments repeatable and portable, meaning you configure it once, use it many more times.

In a simplified way, what Vagrant does is create a virtual machine(VM) with OS you indicate, sync the VM filesystem with its host filesystem, so that you can work as usual in your OS, i.e, Linux Mint, and run your application inside the virtual machine. This way, you can also keep your machine from being polluted with software related to an specific application.

Provisioning

Vagrant virtual machines can be provisioned with several tools:

Shell Scripts
Ansible
Chef
Salt
Puppet
even Docker(acting as a Docker host)

Docs

Vagrant provisioning

Brew World

In the Linux world, we normally have tools to install software. In Debian we have dpkg, in Ubuntu distros it is apt-get, in Redhat based distros it’s yum.

All nice but when we are given a MacBook Pro to work on in web development we don’t have any of them. Also happens that in the Mac world many software can be installed via GUI but some don’t.

MacOS counts with MacPorts but in many articles you’ll found out that Homebrew is favoured over MacPorts. That’s why Homebrew have a section this handbook.

Homebrew

Homebrew is a package manager for the macOS operating system. It’s easy to install and use.

To install run this script in a terminal window:

1 $ /usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/\
2 master/install)"

To verify it installed run:

1 $ brew help

Install X with Brew

Let’s see how to install some popular software using Homebrew in MacOS.

RVM dependencies

The Ruby Version Manager need many dependencies to be installed. Usually, the command rvm requirements would take care but when not you can install many of its dependencies as follows:

1 $ brew install autoconf
2 $ brew install automake
3 $ brew install libtool
4 $ brew install apple-gcc42
5 $ brew install libyaml
6 $ brew install libxslt
7 $ brew install libksba
8 $ brew install openssl

htop

htop is top on stereoids. Definetely, a most for the lovers of termianl process viewers.

To install with Brew just:

1 $ brew install htop

PostgreSQL

The popular database can also be installed with Homebrew with just one command:

1 $ brew install postgresql

Redis

The popular datastore for caches and more can be installed pretty easily:

1 $ brew install redis

WKHTMLTOPDF

A powerful open-source software to generate PDF documents from HTML ones.

In MacOS can be installed using Homebrew:

1 $ brew install wkhtmltopdf

jq

Not to be confused with the JavaScript library jQuery. jq is a software that simplifies handling JSON data in the command line.

1 $ brew install jq

It’s very powerful and in this gist you can see how I used to process some JSON from AWS Code Deploy service.

Pandoc

Not so DevOps related but one of the tools I used to build this ebook is Pandoc. A very powerful, cross-platform software to generate any kind of document from any kind of document.

In MacOS can be installed as follows:

1 $ brew install pandoc

Relevant Links

Additional Resources

In this section you can find hand-picked links about the DevOps culture, Linux, and related stuff. Any topic that is not fully explained will have a link here to a blog post, documentation or somewhere else you could have more detailed information.

Books

Some useful books to further advance your Linux or backend tools/languages knowledge. Most of them are for free.

Guides

More comprenhensive and/or explained guides might be placed here.

Links & Resources

Repos, tutorials, gists, etc.

Linux

Server

Server Setups

Others

CI = Continuous Integration CD = Continuous Delivery CD = Continuous Deployment

Install RVM in Vagrant

Concepts

Problems and Issues?

If you get any kind of error, have doubts or would like to suggest modifications, go to this handbook repo in GitHub and open an issue describing what you mean.

Go to Backend Handbook in GitHub

Table of Contents

Introduction

Objective

About the Author

Setup

Linode

Create New non-root user

Add a user

References

Set up User SSH Access

Generate a New SSH Key

References

Harden SSH Service from Configuration File

Open the Configuration File

Change the Default Port

Deactive Root Access

Deactivate Password Access

Restart SSH Service Configuration

References

Dependencies

Remember # and $ symbols

Library Essentials

What are they for?

How to install them?

Docs

Database Systems and Datastores

PostgreSQL

What is this for?

How to install it?

Verify

Docs

MySQL

What is this for?

How to install it?

Docs

Redis

What is this for?

How to Install Redis

Ubuntu 14.04+

MacOS

From Source

Verify Installation

Docs

Web Servers and App Servers

Nginx Web Server

What is this for?

How to install it?

Verify Installation

Docs

Phusion Passenger

What is this for?

How to install it?

Verify Installation

Docs

Libraries

NodeJS

What is this for?

How to install it?

Verify Installation

Docs

NVM - Node Version Manager

What is this for?

How to install it?

Verify Install

Docs

RVM

What is this for?

How to install it?

Bonus: install ruby

Verify Installation

Docs

WKHTMLTOPDF

What is this for?

How to install it?

Verify Installation

Docs

Image Magick

What is this for?

How to install it?

Docs