The Infrastructure Cookbook

The Infrastructure Cookbook

Modern Infrastructure by Example

About the Book

There seems to be a sudden flood of software designed to make managing web infrastructure better, CoreOS is rethinking how we approach our servers at the operating system level, Logstash allows us to turn our logs into an asset, and Sensu is evolving monitoring to deal with the profusion of services used to provide a modern web application.

This book will show you how to put all these things together to provide a coherant whole. Each chapter will walk through setting up one element of that infrastructure, show you what it is and how it works, and then point you in the right direction to find out more. By the end of the book you'll know how to run a cluster of CoreOS servers, deploy software to that cluster, and ensure that everything keeps running smoothly (or at least that you'll be alerted to it if it isn't).

Planned Outline

  • What is this book?
  • Who is it for?
  • Why did I write it?
  • Approach taken
    • Each chapter works through a particular aspect of hosting a site.
    • Can be used as a reference afterwards.
    • Best to at least skim through first time.
    • Principles
  • What will we end up with?
Chapter 1 - Laying the Foundations
  • What is CoreOS?
  • Why use it?
  • Starting a CoreOS cluster.
  • A quick introduction to Docker.
  • Starting our first container.
Chapter 2 - etcd, confd, and other things ending in d
  • What are etcd and confd?
  • Why use them?
  • Starting etcd.
  • Using etcd.
    • Registering a container.
    • Storing other config.
  • Using confd
    • Pointing a service at other things.
    • Updating config files.
  • Further reading.
Chapter 3 - Logging with ELK
  • What is ELK?
  • Why use it?
  • Setting up RabbitMQ.
    • Register with etcd.
    • Configure with confd.
  • Setting up Elasticsearch.
    • Register with etcd.
    • Configure with confd.
  • Setting up Logstash.
    • Relaying log data to Logstash.
    • Revisit RabbitMQ to ship logs.
    • Revisit CoreOS to ship logs.
  • Setting up Kibana.
  • Scaling.
  • Further reading.
Chapter 4 - Monitoring with Sensu
  • What is Sensu?
  • Why use it?
  • Setting up Sensu.
    • Deploy a Redis container.
    • Configure RabbitMQ and friends with confd.
    • Starting the dashboard.
  • How we're going to configure Sensu
    • Work this one out - need a way to pull service config from confd.
    • Standalone checks look like the way forward here
    • Map alert levels to actual alerting methods, eg CRIT -> SMS, WARN -> Hipchat
  • Using Sensu
    • Monitor CoreOS servers
    • Monitor RabbitMQ
    • Monitor ELK services
    • Deregistering containers?
  • Logging config.
  • Further reading.
Chapter 5 - Metrics with Statsd and Librato
  • What is Statsd?
  • Why use it?
    • Why Librato, rather than Graphite?
    • Why indeed? Maybe revisit this.
  • Setting up Statsd
    • Config from confd
    • Register with etcd
  • Sending metrics to Statsd
    • Sensu
    • May need to write a plugin for this.
    • Other applications
Chapter 6 - The Base Container

This is probably much to far back, but not sure we've built up enough knowledge until we get here.

  • What is a base container?
  • Why are we building one?
  • What is it for?
  • Building the container
    • Service registration
    • Logstash forwarder
    • Adding new Logstash config
    • Sensu client
    • Adding new Sensu checks
    • Running the actual services you want
Chapter 7 - Deploy a Web Service
  • What is the service? (I'm Here - small person tracking web service)
    • Doesn't require a database, just Redis, which we've already configured once.
    • PUT your location (eg "Home") to an endpoint
    • GET the endpoint to find out where you are
    • GET / to find out where everyone is
  • Building the container
  • Using the service
  • Metrics
    • Rack::StatsD
    • Track checkin counts
  • Logging
  • Backups - don't keep all your eggs in one basket.
Chapter 8 - Load Balancing

Need to think this one through a bit:

  • Nginx/VulcanD? Which one?
  • Platform layer load balancers like ELB?
  • Running multiple instances of a container.
Chapter 9 - Postgres
  • What is Postgres?
  • Why use it?
  • Container
    • Go into detail on Docker volumes.
  • Using the service.
  • Monitoring config.
  • Logging config.
  • Backups.
  • Redundancy.
  • Further reading.
Chapter 10 - A database backed web service
  • Something Django based? Maybe Node? We'll see.
  • Puts everything learnt previously together to deploy a full database backed web service.
  • Load balance it as well.
  • Share this book

  • Categories

    • Software

About the Author

Jon Wood
Jon Wood

I've been building servers and hosting websites for a pretty large chunk of my life, starting out with a few friend's personal websites running from a home server in my bedroom I've always had a fascination with how the Internet works. Since those days of being lulled to sleep by a server's cooling fan I've worked for companies big and small, writing software, and running their web infrastructure. I've done everything from brochureware websites to building infrastructure to monitor thousands of devices. These days I work at Hubbub helping high street shops compete with multinational corporations.

Causes Supported

Code Club

A nationwide network of volunteer-led after school coding clubs for children aged 9-11.

We create projects for our volunteers to teach at after school coding clubs or at non-school venues such as libraries. The projects we make teach children how to program by showing them how to make computer games, animations and websites. Our volunteers go to their local club for an hour a week and teach one project a week. Each term the students will progress and learn more whilst at the same time using their imaginations and making creative projects. Terms 1 & 2 use Scratch to teach the basics of programming. Term 3 teaches the basics of web development using HTML and CSS. Term 4 teaches Python and so on. We’d like to put a Code Club in every single primary school in the country. There are over 21,000 primary schools in the UK, it’s a big task but we think we can do it!

The Leanpub 60 Day 100% Happiness Guarantee

Within 60 days of purchase you can get a 100% refund on any Leanpub purchase, in two clicks.

Now, this is technically risky for us, since you'll have the book or course files either way. But we're so confident in our products and services, and in our authors and readers, that we're happy to offer a full money back guarantee for everything we sell.

You can only find out how good something is by trying it, and because of our 100% money back guarantee there's literally no risk to do so!

So, there's no reason not to click the Add to Cart button, is there?

See full terms...

80% Royalties. Earn $16 on a $20 book.

We pay 80% royalties. That's not a typo: you earn $16 on a $20 sale. If we sell 5000 non-refunded copies of your book or course for $20, you'll earn $80,000.

(Yes, some authors have already earned much more than that on Leanpub.)

In fact, authors have earnedover $13 millionwriting, publishing and selling on Leanpub.

Learn more about writing on Leanpub

Free Updates. DRM Free.

If you buy a Leanpub book, you get free updates for as long as the author updates the book! Many authors use Leanpub to publish their books in-progress, while they are writing them. All readers get free updates, regardless of when they bought the book or how much they paid (including free).

Most Leanpub books are available in PDF (for computers) and EPUB (for phones, tablets and Kindle). The formats that a book includes are shown at the top right corner of this page.

Finally, Leanpub books don't have any DRM copy-protection nonsense, so you can easily read them on any supported device.

Learn more about Leanpub's ebook formats and where to read them

Write and Publish on Leanpub

You can use Leanpub to easily write, publish and sell in-progress and completed ebooks and online courses!

Leanpub is a powerful platform for serious authors, combining a simple, elegant writing and publishing workflow with a store focused on selling in-progress ebooks.

Leanpub is a magical typewriter for authors: just write in plain text, and to publish your ebook, just click a button. (Or, if you are producing your ebook your own way, you can even upload your own PDF and/or EPUB files and then publish with one click!) It really is that easy.

Learn more about writing on Leanpub