The Infrastructure Cookbook por Jon Wood [Leanpub PDF/iPad/Kindle]
The Infrastructure Cookbook
The Infrastructure Cookbook
The Infrastructure Cookbook

This book is 5% complete

Last updated on 2014-06-26

About the Book

There seems to be a sudden flood of software designed to make managing web infrastructure better, CoreOS is rethinking how we approach our servers at the operating system level, Logstash allows us to turn our logs into an asset, and Sensu is evolving monitoring to deal with the profusion of services used to provide a modern web application.

This book will show you how to put all these things together to provide a coherant whole. Each chapter will walk through setting up one element of that infrastructure, show you what it is and how it works, and then point you in the right direction to find out more. By the end of the book you'll know how to run a cluster of CoreOS servers, deploy software to that cluster, and ensure that everything keeps running smoothly (or at least that you'll be alerted to it if it isn't).

Planned Outline

Introduction

  • What is this book?
  • Who is it for?
  • Why did I write it?
  • Approach taken
    • Each chapter works through a particular aspect of hosting a site.
    • Can be used as a reference afterwards.
    • Best to at least skim through first time.
    • Principles
  • What will we end up with?

Chapter 1 - Laying the Foundations

  • What is CoreOS?
  • Why use it?
  • Starting a CoreOS cluster.
  • A quick introduction to Docker.
  • Starting our first container.

Chapter 2 - etcd, confd, and other things ending in d

  • What are etcd and confd?
  • Why use them?
  • Starting etcd.
  • Using etcd.
    • Registering a container.
    • Storing other config.
  • Using confd
    • Pointing a service at other things.
    • Updating config files.
  • Further reading.

Chapter 3 - Logging with ELK

  • What is ELK?
  • Why use it?
  • Setting up RabbitMQ.
    • Register with etcd.
    • Configure with confd.
  • Setting up Elasticsearch.
    • Register with etcd.
    • Configure with confd.
  • Setting up Logstash.
    • Relaying log data to Logstash.
    • Revisit RabbitMQ to ship logs.
    • Revisit CoreOS to ship logs.
  • Setting up Kibana.
  • Scaling.
  • Further reading.

Chapter 4 - Monitoring with Sensu

  • What is Sensu?
  • Why use it?
  • Setting up Sensu.
    • Deploy a Redis container.
    • Configure RabbitMQ and friends with confd.
    • Starting the dashboard.
  • How we're going to configure Sensu
    • Work this one out - need a way to pull service config from confd.
    • Standalone checks look like the way forward here
    • Map alert levels to actual alerting methods, eg CRIT -> SMS, WARN -> Hipchat
  • Using Sensu
    • Monitor CoreOS servers
    • Monitor RabbitMQ
    • Monitor ELK services
    • Deregistering containers?
  • Logging config.
  • Further reading.

Chapter 5 - Metrics with Statsd and Librato

  • What is Statsd?
  • Why use it?
    • Why Librato, rather than Graphite?
    • Why indeed? Maybe revisit this.
  • Setting up Statsd
    • Config from confd
    • Register with etcd
  • Sending metrics to Statsd
    • Sensu
    • May need to write a plugin for this.
    • Other applications

Chapter 6 - The Base Container

This is probably much to far back, but not sure we've built up enough knowledge until we get here.

  • What is a base container?
  • Why are we building one?
  • What is it for?
  • Building the container
    • Service registration
    • Logstash forwarder
    • Adding new Logstash config
    • Sensu client
    • Adding new Sensu checks
    • Running the actual services you want

Chapter 7 - Deploy a Web Service

  • What is the service? (I'm Here - small person tracking web service)
    • Doesn't require a database, just Redis, which we've already configured once.
    • PUT your location (eg "Home") to an endpoint
    • GET the endpoint to find out where you are
    • GET / to find out where everyone is
  • Building the container
  • Using the service
  • Metrics
    • Rack::StatsD
    • Track checkin counts
  • Logging
  • Backups - don't keep all your eggs in one basket.

Chapter 8 - Load Balancing

Need to think this one through a bit:

  • Nginx/VulcanD? Which one?
  • Platform layer load balancers like ELB?
  • Running multiple instances of a container.

Chapter 9 - Postgres

  • What is Postgres?
  • Why use it?
  • Container
    • Go into detail on Docker volumes.
  • Using the service.
  • Monitoring config.
  • Logging config.
  • Backups.
  • Redundancy.
  • Further reading.

Chapter 10 - A database backed web service

  • Something Django based? Maybe Node? We'll see.
  • Puts everything learnt previously together to deploy a full database backed web service.
  • Load balance it as well.
  • Share this book

About the Author

Jon Wood
Jon Wood

I've been building servers and hosting websites for a pretty large chunk of my life, starting out with a few friend's personal websites running from a home server in my bedroom I've always had a fascination with how the Internet works. Since those days of being lulled to sleep by a server's cooling fan I've worked for companies big and small, writing software, and running their web infrastructure. I've done everything from brochureware websites to building infrastructure to monitor thousands of devices. These days I work at Hubbub helping high street shops compete with multinational corporations.

Causes Supported

Code Club

https://www.codeclub.org.uk

A nationwide network of volunteer-led after school coding clubs for children aged 9-11.

We create projects for our volunteers to teach at after school coding clubs or at non-school venues such as libraries. The projects we make teach children how to program by showing them how to make computer games, animations and websites. Our volunteers go to their local club for an hour a week and teach one project a week. Each term the students will progress and learn more whilst at the same time using their imaginations and making creative projects. Terms 1 & 2 use Scratch to teach the basics of programming. Term 3 teaches the basics of web development using HTML and CSS. Term 4 teaches Python and so on. We’d like to put a Code Club in every single primary school in the country. There are over 21,000 primary schools in the UK, it’s a big task but we think we can do it!

The Leanpub Unconditional, No Risk, 100% Happiness Guarantee

Within 45 days of purchase you can get a 100% refund on any Leanpub purchase, in two clicks.
See full terms

Write and Publish on Leanpub

Authors and publishers use Leanpub to publish amazing in-progress and completed ebooks, just like this one. You can use Leanpub to write, publish and sell your book as well! Leanpub is a powerful platform for serious authors, combining a simple, elegant writing and publishing workflow with a store focused on selling in-progress ebooks. Leanpub is a magical typewriter for authors: just write in plain text, and to publish your ebook, just click a button. It really is that easy.

Learn more about writing on Leanpub