There seems to be a sudden flood of software designed to make managing web infrastructure better, CoreOS is rethinking how we approach our servers at the operating system level, Logstash allows us to turn our logs into an asset, and Sensu is evolving monitoring to deal with the profusion of services used to provide a modern web application.

This book will show you how to put all these things together to provide a coherant whole. Each chapter will walk through setting up one element of that infrastructure, show you what it is and how it works, and then point you in the right direction to find out more. By the end of the book you'll know how to run a cluster of CoreOS servers, deploy software to that cluster, and ensure that everything keeps running smoothly (or at least that you'll be alerted to it if it isn't).

Planned Outline

Introduction

• What is this book?
• Who is it for?
• Why did I write it?
• Approach taken
  • Each chapter works through a particular aspect of hosting a site.
  • Can be used as a reference afterwards.
  • Best to at least skim through first time.
  • Principles
• What will we end up with?

Chapter 1 - Laying the Foundations

• What is CoreOS?
• Why use it?
• Starting a CoreOS cluster.
• A quick introduction to Docker.
• Starting our first container.

Chapter 2 - etcd, confd, and other things ending in d

• What are etcd and confd?
• Why use them?
• Starting etcd.
• Using etcd.
  • Registering a container.
  • Storing other config.
• Using confd
  • Pointing a service at other things.
  • Updating config files.
• Further reading.

Chapter 3 - Logging with ELK

• What is ELK?
• Why use it?
• Setting up RabbitMQ.
  • Register with etcd.
  • Configure with confd.
• Setting up Elasticsearch.
  • Register with etcd.
  • Configure with confd.
• Setting up Logstash.
  • Relaying log data to Logstash.
  • Revisit RabbitMQ to ship logs.
  • Revisit CoreOS to ship logs.
• Setting up Kibana.
• Scaling.
• Further reading.

Chapter 4 - Monitoring with Sensu

• What is Sensu?
• Why use it?
• Setting up Sensu.
  • Deploy a Redis container.
  • Configure RabbitMQ and friends with confd.
  • Starting the dashboard.
• How we're going to configure Sensu
  • Work this one out - need a way to pull service config from confd.
  • Standalone checks look like the way forward here
  • Map alert levels to actual alerting methods, eg CRIT -> SMS, WARN -> Hipchat
• Using Sensu
  • Monitor CoreOS servers
  • Monitor RabbitMQ
  • Monitor ELK services
  • Deregistering containers?
• Logging config.
• Further reading.

Chapter 5 - Metrics with Statsd and Librato

• What is Statsd?
• Why use it?
  • Why Librato, rather than Graphite?
  • Why indeed? Maybe revisit this.
• Setting up Statsd
  • Config from confd
  • Register with etcd
• Sending metrics to Statsd
  • Sensu
  • May need to write a plugin for this.
  • Other applications

Chapter 6 - The Base Container

This is probably much to far back, but not sure we've built up enough knowledge until we get here.

• What is a base container?
• Why are we building one?
• What is it for?
• Building the container
  • Service registration
  • Logstash forwarder
  • Adding new Logstash config
  • Sensu client
  • Adding new Sensu checks
  • Running the actual services you want

Chapter 7 - Deploy a Web Service

• What is the service? (I'm Here - small person tracking web service)
  • Doesn't require a database, just Redis, which we've already configured once.
  • PUT your location (eg "Home") to an endpoint
  • GET the endpoint to find out where you are
  • GET / to find out where everyone is
• Building the container
• Using the service
• Metrics
  • Rack::StatsD
  • Track checkin counts
• Logging
• Backups - don't keep all your eggs in one basket.

Chapter 8 - Load Balancing

Need to think this one through a bit:

• Nginx/VulcanD? Which one?
• Platform layer load balancers like ELB?
• Running multiple instances of a container.

Chapter 9 - Postgres

• What is Postgres?
• Why use it?
• Container
  • Go into detail on Docker volumes.
• Using the service.
• Monitoring config.
• Logging config.
• Backups.
• Redundancy.
• Further reading.

Chapter 10 - A database backed web service

• Something Django based? Maybe Node? We'll see.
• Puts everything learnt previously together to deploy a full database backed web service.
• Load balance it as well.