\nNotes

1We were inspired by Allan Kelly’s Dialogue Sheets for agile retrospectives and recommend that you try them out! Thanks to Allan for his input to the Run Book dialogue sheets. See Kelly2016.↩

\n\n\n

",{"_295":296,"_43":44,"_47":48,"_45":46,"_75":76,"_297":298,"_353":293},"previewContent","

\n Team Guides for Software\n
\n Foreword\n
\n Introduction\n
- \n What is software operability and why should we care?\n
- \n Where can operability techniques be used?\n
- \n How to use this book\n
- \n Terminology:\n
- \n ‘Slow-burner’ organisational changes:\n
- \n What is covered in this book\n
- \n Why we wrote this book\n
- \n Feedback and suggestions\n
\n
\n 1. What does good operability look like?\n
\n 2. Core practices for good software operability\n
\n 3. Use Run Book collaboration to increase operability and prevent operational issues\n
\n
\n 4. Use modern log aggregation for deep operational insights\n
\n 5. Use Deployment Verification Tests and Endpoint Healthchecks for rapid feedback on environments\n
\n 6. Use information radiators and dashboards to drive effective behaviour and good psychological responses\n
\n 7. Make operability part of the software product\n
\n 8. Appendix\n
- \n 8.1 Adapt your logging techniques to the technology characteristics\n
- \n 8.2 Understand how the complexity of modern distributed systems drives a need for a focus on operability\n
\n
\n Terminology\n
\n References and further reading\n
\n
\n Run Book template\n
- \n Service or system overview\n
- \n System characteristics\n
- \n Required resources\n
- \n Security and access control\n
- \n System configuration\n
- \n System backup and restore\n
- \n Monitoring and alerting\n
- \n Operational tasks\n
- \n Maintenance tasks\n
- \n Failover and Recovery procedures\n
\n
\n About the authors\n
- \n Matthew Skelton\n
- \n Alex Moore\n
- \n Rob Thatcher\n
\n
\n Conflux Books\n
\n Notes\n

\nTeam Guides for Software\n

\n\n

Pocket-sized insights for software teams

\n\n

The Team Guides for Software series takes a team-first approach to software systems with the aim of empowering whole teams to build and operate software systems more effectively. The books are written and curated by experienced software practitioners and emphasise the need for collaboration and learning, with the team at the centre.

\n\n\n

\n\n\n

Titles in the Team Guides for Software series include:

\n\n

\nSoftware Operability by Matthew Skelton, Alex Moore, and Rob Thatcher
\nMetrics for Business Decisions by Mattia Battiston and Chris Young
\nSoftware Testability by Ash Winter and Rob Meaney
\nSoftware Releasability by Manuel Pais and Chris O’Dell

\n\n\n\n\n

\nForeword

\n\n\n

\n\n\n\n

\n\n\n

(Foreword is not available yet.)

\n\n\n

\nIntroduction

\n\n\n

\n\n\n\n

\n\n\n

\nWhat is software operability and why should we care?

\n\n

Software operability is the measure of how well a software system works when operating ‘live’ in production, whether that is the public cloud, a co-located datacentre, an embedded system, or a remote sensor forming part of an Internet of Things (IoT) network. We say that a software system with good operability works well and is operable. A highly operable software system is one that minimizes the time and effort needed for unplanned interventions (whether manual or automated) in order to keep the system running.

\n\n

All too often teams and individuals ignore or downplay aspects of the live/production environment and operating procedures when building software, only for those aspects to cause problems when the software goes live. A focus on operability helps address these operational concerns and both prevent problems happening in the first place and also make our software systems more resilient to unexpected operating conditions.

\n\n

\n
“The most amazing abstractions, cleanest code, or beautiful algorithms are meaningless if your code doesn’t run well on production.” David Copeland (@davetron5000) [Copeland2013]
\n

\n\n

Software with a high level of operability is easy to deploy, test, and interrogate in Production. It provides us with the right amount of good-quality information about the state of the service being provided and exhibits predictable and non-catastrophic failure modes when under high load or abnormal conditions, even if those conditions were never foreseen. Systems with good software operability also lend themselves to rapid diagnosis and transparent recovery following a problem, because they have been built with operational criteria as first-class concerns.

\n\n\n\n\n

\"Electric — Electric trains in Mallorca, Spain, built in 1929 by Siemens and still running in 2014: an example of a physical system with good operability - Photo Copyright (c) Matthew Skelton 2012

\n\n\n

Software systems which follow operability good practice will tend to be simpler to operate and maintain and will make anticipation and diagnosis of errors straightforward [Crowley2012]. Good operability leads to a reduced lifetime cost of ownership and fewer operational problems compared to those software systems whose owners have prioritised functionality heavily over operational criteria.

\n\n

\nWhere can operability techniques be used?

\n\n

The techniques and approaches we explore in this book work for many kinds of software systems: cloud-native, traditional on-premises enterprise IT, high-frequency/low-latency, mobile apps, desktop client-server, embedded systems, IoT devices, wearables, and industrial/medical applications. We’re fairly certain that the techniques also apply to nuclear power stations and space rockets, but these sectors are out of our experience! The important thing here is that these techniques are not specific to any particular set of technologies but work in many different contexts (sometimes requiring some adjustments, sometimes not).

\n\n\n\n

\nHow to use this book

\n\n

Chapter 1 provides an overview of software operability: what it is, why we need to focus on operability, and what we can expect to gain from it.

\n\n

Chapters 2-7 provide specific tried-and-tested techniques and practices for enhancing operability for teams building and running software systems incuding:

\n\n

Add ‘hooks’ into software components for operational checks
Have software development teams write a draft runbook
Avoid expensive Production-only tooling
Ensure that failure responses are gradual, graceful, and graph-able
Treat logging as a first-class concern and a means of communication
Have the product owner and developers on call for Production incidents
Make operations activities more visible, for example using Kanban boards, ChatOps and graphing/alerting on operations (such as server restarts, load-balancer workarounds, etc.)

\n\n

Each chapter is readable independently, containing the necessary level of detail to be understood and actionable on its own, without requiring any of the other chapters in the book to be read first (although certainly reading the full book will provide a more comprehensive understanding of the concepts and practices and their inter-relations).

\n\n

\nRecommended reading for Product Managers

\n\n

We suggest that Product Managers read the chapters in this order, starting with the ‘why’ of operability:

\n\n

Chapter 6 on operability for product management
Chapter 1 on good operability
Chapter 2 on core practices
then other chapters as needed

\n\n

Chapter 7 on Team Topologies could be useful in the context of a move towards a more flow-based organisational operating model.

\n\n

\nRecommended reading for Software and System Architects

\n\n

Software Archtiects and System Architects can nudge teams in healthy directions by demonstrating and advocating for the use of good operability techniques. We recommend that archtiects read the chapters in this order:

\n\n

Note: it is becoming good practice for architects to act as an Enabling team rather than a separate ‘ivory tower’ silo. See Chapter 7 on Team Topologies for details.

\n\n

\nRecommended reading for Team Leads and engineers

\n\n

Team Leads and engineers are likely aware of many of the practices in this book already but the way in which the tools and techniques are used may be new. We recommend that Team Leads read the chapters in this order to internalise the nuance of the operability techniques:

\n\n

Remember that amost all the operability techniques in this book are focused on building shared awaressness within and across teams (a social practice), not just the use of a tool (a technical practice).

\n\n

\nRecommended reading for Enabling teams

\n\n

For people within Enabling teams, we recommend reading the chapters in this order:

\n\n

The focus of an Enabling team is to help other teams adopt and understand better appraoches, so it’s vital that people in Enabling teams internalize the intent of the practices in this book: reduce operational overhead, improve outcomes for users, reduce rework, increase the viability of each separate service or application.

\n\n

\nTerminology:

\n\n

Terminology may seem a lesser concern but in fact extended use of incorrect terms can actually lead to the wrong values and assumptions setting in.

\n\n

We recommend:

\n\n

Avoid the term ‘non-functional requirements’; use ‘operational features’ instead
Avoid a ‘production-ization’ or ‘hardening’ development phase; build in operational excellence from the beginning instead

\n\n\n\n

\n‘Slow-burner’ organisational changes:

\n\n

While we’ve tried to provide very practical approaches to improve operability in this book, we also acknowledge the need to change organizational culture, in particular:

\n\n

Recognise that today’s distributed, multi-component, multi-dependencies software systems demand team members with a deep skillset which takes time and dedication to either find or develop in-house
Treat software operations as a high-skill, value-add activity, not support tasks

\n\n

\nWhat is covered in this book

\n\n

We have not tried to cover every possible detail related to software operability; for a comprehensive guide to software operability, we recommend reading the books Patterns for Performance and Operability: Building and Testing Enterprise Software by Ford et al (Ford2008), Release It by M. Nygard (Nygard2007), and Continuous Delivery by Jez Humble and Dave Farley (HumbleFarley2010).

\n\n

What this guide does provide is a set of hands-on practices based on real-world, tried-and-tested experience across multiple organizations for teams to adopt (and adapt) in order to promote and enhance software operability.

\n\n

In short, if you build or run software systems and care about how well they work, then this book is for you!

\n\n

\nWhy we wrote this book

\n\n

Over the course of several years, working in both permanent roles and across numerous engagements with clients as consultants, it became increasingly obvious to us that many organisations were missing some almost intangible but fundamental understanding of what it takes to enhance the performance, deployability, reliability and all-round “working well”-ability of their software systems.

\n\n

Many discussions and debates about the nuances of these ideas and concepts ensued, and shaped the content of this book over the course of several years. Some of these themes were reflected in the emerging ‘DevOps’ movement, but others are more related to critical thinking, engineering practice, and understanding human factors.

\n\n

Approaching the question from our separate perspectives of Software Development (Matthew), DevOps (Alex), and IT Operations (Rob) and enabled us to consider a wide range of team drivers, concerns and needs, leading us (we hope) to greater understanding of the symbiosis of development and operations teams, and towards developing methods and techniques for enhancing this relationship.

\n\n

In the end, we turned a lot of that discussion and thinking into this book and other articles in an effort to share some of the insights gained from our years of experience and to offer some help and hopefully practical advice to developers, operations staff, product managers, owners and others who were interested in getting the most usable software platforms they could.

\n\n

\nFeedback and suggestions

\n\n

We’d welcome feedback and suggestions for changes.

\n\n

Please contact us at info@operabilitybook.com or on the Leanpub discussion at https://leanpub.com/SoftwareOperability/feedback

\n\n

Matthew Skelton, Alex Moore, and Rob Thatcher - December 2019

\n\n\n

\n\n\n\n

\n\n\n\n\n\n

\n1. What does good operability look like?

\n\n\n

\n\n\n\n

\n\n\n

(This chapter is not available in this edition of the book.)

\n\n\n

\n2. Core practices for good software operability

\n\n\n

\n\n\n\n

\n\n\n

(This chapter is not available in this edition of the book.)

\n\n\n

\n3. Use Run Book collaboration to increase operability and prevent operational issues

\n\n\n

\n\n\n\n

\n\n\n\n\n\n

\n\n\n\n

\n\n\n

\n3.1 Operational aspects are very similar across many software systems

\n\n

It’s quite common for software teams to think that the systems they work on are unique or somehow special and that there is little they can learn from other systems. The reality is that the vast majority of business systems on which you might work as a software team have almost identical operational needs; the technologies and details change, certainly, but the make-or-break operational requirements are remarkably similar across most systems.

\n\n

For example, all software systems have service level requirements, even if these are not identified or articulated. By extension, these systems also have an implicit or explicit SLA, even if it’s just a (physics-defying) agreement that “the system needs to work all the time without failure”. Interestingly, non-trivial systems generally have several run-time dependencies (not all of which might be known), and these systems all need to be monitored for healthy operation (especially when one or more of those dependencies fail). Thankfully, this means we can reuse existing approaches across different systems!

\n\n

Matthew writes:

\n\n

Some years ago, I led a team working on a major system for a well-known financial organisation in London. We were one supplier out of four contributing to the system, and we ran fortnightly, extended ‘dress rehearsal’ sessions together to test the operational readiness of all parts of the system. I remember that many of the software developers (myself included) were surprised at the things we needed to test: SAN controller failovers, content switch rulesets, SSL offloading, a separate management NIC & networks, scheduled data cleardown, etc.

\n\n

As a result of jointly testing for operability like this, we were able to co-design the system and build in operability well before the system went live.\nWorking directly alongside Ops people (network engineers, data architects, etc.) really helped to highlight the operability considerations to the software developers, and we addressed many potential operational problems early on.

\n\n

At a high level, to make typical software systems work in Production, we need to consider things like:

\n\n

The purpose and remit of the system - who ‘owns’ the system?
The characteristics of the system (data flows, network topology, etc.)
Required resources (CPU, storage, etc.)
\nSecurity (access, encryption, etc.)
System configuration\n
\nBackup and restore of data
Monitoring & alerting
Regular operational tasks (including troubleshooting)
\nPatching and cleardown
\nFailover and recovery

\n\n

Some systems have smaller requirements in some areas compared to other systems, but we need to consider all of these things (and more), whether the system is a cloud-based flight-booking system, an on-premise industrial control unit, or an array of IoT devices forming part of a Smart City deployment. Many software teams find this list of operational aspects quite daunting, but don’t worry - the techniques in this chapter will help you to understand and deal with all these operational concerns.

\n\n\n\n

In modern software systems, we expect almost all operational tasks to be automated. Old-style run books with step-by-step instructions for a low-skilled IT support person to follow have no place in today’s software systems (see Goldschrafe2011). When we say ‘Run Book’ here, we mean a set of common checks and prompts based on sound industry experience - plus high-level answers - that help teams to discover operational features of their software systems and thereby improve operability, not a giant document which might replace or even prevent automation and monitoring. We explore some examples of such common checks and prompts in the next section and in the Appendix (Run Book template).

\n\n

\n3.2 Use a Run Book template as a common baseline for operational aspects

\n\n

A good way to start improving operability is by collaborating on filling out a Run Book template. A Run Book template is a set of headings and questions that together cover the operational concerns that will need to be addressed in most business software systems. For instance, there are typically headings like these:

\n\n\n\n

The point of this Run Book collaboration is not to produce a document but to explore and understand the expected runtime behaviour of the system.

\n\n

You can use the Run Book template like this:

\n\n

Start by an overview of the service or system, then clarify with the team what each section in the template means.
Fill in some details for sections that you know about.
If you don’t know the details for a section, mark it as “not defined” - these areas of the system are likely to have operability gaps. Discuss these sections with people outside the team.
If any section is simply not relevant for your system, mark it as “not relevant”.

\n\n

Be suspicious of too many sections marked as “not relevant”, because there is probably some need for that operational task or feature, even if it is handled by a third party supplier. Check with members of the operations team before you assume that something is not relevant!

\n\n

\nSoftware developers and testers complete the first draft of the Run Book

\n\n

You’ll probably get the best outcomes from having the software development team own and drive the activities around the Run Book information, seeking input from IT Ops people (and others in the organization with operational awareness) to fill in gaps in knowledge.

\n\n\n\n

Because the software development team needs to collaborate with the operations team in order to define and complete the various Run Book details, the operations team also gains early insight into the software. This helps to set up channels and patterns of communication, trust, and collaboration, which help to improve the quality and operability of the software system early on in the process.

\n\n

In practice, after filling in the Run Book template, you’ll want to automate many of the checks and procedures (rather than leaving them in a wiki or document) using techniques such as Deployment Verification Tests or DVTs - see Chapter 2. The Run Book collaboration activities should help you and your team discover things about how the system will work, along with any gaps in knowledge, implementation, or operational readiness that need to be addressed.

\n\n

This work on operational aspects should be a regular team activity - see Chapter 7 for more details on how to make operational features a normal part of your team’s work. By working on operational aspects of the software every sprint/iteration/week, we keep ourselves ‘tuned in’ to the practical consequences of product decisions, avoiding a build-up of poor design for the future, and so reducing runtime risk and ‘feature friction’.

\n\n\n\n

\nRun Book template example

\n\n

During our work with many different organisations and teams over the years, we have gathered a set of common operational concerns that span many different kinds of software systems across many industry sectors. This Run Book template is available on Github or via the shortlink runbooktemplate.info:

\n\n\n

\"Run — Run Book template on Github - *runbooktemplate.info*

\n\n\n

The template contains more than 50 headings that relate to the operation of modern software systems, along with some sample responses. You can print the template directly from Github, fork the repository to modify it for your own organisation, or use a Run Book Dialogue Sheet. However you use the template, treat the resulting information as a starting point for discussions about operational readiness, not as a finished document.

\n\n\n\n

In our experience, you will need to address the majority of the points in the Run Book template, if only to confirm that “this section definitely does not apply here” - a valuable realisation. Each section has a description to set the context and explain why it’s needed.

\n\n\n\n

\n3.3 Use a Run Book Dialogue Sheet to facilitate discovery and avoid ‘documentation fallacy’

\n\n

We have found that a very effective way of discovering the operational aspects of a software system is to use a Run Book dialogue sheet¹. These are large (A1 size) sheets of paper that cover a good-sized table and show all the headings from the Run Book template, enabling the whole team to interact with the sheet around the table:

\n\n\n

\"A — A Run Book dialogue sheet - download from runbooktemplate.info

\n\n\n

This particular example is based on the Run Book template hosted at runbooktemplate.info - we’ve used a Creative Commons SA license on the dialogue sheet so you can modify it and create your own version if you like.

\n\n

You can use the Run Book dialogue sheet like this:

\n\n

Find the most recent version of the A1-size PDF at runbooktemplate.info.
Save and print the PDF at A1 size.
Bring together the whole team (Product Owner, UX, Developers, Testers, Build & Release, and Operations people) in the same room.
Talk through each heading on the sheet, capturing useful details using a marker pen on the dialogue sheet. We find that it’s best to start with the Service or system overview section so that the purpose of the system is well-understood by everyone.
Continue with other headings on the sheet until all sections have been either:\n
- Completed
- or Marked as “Not relevant”
- or Marked as “Not defined”
\n
Take a photograph of the finished dialogue sheet for reference - you will want to return to the details at a later date!
(Optional, but recommended) Place the dialogue sheet on the wall next to your team area so it can stimulate discussions with people who walk past it or when new team members join the team.

\n\n\n

\"A — A Run Book dialogue sheet in action - note that some answers are explicitly *not defined*, indicating possible operability gaps

\n\n\n

We recommend spending between 4 hours and 2 days on the Run Book dialogue sheet per system to begin with; some systems may need even more time. One very valuable output from these sessions is to discover the headings in the dialogue sheet that end up being not defined (rather than not relevant). You likely have gaps in operability if an operational aspect of the system is not defined, so there is a good chance that errors or bugs will arise around this aspect. Use this as a ‘signal’ that more clarity is needed (by asking someone external to the team, for example).

\n\n

Using Run Book dialogue sheets helps to emphasise the importance of discovering (and rediscovering) operational aspects of the system in a collaborative way. If the whole team has helped shape the operability of the system, there is less need to begin ‘documenting’ operational requirements in a static wiki or Word document that will quickly become outdated (while providing a false sense of security). The ‘documentation fallacy’ happens when people rely on those static documents as a safety net, only to find out in the worst moment (during an incident) that those documents have long drifted from the reality of our ever evolving systems.

\n\n\n\n\n

\n3.4 Assess operability on a regular basis: every sprint, iteration, or week

\n\n

To maintain a high state of operational readiness in our software, we need to assess operability on a regular basis within the team. A straightforward way to do this is to include a brief exercise in every sprint/iteration planning meeting or retrospective:

\n\n

Place the most recently-completed Run Book dialogue sheet on the table (or print out a blank Run Book template)
Assign one person to each of the 10 or so areas of focus on the dialogue sheet:\n
- Service or system overview
- System characteristics
- Required resources
- etc.
\n
Give each person a different coloured sticky note (or combination of sticky note and letter/shape drawn on the sticky note)
For each story card completed (or planned), give everyone (say) 1 minute to assess whether the feature described by the story might affect their area of focus.
If someone thinks that their area might be affected, they place a sticky note (plus annotation if needed) onto the story card.

\n\n

Any story cards with sticky notes are then discussed and (if necessary) taken to other teams to discuss operability concerns:

\n\n\n

\"Checking — Checking operability of recent changes using the Run Book dialogue sheet and stickies

\n\n\n

Doing this operability assessment exercise should take around 5-10 mins each time once team members are familiar with the criteria and motivations for doing it. If you run the exercise during planning sessions you will get leading indicators of possible operability problems (but with uncertainty), whereas if you run this during a retrospective, you will get lagging indicators, although with more detail. Running the exercise during both retrospectives and planning sessions gives the best outcomes; you can compare expected effects on operability with the actual!

\n\n

You can also run the exercise on a weekly or bi-weekly basis using Kanban approaches; just take the cards representing ‘done’ or ‘waiting’ (or both!) and assess operability against these tasks.

\n\n

\n3.5 Summary

\n\n

The operability of a software system should be the responsibility of the team building the software. Usually, this means that the team needs to collaborate with operations people (or with operational experience) in order to explore and define the operational characteristics of the system. Proven techniques like Run Book collaboration can really help to bring together all the people needed to identify these operational criteria, as well as build trust between teams. Practical tools like Run Book templates and Run Book dialogue sheets provide a light framework for teams to discover and assess operability in their software on an ongoing basis.

\n\n\n

\n\n\n\n

\n\n\n\n

\n4. Use modern log aggregation for deep operational insights

\n\n\n

\n\n\n\n

\n\n\n

(This chapter is not available in this edition of the book.)

\n\n\n

\n5. Use Deployment Verification Tests and Endpoint Healthchecks for rapid feedback on environments

\n\n\n

\n\n\n\n

\n\n\n

(This chapter is not available in this edition of the book.)

\n\n\n

\n6. Use information radiators and dashboards to drive effective behaviour and good psychological responses

\n\n\n

\n\n\n\n

\n\n\n

(This chapter is not available in this edition of the book.)

\n\n\n

\n7. Make operability part of the software product

\n\n\n

\n\n\n\n

\n\n\n

(This chapter is not available in this edition of the book.)

\n\n\n

\n8. Appendix

\n\n\n

\n\n\n\n

\n\n\n

\n8.1 Adapt your logging techniques to the technology characteristics

\n\n

Broadly speaking, there are four main dimensions that shape the detail of how we use logging and metrics:

\n\n

The end-to-end connection quality (bandwidth, reliability) between the originating node and the collecting/ingest system - “how much data can we transmit?”
The processing power and storage on the originating node - “how much data can we store and forward from the node?”
What permissions we have to transmit data from the originating node - “how far can we push the data?”
What is the cost to transmit data - “how much per Gigabyte?”

\n\n

For example, consider a stock control system for a large manufacturer running in a secure data centre. Although the software system has access to Gigabit-speed internet bandwidth via fiber connectivity (high connection quality - #1 above), and runs on x86 VMs (good processing power - #2 above), the company refuses to allow log data and metrics to be shipped outside the data centre (limited ability to “push” the data - #3). In this case, the log aggregation system and metrics collector need to be run inside the same data centre and likely have some resource constraints (storage and processing power) so the emission of log and metrics data needs to be controlled.

\n\n

Conversely, a building management system that uses smart sensors deployed in offices that communicate with a central web-based system using the customers’ internet connections as part of the provided service would have very different constraints: the sensors might be reasonably-capable ARM-based devices with plenty of onboard storage, and we can assume that the network connection is rapid, but we should take care not to swamp the customers’ connections with too much data, as this may cost them additional fees.

\n\n

A SaaS-based online shopping website running in 2024 likely has excellent bandwidth, capable runtime hardware nodes, permission to send data to a collector/aggregator, and reasonable bandwiidth costs.

\n\n

\nLogging and metrics for applications running on x86-based machines

\n\n

If your software runs directly on a machine that resembles a commodity x86-based server or desktop machine (whether 32-bit or 64-bit, physical or virtual), then it is usually safe to assume that:

\n\n

the machine has plenty of RAM and CPU power, and probably multiple CPU cores
there is ample local storage for files (including log files)
the machine has a fast Ethernet connection (1Gbit or more)

\n\n

In this context, a reasonable model for dealing with log data and metrics is to use a locally-installed log agent (a separate daemon/service) that watches log files written to local storage and then forwards these logs to a central location. Metrics can be derived from log data or can be emitted directly from the application with an assumption that the metrics will be collected more or less immediately by the central server.

\n\n

\nLogging and metrics for containers and containerized applications

\n\n

If you software runs in a container (Docker, LXC, etc.), good practice is to avoid writing to the filesystem locally inside the container. Storage for containers is heavily virtualised and not optimal for use by many hundreds of containers simultaneously.

\n\n

Instead, containerized applications should write logs to STDOUT / STDERR, relying on the container fabric or helper containers to listen for this STDOUT output and forward the log messages to a central collector. Generally, the log listeners will decorate the log messages with details of the container ID and type, providing more context for the log messages when they appear in the central aggregator.

\n\n

Time-series metrics for containerized applications can rapidly saturate a network due to the large number of running containers and the subsequent network traffic that can ensure from metrics transmission. In these cases, metrics can be most effectively managed by using a pre-aggregation helper.

\n\n

The containerized applications send their metrics to a aggregator container running on the same host (thereby using only the host’s network card and not the switched network itself). The aggregator then decorates the application metrics with system-level metrics and (possibly) compresses or collates the application metrics before sending them to the central collector.

\n\n\n\n

\nLogging and metrics for Serverless / Function-as-a-Service (FaaS) and Platform-as-a-Service (PaaS)

\n\n

For Serverless or Function-as-a-Service applications (such as AWS Lambda, Azure Functions, and Google Cloud Functions) and applications using a Platform-as-a-Service framework like CloudFoundry or OpenShift, the logging and metrics options are typically (deliberately) restricted, with the platform or runtime hiding the implementation details.

\n\n

For FaaS and PaaS applications, you will typically either use a library specific to the platform, or send data to a default endpoint (STDOUT or localhost) for the platform to collect. For Serverless/FaaS systems, a “log reflector” approach using one or more additional helping functions seems to work well.

\n\n

\nLogging and metrics for IoT / connected devices and embedded systems

\n\n

With the increase in internet-connected devices (“Internet of Things”), autonomous vehicles, medical and agricultural devices, building sensors, and industrial automation, we need to consider sensible approaches to log aggregation and metrics for IoT devices. Such systems are typically characterised by:

\n\n

Limited compute resources (slow CPU, limited RAM, storage that is slow and/or limited in size, limited bandwidth)
Only occasional network connectivity, often by design for some devices
Protocols other than HTTP as the initial transport mechanism (MQTT, LoRaWAN, etc.)

\n\n

In these cases, we need to adopt a pragmatic approach to logging and metrics that still gives us the benefits of rich information and invaluable insights into the operational effectiveness of the device or system without compromising CPU/RAM/network limits.

\n\n

We should be highly selective about what data we send and when we send it from the remote device to any data collection endpoint; if possible, send only essential information in a compressed form. On some IoT networks - such as LoRaWAN - data transmission bandwidth is extremely restricted, so use the bandwidth wisely. It is also important to limit the size of on-device storage used for log and metrics data: overwrite old data if necessary.

\n\n

\nLogging and metrics for mobile apps

\n\n

Personal mobile devices (phones, tablets, and similar devices) sit between x86 machines and IoT devices in terms of storage and processing capabilities. When generating and sending log data and metrics from mobile devices, we need to expect that:

\n\n

The network connectivity is intermittent
The network connectivity is expensive for the user
The device has limited storage space remaining

\n\n

This means we should be quite selective about what data we send and when we send it from the mobile device to any remote data collection endpoint; wait for a wifi connection (rather than cellular data) to send log data and metrics, and even then send only essential information in a compressed form. It is also important to limit the size of on-device storage used for log and metrics data, overwriting old data if necessary.

\n\n

\nGuidelines for logging and metrics

\n\n

Whatever runtime system you are working with, you should adhere to \nsome basic guidelines for logging and metrics:

\n\n

Obfuscate sensitive information when logging, such as credit card numbers
Encode raw user input before writing to a log - this avoids “log forging” (see 9 Logging Sins by Eugen Parachiv)
Avoid excessive logging and metrics output. Through collaboration with other teams, output only the data that is likely to be needed. Use structured output, not raw string data.
Make log messages and metrics names specific. Do not assume that other people will understand the context for that message or metric.

\n\n

See the article Lies My Parents Told Me (About Logs) for more detailed guidance on logging in particular.

\n\n

\n8.2 Understand how the complexity of modern distributed systems drives a need for a focus on operability

\n\n

\n
“As systems become more complex, this reductionist way of understanding them fails; they behave in ways that cannot feasibly be predicted from under-standing of the individual parts, or were not expected by the system designer who assembled the parts, or both.” – Jeffrey Mogul, p.293
\n

\n\n

Modern computer systems are typically highly-distributed, multi-node systems that act as part of a larger service. In order to cope with the many different failure modes of these kind of systems, we need to address operability as first-class concern.

\n\n

Many professional and college/university courses in software engineering, computer science, and programming have only recently begun to tackle the distributed nature of modern software, leaving many software professionals unaware of the need for operability as a foundation for effective distributed software systems.

\n\n

This effect has been exacerbated by the way in which people have misinterpreted the lack of an operability focus in the Agile Manifesto to mean that operability (and operational concerns in general) are not important for Agile software development. The Agile manifesto was written at a time when most software development was for user-installed desktop PC software whose operational needs were very simple (a single computer). Software in 2024 and beyond needs an additional focus on operability above and beyond the user focus of the Agile Manifesto.

\n\n

As the Internet of Things (IoT) drives a proliferation of network-connected devices for both consumer and industrial applications, the potential for unpredictable side-effects in software interactions increases (see Leveson2017a). A strong commitment to operability as a core aspect of modern software is crucial to ensuring that these interconnected systems work effectively.

\n\n\n

\n\n\n\n

\n\n\n\n\n\n

\nTerminology

\n\n\n

\n\n\n\n

\n\n\n

API: Application Programming Interface - a way to interact with software from other software (rather than via a screen)
Blast radius: the extent of systems affected by a fault
CAB: Change Advisory Board - a change management group that is supposed to advise on service changes
Enabling team: a team of experts that uplifts capabilities within other teams and detects capability gaps across multiple teams. Taken from Team Topologies.
Error Budget: a budget (in seconds, minutes, or hours) of downtime that is acceptable for a given service. Used as part of an SRE approach.
FaaS: Function as a Service
Feature Friction: a term characterised by Matthew Skelton, this is the (usually increasing) “effort needed to add a feature non-invasively” (Michael Feathers). Sometimes (incorrectly) called Technical Debt.
Gbit: Gigabit
GB: Gigabyte
IaaS: Infrastructure as a Service
IoT: Internet of Things
MB: Megabyte
NIC: Network Interface Card
Operability: a measure of how well something (such as software) works in a Production/Live system.
PaaS: Platform as a Service
RAM: Random Access Memory
RAS model: the measures of operability used by IBM since the 1960s of Reliability, Availability, Serviceability
RBAC: role-based access control
Run Book: Collection of details about how a software system works in Production. Also known as System Operation Manual.
SaaS: Software as a Service
SRE: Site Reliability Engineering, and approach to reliability popularized by Google.
SLA: Service Level Agreement
TB: Terabyte
UX: User Experience
Zero-Day Vulnerability: a public security flaw announced with 0 days’ warning to vendors

\n\n\n

\n\n\n\n

\n\n\n\n

\nReferences and further reading

\n\n\n

\n\n\n\n

\n\n\n

\n Introduction\n

\n\n

Ford2008 - C. Ford, I. Gileadi, S. Purba, and M. Moerman, Patterns for Performance and Operability: Building and Testing Enterprise Software. Boca Raton: Auerbach Publications, 2008.

\n\n

HumbleFarley2010 - J. Humble and D. Farley, Continuous Delivery: Reliable Software Releases Through Build, Test, and Deployment Automation, 1 edition. Upper Saddle River, NJ: Addison Wesley, 2010.

\n\n

Nygard2007 - M. T. Nygard, Release It!: Design and Deploy Production-Ready Software, 1 edition. Raleigh, N.C: Pragmatic Bookshelf, 2007.

\n\n

\n Chapter 1 - What does good operability look like?\n

\n\n

Beyer2016 - Beyer, Betsy, Jennifer Petoff, Chris Jones, and Niall Richard Murphy. Site Reliability Engineering. 1 edition. Beijing ; Boston: O’Reilly, 2016.

\n\n

Black2012 - Black, Edwin. “IBM’s Role in the Holocaust – What the New Documents Reveal.” Huffington Post (blog), February 27, 2012. https://www.huffingtonpost.com/edwin-black/ibm-holocaust_b_1301691.html

\n\n

Ford2008 - Ford, Chris, Ido Gileadi, Sanjiv Purba, and Mike Moerman. Patterns for Performance and Operability: Building and Testing Enterprise Software. 1 edition. Boca Raton: Auerbach Publications, 2008.

\n\n

Gawande2011 - A. Gawande, The Checklist Manifesto: How to Get Things Right, Reprint edition. New York: Picador, 2011.

\n\n

Leveson2017b - Leveson, Nancy G. 2017. Engineering a Safer World: Systems Thinking Applied to Safety. Reprint edition. Cambridge, Massachusetts London, Englang: MIT Press.

\n\n

Nygard2018 - Nygard, Michael T. Release It! Design and Deploy Production-Ready Software. 2nd ed. edition. Raleigh, North Carolina: O′Reilly, 2018.

\n\n

\n Chapter 2 - Core Operability Practices\n

\n\n

Allspaw2010 - John Allspaw, and Jesse Robbins. 2010. Web Operations. O’Reilly Media. http://shop.oreilly.com/product/0636920000136.do

\n\n

Binette2018 - Binette, Elisa. 2018. “Your Guide to Setting SLOs and SLIs.” New Relic Blog (blog). October 31, 2018. https://blog.newrelic.com/engineering/best-practices-for-setting-slos-and-slis-for-modern-complex-systems/

\n\n

Cohn2008 - M. Cohn, “Non-functional Requirements as User Stories”, 21-Nov-2008. [Online]. Available: http://www.mountaingoatsoftware.com/blog/non-functional-requirements-as-user-stories [Accessed: 12-May-2014].

\n\n

Cohn2016 - Cohn, Mike. “What Are Story Points?” Mountain Goat Software. August 23, 2016. https://www.mountaingoatsoftware.com/blog/what-are-story-points

\n\n

Davies2010 - R. Davies, ‘Non-Functional Requirements: Do User Stories Really Help?’, 2010. [Online]. Available: http://www.methodsandtools.com/archive/archive.php?id=113 [Accessed: 12-May-2014]

\n\n

Geckoboard2020 - Laura Kukula | 9 dashboard design principles: see them in action with real examples. 20 February 2020. https://www.geckoboard.com/blog/9-dashboard-design-principles-see-them-in-action-with-real-examples/

\n\n

Gilb2009 - T. Gilb, “Tom Gilb & Kai Gilb - Helping you deliver Value to your Stakeholders | Are non-functional requirements functional? : Tom Gilb and Kai Gilb’s blog”, 18-Jan-2009. [Online]. Available: http://www.gilb.com/blogpost70-Are-non-functional-requirements-functional [Accessed: 12-May-2014].

\n\n

Humble2010b - Humble, Jez, and David Farley. 2010. “Continuous Delivery: Anatomy of the Deployment Pipeline.” http://www.informit.com/articles/article.aspx?p=1621865

\n\n

Kubernetes2018 - Kubernetes. 2018. “Configure Liveness and Readiness Probes.” July 2018. https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-probes/

\n\n

LonWorks2009 - Echelon Corporation. 2009. “Introduction to the LonWorks® Platform.” http://downloads.echelon.com/support/documentation/manuals/general/078-0183-01B_Intro_to_LonWorks_Rev_2.pdf

\n\n

Mitchell2017 - Ian Mitchell, “Walking Through a Definition of Done.” Scrum.Org. May 17, 2017. http://www.scrum.org/resources/blog/walking-through-definition-done

\n\n

Murphy2016 - Murphy, Niall, Betsy Beyer, Chris Jones, and Jennifer Petoff. 2016. Site Reliability Engineering. O’Reilly Media. http://shop.oreilly.com/product/0636920041528.do

\n\n

SAFe2014 - ‘Scaled Agile Framework’. [Online]. Available: http://scaledagileframework.com/ [Accessed: 12-May-2014].

\n\n

SkeltonPaisLoggin2016 - Matthew Skelton and Manuel Pais. 2016. ‘Why and How to Test Logging’. InfoQ. https://www.infoq.com/articles/why-test-logging/

\n\n

\n Chapter 3 - Use Run Book collaboration to increase operability and prevent operational issues\n

\n\n

Gawande2011b - A. Gawande, The Checklist Manifesto: How to Get Things Right, Reprint edition. New York: Picador, 2011.

\n\n

Goldschrafe2011 - J. Goldschrafe, 2011 ‘Runbooks are stupid and you’re doing them wrong’ 19-08-2011. [Online] http://holyhandgrenade.org/blog/2011/08/runbooks-are-stupid-and-youre-doing-them-wrong/ [Accessed 27-Oct-2016]

\n\n

Kelly2016 - A. Kelly, 2016 ‘Dialogue Sheets’. [Online] https://www.softwarestrategy.co.uk/dialogue-sheets/ [Accessed 24-Oct-2016]

\n\n

\n Chapter 4 - Use modern log aggregation for deep operational and insights\n

\n\n

Black2016 - David Black, 2016 - “A “Log Reflector” for AWS Lambda” http://blog.davidablack.net/2016/08/17/a-log-reflector-for-aws-lambda/

\n\n

Boswell2017 - Drew Boswell, 2017 - ‘A Million Metrics per Second’ [Online] https://medium.com/swissquote-engineering/a-million-metrics-per-second-17a4c7274062 [Accessed 28-Jan-2018]

\n\n

Bourgon2017 - Peter Bourgon, 2017 - ‘Metrics, tracing, and logging’ http://peter.bourgon.org/blog/2017/02/21/metrics-tracing-and-logging.html

\n\n

Cheney2015 - Dave Cheney, 2015 - “Let’s Talk About Logging” https://dave.cheney.net/2015/11/05/lets-talk-about-logging

\n\n

Cui2017 - Yan Cui, 2017 - “Yubl’s Road to Serverless, Part 3 - Ops” https://hackernoon.com/yubls-road-to-serverless-part-3-ops-6c82139bb7ee

\n\n

DeBortoli2017 - Alberto De Bortoli, 2017 - “A better local and remote logging on iOS with JustLog” https://tech.just-eat.com/2017/01/18/a-better-local-and-remote-logging-on-ios-with-justlog/

\n\n

Degioanni2015 - Degioanni, Loris. 2015. “How to Collect StatsD Metrics in Containers.” Sysdig. June 3. https://sysdig.com/blog/how-to-collect-statsd-metrics-in-containers/

\n\n

GrahamCumming2017 - John Graham-Cumming, 2017 ‘Incident report on memory leak caused by Cloudflare parser bug’ https://blog.cloudflare.com/incident-report-on-memory-leak-caused-by-cloudflare-parser-bug/

\n\n

Golubenco2016 - Tudor Golubenco, 2016 “Structured logging with Filebeat” https://www.elastic.co/blog/structured-logging-filebeat

\n\n

Kreps2013 - Jay Kreps, 2013 ‘The Log: What every software engineer should know about real-time data’s unifying abstraction’ https://engineering.linkedin.com/distributed-systems/log-what-every-software-engineer-should-know-about-real-time-datas-unifying

\n\n

Majors2017 - Charity Majors, 2017. “Lies My Parents Told Me (About Logs)” [Online] https://honeycomb.io/blog/2017/04/lies-my-parents-told-me-about-logs/ [Accessed 28-Jan-2018]

\n\n

OpenTracing2017 - OpenTracing. 2017. “Introduction · Opentracing.” 2017. http://opentracing.io/documentation/#what-is-a-trace

\n\n

Paraschiv2017 - Eugen Paraschiv, 2017 ‘9 Logging Sins in Your Java Applications’. [Online] https://dzone.com/articles/9-logging-sins-in-your-java-applications [Accessed 28-Jan-2018]

\n\n

Pivotal2017 - Pivotal, 2017 “Monitoring and Troubleshooting Apps with PCF Metrics | Pivotal Web Services Docs.” Pivotal Web Services Documentation. http://docs.run.pivotal.io/metrics/using.html#trace.

\n\n

Rapid72016 - Rapid7, 2016 “Logging Mosquitto Server logs (from Raspberry Pi) to Logentries” https://blog.rapid7.com/2016/10/07/logging-mosquitto-server-logs-from-raspberry-pi-to-logentries/

\n\n

Reselman2016 - Bob Reselman, 2016 ‘The Value of Correlation IDs’ https://blog.logentries.com/2016/12/the-value-of-correlation-ids/

\n\n

Skelton2012 - Matthew Skelton, 2012 “Tune Logging Levels In Production Without Recompiling Code” https://blog.matthewskelton.net/2012/12/05/tune-logging-levels-in-production-without-recompiling-code/

\n\n

Sridharan2017 - Cindy Sridharan, 2017 - “Logs and Metrics” https://medium.com/@copyconstruct/logs-and-metrics-6d34d3026e38

\n\n

Swan2017 - Chris Swan, 2017 “Operational Considerations for Containers”. [Online] https://www.infoq.com/presentations/containers-operations [Accessed 28-Jan-2018]

\n\n

Turnbull2015 - James Turnbull, 2015 ‘Structuted Logging’. [Online] https://kartar.net/2015/12/structured-logging/ [Accessed 28-Jan-2018]

\n\n

\n Chapter 5 - Use well-defined readiness checks to increase operational confidence\n

\n\n\n

TBC

\n\n

\n Chapter 6 - Use operability as a differentiating aspect of your software\n

\n\n

KeepItUsable - “Personas: Why is it important to understand your users?”. https://www.keepitusable.com/blog/personas-why-is-it-important-to-understand-your-users/

\n\n

\n Chapter 7 - Use patterns and principles from Team Topologies to inform the approach to operability\n

\n\n\n

TBC

\n\n

TBC

\n\n

\n Appendix\n

\n\n\n

AgileManifesto2001 - Manifesto for Agile Software Development. 2001. http://agilemanifesto.org/

\n\n

Leveson2017a - Leveson, Nancy G. 2017. Engineering a Safer World: Systems Thinking Applied to Safety. Reprint edition. Cambridge, Massachusetts London, Englang: MIT Press.

\n\n

Mogul2006 - Mogul, Jeffrey C. 2006. “Emergent (Mis)Behavior vs. Complex Software Systems.” In Proceedings of the 1st ACM SIGOPS/EuroSys European Conference on Computer Systems 2006, 293–304. EuroSys ’06. New York, NY, USA: ACM. https://doi.org/10.1145/1217935.1217964

\n\n\n

\n\n\n\n

\n\n\n\n

\nRun Book template

\n\n\n

\n\n\n\n

\n\n\n\n\n

\nService or system overview

\n\n

Service or system name:

\n\n

\nBusiness overview

\n\n

\n
What business need is met by this service or system? What expectations do we have about availability and performance?
\n

\n\n

(e.g. Provides reliable automated reconciliation of logistics transactions from the previous 24 hours)

\n\n

\nTechnical overview

\n\n

\n
What kind of system is this? Web-connected order processing? Back-end batch system? Internal HTTP-based API? ETL control system?
\n

\n\n

(e.g. Internal API for order reconcilation based on Ruby and RabbitMQ, deployed in Docker containers on Kubernetes)

\n\n

\nService Level Agreements (SLAs)

\n\n

\n
What explicit or implicit expectations are there from users or clients about the availability of the service or system?
\n

\n\n

(e.g. Contractual 99.9% service availability outside of the 03:00-05:00 maintenance window)

\n\n

\nService owner

\n\n

\n
Which team owns and runs this service or system?
\n

\n\n

(e.g. The *Sneaky Sharks* team (Bangalore) develops and runs this service: sneaky.sharks@company.com / *#sneaky-sharks* on Slack / Extension 9265)

\n\n

\nContributing applications, daemons, services, middleware

\n\n

\n
Which distinct software applications, daemons, services, etc. make up the service or system? What external dependencies does it have?
\n

\n\n

(e.g. Ruby app + RabbitMQ for source messages + PostgreSQL for reconciled transactions)

\n\n

\nSystem characteristics

\n\n

\nHours of operation

\n\n

\n
During what hours does the service or system actually need to operate? Can portions or features of the system be unavailable at times if needed?
\n

\n\n

\nHours of operation - core features

\n\n

(e.g. 03:00-01:00 GMT+0)

\n\n

\nHours of operation - secondary features

\n\n

(e.g. 07:00-23:00 GMT+0)

\n\n

\nData and processing flows

\n\n

\n
How and where does data flow through the system? What controls or triggers data flows?
\n

\n\n

(e.g. mobile requests / scheduled batch jobs / inbound IoT sensor data )

\n\n

\nInfrastructure and network design

\n\n

\n
What servers, containers, schedulers, devices, vLANs, firewalls, etc. are needed?
\n

\n\n

(e.g. ‘10+ Ubuntu 14 VMs on AWS IaaS + 2 AWS Regions + 2 VPCs per Region + Route53’)

\n\n

\nResilience, Fault Tolerance (FT) and High Availability (HA)

\n\n

\n
How is the system resilient to failure? What mechanisms for tolerating faults are implemented? How is the system/service made highly available?
\n

\n\n

(e.g. 2 Active-Active data centres across two cities + two or more nodes at each layer)

\n\n

\nThrottling and partial shutdown

\n\n

\n
How can the system be throttled or partially shut down e.g. to avoid flooding other dependent systems? Can the throughput be limited to (say) 100 requests per second? etc. What kind of connection back-off schemes are in place?
\n

\n\n

\nThrottling and partial shutdown - external requests

\n\n

(e.g. Commercial API gateway allows throttling control)

\n\n

\nThrottling and partial shutdown - internal components

\n\n

(e.g. Exponential backoff on all HTTP-based services + /health healthcheck endpoints on all services)

\n\n

\nExpected traffic and load

\n\n

\n
Details of the expected throughput/traffic: call volumes, peak periods, quiet periods. What factors drive the load: bookings, page views, number of items in Basket, etc.)
\n

\n\n

(e.g. Max: 1000 requests per second with 400 concurrent users - Friday @ 16:00 to Sunday @ 18:00, driven by likelihood of barbecue activity in the neighborhood)

\n\n

\nHot or peak periods

\n\n

\nWarm periods

\n\n

\nCool or quiet periods

\n\n

\nEnvironmental differences

\n\n

\n
What are the main differences between Production/Live and other environments? What kinds of things might therefore not be tested in upstream environments?
\n

\n\n

(e.g. Self-signed HTTPS certificates in Pre-Production - certificate expiry may not be detected properly in Production)

\n\n

\nTools

\n\n

\n
What tools are available to help operate the system?
\n

\n\n

(e.g. Use the queue-cleardown.sh script to safely cleardown the processing queue nightly)

\n\n

\nRequired resources

\n\n

\n
What compute, storage, database, metrics, logging, and scaling resources are needed? What are the minimum and expected maximum sizes (in CPU cores, RAM, GB disk space, GBit/sec, etc.)?
\n

\n\n

\nRequired resources - compute

\n\n

(e.g. Min: 4 VMs with 2 vCPU each. Max: around 40 VMs)

\n\n

\nRequired resources - storage

\n\n

(e.g. Min: 10GB Azure blob storage. Max: around 500GB Azure blob storage)

\n\n

\nRequired resources - database

\n\n

(e.g. Min: 500GB Standard Tier RDS. Max: around 2TB Standard Tier RDS)

\n\n

\nRequired resources - metrics

\n\n

(e.g. Min: 100 metrics per node per minute. Max: around 6000 metrics per node per minute)

\n\n

\nRequired resources - logging

\n\n

(e.g. Min: 60 log lines per node per minute (100KB). Max: around 6000 log lines per node per minute (1MB))

\n\n

\nRequired resources - other

\n\n

(e.g. Min: 10 encryption requests per node per minute. Max: around 100 encryption requests per node per minute)

\n\n

\nSecurity and access control

\n\n

\nPassword and PII security

\n\n

\n
What kind of security is in place for passwords and Personally Identifiable Information (PII)? Are the passwords hashed with a strong hash function and salted?
\n

\n\n

(e.g. Passwords are hashed with a 10-character salt and SHA265)

\n\n

\nOngoing security checks

\n\n

\n
How will the system be monitored for security issues?
\n

\n\n

(e.g. The ABC tool scans for reported CVE issues and reports via the ABC dashboard)

\n\n

\nSystem configuration

\n\n

\nConfiguration management

\n\n

\n
How is configuration managed for the system?
\n

\n\n

(e.g. CloudInit bootstraps the installation of Puppet - Puppet then drives all system and application level configuration except for the XYZ service which is configured via App.config files in Subversion)

\n\n

\nSecrets management

\n\n

\n
How are configuration secrets managed?
\n

\n\n

(e.g. Secrets are managed with Hashicorp Vault with 3 shards for the master key)

\n\n

\nSystem backup and restore

\n\n

\nBackup requirements

\n\n

\n
Which parts of the system need to be backed up?
\n

\n\n

(e.g. Only the CoreTransactions database in PostgreSQL and the Puppet master database need to be backed up)

\n\n

\nBackup procedures

\n\n

\n
How does backup happen? Is service affected? Should the system be [partially] shut down first?
\n

\n\n

(e.g. Backup happens from the read replica - live service is not affected)

\n\n

\nRestore procedures

\n\n

\n
How does restore happen? Is service affected? Should the system be [partially] shut down first?
\n

\n\n

(e.g. The Booking service must be switched off before Restore happens otherwise transactions will be lost)

\n\n

\nMonitoring and alerting

\n\n

\nLog aggregation solution

\n\n

\n
What log aggregation & search solution will be used?
\n

\n\n

(e.g. The system will use the existng in-house ELK cluster. 2000-6000 messages per minute expected at normal load levels)

\n\n

\nLog message format

\n\n

\n
What kind of log message format will be used? Structured logging with JSON? log4j style single-line output?
\n

\n\n

(e.g. Log messages will use log4j compatible single-line format with wrapped stack traces)

\n\n

\nEvents and error messages

\n\n

\n
What significant events, state transitions and error events may be logged?
\n

\n\n

(e.g. IDs 1000-1999: Database events; IDs 2000-2999: message bus events; IDs 3000-3999: user-initiated action events; …)

\n\n

\nMetrics

\n\n

\n
What significant metrics will be generated?
\n

\n\n

(e.g. Usual VM stats (CPU, disk, threads, etc.) + around 200 application technical metrics + around 400 user-level metrics)

\n\n

\nHealth checks

\n\n

\n
How is the health of dependencies (components and systems) assessed? How does the system report its own health?
\n

\n\n

\nHealth of dependencies

\n\n

(e.g. Use /health HTTP endpoint for internal components that expose it. Other systems and external endpoints: typically HTTP 200 but some synthetic checks for some services)

\n\n

\nHealth of service

\n\n

(e.g. Provide /health HTTP endpoint: 200 –> basic health, 500 –> bad configuration + /health/deps for checking dependencies)

\n\n

\nOperational tasks

\n\n

\nDeployment

\n\n

\n
How is the software deployed? How does roll-back happen?
\n

\n\n

(e.g. We use GoCD to coordinate deployments, triggering a Chef run pulling RPMs from the internal yum repo)

\n\n

\nBatch processing

\n\n

\n
What kind of batch processing takes place?
\n

\n\n

(e.g. Files are pushed via SFTP to the media server. The system processes up to 100 of these per hour on a cron schedule)

\n\n

\nPower procedures

\n\n

\n
What needs to happen when machines are power-cycled (or cold-rebooted)?
\n

\n\n

(e.g. *** WARNING: we have not investigated this scenario yet! ***)

\n\n

\nRoutine and sanity checks

\n\n

\n
What kind of checks need to happen on a regular basis?
\n

\n\n

(e.g. All /health endpoints should be checked every 60secs plus the synthetic transaction checks run every 5 mins via Pingdom)

\n\n

\nTroubleshooting

\n\n

\n
How should troubleshooting happen? What tools are available?
\n

\n\n

(e.g. Use a combination of the /health endpoint checks and the abc-*.sh scripts for diagnosing typical problems)

\n\n

\nMaintenance tasks

\n\n

\nPatching

\n\n

\n
How should patches be deployed and tested?
\n

\n\n

\nNormal patch cycle

\n\n

(e.g. Use the standard OS patch test cycle together with deployment via Jenkins and Capistrano)

\n\n

\nZero-day vulnerabilities

\n\n

(e.g. Use the early-warning notifications from UpGuard plus deployment via Jenkins and Capistrano)

\n\n

\nDaylight-saving time changes

\n\n

\n
Is the software affected by daylight-saving time changes (both client and server)?
\n

\n\n

(e.g. Server clocks all set to UTC+0. All date/time data converted to UTC with offset before processing)

\n\n

\nData cleardown

\n\n

\n
Which data needs to be cleared down? How often? Which tools or scripts control cleardown?
\n

\n\n

(e.g. Use abc-cleardown.ps1 run nightly to clear down the document cache)

\n\n

\nLog rotation

\n\n

\n
Is log rotation needed? How is it controlled?
\n

\n\n

(e.g. The Windows Event Log *ABC Service* is set to a maximum size of 512MB)

\n\n

\nFailover and Recovery procedures

\n\n

\n
What needs to happen when parts of the system are failed over to standby systems? What needs to during recovery?
\n

\n\n

\nFailover

\n\n

\nRecovery

\n\n

\nTroubleshooting Failover and Recovery

\n\n

\n
What tools or scripts are available to troubleshoot failover and recovery operations?
\n

\n\n

(e.g. Start with running SELECT state__desc FROM sys.database__mirroring__endpoints on the PRIMARY node and then use the scripts in the *db-failover* Git repo)

\n\n\n

\n\n\n\n

\n\n\n\n

\nAbout the authors

\n\n\n

\n\n\n\n

\n\n\n

\nMatthew Skelton

\n\n\n

Matthew Skelton is co-author of the award-winning and ground-breaking book Team Topologies, and CEO & Principal at Conflux. The Team Topologies book was rated one of the â€˜Best product management books of all timeâ€™ by Book Authority and is widely used by organizations worldwide to transform the way they deliver value.

\n\n

Matthew is one of the foremost leaders in modern organizational dynamics for fast flow, drawing on Team Topologies and related practices to support organizations with transformation towards a sustainable fast flow of value and true business agility.

\n\n

A Chartered Engineer (CEng), Matthew brings together principles and practices from multiple disciplines for a holistic approach to digitally-enriched operating models. He combines his experience as a leader and software architect in multiple contexts (GOV.UK, ecommerce, financial services, telecoms, pharma, retail, robotics, etc.) with a strong interest in the human side of organizations for a compassionate and humanistic approach to organizational effectiveness.

\n\n

LinkedIn: https://www.linkedin.com/in/matthewskelton/

\n\n

\nAlex Moore

\n\n\n

Alex is a DevOps Strategy Consultant at BJSS. She is a keen promoter of all things DevOps and Operability. She has a background in systems engineering, IT and software support, mainly in the Financial Services sector.

\n\n

She is passionate about collaboration and can often be found at Tech Community events such as Leeds DevOps, Leeds Test Atelier and DevOpsDays.

\n\n

When she occasionally stops talking about DevOps, she loves to get her car out on the track or listen to live music.

\n\n

LinkedIn: https://www.linkedin.com/in/alexhelenmoore/

\n\n

\nRob Thatcher

\n\n\n

\n\n\n

Rob Thatcher has substantial experience helping organisations to build effective technical operations, support, and delivery teams, and design and operate effective IT architectures. With Director-level experience in the financial services sector, his focus now on building teams through mentoring and coaching with a pragmatic approach to help people build confidence, knowledge and capabilities.

\n\n

LinkedIn: https://www.linkedin.com/in/robthatchr/

\n\n\n

\n\n\n\n

\n\n\n\n

\n $\"\"$ \n

\n \n

\n\n\n\n

\nConflux Books

\n\n

Books for technologists by technologists

\n\n

Our books help to accelerate and deepen your learning in the field of software systems. We focus on subjects that don’t go out of date: fundamental software principles & practices, team interactions, and technology-independent skills.

\n\n

Current and planned titles in the Conflux Books series include:

\n\n

\nBuild Quality In edited by Steve Smith and Matthew Skelton (B01)
\nBetter Whiteboard Sketches by Matthew Skelton (B02)
\nInternal Tech Conferences by Victoria Morgan-Smith and Matthew Skelton (B03)
\nTechnical Writing for Blogs and Articles by Matthew Skelton (B04)

\n\n\n\n\n

\n\n\n\n\n

\n\n\n

Build Quality In - a book of Continuous Delivery and DevOps experience reports. Edited by Steve Smith and Matthew Skelton. Conflux Books, April 2015

\n\n

\n\n\n

Internal Tech Conferences by Victoria Morgan-Smith and Matthew Skelton. Conflux Books, April 2019

\n\n

\n\n\n

\n\n\n

Better Whiteboard Sketches by Matthew Skelton. Conflux Books, August 2019

\n\n

\n\n\n

Team Guides for Software: Software Operability, Metrics for Business Decisions, Software Testability, Software Releasability. Skelton Thatcher Publications, 2016-2019

\n\n

\nNotes

1We were inspired by Allan Kelly’s Dialogue Sheets for agile retrospectives and recommend that you try them out! Thanks to Allan for his input to the Run Book dialogue sheets. See Kelly2016.↩

\n\n\n

\n\n

\n\n","tocStructure",[299,300,301,302,303,304,305,306,307,308,309,310,311,312,313,314,315,316,317,318,319,320,321,322,323,324,325,326,327,328,329,330,331,332,333,334,335,336,337,338,339,340,341,342,343,344,345,346,347,348,349,350,351,352],{"_86":87,"_43":88,"_80":-5,"_89":90,"_82":91,"_107":-5,"_108":-5,"_94":-5},{"_86":87,"_43":96,"_80":-5,"_89":97,"_82":91,"_107":-5,"_108":-5,"_94":-5},{"_86":87,"_43":100,"_80":-5,"_89":101,"_82":91,"_107":-5,"_108":-5,"_94":-5},{"_86":104,"_43":105,"_80":-5,"_89":106,"_82":91,"_107":-5,"_108":-5,"_94":-5},{"_86":104,"_43":110,"_80":-5,"_89":111,"_82":91,"_107":-5,"_108":-5,"_94":-5},{"_86":104,"_43":113,"_80":-5,"_89":114,"_82":91,"_107":-5,"_108":-5,"_94":-5},{"_86":104,"_43":116,"_80":-5,"_89":117,"_82":91,"_107":-5,"_108":-5,"_94":-5},{"_86":104,"_43":119,"_80":-5,"_89":120,"_82":91,"_107":-5,"_108":-5,"_94":-5},{"_86":104,"_43":122,"_80":-5,"_89":123,"_82":91,"_107":-5,"_108":-5,"_94":-5},{"_86":104,"_43":125,"_80":-5,"_89":126,"_82":91,"_107":-5,"_108":-5,"_94":-5},{"_86":104,"_43":128,"_80":-5,"_89":129,"_82":91,"_107":-5,"_108":-5,"_94":-5},{"_86":87,"_43":131,"_80":-5,"_89":132,"_82":91,"_107":-5,"_108":-5,"_94":-5},{"_86":87,"_43":135,"_80":-5,"_89":136,"_82":91,"_107":-5,"_108":-5,"_94":-5},{"_86":87,"_43":139,"_80":-5,"_89":140,"_82":91,"_107":-5,"_108":-5,"_94":-5},{"_86":104,"_43":143,"_80":-5,"_89":144,"_82":91,"_107":-5,"_108":-5,"_94":-5},{"_86":104,"_43":146,"_80":-5,"_89":147,"_82":91,"_107":-5,"_108":-5,"_94":-5},{"_86":104,"_43":149,"_80":-5,"_89":150,"_82":91,"_107":-5,"_108":-5,"_94":-5},{"_86":104,"_43":152,"_80":-5,"_89":153,"_82":91,"_107":-5,"_108":-5,"_94":-5},{"_86":104,"_43":155,"_80":-5,"_89":156,"_82":91,"_107":-5,"_108":-5,"_94":-5},{"_86":87,"_43":158,"_80":-5,"_89":159,"_82":91,"_107":-5,"_108":-5,"_94":-5},{"_86":87,"_43":162,"_80":-5,"_89":163,"_82":91,"_107":-5,"_108":-5,"_94":-5},{"_86":87,"_43":166,"_80":-5,"_89":167,"_82":91,"_107":-5,"_108":-5,"_94":-5},{"_86":87,"_43":170,"_80":-5,"_89":171,"_82":91,"_107":-5,"_108":-5,"_94":-5},{"_86":87,"_43":174,"_80":-5,"_89":175,"_82":91,"_107":-5,"_108":-5,"_94":-5},{"_86":104,"_43":178,"_80":-5,"_89":179,"_82":91,"_107":-5,"_108":-5,"_94":-5},{"_86":104,"_43":181,"_80":-5,"_89":182,"_82":91,"_107":-5,"_108":-5,"_94":-5},{"_86":87,"_43":184,"_80":-5,"_89":185,"_82":91,"_107":-5,"_108":-5,"_94":-5},{"_86":87,"_43":188,"_80":-5,"_89":189,"_82":91,"_107":-5,"_108":-5,"_94":-5},{"_86":104,"_43":100,"_80":-5,"_89":192,"_82":91,"_107":-5,"_108":-5,"_94":-5},{"_86":104,"_43":194,"_80":-5,"_89":195,"_82":91,"_107":-5,"_108":-5,"_94":-5},{"_86":104,"_43":197,"_80":-5,"_89":198,"_82":91,"_107":-5,"_108":-5,"_94":-5},{"_86":104,"_43":200,"_80":-5,"_89":201,"_82":91,"_107":-5,"_108":-5,"_94":-5},{"_86":104,"_43":203,"_80":-5,"_89":204,"_82":91,"_107":-5,"_108":-5,"_94":-5},{"_86":104,"_43":206,"_80":-5,"_89":207,"_82":91,"_107":-5,"_108":-5,"_94":-5},{"_86":104,"_43":209,"_80":-5,"_89":210,"_82":91,"_107":-5,"_108":-5,"_94":-5},{"_86":104,"_43":212,"_80":-5,"_89":213,"_82":91,"_107":-5,"_108":-5,"_94":-5},{"_86":104,"_43":215,"_80":-5,"_89":216,"_82":91,"_107":-5,"_108":-5,"_94":-5},{"_86":87,"_43":218,"_80":-5,"_89":219,"_82":91,"_107":-5,"_108":-5,"_94":-5},{"_86":104,"_43":222,"_80":-5,"_89":223,"_82":91,"_107":-5,"_108":-5,"_94":-5},{"_86":104,"_43":225,"_80":-5,"_89":226,"_82":91,"_107":-5,"_108":-5,"_94":-5},{"_86":104,"_43":228,"_80":-5,"_89":229,"_82":91,"_107":-5,"_108":-5,"_94":-5},{"_86":104,"_43":231,"_80":-5,"_89":232,"_82":91,"_107":-5,"_108":-5,"_94":-5},{"_86":104,"_43":234,"_80":-5,"_89":235,"_82":91,"_107":-5,"_108":-5,"_94":-5},{"_86":104,"_43":237,"_80":-5,"_89":238,"_82":91,"_107":-5,"_108":-5,"_94":-5},{"_86":104,"_43":240,"_80":-5,"_89":241,"_82":91,"_107":-5,"_108":-5,"_94":-5},{"_86":104,"_43":243,"_80":-5,"_89":244,"_82":91,"_107":-5,"_108":-5,"_94":-5},{"_86":104,"_43":246,"_80":-5,"_89":247,"_82":91,"_107":-5,"_108":-5,"_94":-5},{"_86":104,"_43":249,"_80":-5,"_89":250,"_82":91,"_107":-5,"_108":-5,"_94":-5},{"_86":87,"_43":252,"_80":-5,"_89":253,"_82":91,"_107":-5,"_108":-5,"_94":-5},{"_86":104,"_43":61,"_80":-5,"_89":256,"_82":91,"_107":-5,"_108":-5,"_94":-5},{"_86":104,"_43":72,"_80":-5,"_89":258,"_82":91,"_107":-5,"_108":-5,"_94":-5},{"_86":104,"_43":67,"_80":-5,"_89":260,"_82":91,"_107":-5,"_108":-5,"_94":-5},{"_86":87,"_43":262,"_80":-5,"_89":263,"_82":91,"_107":-5,"_108":-5,"_94":-5},{"_86":87,"_43":266,"_80":-5,"_89":267,"_82":91,"_107":-5,"_108":-5,"_94":-5},"publicChapterContent","nextChapter","sections",[],"routes/_store+/_layout","routes/_store+/_readers+/_layout","actionData","errors"]

\nNotes

Table of Contents

\nTeam Guides for Software\n

\nForeword

\nIntroduction

\nWhat is software operability and why should we care?

\nWhere can operability techniques be used?

\nHow to use this book

\nRecommended reading for Product Managers

\nRecommended reading for Software and System Architects

\nRecommended reading for Team Leads and engineers

\nRecommended reading for Enabling teams

\nTerminology:

\n‘Slow-burner’ organisational changes:

\nWhat is covered in this book

\nWhy we wrote this book

\nFeedback and suggestions

\n1. What does good operability look like?

\n2. Core practices for good software operability

\n3. Use Run Book collaboration to increase operability and prevent operational issues

\n3.1 Operational aspects are very similar across many software systems

\n3.2 Use a Run Book template as a common baseline for operational aspects

\nSoftware developers and testers complete the first draft of the Run Book

\nRun Book template example

\n3.3 Use a Run Book Dialogue Sheet to facilitate discovery and avoid ‘documentation fallacy’

\n3.4 Assess operability on a regular basis: every sprint, iteration, or week

\n3.5 Summary

\n4. Use modern log aggregation for deep operational insights

\n5. Use Deployment Verification Tests and Endpoint Healthchecks for rapid feedback on environments

\n6. Use information radiators and dashboards to drive effective behaviour and good psychological responses

\n7. Make operability part of the software product

\n8. Appendix

\n8.1 Adapt your logging techniques to the technology characteristics

\nLogging and metrics for applications running on x86-based machines

\nLogging and metrics for containers and containerized applications

\nLogging and metrics for Serverless / Function-as-a-Service (FaaS) and Platform-as-a-Service (PaaS)

\nLogging and metrics for IoT / connected devices and embedded systems

\nLogging and metrics for mobile apps

\n8.2 Understand how the complexity of modern distributed systems drives a need for a focus on operability

\nTerminology

\nReferences and further reading

\nIntroduction\n

\nChapter 1 - What does good operability look like?\n

\nChapter 2 - Core Operability Practices\n

\nChapter 3 - Use Run Book collaboration to increase operability and prevent operational issues\n

\nChapter 4 - Use modern log aggregation for deep operational and insights\n

\nChapter 5 - Use well-defined readiness checks to increase operational confidence\n

\nChapter 6 - Use operability as a differentiating aspect of your software\n

\nChapter 7 - Use patterns and principles from Team Topologies to inform the approach to operability\n

\nAppendix\n

\nRun Book template

\nService or system overview

\nBusiness overview

\nTechnical overview

\nService Level Agreements (SLAs)

\nService owner

\nContributing applications, daemons, services, middleware

\nSystem characteristics

\nHours of operation

\nHours of operation - core features

\nHours of operation - secondary features

\nData and processing flows

\nInfrastructure and network design

\nResilience, Fault Tolerance (FT) and High Availability (HA)

\nThrottling and partial shutdown

\nThrottling and partial shutdown - external requests

\nThrottling and partial shutdown - internal components

\nExpected traffic and load

\nHot or peak periods

\nWarm periods

\nCool or quiet periods

\nEnvironmental differences

\nTools

\nRequired resources

\nRequired resources - compute

\nRequired resources - storage

\nRequired resources - database

\nRequired resources - metrics

\nRequired resources - logging

\nRequired resources - other

\n Introduction\n

\n Chapter 1 - What does good operability look like?\n

\n Chapter 2 - Core Operability Practices\n

\n Chapter 3 - Use Run Book collaboration to increase operability and prevent operational issues\n

\n Chapter 4 - Use modern log aggregation for deep operational and insights\n

\n Chapter 5 - Use well-defined readiness checks to increase operational confidence\n

\n Chapter 6 - Use operability as a differentiating aspect of your software\n

\n Chapter 7 - Use patterns and principles from Team Topologies to inform the approach to operability\n

\n Appendix\n