Table of Contents
- 1. We have a failure to communicate
- 2. A shared vocabulary
- 3. The C4 model
4. Software documentation as a guidebook
- 4.1 The code doesn’t tell the whole story
- 4.2 Our duty to deliver documentation
- 4.3 Lightweight, supplementary documentation
- 4.4 1. Maps
- 4.5 2. Sights
- 4.6 3. History and culture
- 4.7 4. Practical information
- 4.8 Describe what you can’t get from the code
- 4.9 Product vs project documentation
- 4.10 Keeping documentation up to date
- 4.11 Documentation length
- 4. Software documentation as a guidebook
I graduated from university in 1996, a time when CASE and modeling tools were popular and in common use. I remember attending a training course about the Unified Modeling Language and the SELECT tooling soon after I started my professional career. A number of the projects I worked on made extensive use of tools like SELECT and Rational Rose for diagramming and documenting the design of software systems. With buggy user interfaces and ugly diagrams, the tooling may not have been brilliant back then, but it was still very useful if used in a pragmatic way.
I’m very much a visual person. I like being able to visualise a problem before trying to find a solution. Describe a business process to me and I’ll sketch up a summary of it. Talk to me about a business problem and I’m likely to draw a high-level domain model. Visualising the problem is a way for me to ask questions and figure out whether I’ve understood what you’re saying. I also like sketching out solutions to problems, again because it’s a great way to get everything out into the open in a way that other people can understand quickly.
But then something happened somewhere during the early 2000s, probably as a result of the Manifesto for Agile Software Development that had been published a few years beforehand. The way teams built software started to change, with things like diagramming and documentation being thrown away alongside big design up front. I remember seeing a number of software development teams reducing the quantity of diagrams and documentation they were creating. In fact, I was often the only person on the team who really understood UML well enough to create diagrams with it.
Fast forward to 2019; creating software architecture diagrams and documentation seems to be something of a lost art. I’ve been running software architecture training courses for a number of years, part of which is a simple architecture kata where groups of people are asked to design a software solution and draw some architecture diagrams to describe it. Over 10,000 people and 30 countries later, I have gigabytes of anecdotal photo evidence - the majority of the diagrams use an ad hoc “boxes and lines” notation with no clear notation or semantics. Designing software is where the complexity should be, not communicating it.
This book focusses on the visual communication and documentation of software architecture. I’ve seen a number of debates over the years about whether software development is an art, a craft or an engineering discipline. Although I think it should be an engineering discipline, I believe we’re a number of years away from this being a reality. So while this book doesn’t present a formalised, standardised method to communicate software architecture, it does provide a collection of lightweight ideas and techniques that thousands of people across the world find useful. The core of this is my “C4 model” for visualising software architecture, and the “software guidebook”. You’ll also find discussion about notation, the various uses for diagrams, the value of creating a model and tooling.
I’m an independent software development consultant specialising in software architecture; specifically technical leadership, communication and lightweight, pragmatic approaches to software architecture. In addition to being the author of Software Architecture for Developers, I’m the creator of the C4 software architecture model and I built Structurizr, which is a collection of tooling to help you visualise, document and explore your software architecture.
I regularly speak at software development conferences, meetups and organisations around the world; delivering keynotes, presentations and workshops about software architecture. In 2013, I won the IEEE Software sponsored SATURN 2013 “Architecture in Practice” Presentation Award for my presentation about the conflict between agile and architecture. I’ve spoken at events and/or have clients in over thirty countries around the world.
This part of the book is about visualising software architecture using a collection of diagrams to tell different parts of the same overall story.
We’ve reached an interesting point in the software development industry. Globally distributed teams are building Internet-scale software systems in all manner of programming languages, with architectures ranging from monolithic systems through to those composed of dozens of microservices. Agile and lean approaches are now no longer seen as niche ways to build software, and even the most traditional of organisations are seeking fast feedback with minimum viable products to prove their ideas. Techniques such as automated testing and continuous delivery coupled with the power of cloud computing make this a reality for organisations of any size too. But there’s something still missing.
Ask somebody in the building industry to visually communicate the architecture of a building and you’ll likely be presented with site plans, floor plans, elevation views, cross-section views and detail drawings. In contrast, ask a software developer to communicate the software architecture of a software system using diagrams, and you’ll likely get a confused mess of boxes and lines.
I’ve asked thousands of software developers around the world to do just this over the past decade, and continue to do so today. The results still surprise me, with the thousands of photos taken during these software architecture sketching workshops anecdotally suggesting that effective visual communication of software architecture is a skill that’s sorely lacking in the software development industry. We’ve forgotten how to visualise software design, both as post-project documentation and during the software development process.
If you cast your mind back in time, a number of structured processes provided a reference point for both the software design process, and how to communicate the resulting designs. Some well-known examples include the Rational Unified Process (RUP), and the Structured Systems Analysis And Design Method (SSADM). Although the software development industry has moved on in many ways, we seem to have forgotten some of the good things that these prior approaches gave us, specifically relating to some of the artifacts these approaches encouraged us to create.
Of course, the Unified Modelling Language (UML), a standardised notation for communicating the design of software systems, still lives on. However, while you can argue about whether UML offers an effective way to communicate software designs or not, that’s often irrelevant because many teams have already thrown out UML or simply don’t know it. Such teams typically favour informal boxes and lines style sketches instead but often these diagrams don’t make much sense unless they are accompanied by a detailed narrative.
Abandoning UML is all very well but, in the race for agility, many software development teams have lost the ability to communicate visually. The example software architecture sketches (pictured) illustrate a number of typical approaches to communicating software architecture and they often suffer from a number of problems as we’ll see in the next chapter.
This book aims to resolve these problems, by providing some lightweight ideas and techniques that software development teams can use to visualise and document their software. Just to be clear, I’m not talking about detailed modelling, comprehensive UML models, or model-driven development. This is about effectively and efficiently communicating the software architecture of the software that you’re building, with a view to:
- Help everybody understand the “big picture” of what is being built, and how this fits into the “bigger picture”.
- Create a shared vision of what you’re building within the development team.
- Provide a focal point for the development team to remain focussed on what the software is and how it is being built.
- Provide a point of focus for those technical conversations about how new features should be implemented.
- Provide a “map” that can be used by software developers to navigate the source code.
- Help you explain what you’re building to people outside of the development team, whether they are technical or non-technical.
- Fast-track the on-boarding of new software developers into the team.
Furthermore, any diagrams that are created need to reflect reality. Those architecture diagrams that you have on the wall of your office at the moment; do they reflect the system that is actually being built, or are they conceptual abstractions that bear no resemblance to the structure of the code?
Why is this important? In today’s world of agile delivery and lean startups, many software teams have lost the ability to communicate what it is they are building, so it’s no surprise that these same teams often seem to lack technical leadership, direction and consistency. If you want to ensure that everybody is contributing to the same end-goal, you need to be able to effectively communicate the vision of what it is you’re building. And if you want agility and the ability to move fast, you need to be able to communicate that vision efficiently too.
As I mentioned in the introduction, I’ve asked thousands of software developers to draw software architecture diagrams during workshops I’ve run. Sometimes this is done as part of a software architecture kata, where groups of people are tasked with designing a software system. Other times it’s done as part of a diagramming workshop where I ask software developers to draw some pictures to describe the software architecture of a system they are currently working on. Either way, the result is the same - an ad hoc collection of “boxes and lines” diagrams.
The task is literally phrased as “draw one or more software architecture diagrams to describe your software system”. As you can probably imagine, the resulting diagrams are all very different. Some diagrams show a very high-level of abstraction, others present low-level design details. Some diagrams show static structure, others show runtime and behavioural aspects. Some diagrams show technology choices, most don’t.
When you think about it, this result is unsurprising. Asking people what they found challenging about the exercise reveals that perhaps visual communication of software architecture isn’t something that is proactively being taught. I regularly hear the following questions during the workshops:
- “What types of diagram should we draw?”
- “What notation should we use?”
- “What level of detail should we present?”
- “Who is the audience for these diagrams?”
I run this as a group-based exercise, typically with between two and five people per group. Rather than making the exercise easier, having a group of people with different backgrounds and experience tends to complicate matters, as time is wasted debating how best to complete the task. This is because, unlike the building industry, the software development industry lacks a standard, consistent way to think about, describe and visually communicate software architecture. I believe there are a number of factors that contribute to this:
- In their haste to adopt agile approaches in recent years, many software teams have “thrown out the baby with the bath water”. Modeling and documentation have been thrown out alongside traditional plan-driven processes and methodologies. That may sound a little extreme, but many of the software teams I work with only have a very limited amount of documentation for their software systems.
- Teams that still do see the value in documents and diagrams have typically abandoned the Unified Modeling Language (UML) in favour of an approach that is more lightweight and pragmatic. I’ll discuss UML later in the book, but my anecdotal evidence, based upon meeting and speaking to thousands of software developers, suggests that UML is optimistically only used by a small percentage of software developers.
- There are very few people out there who teach software teams how to effectively model, visualise and communicate software architecture. Based upon running a small number of workshops for computer science undergraduates, this includes lecturers at universities too.
Let’s look at some examples. The small selection of photos that follow are taken from my workshops, where groups have been asked to design a small “financial risk system” for a bank, and draw one or more diagrams to communicate the software architecture of it. The purpose of the financial risk system is to import data from two data sources (a “Trade Data System”, and a “Reference Data System”), merge the datasets, perform some risk calculations and produce a Microsoft Excel compatible report for a number of business users. A subset of those business users can additionally modify some of the parameters that are used during the calculations. You can see the full set of requirements for the financial risk system in the Appendix A.
Regardless of whether this is the only software architecture diagram or one of a collection of software architecture diagrams, this diagram doesn’t tell you much about the solution. Essentially it’s just a shopping list of technologies.
There’s a Unix box and a Windows box, with some additional product selections that include JBoss (a Java EE application server) and Microsoft SQL Server. The problem is, I don’t know what those products are doing, plus there seems to be a connection missing between the Unix box and the Windows box. It’s essentially a bulleted list that’s been presented as a diagram.
When people talk about software architecture, they often refer to “boxes and lines”. This next diagram has boxes, but no lines.
This is a three-tier solution (I think) that uses the Microsoft technology stack. There’s an ASP.NET web application at the top, which I assume is being used for some sort of user interaction, although that’s not shown on the diagram. The bottom section is labelled “SQL Server” and there are lots of separate cylinders. To be honest though, I’m left wondering whether these are separate database servers, schemas or tables.
Finally, in the middle, is a collection of boxes, which I assume are things like components, services, modules, etc. From one perspective, it’s great to see how the middle-tier of the overall solution has been decomposed into smaller chunks, and these are certainly the types of components/services/modules that I would expect to see for such a solution. But again, there are no responsibilities and no interactions. Software architecture is about structure, which is about things (boxes) and how they interact (lines). This diagram has one, but not the other. It’s telling a story, but not the whole story.
This is similar to the previous diagram and is very common, particularly in large organisations for some reason.
Essentially the group that produced this diagram has simply documented their functional decomposition of the solution into a number of smaller things. Imagine a building architect drawing you a diagram of your new house that simply had a collection of boxes labelled “Cooking”, “Eating”, “Sleeping”, “Relaxing”, etc or “Kitchen”, “Dining Room”, “Bedroom”, “Lounge”, etc.
This diagram suffers from the same problem as the previous diagram (no responsibilities and no interactions) plus we additionally have a colour coding to decipher. Can you work out what the colour coding means? Is it related to input vs output functions? Or perhaps it’s business vs infrastructure? Existing vs new? Buy vs build? Or maybe different people simply had different colour pens! Who knows. I often get asked why the central “Risk Assessment Processor” box has a noticeably thicker border than the other boxes. I honestly don’t know, but I suspect it’s simply because the marker pen was held at a different angle.
This is one of my all-time favourites. It was also the one and only diagram that this particular group used to present their solution.
The central spine of this diagram is great because it shows how data comes in from the source data systems (TDS and RDS) and then flows through a series of steps to import the data, perform some calculations, generate reports and finally distribute them. It’s a super-simple activity diagram that provides a nice high-level overview of what the system is doing. But then it all goes wrong.
I think the green circle on the right of the diagram is important because everything is pointing to it, but I’m not sure why. And there’s also a clock, which I assume means that something is scheduled to happen at a specific time.
The left of the diagram is equally confusing, with various lines of differing colours and styles zipping across one another. If you look carefully you’ll see the letters “UI” (User Interface) upside-down. The reason? People were writing from wherever they sat around the table.
This is another very common style of diagram. Next time somebody asks you to produce a software architecture diagram of a system, present them this photo and you’re done!
It’s a very “Software Architecture 101” style of diagram where most of the content is generic. Ignoring the source data systems at the top of the diagram (TDS and RDS); we have boxes generically labelled “transport”, “archive”, “audit”, “report generation”, “error handling” and arrows labelled “error” and “action”. And look at the box in the centre - it’s labelled “business logic”, which is not hugely descriptive!
This diagram is also relatively common. It shows the logical (or conceptual, or functional) building blocks that the software system is comprised of, but offers very little information other than that.
There’s a common misconception that “software architecture” diagrams should be “logical” in nature rather than include any references to technology or implementation details, especially before any code is written. We’ll look at this later in the book.
This diagram is also relatively common. It shows the overall shape of the software architecture (including responsibilities, which I really like) but the technology choices are left to your imagination.
And similarly, this next diagram tells us that the solution is an n-tier Java EE system but, like the previous diagram, it omits some important technology details.
The lines between the web server and the application server have no information about how this communication occurs. Is it SOAP? A JSON web API? XML over HTTPS? Remote method invocation? Asynchronous messaging? It’s not clear.
I’m often told that the financial risk system “is a simple solution that can be built with any technology”, so it doesn’t really matter anyway. I disagree this is the case and the issue of including or omitting technology choices is covered in more detail elsewhere in the book.
This next one is a Java solution consisting of a web application and a number of server-side components. Although it provides a simple high-level overview of the solution, it’s missing some information and you need to make some educated guesses to fill in the blanks.
If you look at the Unix box in the centre of the diagram, you’ll see two smaller boxes labelled “Risk Analysis System” and “Data Import Service”. If you look closely, you’ll see that both boxes are annotated “JAR”, which is the deployment mechanism for Java code (Java ARchive). Basically this is a ZIP file containing compiled Java bytecode. The equivalent in the .NET world is a DLL.
And herein lies the ambiguity. What happens if you put a JAR file on a Unix box? Well, the answer is not very much other than it takes up some disk space. And cron (the Unix scheduler) doesn’t execute JAR files unless they are really standalone console applications, the sort that have a “public static void main” method as a program entry point. By deduction then, I think both of those JAR files are actually standalone applications and that’s what I’d like to see on the diagram. Rather than the deployment mechanism, I want to understand the execution context.
If you’ve heard of “Plain Old C# Objects” (POCOs) or “Plain Old Java Objects” (POJOs), this is the homeless edition. This diagram mixes up a number of different levels of detail.
In the bottom left of the diagram is a SQL Server database, and at the top left of the diagram is a box labelled “Application”. Notice how that same box is annotated (in green) “Console-C#”. Basically, this system seems to be made up of a C# console application and a database. But what about the other boxes?
Well, most of them seem to be C# components, services, modules or objects and they’re much like what we’ve seen on some of the other diagrams. There’s also a “data access” box and a “logger” box, which could be frameworks or architectural layers. Do all of these boxes represent the same level of granularity as the console application and the database? Or are they actually part of the application? I suspect the latter, but the lack of boundaries makes this diagram confusing. I’d like to draw a big box around most of the boxes to say “all of these things live inside the console application”. I want to give those boxes a home. Again, I do want to understand how the system has been decomposed into smaller components, but I also want to know about the execution context too.
This is the middle part of a more complex diagram.
It’s a little like those “choose your own adventure” books that I used to read as a kid. You would start reading at page 1 and eventually arrive at a fork in the story where you decide what should happen next. If you want to attack the big scary creature you’ve just encountered, you turn to page 47. If you want to run away like a coward, it’s page 205 for you. You keep making similar choices and eventually, and annoyingly, your character ends up dying and you have to start over again.
This diagram is the same. You start at the top and weave your way downwards through what is a complex asynchronous and event-driven style of architecture. You often get to make a choice - should you follow the “fail event” or the “complete event”? As with the books, all paths eventually lead to the (SNMP) trap on the left of the diagram.
The diagram is complex, it’s trying to show everything and the single colour being used doesn’t help. Removing some information and/or using colour coding to highlight the different paths through the architecture would help tremendously.
To pick up on something you may have noticed from previous diagrams, I regularly see diagrams that include unlabelled users/actors. Essentially they are faceless clones. I don’t know who they are and why they are using the software.
The final diagram is a great example of why whiteboards are such useful bits of equipment!
All joking aside, these diagrams do suffer from one or more of the following problems:
- Notation (e.g. colour coding, shapes, etc) is not explained or is inconsistent.
- The purpose and meaning of elements is ambiguous.
- Relationships between elements are missing or ambiguous.
- Generic terms such as “business logic” are used.
- Technology choices (or options, if doing up front design) are omitted.
- Levels of abstraction are mixed.
- Too much or too little detail.
- No context or a logical starting point.
In addition, the problems associated with a single diagram are often exacerbated when a collection of diagrams is created:
- The notation (colour coding, line styles, etc) is not consistent between diagrams.
- The naming of elements is not consistent between diagrams.
- The logical order in which to read the diagrams isn’t clear.
- There is no clear transition between one diagram and the next.
The example diagrams typify what I see during my workshops and these types problems are incredibly common. A quick Google image search will uncover a plethora of similar block diagrams that suffer from many of the same problems we’ve seen already. I’m sure you will have seen diagrams like this within your own organisations too.
One of the easiest ways to understand whether a diagram makes sense is to give it to somebody else and ask them to interpret it without providing a narrative. I’m a firm believer that diagrams should be able to stand alone, to some degree anyway. Any narrative should complement the diagram rather than explain it. However, I often hear groups in my workshops say the following:
- “We’ll talk through the diagrams.”
- “This doesn’t make sense, but we’ll explain it during the presentation.”
The assumption that a diagram will be accompanied by a narrative creates a gap between the information captured on the paper and what remains in people’s heads. Diagrams that need explaining have limited value, especially when used for the purpose of creating long-lived documentation.
The diagrams we’ve seen so far have been an ad hoc collection of “boxes and lines”. Although notation is important, one of the fundamental problems I believe we have in the software development industry is that we lack a common, shared vocabulary with which to think about and describe the software systems we build.
Next time you’re sitting in a conversation about software design, listen out for how people use terms like “component”, “module”, “sub-system”, etc. These terms are typically ambiguous. For example, the dictionary definition for the word “component” is “a part of a larger whole”. Imagine that you’re building a web application, which itself uses a database. Given the dictionary definition, both of the following uses of the word “component” are valid.
- “The web application is a component of the entire software system.”
- “The web application is made up of a number of components.”
In essence, the word “component” is being used to describe two very different levels of abstraction.
My goal is to see teams able to discuss the structure of their software systems with a common set of abstractions rather than struggling to understand what the various notational elements are trying to show. Although I would like to see the software development industry create a standard notation that we all understand (like the electrical engineering industry has with circuit diagrams, for example), I don’t think we are there yet. Perhaps a common set of abstractions is more important than a common notation given our industry’s current lack of maturity and engineering discipline.
Most maps are a great example of this principle in action. If you get two different maps of your local area and lay them out side by side, they will both show the major roads, rivers, lakes, forests, towns, districts, schools, churches and so on. Visually though, these maps will probably use different notation in terms of colour-coding, line styles, iconography, etc. In other words, the maps are showing the same things (the same abstractions), but the notation varies. The key to understanding them is exactly that; a key or legend tucked away in a corner somewhere. We can do the same with our software architecture diagrams.
Diagrams are the maps that help software developers navigate a complex codebase.
In order to get to this point though, we need to agree upon some vocabulary. And this is the step that is usually missed during the initial iteration of my software architecture diagramming workshops. Teams charge headlong into the exercise without having a shared understanding of the terms they are using.
“This is a component of our system”, says one developer, pointing to a box on a diagram labelled “Web Application”.
I’ve witnessed groups of people having design discussions using terms like “component” where they are clearly not talking about the same thing. Yet everybody in the group is oblivious to this. Each group needs to agree upon the vocabulary, terminology and abstractions they are going to use. The notation can then evolve.
So, notation aside (we’ll cover that later in the book), my approach to tackling this problem is to introduce a shared vocabulary that we can use to describe our software. The primary aspect I’m interested in is the static structure. And I’m interested in the static structure from different levels of abstraction. Once this static structure is understood and in use, it’s easy to supplement it with other information to illustrate runtime/behavioural characteristics, infrastructure, deployment models, etc.
I like to think of my software system as being a hierarchy of building blocks as follows:
A software system is made up of one or more containers (web applications, mobile apps, desktop applications, databases, file systems, etc), each of which contains one or more components, which in turn are implemented by one or more code elements (e.g. classes, interfaces, objects, functions, etc). And people use the software systems that we build.
A software system is the highest level of abstraction, representing something that delivers value to its users, whether they are human or not.
Put simply, a container represents something that hosts code or data, like an application or a data store. A container is something that needs to be running in order for the overall software system to work. In real terms, a container is something like:
- Server-side web application: A Java EE web application running on Apache Tomcat, an ASP.NET MVC application running on Microsoft IIS, a Ruby on Rails application running on WEBrick, a Node.js application, etc.
- Client-side desktop application: A Windows desktop application written using WPF, a macOS desktop application written using Objective-C, a cross-platform desktop application written using JavaFX, etc.
- Mobile app: An Apple iOS app, an Android app, a Microsoft Windows Phone app, etc.
- Server-side console application: A standalone (e.g. “public static void main”) application, a batch process, etc.
- Microservice: A single microservice, hosted in anything from a traditional web server to something like Spring Boot, Dropwizard, etc.
- Serverless function: A single serverless function (e.g. Amazon Lambda, Azure Function, etc).
- Database: A schema or database in a relational database management system, document store, graph database, etc such as MySQL, Microsoft SQL Server, Oracle Database, MongoDB, Riak, Cassandra, Neo4j, etc.
- Blob or content store: A blob store (e.g. Amazon S3, Microsoft Azure Blob Storage, etc) or content delivery network (e.g. Akamai, Amazon CloudFront, etc).
- File system: A full local file system or a portion of a larger networked file system (e.g. SAN, NAS, etc).
- Shell script: A single shell script written in Bash, etc.
A container is essentially a context or boundary inside which some code is executed or some data is stored. The name “container” was chosen because I wanted a name that didn’t imply anything about the physical nature of how that container is executed1. For example, some web servers run multiple threads inside a single process, whereas others run single threads across multiple processes. When I’m thinking about the static structure of a software system, I don’t want to concern myself with the details of whether a web application is using one operating system process or many when it’s servicing requests. It’s an important detail, but we can get into that later.
It’s also worth noting that each container should be a separately deployable thing. The physical deployment is another important detail that we will look at later, but, in theory anyway, every container can be deployed onto or run on a separate piece of infrastructure; whether that infrastructure is physical, virtual or containerised. The implication here is that communication between containers is likely to require an out-of-process or remote procedure call across the process and/or network boundary.
To give an example, let’s imagine you’re building a website that is comprised of two different web applications (e.g. a desktop version and a mobile version, or an end-user version serving HTML and an API endpoint serving JSON). There are a number of scenarios to consider:
- Each web application is packaged up into separately deployable units (e.g. two Java WAR files, two ASP.NET web applications, etc). This is two containers, regardless of whether both deployable units are actually deployed into the same physical web server (a deployment optimisation).
- Although you think about the two web applications as being logically separate, they are actually inseparable because they are packaged as a single deployment unit (e.g. a single Java WAR file or ASP.NET web application). This is a single container.
The same is true with relational database schemas. I would treat two separate schemas as two separate containers, irrespective of whether they are deployed into the same database server or not.
As a final note, put simply, a container refers to an execution context and it’s a really runtime construct. This means that libraries or modules (e.g. JAR files, DLL files, .NET assemblies, etc) should not be considered as containers unless they are runnable on their own, like a Java or Spring Boot application that is packaged into an executable JAR file, for example.
The word “component” is a hugely overloaded term in the software development industry, but I like to think of a component as being a grouping of related functionality encapsulated behind a well-defined interface. With the C4 model, components are not separately deployable units. Instead, it’s the container that’s the deployable unit. In other words, all of the components inside a container typically execute in the same process space.
Aspects such as how those components are packaged (e.g. one component vs many components per JAR file, DLL, shared library, etc) is an orthogonal concern and, from my perspective, doesn’t affect how we think about components.
Finally, components are made up of one or more code elements constructed with the basic building blocks of the programming language that you’re using; classes, interfaces, enums, functions, objects, etc.
A component is a way to step up one level of abstraction from the code-level building blocks that you have in the technology you’re using. For example:
- Object-oriented programming languages (e.g. Java, C#, C++, etc): A component is made up of classes and interfaces.
- Procedural programming languages (e.g. C): A component could be made up of a number of C files in a particular directory.
- Functional programming languages: A component could be a module (a concept supported by languages such as F#, Haskell, etc), which is a logical grouping of related functions, types, etc.
- Relational database: A component could be a logical grouping of functionality; based upon a number of tables, views, stored procedures, functions, triggers, etc.
If you’re using an object-oriented programming language, your components will be implemented using one or more classes. Let’s look at a quick example to better define what a component is in the context of some code.
The Spring PetClinic application is a sample codebase that illustrates how to build a Java web application using the Spring MVC framework. From a non-technical perspective, it’s a software system designed for an imaginary pet clinic that stores information about pets and their owners, visits made to the clinic, and the vets who work there. The system is only designed to be used by employees of the clinic. From a technical perspective, the Spring PetClinic system consists of a web application and a relational database.
The version of the web application we’ll look at here is a typical layered architecture consisting of a number of web MVC controllers, a service containing “business logic” and some repositories for data access. There are also some domain and util classes too. If you download a copy of the GitHub repository, open it in your IDE of choice and visualise it by reverse-engineering a UML class diagram from the code, you’ll get something like this.
As you would expect, this diagram is showing you all of the Java classes and interfaces that make up the Spring PetClinic web application, plus all of the relationships between them. The properties and methods are hidden on the diagram because they add too much noise to the picture. This isn’t a complex codebase by any stretch of the imagination but, by showing classes and interfaces, the diagram is showing too much detail.
Let’s remove those classes that aren’t useful to having an “architecture” discussion about the system. In other words, let’s only show those classes/interfaces that have some significance from a static structure perspective. In concrete terms, for this specific codebase, it means excluding the model (domain) and util classes.
After a little rearranging, we now have a simpler diagram with which to reason about the software architecture. We can also see the architectural layers again (controllers, services and repositories). But this diagram is still showing code-level elements (i.e. classes and interfaces). In order to zoom up one level, we need to identify which of these code-level elements can be grouped together to form “components”. The strategy for grouping code-level elements into components will vary from codebase to codebase (and we’ll discuss this later in the book) but, for this codebase, the strategy might look like this.
Each of the blue boxes represents what I would consider to be a “component” in this codebase. In summary, each of the web controllers is a separate component, along with the result of combining the remaining interfaces and their implementation classes. If we remove the code level noise, we get a picture like this.
In essence, we’re grouping the classes and interfaces into components, which is a unit of related functionality. You will likely have shared code (e.g. abstract base classes, supporting classes, helper classes, utility classes, etc) that are used across many components, such as the
JdbcPetVisitExtractor in this example. Some can be refactored and moved “inside” a particular component, but some of them are inevitable.
Although this example illustrates a traditional layered architecture, the same principles are applicable regardless of how you package your code (e.g. by layer, feature or component) or the architectural style in use (e.g. layered, hexagonal, ports and adapters, etc). My aim in all of this is to minimise, and in fact remove, the gap between how software developers think about components from a logical and physical perspective. Components should be real things, evident in the code, rather than logical constructs that are used in architecture discussions only.
If you’re familiar with the definition of software architecture from books such as Software Architecture in Practice, you will have noticed that I don’t use the term “module” as a part of the static structure definition. A module typically refers to an implementation unit (e.g. a library or some other collection of programming elements) that may be combined with other modules into a component, which itself is instantiated to create component instances at runtime. While this model makes sense, I find it adds an additional level of detail that is usually unnecessary when thinking about a software system from a “big picture” perspective. For this reason, I’ve deliberately avoided using the term “module” and instead focus on the identification of coarser-grained components within the static structure.
I’ve also avoided using the term “subsystem”, which some people use to refer to a collection of related components or a functional slice of a software system. The problem I have with the term “subsystem” is that it’s often difficult to map this concept onto a real-world codebase. If the concept of components and modules, or systems and subsystems, is useful, then feel free to build that into the shared vocabulary that you create.
Given the degree of hype and discussion around microservices at the moment, it’s worth being explicit about how to describe microservices using the vocabulary we’ve defined so far. Broadly speaking, there are two options.
If your software system has a dependency upon a number of microservices that are outside of your control (e.g. they are owned and/or operated by a separate team), I would treat these microservices as external software systems that you can’t see inside of.
On the other hand, if the microservices are a part of a software system that you are building (i.e. you own them), I would treat them as containers, along with any data stores that those microservices use (these are separate containers). In the same way that a modular monolithic application is a container with a number of components running inside it, a microservice is a container with a (smaller) number of components running inside it. The actual number of components will depend upon the implementation strategy. It could range from the very simple (i.e. one, where a microservice is a container with a single component running inside) through to something like a mini-layered or hexagonal architecture.
I tend to treat the serverless concepts (e.g. Amazon Lambdas, Azure Functions) in the same way as microservices. If you’re building a software system comprised of a number of serverless functions, think of them as containers because they are all separately deployable.
You might also be wondering where platforms, frameworks and libraries fit into all of this. After all, platforms and frameworks are usually something that you build your software on top of, while libraries are things that your software uses. In most cases, these are really just technology choices that components make use of, and are therefore implementation details rather than components in their own right. For example, the components in the Spring PetClinic web application use the Spring framework, but I wouldn’t show the Spring framework as a component. The same is true of the logging library and database driver, etc.
In most cases, good design2 will encourage you to wrap up components from platforms, frameworks and libraries into your own components. Having said that, there are times when your software might use components provided by platforms, frameworks and libraries of course. Understanding how you use such components is the key to understanding how they fit into a static model of your software.
I’ve illustrated the vocabulary that I use here, and it works for the majority of organisations I work with. But, of course, there are no universal rules. Sometimes, rather than introducing something new, it’s easier for an organisation to stick with the vocabulary they are already using, ensuring that it is explicitly defined and understood by everybody.
As an example, one organisation I worked with builds software in C to run on mobile devices. Instead of “container” and “component”, they use the terms “component” and “module” respectively. In this case, a “component” refers to an executable built in C, which in turn is made up of a number of modules. Although the terminology is different, we still have a hierarchical structure that can be used to describe a software system at a number of different levels of abstraction.
With a shared vocabulary in mind, we can now move on to draw some diagrams at varying levels of abstraction to visualise the static structure of a software system. I call this the “C4 model”; (System) Context, Containers, Components and Code.
- System Context: A System Context diagram provides a starting point, showing how the software system in scope fits into the world around it.
- Containers: A Container diagram zooms into the software system in scope, showing the high-level technical building blocks (containers) and how they interact.
- Components: A Component diagram zooms into an individual container, showing the components inside it.
- Code: A code (e.g. UML class) diagram can be used to zoom into an individual component, showing how that component is implemented.
You can think of the C4 diagrams as being a set of maps for a software system, which provides you with the ability to zoom in and out at varying levels of detail. Shneiderman’s mantra is a simple concept for understanding and visualising large quantities of data, but it fits really nicely with the C4 model because it’s hierarchical.
Overview first, zoom and filter, then details-on-demand
My starting point for understanding any software system is to draw a system context diagram. This helps me to understand the scope of the system, who is using it and what the key system dependencies are. It’s usually quick to draw and quick to understand.
Next I’ll open up the system and draw a diagram showing the containers (web applications, mobile apps, standalone applications, databases, file systems, message buses, etc) that make up the system. This shows the overall shape of the software system, how responsibilities have been distributed and the key technology choices that have been made.
As developers, we often need more detail, so I’ll then zoom into each (interesting) container in turn and show the “components” inside it. This is where I show how each application has been decomposed into components, along with a brief note about key responsibilities and technology choices of those components. Hand-drawing the diagrams can become tedious, which is why you should ideally look at tooling to help automate it instead.
I might optionally progress deeper into the hierarchy to show the code-level elements (e.g. classes, interfaces, objects, functions, etc) that make up a particular component. Ultimately though, this detail resides in the code and, as software developers, we can get that on demand via our IDEs.
Next time you’re asked to create some software architecture diagrams (whether that’s to understand an existing system, present a system overview, or do some software archaeology), my advice is to keep Shneiderman’s mantra in mind. Start at the top and work into the detail, creating a story that gets deeper into the detail as it progresses. The different levels of diagrams allow you to tell different stories to different audiences; some of who will be technical, some not.
As a quick note, the C4 model is not a description of a design process, it’s just a collection of diagrams that you can use to describe the static structure of a software system. That said, and we’ll cover this later, while the C4 model describes diagrams covering four levels of abstraction, you don’t necessarily need to create every diagram at every level. My recommendation is that all teams create System Context and Container diagrams, and really think about whether Component and Code diagrams provide enough benefit considering the cost of creating and keeping them up to date.
This part of the book is about that essential topic we love to hate - writing documentation!
“Working software over comprehensive documentation” is what the Manifesto for Agile Software Development says and it’s incredible to see how many software teams have interpreted those five words as “don’t write any documentation”. The underlying principle here is that real working software is much more valuable to end-users than a stack of comprehensive documentation, but many teams use this line in the agile manifesto as an excuse to not write any documentation at all.
Unfortunately not having a source of supplementary information about a complex software system can slow a team down as they struggle to navigate the codebase.
We all know that writing good code is important and refactoring forces us to think about making methods smaller, more reusable and self-documenting. Some people say that comments are bad and that self-commenting code is what we should strive for. However you do it, everybody should strive for good code that’s easy to read, understand and maintain. But the code doesn’t tell the whole story.
Let’s imagine you’ve started work on a new software project that’s already underway. The major building blocks are in place and some of the functionality has already been delivered. You start up your development machine, download the code from the source code control system and load it up into your development environment. What do you do next and how do you start being productive?
If nobody has the time to walk you through the codebase, you can start to make your own assumptions based upon the limited knowledge you have about the project, the business domain, your expectations of how the team builds software and your knowledge of the technologies in use.
For example, you might be able to determine something about the overall architecture of the software system through how the codebase has been broken up into sub-projects, directories, packages, namespaces, etc. Perhaps there are some naming conventions in use. Even from the previous static screenshot of Microsoft Visual Studio, we can determine a number of characteristics about the software, which in this case is an (anonymised) Internet banking system.
- The system has been written in C# on the Microsoft .NET platform.
- The overall .NET solution has been broken down into a number of Visual Studio projects and there’s a .NET web application called “ib.web”, which you’d expect since this is an Internet banking system (“ib” = “Internet Banking”).
- The system appears to be made up of a number of architectural tiers. There’s “ib.web” and “ib.middletier”, but I don’t know if these are physical or logical tiers.
- There looks to be a hierarchical naming convention for projects. For example, “ib.middletier.authentication.lib”, “ib.middletier.messaging.lib” and “ib.middletier.bankingsystem.lib” are class libraries that seem to relate to the middle-tier. Are these simply a logical grouping for classes or something more significant such as higher level components and services?
- With some knowledge of the technology, I can see a “Service References” folder lurking underneath the “ib.web” project. These are Windows Communication Foundation (WCF) service references that, in the case of this example, are essentially web service clients. The naming of them seems to correspond to the class libraries within the middle-tier, so I think we actually have a distributed system with a middle-tier that exposes a number of well-defined services.
A further deep-dive through the code will help to prove your initial assumptions right or wrong, but it’s also likely to leave you with a whole host of questions. Perhaps you understand what the system does at a high level, but you don’t understand things like:
- How the software system fits into the existing system landscape.
- Why the technologies in use were chosen.
- The overall structure of the software system.
- Where the various components are deployed at runtime and how they communicate.
- How the web-tier “knows” where to find the middle-tier.
- What approach to logging/configuration/error handling/etc has been adopted and whether it is consistent across the codebase.
- Whether any common patterns and principles are in use across the codebase.
- How and where to add new functionality.
- How security has been implemented across the stack.
- How scalability is achieved.
- How the interfaces with other systems work.
I’ve been asked to review and work on systems where there has been no documentation. You can certainly gauge the answers to most of these questions from the code but it can be hard work. Reading the code will get you so far but you’ll probably need to ask questions to the rest of the team at some point. And if you don’t ask the right questions, you won’t get the right answers - you don’t know what you don’t know.
I’m also a firm believer that many software teams have a duty to deliver some supplementary documentation along with the codebase, especially those that are building the software under an outsourcing and/or offshoring contract. I’ve seen IT consulting organisations deliver highly complex software systems to their customers without a single piece of supporting documentation, often because the team doesn’t have any documentation. If the original software developers leave the consulting organisation, will the new team be able to understand what the software is all about, how it’s been built and how to enhance it in a way that is sympathetic to the original architecture? And what about the poor customer? Is it right that they should only be delivered a working codebase?
The problem is that when software teams think about documentation, they usually think of huge Microsoft Word documents based upon a software architecture document template from the 1990’s that includes sections where they need to draw UML class diagrams for every use case that their software supports. Few people enjoy reading this type of document, let alone writing it! A different approach is needed. I like to think about supplementary documentation as an ever-changing travel guidebook rather than a comprehensive static piece of history. But what goes into a such a guidebook?
Let’s imagine that I teleported you away from where you are now and dropped you in a quiet, leafy country lane somewhere in the world (picture 1). Where are you and how do you figure out the answer to this question? You could shout for help, but this will only work if there are other people in the vicinity. Or you could simply start walking until you recognised something or encountered some civilisation, who you could then ask for help. As geeks though, we would probably fire up the maps application on our smartphone and use the GPS to pinpoint our location (picture 2).
The problem with picture 2 is that although it may show our location, we’re a little too “zoomed in” to potentially make sense of it. If we zoom out further, eventually we’ll get to see that I teleported you to a country lane in Jersey (picture 3).
The next issue is that the satellite imagery is showing a lot of detail, which makes it hard to see where we are relative to some of the significant features of the island, such as the major roads and places. To counter this, we can remove the satellite imagery (picture 4). Although not as detailed, this abstraction allows us to see some of the major structural elements of the island along with some of the place names, which were perviously getting obscured by the detail. With our simplified view of the island, we can zoom out further until we get to a big picture showing exactly where Jersey is in Europe (pictures 5, 6 and 7). All of these images show the same location from different levels of abstraction, each of which can help you to answer different questions.
If I were to open up the codebase of a complex software system and highlight a random line of code, exploring is fun but it would take a while for you to understand where you were and how the code fitted into the software system as a whole. Most integrated development environments have a way to navigate the code by namespace, package or folder but often the physical structure of the codebase is different to the logical structure. For example, you may have many classes that make up a single component, and many of those components may make up a single deployable unit.
Diagrams can act as maps to help people navigate a complex codebase and this is one of the most important parts of supplementary software documentation. Ideally there should be a small number of simple diagrams, each showing a different part of the software system or level of abstraction. Visualisation of software architecture was the focus of the first part of this book.
If you ever visit Jersey, and you should because it’s beautiful, you’ll probably want a map. There are visitor maps available at the ports and these present a simplified view of what Jersey looks like. Essentially the visitor maps are detailed sketches of the island and, rather than showing every single building, they show an abstract view. Although Jersey is small, once unfolded, these maps can look daunting if you’ve not visited before, so what you ideally need is a list of the major points of interest and sights to see. This is one of the main reasons that people take a travel guidebook on holiday with them. Regardless of whether it’s physical or virtual (e.g. an e-book on your smartphone), the guidebook will undoubtedly list out the top sights that you should make a visit to.
A codebase is no different. Although we could spend a long time diagramming and describing every single piece of code, there’s really little value in doing that. What we really need is something that lists out the points of interest so that we can focus our energy on understanding the major elements of the software without getting bogged down in all of the detail. Many web applications, for example, are actually fairly boring and rather than understanding how each of the 200+ pages work, I’d rather see the points of interest. These may include things like the patterns that are used to implement web pages and data access strategies along with how security and scalability are handled.
If you do ever visit Jersey, and you really should because it is beautiful, you may see some things that look out of kilter with their surroundings. For example, we have a lovely granite stone castle on the south coast of the island called Elizabeth Castle that was built in the 16th century. As you walk around admiring the architecture, eventually you’ll reach the top where it looks like somebody has dumped a large concrete cylinder, which is not in keeping with the intricate granite stonework generally seen elsewhere around the castle.
As you explore further, you’ll see signs explaining that the castle was refortified during the German occupation in the second world war. Here, the history helps explain why the castle is the way that it is.
Again, a codebase is no different and some knowledge of the history, culture and rationale can go a long way in helping you understand why a software system has been designed in the way it was. This is particularly useful for people who are new to an existing team.
The final thing that travel guidebooks tend to include is practical information. You know, all the useful bits and pieces about currency, electricity supplies, immigration, local laws, local customs, how to get around, etc.
If we think about a software system, the practical information might include where the source code can be found, how to build it, how to deploy it, the principles that the team follow, etc. It’s all of the stuff that can help the software developers, support staff, etc do their job effectively.
Exploring is great fun but ultimately it takes time, which we often don’t have. Since the code doesn’t tell the whole story, some supplementary documentation can be very useful, especially if you’re handing over the software to somebody else or people are leaving and joining the team on a regular basis. My advice is to think about this supplementary documentation as a guidebook, which should give people enough information to get started and help them accelerate the exploration process.
Do resist the temptation to go into too much technical detail though because the technical people that will understand that level of detail will know how to find it in the codebase anyway. As with everything, there’s a happy mid-point somewhere. The following headings describe what you might want to include in a software guidebook:
- Functional Overview
- Quality Attributes
- Software Architecture
- Infrastructure Architecture
- Operation and Support
- Development Environment
- Decision Log
There are, of course, a number of different documentation templates available, and this is my starting point for my own documentation. I would also recommend taking a look at arc42, which captures the same sort of information in a slightly different format, and the “Building Block View” complements the C4 model nicely.
As a final note, the style of documentation that I’m referring to here is related to the product being built rather than the project that is creating/changing the product. A number of organisations I’ve worked with have software systems approaching twenty years old and, although they have varying amounts of project-level documentation, there’s often nothing that tells the story of how the product works and how it’s evolved. Often these organisations have a single product (software system) and every major change is managed as a separate project. This results in a huge amount of change over the course of twenty years and a considerable amount of project documentation to digest in order to understand the current state of the software. New joiners in such environments are often expected to simply read the code and fill in the blanks by tracking down documentation produced by various project teams, which is time-consuming to say the least!
I recommend that software teams create a single software guidebook for every software system that they build. This doesn’t mean that teams shouldn’t create project-level documentation, but there should be a single place where somebody can find information about how the product works and how it’s evolved over time. Once a single software guidebook is in place, every project/change-stream/timebox to change that system is exactly that - a small delta. A single software guidebook per product makes it much easier to understand the current state and provides a great starting point for future exploration.
We’ll talk about tooling later but, from a process perspective anyway, my approach to documentation is “little and often”. If you have a “definition of done” for your work items, simply add another line to the bottom that says something like, “Documentation and diagrams updated”. From my own experience, doing a number of small doucmentation updates is much more palatable than sitting down for a few weeks with the sole purpose of writing doucmentation!
How long should documentation be? I get asked this question a lot. And rather than talk about numbers of pages, what I’m really looking for is something that I can read in a couple of hours, over a coffee or two, to get a really good starting point for exploring the code.
1I do appreciate that the term “container” is now in widespread use because of containerisation and technologies like Docker. Feel free to use something like “runtime context”, “execution environment” or “deployable unit” instead if you’d prefer to avoid the term “container” when discussing software architecture.↩
2Many software architects will often try to decouple their code from that provided by platforms, frameworks and libraries so that decisions can deferred and technologies changed if necessary.↩