Chapter 1: Agile & Software Architecture

Software development is a pretty new discipline. It started in theory with Charles Babbage and his Differential Engine, and Lady Augusta Ada Byron, Countess of Lovelace, becoming the first programmer in history, although she never had the actual computer to execute her programs!

From these early days to Word War II, computing turned from theory to a mean to push mathemathics to a larger scale, and remained as an academic endeavor until the early 50’s, with the Univac. Later that decade IBM starts producing their first commercials computers and programmers start spreading from mathematicians to engineers, and a new species was born: the homo-nerdus, also known as “computer programmer”.

Early programming methods were pretty much ad-hoc, and the cycle between coding and compilation could be measured in weeks, from punching the cards (which sometimes the coder could not do on her own) to submitting the batch of cards to the Computer’s Administrators and wait days to receive back a paper listing with the result run or just pages of error information.

As computers became popular in the corporate world and software applications got more complex, the need to stronger engineering practices appeared, and from 70’s to 90’s our industry exported ideas and processes from the previous engineering fields, from electronics to civil construction. The result introduced several problems, basically because programmers got subjected to controls and environments closer to blue-collar workers than designers, most of the time restricting innovation and creativity.

Agile software development crystalized at the start of the XXI century as a reaction to these prescriptive processes, putting the focus back in the non-repetitive side of programming, fostering relationships, team dynamics, collaboration outside of the technical roles, and maximizing feedback loops everywhere.

In the first part of this book I want to provide a short recap of its main topics, just to set expectations, boundaries and terminology I will use in the rest of the book, and as a pivot for further discussion.

First I will review the Agile paradigm from the values and principles and the practices in he most common frameworks like Scrum and Extreme Programming, its Lean cousin Kanban, and then what I understand as the meta-model behind Agile.

Then I will do the same with Software Architecture, focusing mostly in the type of Architecture I will refer to in this book, and pointing to connected disciplines.

Lastly I want to present my view of the main challenges Software Architecture face on Agile teams, projects and organizations, and provide a short overview of the different approaches I present about this topic during the rest of the book.

Acknowledgements

This book is still a work in progress, but luckily I’m not alone. I want to thank several friends who helped me come to the ideas I’m compiling here, and also the ones helping me during writing, by providing feedback about concepts, grammar even my style and humor (or lack of thereof).

I have to thank first Diego Fontdevila who wrote with me the paper where I started writing down the ideas and experiences I gathered over the previous ten years (Software Architecture in the Agile Life Cycle, Architecture Journal #23).

Several years before the paper, I worked and taught Software Architecture with Diego González and Juan Ladetto, whose insights and ideas were really influential in many of my choices and research.

Also, I feel in debt with Ted Neward, whose book “Effective Enterprise Java” had a huge impact in the way I see the field beyond any technology, and of course, because he started the idea of Architectural Katas, which I later adopted and customized to run in several events and also as part of my workshops.

I also need to thank the whole Southworks crew, in special Johnny Halife, Matias Woloski and my old-time friend Angel “Java” López. I always miss the awesome discussions we had, full of ideas and practical wisdom. Also, I enjoyed working and discussing with the team at Microsoft Patterns & Practices, one of Southworks most important customers, in particular Eugenio Pace and Scott Densmore.

As I joined Kleer and designed my Agile Architecture Workshop, I had great input from several of my colleagues at Kleer, especially Juan Gabardini, Luis Mulato and Yamit Cárdenas, and colleagues from other companies like Raúl de Villa Cano, Marcelo Oks, Andrés Joaquín, Adrián Anacleto, Santiago Blanco and many more I’m probably forgetting.

Lastly, I want to thank the first reviewers of this book, Nico Páez and Scott Baldwin.

Agile Recap

As Agile Methodologies decreases focus on written documentation in favor of oral and visual communication, I will start using my colleague Pablitux’s poster to review the Agile Manifesto:

The text of the Manifesto says:

We are uncovering better ways of developing software by doing it and helping others do it.

Through this work we have come to value:

Individuals and interactions over processes and tools

Working software over comprehensive documentation

Customer collaboration over contract negotiation

Responding to change over following a plan

That is, while there is value in the items on the right, we value the items on the left more.

Let’s re-read these values from an architectural viewpoint, and ask some questions:

• Who are the Individuals in this case?
• Isn’t architecture mostly about processes and tool?
• Can we build working architecture without comprehensive documentation?
• Who are our customers when we build (or install) architecture components?
• Do we have a contract?
• Are we capable of changing our architecture when needed, instead of sticking to our prescribed model?

My personal view on Agile

After almost 20 years working under the agile paradigm, starting with the early XP days by the late 90s, as you might expect, I have some strong opinions about this model and its impact in the industry and the community.

I don’t think my ideas are particularly original or important, but as the book moves into the model I found to put them to work into software architecture, I think it is important to tell you about some of them, even if this serves to set some disagreement early on.

So here I go:

Agile is a cultural approach

I find the frequent usage of “Agile Methodologies” somewhat unfortunate. Some people still thinks that Agile is just a different way to run projects, but it is way beyond this. As Kent Beck used to say about XP, it is about social change.

As pretentious as social change might sound, Agile is mostly about moving from centralized decision-making and high-specialization to an environment of shared decision-making and high-collaboration. This is somehow subversive in the sense of potentially breaking chain of commands, which quickly push for a different distribution of money, power, knowledge and so on.

Of course, this is an ideal, but it is there, and I believe that continuous improvement only can get you close to that. Otherwise, improvement will slow down and probably stale.

Agility requires Quality

Quality in execution comes from avoiding shortcuts and being transparent about what you do, without looking for someone to blame when things fail, but looking for underlying problem and fixing root causes.

Transparency, introspection and improvement are core values of Agile, and that’s what one things brings to the other.

Moreover, Technical Excellence is one of the principles stated in the Manifesto, and is one of the distinctive traits one consistently finds in succeeding Agile teams. Indeed, many teams adopting Agile can get great results at first, but quickly get in trouble when neglecting technical improvement. This pattern was so common that even Ken Schwaber (one of the fathers of Scrum) started using the term “Flaccid Scrum” to describe the problem.

Software Architecture Recap

Software Development is a Complex human activity.

Angel “Java” López

Of course I won’t dare to define Software Architecture on my own, as this is an often debated topic, and I’m not planning to add a lot of heat to this.

Instead, I will try to present some commonly used definitions from several recognized authors, and try to get some basic ideas to build upon during the rest of the book.

First, a meta-discussion on the topic from Luke Hohmann’s “Beyond Software Architecture” (Addison Wesley, 2003):

Software architecture is a complex topic. Because of its complexity, our profession has produced a variety of definitions, each more or less useful depending on your point of view. Here is a definition from my first book, Journey of the Software Professional:

A system architecture defines the basic ‘‘structure” of the system (e.g., the high-level modules comprising the major functions of the system, the management and distribution of data, the kind and style of its user interface, what platform(s) will it run on, and so forth).

This definition is pretty consistent with many others for example, [Bass], [Lerman], and [POSA]. However, it lacks some important elements, such as specific technology choices and the required capabilities of the desired system. A colleague of mine, Myron Ahn, created the following definition of software architecture. It is a bit more expansive and covers a bit more ground than my original (2002, personal communication):

Software architecture is the sum of the nontrivial modules, processes, and data of the system, their structure and exact relationships to each other, how they can be and are expected to be extended and modified, and on which technologies they depend, from which one can deduce the exact capabilities and flexibilities of the system, and from which one can form a plan for the implementation or modification of the system.

We could extend these definitions from the technical point of view, but this wouldn’t provide a lot of value. More than any other aspect of the system, architecture deals with the ‘‘big picture.” The real key to understanding it is to adopt this big picture point of view.

Another meta-vision comes from Ian Gorton’s “Essential Software Architecture” (Springer, 2006):

1.2 Definitions of Software Architecture

Trying to define a term such as software architecture is always a potentially dangerous activity. There really is no widely accepted definition by the industry. To understand the diversity in views, have a browse through the list maintained by the Software Engineering Institute2. There’s a lot. Reading these reminds me of an anonymous quote I heard on a satirical radio program recently, which went something along the lines of ‘the reason academic debate is so vigorous is that there is so little at stake’. I’ve no intention of adding to this debate. Instead, let’s examine three definitions. As an IEEE member, I of course naturally start with the definition adopted by my professional body:

“Architecture is defined by the recommended practice as the fundamental organization of a system, embodied in its components, their relationships to each other and the environment, and the principles governing its design and evolution.” [ANSI/IEEE Std 1471-2000, Recommended Practice for Architectural Description of Software-Intensive Systems]

This lays the foundations for an understanding of the discipline. Architecture captures system structure in terms of components and how they interact. It also defines system-wide design rules and considers how a system may change. Next, it’s always worth getting the latest perspective from some of the leading thinkers in the field.

“The software architecture of a program or computing system is the structure or structures of the system, which comprise software elements, the externally visible properties of those elements, and the relationships among them.” [L.Bass, P.Clements, R.Kazman, Software Architecture in Practice (2nd edition), Addison-Wesley 2003]

This builds somewhat on the above ANSI/IEEE definition, especially as it makes the role of abstraction (i.e. externally visible properties) in an architecture and multiple architecture views (structures of the system) explicit. Compare this with another, from Garlan and Shaw’s early influential work:

“[Software architecture goes] beyond the algorithms and data structures of the computation; designing and specifying the overall system structure emerges as a new kind of problem. Structural issues include gross organization and global control structure; protocols for communication, synchronization, and data access; assignment of functionality to design elements; physical distribution; composition of design elements; scaling and performance; and selection among design alternatives.” [D. Garlan, M. Shaw, An Introduction to Software Architecture, Advances in Software Engineering and Knowledge Engineering, Volume I, World Scientific, 1993]

It’s interesting to look at these, as there is much commonality. I include the third mainly as it’s again explicit about certain issues, such as scalability and distribution, which are implicit in the first two. Regardless, analyzing these a little makes it possible to draw out some of the fundamental characteristics of software architectures.

Finally, Stephen Albin comes to his own terms in “The Art of Software Architecture” (Wiley, 2003):

Software architecture involves the integration of software development methodologies and models, which distinguishes it from particular analysis and design methodologies. The structure of complex software solutions departs from the description of the problem, adding to the complexity of software development. Software architecture is a body of methods and techniques that helps us to manage the complexities of software development.

Software architecture is a natural extension of the software engineering discipline. In early literature it was simply referred to as programming in the large. Software architecture presents a view of a software system as components and connectors. Components encapsulate some coherent set of functionality. Connectors realize the runtime interaction between components. The system design achieves certain qualities based on its composition from components and connectors. The architecture of a software system can be specified in a document called the architectural description. Software architecture design is not entirely different from existing software design methodologies. Rather it complements them with additional views of a system that have not been traditionally handled by methodologies like object-oriented design. We will learn that software architecture fits within a larger enterprise architecture that also encompasses business architecture, information technology architecture, and data architecture.

Agile Architecture Challenges

Now that I discussed the Agile paradigm and Software Architecture, let’s talk about the intersection between the two.

Even while agile methodologies are getting widely accepted in the development world, there is still a lot of debate about how to apply them to the architectural space. One of the most conflictive issues stems around “big design upfront,” which is strongly discouraged by agile practitioners, and the traditional approach to architectural design.

In our original paper from 2010 ¹, Diego Fontdevila and I proposed a set of team dynamics, conceptual practices, and specific technologies to embed software architecture within the agile approach—keeping up the shared goals of technical excellence, streamlined development practices, and a constant and ever- increasing flow of business.

Architectural Dynamics in Agile Teams

One of the 12 principles of the Agile Manifesto states that “the best architectures, requirements, and designs emerge from self-organizing teams.” We took this to heart —especially, the reference to our shared specialization.

While architecture is an activity that is historically performed with an emphasis on the early stages of a project, the main focus of agile development is on emergent design and iterative production— creating a series of interesting challenges down the road.

First of all, agile makes a big push toward shared responsibility and, thus, dilutes the traditional role of the architect as the one who “defines” the higher-level design of a solution. In this new approach, architecture (as most other development activities) is something that is performed by the whole team -—preserving its multidisciplinary nature. This does not imply that the architect profile goes away, as with all the other roles; it means that while someone contributes with a broader and probably more experienced perspective (usually leading in this aspect), the whole team participates and understands the implications of the design decisions that it makes, and continuously evaluates them.

In our experience, key considerations —-such as the modularity strategy, how communication is handled within and outside the application, and how data and services are accessed and abstracted—- are successfully defined and implemented when the whole development team establishes a consensus about these issues. In this way, team members fully understand the consequences of the selected alternatives, remain aware of their initial assumptions throughout the solution life cycle, and quickly raise concerns when their validity is affected.

Most of these challenges are usually tackled by folding architectural discussion and revision into the regular meetings that take place over the course of an iteration—such as planning and review meetings, and frequent sync-ups and design meetings with plenty of white boarding and open talk. It is also worthwhile to have the most important guidelines permanently exposed in an informative space, including diagrams, checklists or reference charts around the walls, and semipermanent flip charts that are used as posters.

This article does not cover in detail specific techniques that apply to coordinating several subteams; mainly, it mirrors the standard guidelines about “Scrum of Scrums”. The addition to such activities is a stronger focus on the preservation of conceptual integrity—thus, planning frequent high-level design meetings between teams. Again, these meetings should avoid becoming architect meetings; while the contribution of team members who have a stronger architectural background is obviously important, it is very important for other members to participate. Even the less experienced team members can provide a somewhat naïve perspective to some discussion—promptly flagging complexity excesses that are a professional malady among us architects.

To close on the team dynamics, as the agile perspective goes over the standard view of the development team and extends to customers, operations personnel, and other stakeholders, expectation management is a big deal also for the solution architecture. In the strategy I present in this book there is a strong emphasis on mapping the needs and goals of these actors to the architectural constraints and converting the most important ones into strong metrics to be evaluated.