Preface

Aims of the Pocketbook

This little Pocketbook has two aims.

The first aim is to provide a brief introduction to the discipline called testing. Have you just been told you are responsible for testing something? Perhaps it is the implementation of a computer system in your business. But you have never done this sort of thing before! Quite a lot of the information and training on testing is technical, bureaucratic, complicated or dated. This Pocketbook presents a set of Test Axioms that will help you determine what your mission should be and how to gain commitment and understanding from your management. The technical stuff might then make more sense to you¹.

The second aim of this Pocketbook is to provide a handy reference, an aide memoire or prompter for testing practitioners. But it isn’t a pocket dictionary or summary of procedures, techniques or processes. When you are stuck for what to do next, or believe there’s something wrong in your own or someone else’s testing or you want to understand their testing or improve it, this Pocketbook might be of help. It will prompt you to ask some germane questions of yourself, your team, your management, stakeholder or supplier.

Testing is easy, isn’t it?

Some people think so. These people have never had to think about it seriously beyond it being a task on a project plan to be completed as quickly as possible before ‘going live’ (whatever that means). But there are good reasons why everyone should think testing is easy, or at least is second-nature. You see, everyone became a tester² before they could walk or talk.

Just about everything a baby can grasp is put into their mouths within seconds. It is tasted. It is tested. Babies and children learn incredibly rapidly through play. Play and testing are very closely related. At some level, all ‘good’ testing requires curiosity and imagination. So what could be hard about play?

Many things make testing hard. As we grow and mature, curiosity and imagination are often stifled and our natural testing skills are dulled. Usually, we don’t test for our own benefit; rather, we test on behalf of other people whose needs, preferences, prejudices and assumptions mislead us. The things we are asked to test are not simple; systems, even small ones, can be immensely complex to understand, to configure and to interpret.

Context-neutral

Although my own background is in software testing, I have tried throughout the Pocketbook to avoid the use of terms like software, programmer, program, module etc. because testing is about so much more than software. Instead of software, I have used the word system to label what we test. I’ve tried to make the text as generic as possible so that it supports the testing of any system. Does this mean the Pocketbook describes the approach to testing anything? Well, that would be neat but highly unlikely³.

In my career, I’ve tested programs, software systems, interfaces, hardware, firmware, buildings, furniture, telephones, concrete and people. This Pocketbook attempts to describe the thought processes associated with testing systems in the broadest sense and is not specific to any one milieu, technology, organisation, culture. It is context-neutral.

By the way, I’ll discuss testing mainly in the context of projects, but testing isn’t restricted to systems development projects. Testing might also take place in a maintenance or business-as-usual environment to ensure repairs or other changes to production systems work correctly. Testing might also be used to explore how a system works for the purpose of an evaluation.

This Pocketbook won’t make you a demon tester. It won’t teach you anything about test process, test case design or methodology. What it will do is introduce (or remind you of) the pre-requisites, thinking processes and decision-making you need to do a half-decent job.

Test Axioms

I introduced the idea of Test Axioms in posts on my blog in the spring of 2008⁴. Over a few months their definitions evolved and in May 2008 I summarised the thinking behind them and tabulated 16 proposed axioms. Further evolution has occurred and with some changes, they have been used in this Pocketbook.

There was quite a reaction to the proposed axioms. Some people rejected the idea, saying there were no such things. Others were more supportive and offered new axioms or alternate definitions. You can see these posts and links to other peoples’ comments on my blog.

I believe that there are a set of rules or principles that provide a framework for all testing. But there is no single agreed definition of test. This is mostly because every consultant and author has tended to write their own definition to suit their own purposes (and I am as guilty as the rest of them). As an industry, we are hamstrung because of this. Our clients are confused, and we get distracted by discussions on definitions because of not-invented-here mentalities and our competitive instincts. Context-neutral definitions are proposed in Chapter 1.

An axiom is something believed to be true, but cannot be proven in any practical way. It could be disproven by experiment or experience and we should be prepared to be proven wrong and welcome attempts to do this⁵.

But some people object to the notion of Test Axioms and say that nothing in testing is certain. There are no axioms. All testing rules, principles, techniques, approaches etc. are heuristic. Heuristics have value in some contexts, but are limited in application, usefulness, accuracy etc. in other contexts. They are limited or fallible in known ways.

Here is my proposal for Test Axioms restated in a different way. I have tried to define them in a way that testers can, for all practical purposes, regard them as axiomatic. If anyone devises a testing context where the axioms are violated, we need to think again: Perhaps the axiom should be scrapped or changed or its scope of applicability defined.

Reviewers have generously commented on the early drafts of the book and provided many suggestions and helpful criticisms. But so far, I have not received any concrete examples that invalidate the Test Axioms as stated in the Pocketbook.

The Test Axioms are an attempt to provide a context-neutral set of rules for testing that identify the critical thinking processes and motivations for all test approaches.

Stakeholder obsessed

Testing is an information or intelligence-gathering⁶ activity performed on behalf of (people who I will call) testing stakeholders. The manager who asked you to test could be your most important stakeholder (ask them!) They think testing is important enough to get someone as important as you involved – but might not be able to articulate why they see it as an important role.

You might use this Pocketbook to help you have a rational dialogue with them, to home in on the important issues you need to resolve to enable you to do the right job.

By the way, if you are testing the products of your own efforts, you could be your own stakeholder. Your approach to testing your own products or systems will be focused on what you and others, as stakeholders, want to learn about those products or systems.

Most systems that need testing have stakeholders whose interests do not coincide perfectly. We cannot test everything, so we need to help them to make choices. We need to develop a good relationship with stakeholders to build consensus, buy-in and trust in our test approach. Since most stakeholders are non-technical, the language we use must be simple and direct. The Test Axiom definitions are just that.

Structure

Chapter 1 sets the scene and introduces the foundations of testing. Testing is a natural human activity; we test systems; the purpose of testing is presented. If we take an axiomatic stance to testing, we need context-, process-, technology- (and other things) –neutral definitions of test and testing. My chosen definitions close the chapter.

Chapter 2 introduces the First Equation of Testing as a way of understanding how an approach to testing can be formulated. The fundamental development process is introduced. Test Axioms, Context, Values and Thinking are presented as the building blocks of a test approach.

In Chapters 3, 4, and 5 the Test Axioms are presented under chapter headings Stakeholder, Design and Delivery respectively. Each axiom is discussed to present the reasons why the axiom exists and what its impact on a test approach can be. Some relationships between the axioms are identified.

Chapter 6 discusses how testing improvement can be achieved. The meaning of improvement and brief overviews of model-based and goal-directed improvement approaches are provided. The change-management process is introduced and a popular ‘8-step’ process provides an example approach.

Chapter 7 provides some brief guidelines for conducting an assessment of testing in an organisation. The axioms are then listed, one per page complete with ‘seed’ questions for each that can help your investigations. (They could also help you to challenge your own test approach).

Chapter 8 presents some closing remarks and summarises how the Test Axioms can be used to define test approaches.

An index is included at the end of the Pocketbook. Definitions of key terms are identified by bold page numbers in the main text.

Acknowledgements

Although the writing of this Pocketbook is a solo effort, it is the concentrated result of many testing assignments starting in 1992, countless conversations with my friends in testing at the UK Test Management Forum, EuroSTAR, STAR East and West, LEWT, software testing specialist interest groups SIGIST (UK), TestNet (Holland), SAST (Sweden) and Den Norske Dataforening (Norway) and tough questioning in training courses, blogs and email discussions. You all know who you are :O).

In particular, I’d like to thank all of the Tester Retreat attendees, especially my good friends Isabel Evans, Peter Morgan, Graham Thomas and Neil Thompson for inspiring the initial work on Test Axioms and providing feedback and guidance.

I invited a lot of people to review the Pocketbook and most reviewers responded quickly and with a lot of comments⁷. I apologise for taking so long to use them. All of the comments were considered – some conflicted with each other and some conflicted with my aim of being context-neutral, so not every contribution made it into the final draft.

Many, many thanks to: Julien Bensaid, Bart Broekman, Bob van de Burgt, Fiona Charles, Graham Dwyer, Julia Gerrard, Derk-Jan de Grood, Tim Koomen, Alon Linetzki, Richard Marsden, Henk van Merode, Tone Molyneux, Maurice Siteur, Michael Stahl, John van Veen, Jeffrey Wannee, Adam White, Barbara van Wijk, Maarten Woolthuis and all those people who reviewed the first draft, gave verbal feedback and support. If I’ve omitted your name – many apologies and please let me know so I can thank you in future editions.

Thanks and much love, as always, to my (tolerant and smarter-than-me) family, Julia, Lizzie and Max.

Any errors or omissions are my fault entirely. This is a work in progress. Please let me know how I can improve it. My email address is paul@gerrardconsulting.com.

Further information can be found on the book’s website:

http://testers-pocketbook.com

Before we begin…