1. What do we Mean by Testing?

We Test Systems

The word system conjures up different images to different people. To people in the IT community, a system is one or more computer applications (or sub-systems) that provide services to customers, partners or end-users. A software company might view their software product, which is a component or sub-system of larger systems as a system. But a software-centric view is rather too narrow for our purposes.

A systems approach takes a much broader view and a system might include the computer hardware, software and infrastructure, but might also include the procedures, partners, management, users and many other assets.

We’ll use a rather abstract definition of system from now on.

A system is a combination of related parts organized into a complex whole.

A system can be anything from a mobile phone to a guided missile; a web site to a global entertainment network; an electric shaver to an airport; a team, company, culture or society.

Every system needs some testing; every test requires some disciplined thinking; testers are the people who think, and make testing happen.

We are all Testers; We are all Tested

We are all testers. We test all the time. We test drive a car before we buy it. We test our food before we swallow it. We re-read our emails before we send them. We download software and try it out before we pay for it. We try on clothes and ask for a friend’s opinion before we commit. We visit and inspect houses before make an offer to buy them. We court prospective mates – we test for evidence of fitness for mating and raising offspring!

In every case, our behaviour is affected by the outcome of a test. If the outcome is negative, we don’t buy, commit, swallow or even propose marriage. If we are risk-averse, if the outcome is neutral or uninformative, we might decide we need more information, new tests, and test again.

During the course of our lives, we are all tested or assessed in countless ways. We take school tests, examinations, IQ and aptitude tests; we fill in forms for tax assessments, new bank accounts and credit checks; we are subjected to health checks, eye tests and driving tests. Interns and apprentices are hired on the basis of successful completion of a probationary period.

We might not like it but on the successful outcome of these tests depend our grades, licence to drive, job offers and many other aspects of our very livelihood.

Everyone has their standards, and we all assess other people and things against them all the time⁸.

The Purpose of Testing

At the most fundamental level, the purpose of testing is to gather information to learn about some aspect of a system and potentially make a decision based on the outcome of one or more tests. Consequently, testing is an information business, a people business and a business business. Let me explain.

Testers (or system builders⁹, when they are testing) do not build; they do not put defects in and do not take defects out. In this respect, testers do not improve quality or add value to the systems they test. They are however, responsible for providing the most valuable information required by developers (to fix defects), project management (to understand and manage achievement) and stakeholders (to be assured). In this one respect, testing is all-powerful – it is the single source of knowledge of achievement in systems projects¹⁰.

Testing is a people business. Most testers need excellent interpersonal skills, particularly communication skills. Testers and those who manage them need at one time or another to communicate with, negotiate with, influence and advise end users, developers, technical/environmental support, business and stakeholder management, internal and external auditors, outsource companies, customers, client services, and product managers. Interpersonal skills are, in many ways, more important than technical skills for testers.

A business business? Testing exists to detect defects and so protect end users, but there is a higher goal: to inform stakeholder and management decision making. The stakeholders who must decide to release, accept or reject a system and the managers who must close, delay, re-think or re-plan systems projects are critically dependent on the information produced by testers. In this respect, testers are a great ally of their stakeholders.

When we execute a test, if our aim is to understand aspects of the system under test to make a decision, we can treat the test as an information gathering or learning process. We design tests to gather specific information that we do not currently have.

The Definition of Test

It’s about time I defined the most important little word in this Pocketbook.

This Pocketbook is about that word – test – used as a noun and as a verb. It’s about testing as an activity and the outcome of that activity. It’s about the people or organisations who commission that activity and those who use the results. Very much it’s about the people who call themselves testers and the complex systems on which we work.

It’s a small word, but it turns out to be a large subject. So let’s get our definition right.

We need a definition of test that is context-neutral so I looked up test in the dictionary.com website. Of the many pages of references to the word test and its applications in many areas, the definition from the American Heritage Dictionary is the most appropriate.

Test: (noun) a procedure for critical evaluation; a means of determining the presence, quality, or truth of something; a trial

This statement seems to capture the essence of what is meant by a test – but there are three variations. Well this isn’t so bad I think, as all three taken together give us the foundations we need. Let’s take a closer look at each one.

A procedure for critical evaluation

Critical evaluation involves a skilful judgement as to the truth or merit of something. A test is a procedure, usually with a series of steps that include some form of preparation, execution, results gathering, analysis and interpretation. This isn’t a definitive description of a test procedure. There could be more steps and one could break these main steps down further.

The procedure doesn’t necessarily require prepared documentation, but many tests are so documented¹¹. The important thing is that there is a perspicacious thought process at the heart of a test.

This thought process is driven by the need to evaluate the system under test with respect to its adequacy, consistency, behaviour, accuracy, reliability or any other significant aspect or property.

A means of determining the presence, quality, or truth of something

A test could determine the presence (or absence) of something easily enough, but quality is a different matter: the term is loaded with emotional connotations, but we are rescued by the dictionary.

A quality can be, “an essential or distinctive characteristic, property, or attribute”¹². Now we can see that a test can reveal these properties.

Can a test determine the truth of something? Well this makes good sense too. Typically, we need to test an assertion such as, “this system meets some requirement”¹³ or “this system behaves in such a way” or “this system is acceptable” and so on. There’s a certain amount of subjective judgement involved but we can see that a test or tests could provide evidence for someone to exercise that judgement and make a decision.

A trial

The notion of a trial implies that the process of testing a system will help us to evaluate that system with respect to its qualities. The purpose of such an evaluation is normally to make a decision.

The decision might be to accept or reject the system, but it might also be to expose its shortcomings so they can be remedied in some way. A test might also influence an individual or organisation to change direction – to rethink a design; to relax or change a requirement; to scrap a component and start again; to buy rather than build or build rather than buy.

A natural way of looking at a system under test is that it is on trial, and will be judged in some way.

The definition of testing

From our definition of the noun test, we can derive a verb easily enough.

Test: (verb) to critically evaluate; to determine the presence, quality, or truth of something; to conduct a trial.

So far so good¹⁴.