What should be tested? Tests vs real world

We live in an era of unusually dynamic development of technology. Things are getting smaller, more, and more accurate right before our eyes. There is progress in every field of science.

We can cure more effectively, anticipate disasters, and warn people about them. We can prevent a lot of things that once tormented mankind. We improve our lives in harmony with Mother Earth.

Technology gives us the possibility of global communication. It only takes an eye blink to travel from one place to another. Thanks to the global network and its availability, smart minds from around the world can exchange data, results, and ideas. Because of this possibility we are able to check the weather, make a beautiful photo, and share it with friends all around the world. Everything thanks to a device that fits into your hand.

Behind the phenomena are people with passion and knowledge who want to develop the world. They create, invent, and design new products to make our lives easier. However, each of them, before putting their work into our hands, must make sure that it works as planned. If you were not sure that everything was solid, resistant, and safe, the astronauts definitely did not want to board the shuttle of the Apollo 11 mission. (I would skip this one because it’s not clearly written) 

And here is where the testers come along.

What is testing?

Testing is a comprehensive issue and covers many fields of science. We can test air quality, the efficiency of solar panels, a new treatment method, software. Testing consists of individual components, tests. So, what is a test? According to the dictionary definition: 

Take measures to check the quality, performance, or reliability of (something), especially before putting it into widespread use or practice 

The word refers to many things and could refer to other fields of science. In the article, we will refer to the cases of testing the software and devices on which the software is located.

Goals of testing

To answer the question "What should be tested?" we should start with general testing goals. In the IT sector, we have a lot of diversity when it comes to languages in which the code is written. Languages differ in syntax, complexity, and purpose. For example, it would be difficult to write a batch code per processor in HTML. However, each software must be tested before it goes to the client. The question arises: should the testing criteria of different software groups be the same? The answer is no. However, if there are different criteria, are the goals changing?

The answer to this question is as follows: the goals do not change. The main purpose of software testing is to detect faults, program imperfections, incompatibilities with the expected results. Their detection can prevent many dangers: from losing customer's trust, even causing damage to the health of the person who uses this software.

Often it is time-consuming to correct the program and it may affect the customer's receipt of the product after the set deadline. However, it is worth taking care of its quality because it can be an asset in the overall assessment of trust in the supplier.

Rules of testing

We can distinguish seven basic principles of testing:

• Exhaustive tests are not possible

It would be an ideal situation if our application would be tested in every possible way and in all conditions. Then we would be sure that our product has no right to be advertised. Is it possible? No. It would take too much time and time is usually limited. The solution is to focus on those parts that are most frequently used and vulnerable.

• Defect clustering

At some stage, the software becomes very complex and each developer knows the situation in which one small error causes an entire avalanche of bugs. The best practice is to identify very risky modules and focus on them.

• Pesticide paradox

This rule applies to the situation in which we have a field, and that the plants growing on our field could yield a good crop, we must provide them with protection against pests. However, when we will try to remove pests with only one insect repellent, they will finally become resistant to it. Similarly with tests, if the test will be carried out in the same form, the program will finally become resistant to it. It is therefore important to design new tests or edit the old ones.

• Testing shows the presence of defects

The description of this principle is very simple: it is virtually impossible to find an application that has no errors. And testing has these glitches to reveal. When the application passes all the tests positively, there is a very good chance that some bugs still exist.

• Absence of an error

This principle of testing refers to the belief that if our software has passed all the tests and challenges, it is ready to be released to the market. The fact that the project, program, or website does not have any defects does not mean that it meets the requirements of the users.

• Early testing

It is important that software tests are performed at every stage of its life cycle and not only at the end. The later the error is found, the more likely it is that it will bring more errors. Taking care of it can save you time and resources.

• Testing is context-dependent

Programs, applications, and websites differ from each other depending on their purpose. The blog website will be tested differently from the banking system. It is important to match the way and methods of testing to the level of risk.

Model of testing

The previous paragraphs familiarized us with the goals and principles of testing, these are the bases that every tester should know, remember, and most importantly refer to during their work. In this part I would like to introduce more detailed issues related to software testing. The first thing to know is the V model. The V model is the most commonly used model of software testing and it shows different testing phases in good way. It requires the creation of two groups of people who participate in the project: a team of developers and a team of testers. It consists of verification phases and validation phases separated by the implementation phase.
The verification phases include:

• Requirements analysis

During the analysis of the requirements, it is determined what the ideal product should be capable of. Its functionality, interface, security, and performance are determined. It is saved in the so-called user requirements document. During this stage, acceptance tests are designed.

• System design

During the system design stage, the ways in which a given functionality will be achieved are determined, and whether it is possible to implement it. The document relating to user requirements is edited. System tests are also designed at this stage.

• Architecture design

This planning phase is responsible for setting high-level requirements. Lists of modules, relationships between them, dependencies, databases, technologies, and integration tests are saved.

• Module design

Here, a detailed outline of each module is designed, as well as a way of communication, internal communication interfaces and API, and a way of showing errors. In addition, unit tests are designed.

Then the implementation phase follows. Another is the validation phase, during which the testers take up work, and it consists of:

• Unit testing

The lowest possible level of testing in the V model. It is responsible for verifying whether each module has the right output, how it reacts to an incorrect input and is it safe and secure.

• Integration testing

Integration testing refers to checking whether individual modules communicate with each other and whether they can coexist with each other.

• System testing

System tests are designed by the customer team. It is the client who determines what the requirements are, what the program should be able to do. All applications should show functionality, resistance, and safety of use. Performance tests, strength tests, regression, and so on are used here.

• User acceptance testing

The last stage of testing. The tests composed by end-users are validated here. Realistic inputs are used. This is to show whether the product is ready to be launched on the market.

Model V is just one of many models. These include primarily prototyping, the RAD model, Rational Unified Process (RUP) and agile, iterative-incremental models, and many, many others. Model V has been presented here to show different types of tests, at various stages of the program's life cycle. The choice was subjective.

What should be tested?

I think that after reading everything above, we can conclude that the answer to the title question is not trivial. We know that there are many different stages, we know what the general principles are and why tests are needed at all. However, what should be tested?

It should be remembered that it is very difficult, if not impossible, to anticipate every possibility with which the customer will use the final product. In the case of devices, even the ambient temperature can have a big impact. Due to the multitude of factors, testing everything in every possible configuration takes an infinite time.

In my opinion, each project has different goals, different requirements, and a different degree of risk towards the final recipient. After gathering all this into one, the final picture of our work emerges, and based on this image we should edit the appropriate tests. For a blog website, they will concern the correctness of adding comments and the speed of switching between subpages. For the shuttle, it will be the strength of the material. In my opinion, the requirements that the product should meet and the safety of its use should be tested. 

Testing vs real life

Testing is an inseparable part of every project, it depends on the validation phase whether the product will be implemented by the customer. Test design starts at the beginning of the project from the so-called acceptance tests, by system tests for integration tests and unit tests.

To be able to compare testing with real conditions, we should ask ourselves: what are the real conditions? Real conditions are those in which our program will be used by end-users. 

The customer defines how the end-user will use the program. Based on this definition, tests are designed starting from those at the highest level, that is the acceptance tests. Based on the highest level, testers design what is below and they determine how the whole machine should work. Its weaknesses are identified and protected accordingly. Individual modules are developed. Testing aims to show that the software is consistent and robust, and works in a certain way.

Every stage of validation tries to imitate the real conditions. There are test cases that imitate normal use but there are also others that expose the program to overload. Different variations of input, different numbers of events, different combinations. However, we must remember a few principles: the absence of an error and exhaustive tests are not possible.

For example, a car manufacturer must be sure that every component of the car works properly, is in place, and cooperates properly with other elements. The person must know that each component meets certain requirements. All parts are put together into a whole car, which interface is the possibility of driving. However, as testers we cannot say where the rider will drive, we can only check its resistance to the range of conditions that might be met. In real life, we are dealing with many combinations so that the simulation would last indefinitely. Therefore, the most frequent and the most critical ones are selected in the testing process.

About the author

I have studied physics with programming specialization at the University of Wroclaw. During the studies, I began a part-time job in an embedded focused company, where I learned the software and hardware skills. After almost two years, I decided to change the way of my life and try as a tester in Nokia. Now with our team, we verify the working features. Personally, I am interested in low-level programming, mathematics, and telecommunication. After work, I love road cycling.