Ensuring Full Test Coverage with Agile Testing Sandboxes

Ensuring Full Test Coverage with Agile Testing Sandboxes

Testing has always been a critical part of the development lifecycle but never before has it become so critical due to the pressure of delivering high quality software and avoiding a roasting on social media when applications go wrong. Organizations are spending more time and money on testing than ever before, however, are they actually covering everything in their tests?

In our experience we have seen organizations running hundreds, thousand or even tens of thousands of tests, however, often many of the tests simply test the same thing. It’s a little like a jigsaw where many of the pieces are missing:

ostia full coverage-001

So what sort of tests are missing? The following is a non-exhaustive list of ‘missing pieces’ we have found in test coverage that we have reviewed:

  • Lack of testing of the ‘unhappy path’. It is estimated that 20% of application development consists of ‘solving the problem’ while the other 80% deals with what happens if things go wrong. Yet most tests we find deal with the ‘happy path’ to ensure that this works. In today’s world, how an application deals with errors is potentially as important as the functioning application.
  • Lack of testing of timeout conditions. What happens if a service does not respond in the agreed time? Many applications retry but don’t always test that this functions correctly.
  • Lack of testing of edge data conditions. Often tests do not include testing of what might happen with the first or last character of the alphabet, or negative or positive numbers.
  • Lack of testing of bad data conditions. How will the application deal with data it is not expecting?

The primary reason that this occurs is because testing occurs with shared test environments so setting up tests to cover these issues is difficult. Breaking a test environment to test the unhappy path of any application could have serious consequences for other users of the environment. This is where agile testing sandboxes can help:

ostia full coverage-002

An Agile Testing Sandbox simulates the test environment upon which an application is dependent for testing but runs independent of the real test environment on Cloud, Docker or commodity hardware and software. Consider a sandbox to be like a flight simulator for your testing environments. This offers fully functional versions of the application, services and synthetic data that the testing teams can use for their continuous testing. These can be created on demand per tester so no further sharing of environments is required and test coverage can be brought to 100%.

This results in:

  • Testing sandboxes that can support error results for testing while avoiding any impact on other users of the application.
  • The sandbox can be set up to respond outside of the agreed levels of service in the Service Level Agreement (SLA) to test conditions like timeouts etc. This can be used to test retry operations and what happens if the application simply does not respond.
  • The sandbox can be set up to ensure that data is available for the edge data conditions so that applications can be sure that they can correctly deal with the limits of data that they must be able to process.
  • Following on from the previous point, the sandbox can also be set up to return bad data to ensure that the application can deal with bad data should it be returned.

These sandboxes are not simply built and thrown away after a project. When a project is tested in this way, the sandbox environment becomes a valuable asset that can be reused again and again in future related or even unrelated test projects.

Portus EVS enables the creation of sandbox environments in days enabling organizations to ensure that they are getting 100% test coverage in a cost effective way thus delivering quality applications to their customers.


Written by : Ostia Solutions