Introduction
Continuous Testing is the testing strategy to fast-track the testing required for achieving rapid software development using Agile and DevOps methodologies.
What is Continuous Testing?
Continuous Testing is the process of testing at all stages of software development – one after the other – without any human intervention.
These different stages could be stages of tests: unit tests, integration tests, acceptance tests and performance tests as well as stages of environments such as Dev, QA, Staging and Production.
For example, when a code is integrated into the centralized source code repository, unit tests should be automatically executed followed by integration tests.
If integration tests pass, automated acceptance tests can be run followed by performance tests.
Similarly, a code change is first tested in a dev environment using automated test scripts followed by a QA environment. If all works well in the QA environment, the change can be automatically progressed to the staging environment and finally into the production environment.
Unit tests and dev environments are usually taken care of by developers, whereas all other tests and environments are mainly considered the responsibility of testing and DevOps teams.
Why is Continuous Testing Needed?
In today’s world, software applications have become a key differentiator for businesses. Impact of failed software releases and security breaches in software applications have been seen to impact their market values. Therefore, software quality has become all the more important, it has become a pressing need.
With the crucial role software applications are playing these days in businesses, the need to rapidly develop and adapt the software as per the new requirements has also emerged strongly. While a lot of new software development methodologies could be used to fast-track development work, not much progress could be made in accelerating the testing work. Testing was essentially becoming a bottleneck in achieving quicker software delivery.
To achieve faster software delivery, if a shorter version of testing was adopted, it increased quality issues and business risks. Longer versions of testing were becoming unachievable with quicker software releases. This is where Continuous Testing can be used to match the speed required for faster software development and delivery.
What is DevOps?
DevOps is the methodology which combines development and operations to achieve shorter software delivery cycles through the CI/CD pipeline. Shorter software delivery cycles enable teams to quickly release small product changes to the users which can make gathering feedback related to a particular change more effective and easier.
Prior to the DevOps world, releasing even a small change in the product would require a fixed elongated time and process before it could reach the users. DevOps is a wide domain of tools and practices, but for the sake of simplicity it can be broadly described as follows:
1. Continuous Integration: Developers integrate their code with the source code repository on a regular basis. Whenever they integrate, the entire mainline is built followed by unit testing and code quality checking.
2. Continuous Testing: Pre-defined sets of test cases are executed on the binaries obtained from Continuous Integration to ensure quality and assess associated business risks.
3. Continuous Delivery: Binaries that pass through Continuous Testing can be considered good enough to be released anywhere anytime, essentially these are the Release Candidates of the product.
4. Continuous Deployment: Binaries that get a go-ahead for release are actually deployed into production environments.
Why is Continuous Testing Important for DevOps?
DevOps is all about enabling continuity, be it code integrations or running tests or deploying in various environments. Continuous Testing is the bridge that joins Continuous Integration with Continuous Delivery and Deployment to make an effective continuous software delivery pipeline.
Continuous Testing makes sure that a change that is to be considered for Continuous Delivery / Deployment meets the expected quality criteria. It provides a means to assess the overall health of the product after a change, which can be further used to assess the associated business risks.
Continuous Testing is the way to ensure that a change that doesn’t meet the expected quality criteria, cannot and should not be released to the users.
Continuous Testing vs. Automation Testing: What’s the Difference?
Traditional test automation approach can be termed as reactive whereas Continuous Testing approach can be termed as proactive.
Traditionally, test automation starts after a feature is developed and integrated into a centralized source code repository. Once a developer’s task completes, the tester’s task of writing automation scripts begins.
Considering that purpose of these automation scripts may range from functional tests to integration tests to user-acceptance tests to performance tests and even more, creating these scripts is expected to take considerable time which adds a big chunk in the overall time required to deliver the feature/software.
No matter how best the tester tries, he/she always ends up being seen as the laggard who is trying to catch up with the expected need and speed of software delivery in the modern world. This is what changes with Continuous Testing.
In continuous testing, test scripts are written before the feature development. As soon as a developer integrates his/her feature source code into the centralized source code repository, the code is automatically built, and automated run of various test scripts starts to assess the quality of the code.
Continuous testing leverages test automation for better efficiency and optimization. It lifts the burden of being laggards from testers and makes them the accelerators for the desired speed of software delivery.
Benefits of Continuous Testing
Continuous testing is known to provide various benefits over traditional test automation. Some of them are:
1. Early Feedback: As soon as a build is available from a centralized source code repository after a feature work has been integrated, the quality assessment starts, and this assessment helps in providing early feedback on the feature work.
One doesn’t have to wait for human interventions to get feedback. This early feedback helps in making timely assessments for associated business risks which can vary from deteriorated user experience to failed software releases.
2. Faster Delivery: Since all processes that take place after developer’s code integration into centralized source code repository are taken care of and automated even before the development work is complete, it helps in saving a lot of time thus enabling faster delivery of features and software.
3. Better Quality Control: With Continuous Testing, teams can execute a much better control of the quality process. Various levels of test-suites can be used to gate various levels of quality in the product.
Automated controls can be implemented to ensure that a change that fails to pass through one level of quality doesn’t make it to the next level. For example, a change that fails integration tests can be prevented from running through performance tests.
4. Continuous Improvement: With continuous testing in place, the efforts required to improve a piece of work get reduced a lot since the required pieces of testing are already in place and can be executed without much need of human intervention.
For example, if you are working on improving the performance of a product and performance tests are already part of the product’s Continuous Testing, you can keep on doing your improvisations without much intervention from the testing team.
Without Continuous Testing, for every improvement you plan to make, you would have to make it prioritized with the testing team as well, causing a lot of overhead and delay along with a loss in the testing team’s bandwidth in doing all these iterations.
This advantage of Continuous Testing goes a long way in making teams feel more confident and encouraged to improve continuously.
5. Quality with Velocity: Today, software applications are needed to be developed and delivered in a rapid mode. Traditional testing methodologies were becoming a bottleneck for this model. Continuous Testing works well here as it provides quality with velocity thus making rapid development and delivery possible.
Challenges with Continuous Testing
Incorporating Continuous Testing in the software development cycle does come with its own challenges, such as:
1. Cultural Shift: Continuous Testing demands a big cultural shift in the way organizations work. Organizations and teams are used to working in a certain way such as waterfall and agile where testing team’s work begins after the development or with development.
For Continuous Testing, the testing team’s tasks need to be completed before actual development tasks. To achieve this, the testing team needs to be involved at the early stages of requirements and design discussions.
The testing team needs to be seen as a key stakeholder who needs to know what the plan is as well as why the plan is important to achieve so that the associated business risks can be timely assessed and mitigated.
2. Upskilling: Testing teams need a lot of upskilling to become the first-class citizens of software developers whose thoughts and opinions matter in design discussions and planning rather than the second-class citizens they are used to being, working with whatever has been decided and developed.
3. Futuristic Thinking: To start ahead and finish ahead of actual development, testing teams need to have a futuristic thought process. Preparing to test something that doesn’t even exist yet, is difficult. To achieve this, it is important that there is agreement on designs, architecture and interfaces. Major changes in any/all of these can put Continuous Testing at risk.
4. Infrastructure: Depending on the type of project, testing teams may need access to different kinds of infrastructure to set up different test environments such as QA Test Environment, Stage Test Environment, Production Test Environment. Also, the need for infrastructure can vary from different hardware to different software applications. Ensuring the availability of such a wide range of infrastructure can be a problem.
5. Test case Effectiveness and Optimization: Continuous Testing is very susceptible to elongated testing cycles that may result by running the extensive test suites that have been created so far. Long-running test cycles can defeat the whole purpose of Continuous Testing.
For Continuous Testing to be effective, test cycles need to be short so that they can be run for every new code change and build. To achieve this, test-suites need to be reviewed regularly to identify the most relevant and efficient test cases that can provide maximum feedback in the minimum time.
Conclusion
Benefits of Continuous Testing can surely outdo the challenges associated with it, provided it is implemented correctly. Continuous Testing is the need of the hour to make testing match pace with development and deployment practices for achieving faster software release cycles. Continuous Testing makes Agile and DevOps methodologies more effective and efficient in achieving rapid software developments.
Check out how Testsigma performs continuous testing
