What is Regression Testing?
Regression testing is the process of verifying that if a change is made in one component of the product, the other components of the product continue to behave like it was before. It is a means to ensure that change in one functionality of the product doesn’t result in unintentional changes in other functionalities of the product.
This change could be anything: a defect/bug fix, code refactoring, a feature enhancement, change in system configuration or run environment or even upgrading a dependent library. Any change that can impact the behavior of the product calls for regression testing.
Need for Regression Testing
Different modules of a product work together to achieve the desired behavior. A change in any of the modules often has the ability to impact its interaction with other modules of the product, which may or may not be foreseen. This is the primary reason that regression testing is needed.
This need becomes even more pressing if the product design is not very clean and the interaction of various modules is not thoughtfully handled.
Significance/Advantages of Regression Testing
Regression testing helps in identifying unforeseen changes in a product’s behavior after a planned change in its source code or configurations or system environment or dependency libraries.
Regression testing is the way to ensure that even after the change, the product works the way it should and no new defects are introduced due to the change implemented. Regression testing serves a crucial role in maintaining the quality of a product.
A feature enhancement that breaks existing features of the product or a bug fix that introduces new bugs as side-effects does no good to the product and therefore, regression testing makes sure that things that are seemingly good are actually good for the product and working.
Test Plan/Test Cases Selection for Regression Testing
A regression test plan could be a set of functional tests and/or a set of unit tests, whose success would indicate everything is good to go and a failure would indicate a new problem.
What all to cover as a part of these functional and/or unit tests is a tricky problem and solution varies from situation to situation.
A common approach is to test all the modules that don’t have the new change but interact with the module that has the new change. However, a lot of this decision depends on the product design and your experience with the product quality.
For example, if in the past you have seen that changes in one module have had unforeseen impacts on other modules, you know the Regression test plan needs to be a bit more extensive to cover at least sanity test cases for every module and acceptance test cases for the modules that directly interact with the changed module.
On the other hand, if you know that the modules of the product are very well designed and have stood the test of time on quality metrics, you can keep the regression test plan to be short and limited to only the modules that directly interact with the changed module.
Seeking inputs from product experts who know the internal details of the product helps a lot in creating the Test Plan for Regression Testing.
The product experts here could be a developer working on the change, design architect, a quality engineer who is well versed with the product internals or a product manager who has good knowledge about the business requirements.
Their inputs could range from a technical assessment of the impacted modules to a business assessment of the impacted user scenarios and these can serve as valuable information for the regression test plan.
As much as it depends on the product you are working with, the regression test plan also depends a lot on the resources you have at your disposal in terms of time and infrastructure.
If you are working against a strict timeline or a strict budget of resources or both, you know you cannot afford to have a very extensive Regression Test Plan and your need of the hour is a short and concise one.
We can say that choosing a Regression Test Plan is striking the right balance between your available time and resources, guided by your experience with the product quality.
Significance Of Automation in Regression Testing
Test automation is the most celebrated and leveraged in Regression Testing. It just eases off the burden of manually running the Regression test plan.
A pre-existing automation setup not only saves your time from a manual run of the test cases, but it’s also less error-prone and has a baseline for test results against which the new results can be compared and analyzed.
If test automation can be scheduled to run at regular intervals, it can make Regression Testing really easy.
Depending on the time it takes for the product source code to build and the time it takes for the automation test suite to be run, you can configure the frequency of the automated test runs. But having a regular interval for automated test runs helps a lot in catching the problems early.
The smaller the interval is, the easier it is to identify the root cause of the problem. For example, if test automation is configured to run with every new build, as soon as any new change in source code breaks an existing behaviour, it can be noticed and rectified immediately.
If the test automation is configured to run daily, you need to analyze changes made only in a single day to identify the root cause of the problem.
If the frequency of test runs is weekly, the changes to identify a problem span a week’s time. As this frequency interval gets bigger, bigger gets the overhead in identifying the cause of the problem.
Effectively for regression testing, we can say that a team’s manual testing effort becomes inversely proportional to its effort investment in automation test runs.
Ideally, automated test runs with every new build should be the best solution, but we all know that the real world is not ideal and the real world has its own tradeoffs. These tradeoffs could be:
- shortage of human resources in the team to work on test automation
- lack of machine infrastructure to keep automation up and running
- elongated time of builds and test automation runs which make it practically impossible to run the test automation with every new build
- A high number of check-ins in the product source code on a daily basis which can put extensive pressure on the build and test automation infrastructure.
To workaround with these tradeoffs in the real world, the ideal frequency of your test automation runs is usually a balanced combination of your available resources and the urgency of your need for regression testing.
How to decide the entry criteria for Regression Testing?
If test automation is not integrated with the build system and/or automated test runs are not scheduled at regular intervals, Regression Testing needs to be performed manually. In that case, does every code change need to be regressed? The answer is No.
To qualify for Regression Testing, a code change needs to have an impact on other components of the product. To identify this, one needs to analyze the interactions of the changed module with other modules of the product.
Usually, developers making the change are aware of the impact it can have on other modules and the overall product behavior and they should responsibly demand Regression Testing if needed.
Following considerations can be made when deciding the entry criteria for Regression Testing:
- Change made in code is impactful. There is no need to regress modules if they are not impacted by the change.
- Presence of adequate test data and results. Previously executed test cases and/or test plans should be present for Regression Testing so that their results can be used as a benchmark for current behavior.
- Available resources. Time, team members and infrastructure are needed without which Regression Testing cannot be performed.
- Intention to fix issues. If for any reason, there is a lack of intention to fix the issues found during Regression Testing, there is no benefit of investing in Regression Testing.
How to decide the exit criteria for Regression Testing?
Exit criteria for Regression Testing are the criteria which help you decide when your regression testing should be considered as complete.
Deciding the exit criteria for Regression Testing is a fine balance between resources available to you in terms of time and money and your appetite for risk.
If you have the luxury of time and resources at your hand, it’s good to have an extensive regression test plan that can cover as much as possible and be better safe than sorry.
But if you don’t have the luxury of time and /or resources at your hand, you need to strike a fine balance on what all you can cover as part of the Regression Testing and what you are ready to live with some calculated risk.
The risk is calculated using technical as well as business knowledge of the product. Following are some suggestions that can be considered when calculating risk:
- Features of the product that are popular among users should be given priority in Regression Testing.
- Technical areas of the product that have stood the test of time and have known to be good on the quality reports are the ones you can choose to live with some risk.
- Analysis of previously reported bugs by customers as well as internal teams can help in identifying the popular features of the product as well as the weak areas of the product.
- The probability of a feature bug being reported by a user only arises if it is part of a user’s workflow. This implies that product areas that have the most reported user bugs are also the most popular among users.
- Bugs that have been long open and still haven’t been fixed reflect the product areas that have the least impact in the user’s workflows, probably users have found their own workaround solutions for them.
There is no thumb’s rule for deciding the exit criteria, it’s just a balance between what you need to validate for survival and what risk you are ready to live with.
Regression Testing for Legacy codebase vs New codebase
Regression testing can vary a lot for legacy codebases and new codebases. When you are working with a new codebase for a product, you have the opportunity to build the testing framework from an early stage of the product.
This is the time when you can incorporate unit tests along with functional tests in your testing framework and when these tests are run as part of regular automation, they can go a long way in improving the product quality as well as the effort needed for regression testing later.
The sooner you start investing in your test automation, the sooner you start reaping its benefits. The earlier your test automation starts running, the better off you are in maintaining the quality of the product and minimizing the manual effort required for regression testing of every new build.
When you work with a legacy code base, that may or may not have test automation, the situation becomes different. Products that have legacy codebases are usually the ones that have managed to strike a chord with the users for several years, sometimes even several decades.
For most of these products, when they were being developed, dedicated testing effort in terms of time and resources used to be a prominent phase in the software development life cycle and huge manual testing effort used to be done.
But with more evolved software development methodologies being practised now, regression testing for these products becomes challenging.
Their codebases are huge and not easily integrable with modern testing frameworks. In some cases, where such products have had their own test automation frameworks, enhancing those frameworks to achieve the same capabilities as offered by modern test frameworks itself becomes a task.
The legacy test frameworks were sometimes developed with the objective of reducing the manual effort and not to quickly perform the regression testing at regular intervals. And the list of challenges here can go on and on.
Despite all the challenges, legacy products are enhanced for new features, adapted to new run environments and maintained for quality. Because working with legacy products has its own benefits. Legacy products have huge data associated with them, which you can use to your advantage.
With this data, you can easily get information such as:
- what features of the product are popular with the users
- how do various product areas stand on quality metrics
- what are the weak areas of the product in terms of quality
With all this information at your hand, the trick lies in finding the right balance in your regression test plan – the balance in what you need for survival and what risk you are ready to live with, the balance in what your users care about the most and what your users care about the least.
If you can achieve this balance, you know your legacy product already has a loyal customer base that is not going anywhere as long as their expectations with the product are respectfully met.
Conclusion
Regression testing is essential in maintaining a product’s quality. To ensure good quality for the product, you need a good regression test plan with well-defined exit criteria.
To determine the exit criteria, you need to identify what matters most and what matters least for the product. Therefore, to achieve the right regression test plan, take inputs from all stakeholders of the product and find the right balance.
