With every software enhancement, the final product that is being delivered to the customer includes the new feature as well as the existing ones. The focus of Agile teams is to maintain consistency in delivering a satisfactory final product to the customer within the timeline.
This is ultimately achieved by sequencing the testing of software modules that stand important from the customer’s perspective. Hence the need for test case prioritization arises.
The test case prioritization of regression testing is done taking into account the business requirements, previous test cycle experience on functioning of the existing features and the delivery timelines.
In this article, let’s discuss the approach in prioritizing the test cases for regression testing and its importance in the test life cycle.
What is Regression Testing?
Regression Testing is a software testing process to verify that any modification made to the software or the product does not impact the existing functionality of the software.
With the addition of any new piece of code to the software, the ‘as is’ functionality of the product should not be hampered by any means.
Consider a day-to-day example of a mobile banking application:
Post the addition of a new check deposit feature introduced in the application, it is observed that the customer is not able to access the ‘transfer funds’ module in the application.
Here, we wish to avail the benefits of the new feature but at the same time, we also want to use the existing features of the application without discrepancies.
Regression testing is extremely important and plays a vital role in such scenarios.
Under what circumstances, regression testing is needed?
- Whenever there is a new business requirement and the existing code is modified as per the requirement.
- Addition of any new feature or service to the software.
- Post defect fixing and redeploying the code to a specific testable environment.
Multiple ways to perform regression testing
Regression testing is needed whenever there is software enhancement, error correction, modification and deletion of existing features. These code changes can cause the system to work incorrectly.
There are multiple ways in which regression testing can be carried out.
- Re-execution of the entire test suite :
- Selection of regression test suites :
- Prioritization of test cases :
Prioritization of regression test cases is one of the major factors for performing the regression test execution. We can prioritize the test cases depending on the criticality of an application and the business impact of the same. The regression test suite can be extensively reduced by setting priority on the test case selection.
This is one of the methods of regression testing where all existing test cases in the test suite are re-executed. This process requires huge time and resources.
In this process of regression testing, a chunk or portion of the test suite is being selected for execution instead of re-executing the entire test suite.
Why do we need to prioritize test cases for regression testing?
During the test execution process, there are major challenges witnessed in regression testing as listed below :
- Regression testing takes a huge amount of time for bigger projects.
- In some cases, the test execution might even take one complete sprint for completion.
- This increases the test execution time, the individual efforts along with the increased project costs.
- Also, re-executing the same set of test cases repeatedly results in lack of concentration.
- The testers might fail to test the most critical functionality of the application deriving bugs into production.
- This eventually increases the project expenditure.
- Prioritizing the test cases helps the DevOps team to reduce the cost of software test execution phase and the time taken for test execution to ensure the delivery of an exceptional quality product.
- It allows the testers to think and analyze which tests to run to manage the risks for software delivery.
- Test case prioritization offers help in detecting and correcting faults earlier than might otherwise be possible.
- Prioritization assists the team to resolve most critical defects at an early stage of a testing cycle.
- If the test cases are prioritized for regression testing, the DevTestOps team can get a chance to detect and correct the faults at an early stage as possible.
How to prioritize test cases for regression testing?
It is usually observed that a huge number of defects are usually reported by the testing team during the date very close to the production release time.
This creates a negative impression on the customer because, in order to fix these critical bugs, the delivery deadline might need to be extended. This calls for prioritizing test cases for regression testing.
Prioritizing regression test cases can be done in the following ways:
- Select test cases with frequent defects :
- Choose test cases with critical functionalities :
- Select test cases with frequent code changes :
- Cover end-to-end test flows :
- Cover field validation test cases :
- Select a risk-based testing approach :
- High priority: These test cases cover the critical function of the application, defect prone modules and modules that have undergone recent changes.
- Medium priority: These test cases include negative test scenarios. Field validations, error message generation test cases are included under this category.
- Low priority: These test cases cover the remaining functionality of an application (includes UI and less defect prone modules).
- Select a collaborative approach :
This approach works in collaboration between the developers and testers.
The developers and testers as per their knowledge and experience can contribute to prioritizing the test cases for regression.
During the sprint, the team can sync up to work on a regression scrum board specifying the areas worked on by every team member.
For e.g. as per the developer’s knowledge, he mentions the recently added code changes can influence the accounts summary section of any particular user. He then adds this information to the board.
The tester then by looking at the board can make a selection of test cases covering the ‘Accounts Summary’ module and assess the time taken for execution of this module.
This makes the testing more efficient and less time-consuming.
Considering the previous regression test cycle experience and learnings, we can select test cases that resulted in generating frequent defects.
We can select the test cases that are designed to cover the critical functionality of an application. For e.g. In a mobile banking application, we have major functionalities like ‘Transfer funds’, ‘Bill payments’, ‘View service requests’. We can focus and concentrate on testing these functionalities first.
Let’s discuss this with an example — In the mobile banking application under the ‘View service requests’ module, we have the addition of multiple service requests to the existing ones like Credit card limit increase, check book request, account linking request, stop check payment request, etc. With this example, we understand multiple times the code has been modified for this functionality. That’s the reason we need to prioritize and select the test cases covering this functionality.
Here we can cover all the end to end integration test cases in which the happy flows of a module are tested right from the beginning to completion of flow. For e.g. End to end testing for placing a fund transfer request or adding a payee under the bill payments section.
Selection of negative test cases covering the field validations where if we miss inputting the details of a mandatory field in a form, the application displays an error message restricting the user to proceed towards the next section.
In the risk-based approach for agile regression testing, considering the test cases covering the modules where recent code changes have been made, the testers rank the test cases as per their priority which reduces regression time and efforts.
The testing team divides the regression test suite into three categories :
High and medium priority test cases are always part of the agile regression test suite which is scheduled after every sprint.
Low priority test cases can be used in regression testing before the major release.
Hence, assigning priorities helps reduce time and efforts on regression testing in agile with better quality product delivery.
Facets to consider while prioritizing regression test cases
Test case prioritization is an extremely important factor when it comes to software testing. It allows the testing team to test the high priority test cases which help to resolve critical bugs earlier in the testing phase. In order to ensure testing is being carried out with the proper selection of test cases for the purpose of identifying the issues correctly.
In the agile way of working, we can utilize the scrum ceremonies like our refinement sessions.
In this forum, the developers and testers can discuss the code changes to be implemented for a particular functionality.
The developers based on their experience can suggest to the testing team which other modules will be affected by this change. A regression testing tool can also be of help here.
This helps the testing team to take into account the selection of test cases covering that specific module and assign it the highest priority during the test execution phase.
Developers can discuss the core functionality modules which required complex coding which resulted in problems in the past. Considering this past history of issues, testers can choose the test cases accordingly.
Test case prioritization can also be done in :
- The context of the importance of usability of a particular feature in the application by the customer.
- Modules in an application of most frequent use.
- Any new functionality to be taken into account.
- Features that undergo frequent changes.
- Features where failures would be more visible (in the opinion of developers).
Factors that prove the benefits of test case prioritization
Test case prioritization is extremely beneficial in the software testing phase.
This process not only minimizes the extra effort, time and overall cost to the project but also helps the team to focus on critical issues on priority for maintaining the good quality of the software.
- As per the priority, test cases can be arranged in a specific order to detect the fault at an early stage. Also, next time when the regression is planned and the testing team resources are to be shuffled, it becomes easier for the new agile team as well to pick up the test cases as it is maintained in the priority order.
- Test case prioritization ensures that the most important test cases are executed first in the software testing life cycle.
- Test case prioritization assists the testers to find bugs in the application as soon as possible.
- As the high priority test cases are given early preference in execution to find bugs at an early stage, test case prioritization improves the overall performance of regression testing.
How test cases can be prioritized based on certain techniques
So far we have discussed multiple approaches for selecting and prioritizing the test cases for regression testing. However, the selection of test cases requires expertise and knowledge of experienced testers.
Let’s discuss some major techniques of test case prioritization followed by Dev(Test)Ops team :
- As per Customer requirement: In the customer requirement based priority technique, test cases are prioritized based on factors determining the business requirements of customers as documented in the functional requirement document. Test cases are designed as per the customer assigned priority, complexity of a requirement and the requirement volatility. These factors are given a value and test cases with higher factor values are prioritized.
- Code coverage based technique: In the code coverage based technique, prioritization is based on the portion of the code of a program that has been covered during the testing process. These test cases have the capability of testing a major part of code so those are prioritized first.
- Cost-effective: In a cost-effective approach, the test cases are prioritized on the basis of the cost factor. The cost can be the cost of regression testing process, cost of requirement gathering, cost of entire test execution, cost of analysis to select a test case, cost of prioritization of test cases. Test cases requiring less cost will get the higher priority.
- History-based: In history-based approach, test cases are prioritized based on the history of test cases. The test execution results of a particular test case determining the possible chances of failure are selected and prioritized first and so on. The execution history of the test determines the likelihood of the selection in the current testing phase.
- Priority using fault severity: In this approach, the requirements considered are based on fault severity i.e the number of times the fault can occur in the code which makes it important to test that specific code first thus given the higher weight.
- Random prioritization: The test cases here are ordered randomly in the test suite.
The question here is on what basis the weight of the requirement is determined.
Here are the four factors explained in brief :
1. Business value measure: In business value measure, according to the importance of a requirement they are assigned a rank. The most critical requirement is given rank (10) and the least important requirement is given rank (1)
2. Project change volatility: Project change volatility factor depends on the number of times the customer is modifying the business requirement during the software development life cycle. This usually happens due to lack of communication, inputs from the user and unclear requirements.
3. Development complexity: The overall time taken for the development process, the technology on which the feature has been built up decides the complexity of the development phase.
4. Fault proneness of requirement: In this process, considering the historical data on requirement failure reported by the customer we can assign a weight to the requirements and the test cases are then assigned a priority.
The faults are then assigned the severity e.g. complex, moderate, low measure.
More information on test case prioritization can be found here.
Conclusion
Test case prioritization is an effective and efficient technique to deliver better quality software to the customer. Test case prioritization allows the testers to categorize the test cases as per the highest and lowest priority to perform smooth test execution. Test case prioritization helps to deliver good business to the customers by ensuring the most important test cases are executed first.
By this means the issues and defects in the software can be identified and fixed as early as possible. One major achievement in test case prioritization would be to gain the customer confidence and faith on the DevOps team to be able to deliver a less defect prone software into production.
Achieving customer appreciation and confidence in the delivery of the product helps in generating good business to the software industry.
