KPIs or Key Performance Indicators in the software testing industry are some measurable values that are computed to gauge the efficiency and effectiveness of the testing process as a whole.
Though it may seem that measuring the KPIs is a natural thing to do, there is a divide between the people of the software testing community on the use of KPIs. Some say that KPIs can’t help achieve a much-needed balance between time, cost and quality while others swear by their effectiveness.
No two organizations have the same product and hence the development methodology and the testing processes are not same either. So what a QA individual or manager needs, is to understand the process, go through the KPIs that can be measured and figure out the KPIs that should be measured for maximum effectiveness in a particular area.
When are the KPIs not useful?
Although measuring the effectiveness of a process is essential to know if you are doing it right, measuring the testing process via KPIs will not make sense in a scenario where:
- Testing your product has just started
If you are going to launch your product for the first time and testing has just started, there won’t be much to measure. This time will be crucial to put a testing process in place rather than measuring the effectiveness of the testing process. - You are not planning to have a long testing cycle
If you are making a product that would not be changing for a long time after the launch and testing will be a one time process, measurement of the effectiveness of the process would not be beneficial as you won’t have any new testing cycles to improve upon. - You are on a restricted budget
Just like doing any activity, measuring testing KPIs also takes time and effort and consequently cost. So rather than measuring the KPIs, applying a cost-effective testing process should be the main focus when the budget for testing is restricted.
When are the KPIs especially useful?
- You have been working on the the same testing process for a while
When you have already implemented a testing process successfully and have executed it a few times, it is the right time to measure the KPIs to know the areas where your testing process needs improvements. - You have a big testing team
Having a big testing team also means that there will be a big distribution of testing tasks. So to make sure that the tasks are distributed effectively and are conducted efficiently, measuring some testing KPIs will prove beneficial and will also help to keep everyone on track. - You are thinking of introducing new processes for testing
If you are thinking of revamping your testing process, it will prove helpful to have some KPIs measured to the original process. It will help you decide what goals to achieve with the new testing processes.
What are the KPIs?
There is no hard and fast rule to measure all these KPIs and you can also come up with KPIs that are not mentioned in the list. The most common KPIs that are measured in the software testing industry:
- Active Defects
All the defects that are not closed yet are called Active Defects. It could have defects that are new, that are open or that are fixed but not verified. The testing manager needs to decide on a threshold value beyond which immediate action will have to be taken about how to proceed to lower the number of active defects. The general rule is, the lower the number of active defects, the better is the quality of the product at a point in time. - Authored Tests
This KPI is used to measure the number of test cases being designed during a defined interval of time. This also helps in measuring the test cases against requirements and the designed test cases can be further evaluated for inclusion in the regression test suite or ad hoc test suite. - Automated Tests
This KPI is to measure the percentage of test cases automated against the total number of test cases. Usually, a higher percentage means a better probability of catching any breaks during automation runs. The threshold for the percentage of automation should be decided according to the type of product and cost of automation. - Covered Requirements
- This KPI is used to measure the mapping of test cases to requirements. A test manager is required to make sure that all the requirements have corresponding test cases and the action should be taken on any requirements that could not be mapped to any test case and vice versa. The goal is to keep the mapping of requirements to test cases to 100%.
- Code Coverage
This KPI is used by testers to ascertain the percentage of code they are covering with their automated tests. - Defects Fixed Per Day
This is used to measure the effectiveness of the development but again is subjective as some bugs could be harder to fix than others. This could be used to predict the amount of work for the testing team. - Passed Requirements
This KPI comes useful when the release of a product is being planned, if there is any requirement that has not passed testing, the release should be delayed. - Passed Tests
To understand the effectiveness of the test case design process, the number of defects reported via designed test cases are measured. - Rejected Defects
This KPI measures the percentage of defects that are rejected as compared to the total defects reported. If the percentage is higher than the set threshold value then the underlying issue needs to be identified and acted upon. This could mean more training for software testers or better requirement documentation. - Reviewed Requirements
This KPI is to make sure that any requirement that the testing and development team is working on has been reviewed by the subject matter expert and is good to go. Unreviewed requirements could lead to inaccurate development and testing which will prove costly in the long run. - Severe Defects
This KPI aims to keep the number of severe defects in an application at a time under a limit if there are more severe defects than that immediate action is needed. But before using this the testing team needs to be properly trained to identify severe defects correctly. - Test Instances Executed
This KPI is to measure the velocity of test execution at any point of time to make sure that the testing cycle is on track for the release. - Tests Executed
This KPI is to measure the total number of test cases executed on a build including manual as well as automated at any given point of time. This is also a Velocity KPI. - Time schedule and constraint
This KPI is used to measure the average time of test execution. This helps in providing testing time estimates during release planning or while providing the development and testing plans to the project managers. - Defects Closure Rate
This KPI is to measure the efficiency of testers in terms of verifying and closing the fixed defects, this also helps to estimate better for the release cycle.
For an organization looking to measure testing KPIs, it is essential to understand the existing testing process and then measuring the KPIs that will bring out the actual improvements needed. It is important to know what value addition you are looking for, because these KPIs, if not used properly, could easily lead you in the wrong direction.
If you have an organization that has been planning to move to test automation but has been worried about the learning curve and the humongous amount of time needed for implementation of automation, look out for tools that enable to you to automate test cases in as much time as creating them manually will take. This will help improve multiple KPIs such as Automated Tests, Covered Requirements, Passed Requirements, Passed Tests, Test Instances Executed, Tests Executed. Testsigma is one such tool that is quickly getting recognized for the ease of use and the negligible learning curve.
