Automated web testing is not a new concept today. With advancements in web development technologies and how complex web codes are written today, web testing was bound to walk this way sooner or later. The problem with adopting automation is that it is not like a Netflix subscription that we can pay and enjoy cloud services.
Climbing the automated web testing ladder requires planning, careful execution, and a workforce that is familiar with all the intricacies. In addition, all these things require capital that cannot be ignored or marked “miscellaneous”.
The cost is huge and all these things come to a simple yet head-scratching question – will we be able to get a return on this investment?(ROI) Or are we better off without it?
This post will help you answer these questions if you are in such a situation or think you will face such questions in the future. Let’s dissect the reasons affecting the return on investment (ROI) and draw a rough timeline by which the difference between them would become positive.
Factors affecting ROI in automated web testing
While calculating the ROI in automated web testing, we may come through the following factors-
- expansion in the space dimension
- expansion in the time extension
- additional costs in these dimensions and others.
Let’s see them one by one and how they are related to each other.
Expansion in space dimension for ROI
The first thing to understand is how much space will we be increasing when we start thinking about automated web testing. First of all, we need people that can perform automated web testing. It will cost money to hire them and keep them around as well. The more people you hire, the more it will cost you.
So, taking an average salary of $80,000 for one automation tester for one year, hiring three of them would cost you $240,000. Although $240,000 cannot be considered the purest form of investment as manual testers will draw salaries too if given this work. You can subtract the current offering from this amount.
Salary Investments = (Automation tester salary – Manual tester salary) * number of testers
Also, the people doing the hiring for the organization will be paid for their time. The number of people can range from 3 to 8 depending on the lot size.
Hiring Costs = (One day salary of the interviewer) * (number of interviewers) * (number of days interviews take place)
Resource Investments
Next comes resource allocation. Each of your testers will require resources to work on. For example, if you have opted for any cloud-based tool for say CI/CD pipelines then you need to extend the current plan.
Since these testers will access the tool simultaneously, a better plan would be desirable which will cost more money. This can be completely slashed out with manual testers as they will not require cloud-based automation testing frameworks.
Resource investments = Additional tool plan costs
Resource Setup
The last factor in space dimension is the device lab setup required for automation testing. Automation testing is preferred in cases where repetitive tasks are being performed which simply wastes precious time.
For example, a manual tester can also work on a device lab but repeatedly doing the same things on each device is an error-prone mechanism. Hence, this is one of the factors to introduce automation in the system.
If your device labs were already set up and manual testers were working on them (which is never recommended), these costs are saved as they were previous investments. If not, this will be included.
Device lab setup cost – Device 1 cost + Device 2 cost + Device 3 cost…………. Device n cost
You will also require to invest in a central connection to these devices to manage them through a virtual machine etc. This cost would be negligible though in front of actual device costs.
If you are not looking forward to making heavy investments in device setups and are more inclined towards renting one, your costs will come down significantly.
I recommend going through a cloud-based setup that has real devices to operate on. This way you can get all the advantages of real devices and do not have to dig so deep in your pockets.
Cloud-based automation web testing platform is purely a matter of requirements. If your requirements are satisfied by a tool X and a tool Y, you can choose the one asking for less money.
For example, a fantastic cloud-based automation testing tool is Testsigma that hosts real devices that give accurate results and performance metrics of your web application.
You can also perform cross-browser testing and AI-based test script generation with NLP (English-based language) in no time. With so many features and so many devices, this tool just costs $249 per month. Compare this with device setup cost and you will understand the difference.
Cloud-based tool investment = Annual Charges or (Monthly charges) * months
Configuring tools and frameworks
A bitter truth in the testing industry is that test automation takes a lot of time to get started. The configuration time is especially high when you are opting for the online tools and teams make the mistake of not considering it for ROI in automated web testing.
The problem with configurations is that you will never work on a single tool or library etc. The processes are too complex today and even if you take Eclipse in your local system, you know how troublesome it becomes to eradicate all the errors.
Therefore, always take configurational setup time into consideration while calculating the ROI. This cost will just be initial but will impact your total investment number.
Configurational costs = (Salary of automation tester for one day * days taken for configuring the tool) * number of testers working in the configurational setup
While calculating the ROI here, we are not taking into account the training costs for the automated web testing languages and frameworks. This is in assumption to the fact that you will hire people who are already familiar with the language and other requirements (as we considered the hiring costs).
If you already have automation testers but they need to be trained, you can replace them with hiring costs. The net investment should remain the same though.
Expansion in time extension
The above section was all about setting up the resources and investing in them when we start automated web testing. But as the saying goes “time is money”, we spend money every second a tester or a resource is at work.
Test script creation
The first thing starts by creating the test scripts. If this is the first time your team is boarding the automation flight, a lot of the time will be spent on this. This can even extend to a couple of months. The more testers are working on this, the more cost you are going to bear.
Test script creation investment = (Salary of one automation tester for one day * days taken) * number of testers working on test scripts.
How can you tackle this and bring the costs down? The best way is to go for tools that actually do not require any programming languages. For example, Testsigma uses plain English language (such as “Navigate to www.testsigma.com”) instead. These can be created extremely faster.
The second method is to choose a tool that is GUI based and uses drag-and-drop to create tests. This may bring down costs too.
Maintenance
Next comes the maintenance. If the tests have been created and you are working on subsequent iterations (for example), then test scripts will need maintenance.
This maintenance work is equivalent to writing new test cases, debugging some older ones, deleting some test cases, shuffling the code a bit according to the new version etc. The bottom line is maintenance takes considerable time and hence investments.
Maintenance investment = (Salary of one automation tester for one day * days taken for maintenance) * number of testers maintaining the test cases
Maintenance is a direct result of the tool you are using. If you have opted for tools that have the capability of automatically healing the broken test cases then maintenance time can be drastically reduced.
Testsigma comes built with a self-heal feature that looks to detect and rectify any structure-related changes as soon as they occur.
Testsigma is giving 30 days free trial
Both of these processes can easily top the charts for the things that would take most of your time and cost you in automated web testing. There will definitely be other things in between though.
For example, sometimes you might get a failed test case error because of things that were just out of your control. A slow page load, a failed UI element locator, failed loop execution just to name a few.
We can surely take them in this post but the cost is just negligible when we talk about hundreds of thousands of dollars.
How much time do we have to wait?
Up until this post we talked about the investments required in starting the automated web testing and not about the return time on them. If you are opting for automated web testing and use that efficiently for a longer period of time, the return will eventually come.
The point of inflection in the time-cost graph is bound to be achieved. The time, however, depends on you.
The following graph shows the time-cost relationship between the time and the investment returns relationship:
The definitive values are extremely hard to predict due to the number of variables involved. How many testers are working? What kind of project is this? What are your testing methods? Are your testers trained? And so on.
Although, we can roughly calculate the time saved in each test and multiply it with the tests we are executing.
For example, let say a manual tester takes 20 minutes to execute 100 tests. In automation, we need 30 minutes to write the tests and 5 minutes to execute them. The comparison is 20 to 35 which is the initial investment we have talked about in this post.
But in another iteration, a manual tester will again take 20 minutes to execute the same 100 tests. This time though, we have our automated web tests already created and therefore it just takes 5 minutes this time. So, for two iterations, my time becomes 40 to 40 which is equal.
This is the point of inflection shown in the graph above. In the third iteration, automation surpasses with the time being 60 to 45 (manual to automation). Automation will never look back again.
To make things interesting in the above calculation, we can also add the salaries of each of the testers. The manual tester will charge for 60 minutes in three iterations whereas the automation tester for only 30 minutes of test creation. The next 15 minutes are resource costs that are negligible in comparison.
ROI is subjective – Do not copy-paste
ROI is always with reference to the investments. Investments are always with respect to the project and situations. Therefore, if you think company X had to invest $300,000 a year and therefore you will do the same, you are probably wrong.
ROI is purely subjective in nature. As I have repeatedly mentioned, the variables involved changes everything for every organization, and careful planning is the only rescue part.
This post tried to highlight the major investments, their costs of implementation and the return approximations. I hope this post made a fair judgment between the practical and theoretical aspects of ROI in automated web testing. For compliments and complaints, please reach out to us in the comment section!
Blog image credit: Business vector created by stories – www.freepik.com
