adminIt is extremely important to develop a strategic plan when creating test automation scripts. With the push towards Agile and DevOps the mindset has become to automate as much testing as possible. While that idea makes sense, it is important to determine how much test automation is needed. Part of the problem is that those who are managing Agile or DevOps teams have a really different mindset, and they really don’t understand basic testing principles. Instead of taking the time to understand, they force their understanding and push test engineering in the wrong direction.
Test Automation Planning
Test automation is certainly critical for Agile and DevOps. There are things that can and should be automated but before that is done, a few questions must be asked:
- How many times are these test automation scripts going to be executed?
- Is the code stable or is it going to change a lot over the next several sprints?
- Has the test been executed manually and has it passed successfully?
- How much time will it take to maintain the test scripts?
- Do you have the right resources to build and update the test scripts?
- What type of tools are you going to use?
- What type of framework are you going to build?
- How much does the test automation software cost?
These questions will help lay the foundation of your test automation planning. The next step will be to identify the number of test cases that need to be automated.
Testing That Should Not Be Automated
Here is a list of test cases that should not be automated:
- Test cases that will only be executed a few times
- Test cases that require human touch, for example review of a customer bill
- Test cases that require a third party billing system such as a payment gateway that doesn’t have a test environment
- Test cases were the code is unstable and is constantly changing
Test Case Quality
Test automation scripts help with providing test coverage across applications. This allows faster deployments because it doesn’t require manual testing. However, if you create hundreds of test cases and those never identify any defects, is it really worth maintaining and running those over and over again? Probably not. The test cases that are automated should be able to capture defects. Therefore, it is important that you have some really sharp manual testers who can think through the testing scenarios and identify problem areas that need attention. Without that system knowledge, you will not capture defects and it will not be a good use of your resources.
Return on Investment
It is really important to understand how much cost is involved in creating automated tests. If you create thousands of them, how much is it going to take to make changes on those when the GUI or back end systems change? If it is your department or team, as a manager or employee, you really need to be able to answer this challenging question. At every organization, I make sure that I calculate the ROI. There are a few ways to do this. First, you roughly need to know what is the cost of a member of your team or the average rate per contractor or employee. Most organizations will have that already. Second, you will need to determine how many scripts are going to be automated and how many times are they going to be executed. Third, you need to know how much time does it take to execute those test cases manually. Fourth, once the test cases are automated, you need to know how much time does each one take to execute. These items will at least give you a basic ROI. Most ROI numbers are somewhere between 6-12 months. Anything over a year, probably isn’t worth it. There are other variations that are more complex, but this will at least get you started. Knowing this information is certainly powerful, and it will help justify the expense for having automation engineers.
I cannot stress the importance of doing these basic steps. With Agile and DevOps, everything tends to be rushed. Perform this basic analysis first and spend a little time up front to yield large returns in the long run.
Ron Wilson