A while ago James Whittaker talked about building repositories with reusable test cases. The whole discussion is available at the MSDN blogs. I definitely recommend reading it. As applications often have common flaws, it is beneficial to have such a repository. Tweaking the test cases to the application under test is definitely a "must-do". However, once those tweaks are done for the particular software under test, the job gets much easier.
Reflecting at the security industry, it's hard to not notice the need for James's idea to be adopted. In fact, a typical problem report from a security vendor will include the following bits:
- What is the problem?
- How can the problem be reproduced?
- How can the problem be fixed?
2 comments:
I'm not a big fan of reusable test cases due to the poor success rates I've seen so far. Almost every test org I've been in tried at some point to create a super-duper-all-comprehensive automated testing framework by collecting testcases and parameterizing them so they could be executed against almost any target, only to realize that the parameter space is too huge to be practical. Translated in English, you'll always have to do some work to adapt testcases or their execution to the SUT. And in many cases you'll spend more times at filtering in/out testcases and adapting them to the SUT than if you started from scratch.
In this sense, I'm a big fun of bullet lists :-) A bunch of checklists works better than a robot.
@Gabe: actually the idea is more toward providing the actual test case that can be plugged into the customer's testing automation right away.
Post a Comment