Aug 3, 2009

Recycling security test cases

Test cases are precious. In my day to day work, a lot of what we do is built around the notion that a test case either passes or fails. Think about the time when you withdraw money out of your favorite ATM or when your favorite application may have compromised your privacy... it's always the case of a passed or failed test case. Is there a flaw in the system? Can the system under test be put to its knees because of it?

A while ago James Whittaker talked about building repositories with reusable test cases. The whole discussion is available at the MSDN blogs. I definitely recommend reading it. As applications often have common flaws, it is beneficial to have such a repository. Tweaking the test cases to the application under test is definitely a "must-do". However, once those tweaks are done for the particular software under test, the job gets much easier.

Reflecting at the security industry, it's hard to not notice the need for James's idea to be adopted. In fact, a typical problem report from a security vendor will include the following bits:

  1. What is the problem?

  2. How can the problem be reproduced?

  3. How can the problem be fixed?
Maybe we should start swapping the second bit with the actual executable test case? I know it's hard to do this for any type of application, but for the most common type -- Web apps -- it's certainly doable. Just look at tools like Selenium and you'll get the idea.


Gabe said...

I'm not a big fan of reusable test cases due to the poor success rates I've seen so far. Almost every test org I've been in tried at some point to create a super-duper-all-comprehensive automated testing framework by collecting testcases and parameterizing them so they could be executed against almost any target, only to realize that the parameter space is too huge to be practical. Translated in English, you'll always have to do some work to adapt testcases or their execution to the SUT. And in many cases you'll spend more times at filtering in/out testcases and adapting them to the SUT than if you started from scratch.

In this sense, I'm a big fun of bullet lists :-) A bunch of checklists works better than a robot.

radi said...

@Gabe: actually the idea is more toward providing the actual test case that can be plugged into the customer's testing automation right away.