Last updated: April 28, 2006

A Customer's Perspective on Software Testing

by Troy Pearse, Hewlett-Packard Co., Boise, ID

My expectation is that coming into CS481 you do not have an adequate background in software testing.  I expect that you have written a number of programs which must produce a certain result, but that you have never been required to write a document stating how you are going to test your program, implemented hundreds of test cases which are designed to be run many times to validate that your program continually works correctly, and made sure that your program doesn't fail in any conceivable situation.

I expect you to spend time learning about software testing and to spend a considerable amount of effort designing, implementing, running and verifying tests for your project.  I expect far more testing than you have ever done on any project at school.  This project is more than just making sure your program runs on a few select test cases.  You must make sure the tool operates correctly in a wide variety of environments.  You must seek out industrial test cases to prove the robustness of your tool.  You must conceive of possible weaknesses in your code and solution and then write tests to exercise those weaknesses.  Without adequate testing, you do not have an acceptable project.

1) Correctness of results (compare it to known, existing tools).

You need to verify that the results your tool produces are correct.  This will mean doing an extensive amount of manual calculation of results to compare to your test results. Yes, it takes time and isn't fun, but you must do it.  For your specific project I am requiring that you calculate by hand all of the reuse and turmoil metrics between the CS481 func97 tool and the CS481 Jact tool to validate your project, plus one industrial test case (see section number 4).  This is your minimum amount of correctness testing.

2) Robustness and portability of solution (port it everywhere).

It is very important to me that your program(s) work robustly, in a variety of computing environments, and that it does not unexpectedly abort.  You will need to test your program to make sure it can handle valid file system conditions (such as file permissions) as well as potential error conditions.  You will make extra efforts to make sure your tool is portable in ANSI C environments and that it does not use any shell capabilities that are not common across at least the Borne Shell and the Korn Shell.  I prefer a stand-alone program that does not require a multitude of support programs or scripts.  This will make the tool easier to distribute and use.  If other tools or scripts are required for proper execution, you must write tests that validate proper operation with and without those dependencies.

3) Testing completeness and re-testing (regression test all changes – have 1 test per defect).

You must create tests to test all of the different functionalities that your program has, and these tests must be able to be run and verified many times during your project.  You should make an effort to verify that the tests you have written at least exercise every area of the code that has been written.  In addition, every problem that you find with your program must have an automated test written to recreate that problem which can be rerun to validate when the defect is fixed and run on future versions to make sure that or a similar problem hasn't been reintroduced.  I expect you to plan on running your entire set of tests early, and often.

4) Industrial test cases (stress test it with big, bad, and ugly data).

Your program is expected to run on large, difficult to parse, industrial ANSI C files.  You must find industrial test data (on your own) to test with.  I suggest that you locate 2 different versions of a GNU tool, such as GNU Emacs or some other GNU tool, and run your tool on ALL of the different C source files.  Also, as a part of verifying the correctness of your tool, you must choose several representative C source files from one of these tools and validate the results by hand.

Suggested References:

Glenford Myers: The Art of Software Testing

Bill Hetzel: The Complete Guide to Software Testing