Last updated: April 28, 2006

Testing Tips

by Troy Pearse, Hewlett-Packard Co., Boise, ID

Testing Frame of Mind

Testing is a very technical job that requires understanding of the specified functionality, the implementation model being used to program the functionality, and a recognition of limitations or exceptions that may keep the program from working.

Testing Lessons Learned

The following are a set of testing lessons learned over many years working on the front lines.  One of the important learnings from this is that the engineer who applies himself or herself to testing aggressively, will find more defects.  The harder you look, the earlier you look, the more you will find. Think of it as a perpetual Easter egg hunt!  Don't miss any eggs in the tall grass, or you'll be sorry later when you step on a rotten one!!

1.  Defects Happen

Writing code is writing defects.  The more code, the more defects.  Accepting that code has defects is important to writing tests.  Once you know defects are in the code it becomes easier to write tests to find them.

2.  Test Early

Begin testing as soon as possible.  This means planning early freezes that can be tested in some manner, such as individual functional tests, an early SWQA run, or just lab duration tests. The earlier you start testing, the earlier you start finding defects, the more time you have to fix those defects.  Establishing early test mile- stones is also a good way to verify you are on schedule.

Integrating code finds defects.  The earlier you can integrate some functionality, the earlier you find those defects. Plan on early code integrations.

4.  Record Every Quality Issue

Log every issue relating to quality into a defect database so it doesn't get lost.  Don't just fix problems that you notice in your code; log a defect, write a test reproduce the problem, make the test part of your regression test suite to make sure it doesn't show up later, and then fix it.  Defect tracking in a large group is a central way to communicate project issues.

5.  Establish and Test Interfaces

Having a well defined interface between sections of code and then writing test to verify correct interfacing is important. Verify parameters passed and returned across major interface boundaries, and use assertions whenever possible to verify reality.

6.  Test Before Check-In

Running regression tests before checking a file into the project source code repository is important to increase the stability of the project’s development environment.  Think twice about simple changes that you don't think merit a regression test.  In industry even one-line changes, or even changes to comments, break the build and cause other project members wasted time.  Do at least one sanity check before checking in code.

7.  Test One Change At A Time

Avoid the temptation to group multiple changes or fixes to a file, and then do only one check-in. Incremental development and test is important to isolate the cause of problems and to identify the amount of change being done to a file.  Isolating changes may also help the engineer down the line who is trying to isolate and understand your changes and merge them back in to another project.

8.  Plan Alpha and Beta Tests

Look for ways that you can give your code exposure to being run in an environment different than your own.  This includes making it available for internal Alpha use, and external Beta use, Invest time to make sure the product gets out and used, then follow-up to capture failure data.

9.  More Test = More Defects Found

The more tests you write, the more tests you run, the more defects you will find.  It is just that simple.  If you are worried about the number of defects in a certain area, ask yourself how you can write more tests.  Even setting out to write a "duplicate" test can uncover something you forgot, Try a test brainstorming session.  Take the attitude "How could I make this code fail?!!!" And then make it happen.  Look at the source code and poke around in a debugger for ideas.  If you don't see anything move on.  If something looks out of place, get out the shovel and dig away.

10.  Defects Hide Together

When you find some defects in an area of functionality, there are likely more waiting to be found.  Add more tests and tweak existing tests to expose those related defects.  Poke around, identify common root causes to failures and exploit them; think about race conditions and test them to bring the product to its knees!  At the same time, some areas of the code are likely to have fewer defects.  Learning to not spend too much time on them is important, also.

11.  Complex Code Has More Defects

The more complex the code, the more defects it contains.  Where complexity can be defined as a combination of function size, number of branches, amount of complex branch or loop logic, number of parameters used, number of globals used, number of external functions called, and number of functions calling this function. One simple metric of complexity is NU (Maintainability Index), which can be calculated using the NIAS tool that is available through the University of Idaho Software Engineering Test Lab's WWW home page.  Any module with a NU value less that 65 should have additional tests written for it.  Do not ignore the fact that complex code has more defects, and that those defects themselves are complex and often difficult to find.

12.  Touching Complex Code is Dangerous

Every time you touch a complex piece of code, you have the potential to break it.  Before you proceed, determine if there are tests written to test the area of code you are changing, to make sure you didn't break anything.  If you repeatedly have to change a module to get it right, then there are likely defects waiting to happen, ones you discovered, uncovered or created.  Writing new tests is one way to combat touching complex code.  Reviewing changes to complex code with peers is another effective way to find defects early.

13.  Measure Your Test Coverage

There are several test coverage tools that are simple to use and can tell you something about your testing.  Use these tools to review the tests you have written and to give you ideas about new tests to write.  Be careful not to take the narrow approach to just getting a certain kind of test coverage.  Think about what you are trying to test.  Statement coverage is a good bare-minimum goal for all code, but for areas that are more complex try and get double coverage by different tests.  Think about branches, loops, variable initialization and use, and don't forget about testing for exceptions.

14.  Use Tools To Find Suspect Code

Don't wait for a failure.  Use static analysis tools to look for suspicious or problematic code.  This is especially important if you are getting code from an outside source, or porting to a new platform. Start with your compiler warnings and then use other lint-like tools.  Yes, there is a lot of output to deal with, but talk with someone fluent in those tools to get some ideas of what is important.  Try and determine what each diagnostic message is trying to tell you.  If there is a potential ambiguity in the code, try to remove it.  Many execution and portability problems occur from writing code that is ambiguous to the compiler due to some dusty corner of the language that nobody knows.

15.  Peer Review Your Test 

Discussing your tests with a peer can help you identify things you forgot to test.  Try and learn best practices from your team-mates. Good testing means good code, and you want to write good code.