Is Defect a defect when not defined in a Test Script?

This question came up during an email thread that I had with some testers. Most of the testers had the view that a defect can be a defect even if it has not been defined in a test script, while I said that this is not true. The conversation went like this:

Tester: We can not put every scenario in a test script

ME: Yes you can

Tester: Can not because we can not think of scenarios while writing test scripts

ME: Then you should. Testers also think when they are testing, so what stops them from thinking when they are creating test scripts

Tester: It is not possible and also it is not feasible

ME: Not possible, I will disagree. Let us focus on the Not Feasible part

Tester: It becomes very difficult to document everything as then the scripts will run into multiple pages and maintaining them would be an overhead

ME That is not reason enough not to capture all that can be in a test script. You have to accept the fact that a test script can be incorrect. For a defect to be a defect, it has to be mapped against a missing requirement that should come from a document that can a client would have provided. If you can not provide a document/reference for the missing functionality, then there is not reason you can call it a defect. It is fine to log it as a TAR and discuss as a change request, but a defect i do not agree to.

Beyond this point, I started finding testers fumbling for words. No one out there could give me a solid answer to convince me. What are your views on the same? I blogged about why do i do unit testing and why do i feel we can cover all the scope in a test script. We developers do that for Unit testing, why can testers not?

3 thoughts on “Is Defect a defect when not defined in a Test Script?

  1. Between all these we are forgetting why a test script which defines all scenarios is the most fruitful of all

    Read some of those:

    * Writing things down forces you to make decisions on the details. During the “how does this actually work” phase of testing lots of things get tried and tested, but you probably don’t need all of them to build an effective test set. Equally you might find that you haven’t really covered all the bases and need to add a few tests.
    * Once you have written it down anybody can run it. This can be quite important if you are working as part of a team of people where you might not be the one that runs the script next time. Even if it is only you running test, having a script to work from means that you can come back to something in the future and just re-run what you have written without having to re-do the analysis.
    * If it’s written down somebody else can review it. This one is similar to the previous point about anybody being able to run it, but it allows you to take advantage of the opinions of other people. If your testing is all in your head then it is quite difficult for somebody else to give you an opinion on it.
    * It shows what you have done. Test-scripts give other people a chance to appreciate your hard work, hopefully with a big pay rise. They also document the testing that has or hasn’t been done. Without some kind of written record nobody knows what testing has definitely been done on a given version of the software. This may seem like something completely irrelevant at the time but people come back to old versions all the time. Once a program has left testing its biggest test is only starting at the hand of the unsuspecting customers.
    * It makes the ISO/CMM/Tick IT police happy. If this doesn’t mean anything to you consider yourself lucky as you can probably do what makes sense for you (and get away with murder in the process). If do have them you will know that test scripts are considered to be an essential part of the testing process.

  2. @Ashish – Thanks for calling these out. Now that we are talking about “Why Test Scripts” – i will like to add one more point:

    * Test scripts provide a consistent way of doing testing over and over again. Which is the fundamental principle for automating anything. If we can not have that in place, we would never be able to automate a script.

  3. You might want to check out Behavior Driven Development (BDD). They make a pretty compelling argument that not only are scenarios required for regression testing but that they’re also a pretty good way for programmers to actually design their applications – acceptance tests first from the UI on in.

%d bloggers like this: