Testing reports

[John Ralls]

On Apr 11, 2012, at 5:06 AM, Colin Scott wrote:

> 
>> I would argue that the contents of the report are more important 
>> than the formatting, at least in the context of how this thread
>> started.
> 
> Hmmmmm.  I'm not at all sure that I agree.  *WHY* would the report formatting change?
> 
> The fundamental point here is that even if the change is harmless it should only occur as the result of a specific and specifically intended action - anything else must be a bug!!!  Either way, any change in the output should be flagged, either so it may be certified as correct (and then incorporated into the standard against which subseqent tests will be made) or so the cause of the change can be tracked down and fixed.
> 
> There are no circumstances I can envisage where it is proper to ignore *any* change to the output.

Rubbish. Non-trivial programs have both interdependencies between modules and dependencies on external libraries. Those can -- and do -- change, and can cause changes in the behavior or output of a module under test. Well designed and written tests focus developer effort on changes which matter and ignore changes that don't. Tests which fail for insignificant reasons (like case in an SGML tag) waste developer time and are worse than useless.

Regards,
John Ralls