Testing reports

[John Ralls]

On Apr 12, 2012, at 7:57 AM, Colin Scott wrote:

> 
>> ... Tests which fail for insignificant 
>> reasons (like case in an SGML tag) waste developer time and are 
>> worse than useless.
> 
> Changes made in external libraries are just as capable of screwing up your results as changes made anywhere else, and you need to know what changes have occurred, even where those changes are harmless - and if they are they need to be incorporated into your standard.  

Rubbish again. Well, maybe it's true if you're writing embedded controls for critical equipment (say, fly-by-wire for an airplane). In any other case, though, that level of change awareness is too expensive. In commercial work, it would drive the cost of the product beyond what customers would pay; in volunteer work (like Gnucash) it would mean that those tests are ignored -- or the comparison document gets robotically changed to match the new output.

Regards,
John Ralls