spam,Re: Testing reports

[Derek Atkins]

John Ralls <jralls at ceridwen.us> writes:

> On Apr 11, 2012, at 5:06 AM, Colin Scott wrote:
>
>> 
>>> I would argue that case inversion like this doesn't necessarily 
>>> require a good reason, but that's a minor detail.
>> 
>> Are you saying that you are happy for the output to change without that change being the planned result of a deliberate, targetted, action?  I would be *very* unhappy were any program of mine to behave that way!
>
> Speaking for myself, not Derek:
>
> Absolutely. If I rip out the custom HTML generator in favor of building a DOM tree in code, and then use libxml's write_to_string() to create the report, then no, I would expect good tests to tell me if important content had changed but to pass if the only changes were things like the case of the tags or whitespace in text sections. Doing otherwise pretty much describes a brittle test.
>
> To take it a bit further, suppose that libxml on Win32 emits capitalized tags, but on Debian Wheezy it generates lower-case tags.
> I need to have "make distcheck" pass in order to generate the distribution in each case, and having to #ifdef tests for for the OS for something stupid like that would be a major pain.

It's okay, I actually agree with this, so you can speak for me here, too
:)

I think it's important to make sure the content of the report is
correct.  Stuff that does not affect the content (like case of the HTML
tags or extraneous white space) should be ignored.

I do agree with Colin that the tests should be capable of detecting
those kinds of changes.  That's easy -- store a .html file and perform a
"cmp" to see if it changed.  However, just a test would be extremely
brittleand cause way more false positives.

Another thing to keep in mind, Colin, is that the developer working on
the report might not be the one who breaks it.  Moreover, a change over
in some section of the code can cause a test break elsewhere.  Tracking
down why the test breaks can take time.

Even worse, but most importantly, the test failure might not be noticed
right away, so now the developer who notices the test failure now has to
bisect to track down where the failure started and then figure out why.
If a change in a dependency causes the test to fail, that could make
tracking it down even harder, because it could fail in your environment
but pass in mine.

Therefore, it is best, IMNSHO, to spend the time to make the test the
least brittle as possible.  It would be time well spent

This begs the question:  who is going to do the work?

> Regards,
> John Ralls

-derek

-- 
       Derek Atkins, SB '93 MIT EE, SM '95 MIT Media Laboratory
       Member, MIT Student Information Processing Board  (SIPB)
       URL: http://web.mit.edu/warlord/    PP-ASEL-IA     N1NWH
       warlord at MIT.EDU                        PGP key available