Testing reports

[John Ralls]

On Mar 26, 2012, at 1:31 PM, Colin Scott wrote:

> 
>> Do you know how to write automated tests -- in any language -- that 
>> accurately check the output format of an HTML page? Automated in 
>> this case means the test program decides whether the formatting is 
>> correct, no human involved. If you do, or know where I can learn 
>> that, please share.
> 
> Sadly, no.  When I was doing this sort of stuff the output was in text format rather than HTML.  I did invent a rather crude - but nonetheless effective - test rig for a very large suite of programs, including reports, where a program could be run on a standard set of inputs, and diff the output with that from a baseline run.  It wasn't very flexible, but it was pretty effective at its rather simple-minded task, and was particularly helpful for regression testing ...
> 
>> The reports module doesn't have a test environment (and Guile 
>> doesn't provide one until 2.0), but the requirement is for 
>> automated tests that report success or failure to make check.
> 
> known data + known input = known output!
> 
> If something like Guile didn't provide the test facilities I needed, my simple-minded habit was to wrap it up in some suitable envelope (usually a shell script, but occsionally some Perl or similar!), and examine the outputs as in the "equation" above!  My guess is that for this task at any rate you don't think kludgey enough!  :-)

You're getting hung up on Guile. For the HTML output, the test can use any reasonable (meaning runnable in an autotools build environment), but no matter.

The real problem with that is that you demanded format tests, not content tests. Content tests are relatively easy: compare the output text with the expected text, as you say. Format tests (e.g., Is the margin right? Does the paragraph wrap at the right place? Do the columns line up correctly?) are much more difficult without human intervention -- and tests that require human intervention tend not to get run.

Regards,
John Ralls