Testing reports

[Colin Law]

On 27 March 2012 09:38, Derek Atkins <warlord at mit.edu> wrote:
> John Ralls <jralls at ceridwen.us> writes:
>
>> On Mar 26, 2012, at 1:45 PM, Colin Law wrote:
>>
>>> On 26 March 2012 15:15, John Ralls <jralls at ceridwen.us> wrote:
>>>> ...  Do you know how to write automated tests -- in any language --
>>>> that accurately check the output format of an HTML page? Automated
>>>> in this case means the test program decides whether the formatting
>>>> is correct, no human involved. If you do, or know where I can learn
>>>> that, please share.
>>>
>>> Ruby on Rails has various tools that allow automated testing of html
>>> output (designed for testing web apps), but I don't know that much
>>> about them or how they could be used with gnucash.  It would need a
>>> working Ruby environment I believe with the appropriate Ruby Gems
>>> installed and so may not be appropriate.
>>
>> Got a more complete pointer? A particular Gem, maybe? What I found
>> [1][2] don't appear to do post-render checks. Most of the search
>> results focussed on testing the Ruby rather than the HTML. That said,
>> validating the HTML for well-formedness would be a pretty good
>> start. There are C, Perl, and Python tools for doing that. A Ruby tool
>> would be OK, too -- I made the Git tools that we use for synching with
>> the Subversion repository by translating a Ruby Gem and then
>> elaborating it.
>
> The Integration Testing lets you pick apart the HTML output
> post-rendering.

Functional testing includes this also.

> But it's not testing what the user sees, per-se, but
> rather the raw HTML.  So it doesn't answer the "is the margin wide
> enough" question, but it does help you test a format, and makes sure
> that if e.g. you have a table, the expected data is in column #N.

I don't think there is any way of testing what the actual output looks
like, only that the html/css is what you expect.

>
> I think we DO need to have a content tester, so that we can run tests on
> reports that check for specific content/results.  As Colin says, known
> input + known report => known output.  Right now we do NOT have a good
> way to run those kinds of tests, so we would need to write a test
> harness for this.
>
> This would NOT be able to test, as John says, the visual rendering,
> paragraph breaks, margin widths, etc.
>
> Colin, I am sorry that the report formatting changed between 2.2 and
> 2.4.  Some of this probably happened because of the move from GtkHTML ->
> WebKit which enabled CSS, and many reports were updated to support CSS
> instead of the old weird stylesheet stuff.  GnuCash is still in flux,
> there, and I expect that the swap to WebKit might complete in 2.6 (but
> I'm not sure).  I think it's unreasonable to assume that a report will
> *look* the same across all versions and never change across application
> version changes.  However you are right that if we had a test harness we
> might have caught reports that failed to report the correct numbers when
> e.g. accounts were closed.
>
> So let's make some forward progress here.  Colin, can you help us get a
> test harness in place that will test a report for its content?  Clearly
> it means we would need some test fixtures, and some way to run a report
> and gleam the HTML output for testing.  Can you help us here?

Which Colin is this addressed to?  If me then I am afraid I have too
much on at the moment.

Colin L.