Testing reports

[Derek Atkins]

John Ralls <jralls at ceridwen.us> writes:

> On Mar 26, 2012, at 1:45 PM, Colin Law wrote:
>
>> On 26 March 2012 15:15, John Ralls <jralls at ceridwen.us> wrote:
>>> ...  Do you know how to write automated tests -- in any language --
>>> that accurately check the output format of an HTML page? Automated
>>> in this case means the test program decides whether the formatting
>>> is correct, no human involved. If you do, or know where I can learn
>>> that, please share.
>> 
>> Ruby on Rails has various tools that allow automated testing of html
>> output (designed for testing web apps), but I don't know that much
>> about them or how they could be used with gnucash.  It would need a
>> working Ruby environment I believe with the appropriate Ruby Gems
>> installed and so may not be appropriate.
>
> Got a more complete pointer? A particular Gem, maybe? What I found
> [1][2] don't appear to do post-render checks. Most of the search
> results focussed on testing the Ruby rather than the HTML. That said,
> validating the HTML for well-formedness would be a pretty good
> start. There are C, Perl, and Python tools for doing that. A Ruby tool
> would be OK, too -- I made the Git tools that we use for synching with
> the Subversion repository by translating a Ruby Gem and then
> elaborating it.

The Integration Testing lets you pick apart the HTML output
post-rendering.  But it's not testing what the user sees, per-se, but
rather the raw HTML.  So it doesn't answer the "is the margin wide
enough" question, but it does help you test a format, and makes sure
that if e.g. you have a table, the expected data is in column #N.

I think we DO need to have a content tester, so that we can run tests on
reports that check for specific content/results.  As Colin says, known
input + known report => known output.  Right now we do NOT have a good
way to run those kinds of tests, so we would need to write a test
harness for this.

This would NOT be able to test, as John says, the visual rendering,
paragraph breaks, margin widths, etc.

Colin, I am sorry that the report formatting changed between 2.2 and
2.4.  Some of this probably happened because of the move from GtkHTML ->
WebKit which enabled CSS, and many reports were updated to support CSS
instead of the old weird stylesheet stuff.  GnuCash is still in flux,
there, and I expect that the swap to WebKit might complete in 2.6 (but
I'm not sure).  I think it's unreasonable to assume that a report will
*look* the same across all versions and never change across application
version changes.  However you are right that if we had a test harness we
might have caught reports that failed to report the correct numbers when
e.g. accounts were closed.

So let's make some forward progress here.  Colin, can you help us get a
test harness in place that will test a report for its content?  Clearly
it means we would need some test fixtures, and some way to run a report
and gleam the HTML output for testing.  Can you help us here?

> Regards,
> John Ralls
>
> [1] http://guides.rubyonrails.org/testing.html
> [2] http://strugglingwithruby.blogspot.com/2010/04/testing-for-valid-html.html

-derek

-- 
       Derek Atkins, SB '93 MIT EE, SM '95 MIT Media Laboratory
       Member, MIT Student Information Processing Board  (SIPB)
       URL: http://web.mit.edu/warlord/    PP-ASEL-IA     N1NWH
       warlord at MIT.EDU                        PGP key available