Testing reports

[John Ralls]

On Mar 26, 2012, at 8:27 PM, Yawar Amin wrote:

> Hi John,
> 
> On 2012-03-23, at 19:48, John Ralls wrote:
> 
>>> […]
>>> 
>>> So, what's a good test harness for a Scheme report against the GnuCash
>>> backend?
>> 
>> For what value of "good"?
> 
> ‘Good’ meaning something that ensures the contents of a report stay consistent between different versions. For example, the original problem on this thread was that the Profit & Loss report wasn’t showing correct figures after the book was closed. So that was a regression. A test for this report might be able to ensure that its results are the same before and after book closing, no matter what date the closing was done on.
> 
> If I understand unit testing, that should be able to cover cases like these?
> 
> Btw, I wasn’t actually talking about the output’s HTML rendering. Obviously, we’re rendering as a web page; we don’t really expect the result to be pixel-perfect.
> 

A test harness doesn't ensure anything. It facilitates running tests, collecting results, generating reports, and telling build systems (e.g. make) that tests failed. Nicer ones will have extra features, like a way to organize tests hierarchically and a variety of comparison functions. 

No, unit testing is testing each function (unit) in a program, isolated as much as possible from the rest of the program (even to the extent of having stand-ins for functions called by the function-under-test to minimize dependencies) while varying the inputs to probe each execution path through the function. Testing a report's output falls under what is variously called integration, functional, or acceptance testing, which tests whole libraries or programs together to make sure that all of the pieces work together properly. You can use a unit-testing harness to do parts of integration testing, but testing the GUI requires a special harness which simulates user input and instruments in some way the GUI display. The Ruby on Rails testing tutorial [1] has a nice introduction to the various phases of testing (and separates functional and integration testing).

Colin Scott is the one who wanted the output format to be tested. Better to get the basics covered first before we worry about that. Unit testing the reports code would be an excellent start. 

So, back to your original question: There's already a rudimentary test harness in place, along with some even more rudimentary tests. The shortest path would be to use that, and for the moment that's the best choice for integration tests because it's already in place. On the C side, I'm using the GLib testing facility [2]. You can wrap that facility for use in Guile if you want a purer unit-testing experience. Guile 2.0 includes its own unit testing module, but AFAIK Gnucash still chokes on Guile 2, so that would have to be fixed before you could use it.

Regards,
John Ralls
[1] http://guides.rubyonrails.org/testing.html
[2] http://developer.gnome.org/glib/unstable/glib-Testing.html