[GNC-dev] stress-testing reports

[Christopher Lam]

Hi all

To aid in development and refactoring of the numerous reports, I've
developed a report-options stress test.


*Premise:*
Most reports are currently defined in scheme. The reporting engine expects,
at a minimum, 2 functions to be defined - an options generator, and a
renderer.

The options generator will create a blank (gnc:report-options) object,
modify it by adding various options eg (gnc:make-string-option),
(gnc:make-simple-boolean-option), and return the options object to the
engine. This forms the basis of the report options UI.

The renderer will accept a report ID, and the options object. The renderer
will read the options used, query the database, and present a
gnc:html-document object to the engine. I'm not sure what happens
afterwards; the engine will render the html-document in Webkit.

*Refactoring*
Most of the report infrastructure was designed about 15-20 years ago...
this was using scheme/lisp conventions - bearing in mind LISP was designed
in the 1970s, and modern scripting engines such as python etc were nascent
- nonetheless scheme was chosen, for reasons that are apparent to schemers
only -- pythonistas will argue otherwise, but I do now that scheme is a
more flexible language to design financial reports; scheme has also evolved
in the past 15 years to become a powerful compilable language... Guile is
no different and will continue to evolve in a positive direction.

Unfortunately the state of Gnucash scheme code as of 2018 is stuck in the
past; for various reasons they have been kept on life support, limping
forward via every bugzilla report, and very difficult to understand and
create new types of reports. It would be good to refactor the numerous
scheme functions but, as some users have found out, refactoring the cryptic
code with good intentions do create some unusual report crashes.

*Proposal*
I have designed a mechanism whereby most standard reports in Gnucash have
been queried, and their options (string/boolean/etc) dumped, and a
test-stress-options is run continuously... This will repeatedly render the
options with various combinations of options, simply testing whether the
report will actually run. It does not verify the contents of the report.
It's designed to catch errors whereby e.g. finding periodic average on a
regular report is working well, but a refactored report whereby calculating
the average will fail on an empty book causing (total amount / num columns)
will trigger a division by zero.

The test-stress-options will come in 2 guises:

The first will perform a simple-stress-test, whereby all options are set to
successive values... if a report has 3 booleans (t/f) and 1 multichoice
with 4 values (a/b/c/d), it will run them with: (t,t,t,a), (f,f,f,b),
(t,t,t,c), (f,f,f,d) - this is fairly simple.

The second test will perform a combinatorial stress test, whereby the
number of options is sent to a local build of jenny from
http://burtleburtle.net/bob/math/jenny.html (thanks rgmerk); guile will
subprocess "jenny 2 2 2 4" which will generate an efficient list of options
(1a 2b 3a 4a), (1b 2b 3a 4b) etc, which is then interpreted by guile to run
the report repeatedly, hopefully catching more errors.

Either test is first run on an empty book, then on a book with sample
transactions.

On my laptop, a Intel i5, the simple-stress-test compiles in 1s and runs
most standard reports in 5 seconds and the combinatorial-stress-test
compiles in 8s and runs them in 20s. See its output at
https://gist.github.com/christopherlam/c04d43398ecac486d662c4e74eb293ca

Some reports e.g. multicolumn, hello-world, and eguile-based reports refuse
to be run, and are excluded.

IMHO this strategy, although complicated, is a safe and effective way to
ensure that refactoring internal functions do not lead to unexpected report
crashes. I'd think the simple stress test would be sufficient for daily
use, and combinatorics stress test prior to release. Code currently lives
on at https://github.com/Gnucash/gnucash/pull/378

Comments welcome!