Error when running test suite

John Ralls jralls at ceridwen.us
Tue Dec 3 11:00:59 EST 2013


On Dec 3, 2013, at 7:32 AM, Derek Atkins <warlord at MIT.EDU> wrote:

> John Ralls <jralls at ceridwen.us> writes:
> 
>>> There's a difference between a test failing and the environment being
>>> incorrect (e.g. due to a missing locale).  In this case it's a broken
>>> environment -- it's something easily correctable by someone running the
>>> tests so IMHO the failure should not be fatal and instead should tell
>>> the user about their broken environment.  It's not an issue that
>>> requires a developer to look at it (in which case a crash is arguably
>>> better, although personally I'd rather the crash be forced only while
>>> debugging the test rather than while running a put "make check").
>> 
>> Fair enough, though anything short of a hard failure is unlikely to be
>> noticed in the spew of output from 'make check'. OTOH, most folks
>> don't care whether GC works in all locales, only that it works in
>> their locale. It would be a simple enough change to try setlocale for
>> each test locale and then test only the ones that work, while emitting
>> a warning that the others aren't being tested, similar to what we do
>> with dbi backends.
> 
> Also arguably we should upmark failures as part of "make check".  At the
> end of "make check" it should print out a big PASS or FAIL (possibly
> with the # of test failures), and maybe even exit with a non-zero exit
> code.  I'm just not sure how to do that.  (I thought it *used* to do
> that)

"Upmark"?

It's done by the old home-grown test code at the subdirectory level; it returns non-zero if there are failures so that make will abort.
That's not supported by gtester, which just stops when it has a failure.

More sophisticated test runners won't abort at all, but will collect and display the number of passes and fails for the whole project along with a log indicating failures. I don't know of any like that for GLib's test framework, but I haven't looked. There's a huge variety of test runners out there for CppUnit, which is by far the most popular C/C++ unit testing framework. Boost test is another framework I'm considering, largely because it would save a dependency.

Since you have some views on what we need for a test environment, perhaps you could add a "future" section to the Testing wiki page summarizing what you'd like to see.

Regards,
John Ralls


More information about the gnucash-devel mailing list