Gnucash UI dogtail test harness some ideas

[Josh Sled]

ahmad sayed <ahmadsayed83 at yahoo.com> writes:

> 1 - Testplan editor is a common practice in Testing tools, in which user
[...]
>     My suggestion here is to use xml with the following format
>     <Testcase python_method="test_new_account_dialog">
>         Test the new Account dialog
>     </Testcase>

I'm not sure that this XML file is useful.

In the complex case of some XML file that was describing the tests more fully
(or even "formally"), that file could contain a whole mini-language for doing
test Setup ... then Acting to invoke the test ... then Asserting the test's
results.  Those steps would inevitably get more complex.  That mini-language
would need to be mapped into code at some point, and eventually wants to be a
"real" language itself.  Given that python is already a nice, simple
language, and can express whole object-structures literally, why not just use
it directly?


Even in the simple case, the XML file is just a re-iteration of all the test
functions (or methods, whatever they are) that already need to be specified
in the code. If that's true, then how is it more useful than just having that
list in python itself?  E.g....

    class TestNewAccountDialog (TestCase):
        # __init__(), other utility method definitions, &c.
  
        def test_new_account_dialog(self):
            '''Test the New Account dialog'''
            # ...
  
        def test_cancelled_new_account_dialog(self):
            # ...
  
        def test_all_defaults_new_account_dialog(self):
            # ...
  
        def test_no_new_accounts_selected(self):
            '''Now we're getting fancy.'''
            # ...

    def get_new_account_dialog_test_suite():
        '''@return a list of test cases'''
        x = TestNewAccountDialog()
        return [x.test_new_account_dialog,
                x.test_cancelled_new_account_dialog,
                x.test_all_defaults_new_account_dialog,
                x.test_no_new_accounts_dialog]

    def test_main():
        '''trivial (example) test-runner'''
        success,failure,error = 0,0,0
        for test in get_new_account_dialog_test_suite():
            try:
                # clean_slate()
                if test(): success += 1
                else: failure += 1
            catch:
                error += 1
        print success,failure,error


I note that – similar to JUnit – every method (and its docstring) that
started with 'test_' could even be inspected from the TestNewAccountDialog
class itself in order to build the test suite/plan ... thus
get_new_account_dialog_test_suite (and all its brothers) could go away
... which would mean that you would only need to specify the tests once: in
their already-required definition.


I've used testing tools in both the "programatic" and "declared" approach,
and I've found the former to be much more desirable.  Even if you're really
excited about doing an XML-based test-plan, I hope I can convince you to put
that part of the project off until later on, if/when more of the core and
required parts of the test framework are proven out.



> 2 - Generic Validator is a way to write a quick validator, by storing the
> state of the widget or dialog under test, by state I mean all its children
> named and their values, those data will be dumped to a file and stored as
> reference after the first run, and then it will be used as reference for
> later runs.

How much of this validator would be custom work vs. being provided by
dogtail?  I don't have a good understanding of dogtail yet, so please correct
me where I'm wrong, but it looks like dogtail itself can "dump" the state of
a dialog as a tree of named resources, widgets and values... in which case,
one could – for very simple cases – construct an "expected" state file in the
same format, and literally run `diff` between the two files... is that
correct?

In that case, it seems like one would require the following other pieces for
such a mechanism to work:

- filtering:  If you dump a whole dialog, but are only interested in
  checking the validity of a few fields, you need to filter the others so
  that `diff` doesn't report extra differences.

- canonicalizing: If you change the visual layout of the dialog, that
  (probably) shouldn't invalidate the test.  Something needs to bring the
  dumped/asserted state (and perhaps the "expected" state as well) into a
  "canonical" form for a good comparison.

Does this sound right?  Is there anything else?  Does dogtail provide either,
already?

-- 
...jsled
http://asynchronous.org/ - a=jsled; b=asynchronous.org; echo ${a}@${b}
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 188 bytes
Desc: not available
Url : http://lists.gnucash.org/pipermail/gnucash-devel/attachments/20070606/664a40c4/attachment.bin