Gnucash UI dogtail test harness some ideas

[ahmad sayed]

Hi josh,

>> I'm not sure that this XML file is useful.
The idea is not in making the testplan in xml only,  xml is only a suggested approach, the main idea is to separate the testplan from the code 

>> In the complex case of some XML file that was describing the tests more fully
>> (or even "formally"), that file could contain a whole mini-language for doing
>> test Setup ... 

I always prefer to use XML at its descriptive level, xml o describe some thing, which will be what this test case are going to do in formal describtion, and which method in the python code do it, maximum complextity i intend to add is to add keywords, to control which test cases to run like run all testcases that marked as smoke test, all run the whole regression, i didn't have a wide experience but i see this practice in the GUI test suites, the other complexity i expect to add is to make this XML, is putting the test case data in this xml, which will be we have more than one test case in the
testplan file but all of these test cases call the same python method with different data.

>> Even in the simple case, the XML file is just a re-iteration of all the test
>> functions (or methods, whatever they are) that already need to be specified
>> in the code.

I really liked this, but this make writing the test plan part of the code which is good with unit testing.

>> I've used testing tools in both the "programatic" and "declared" approach,
>> and I've found the former to be much more desirable.

Josh, honestly i didn't have your wide experience to discuss  you  in design issues, but i speak according to my small experience in the similar case, also according to my reading regarding similar situation, i read an articles  about testplan data should be separated from the test case code, and tools the provide this separation considered better.

>> I hope I can convince you to put
>> that part of the project off until later on, if/when more of the core and
>>> required parts of the test framework are proven out.

Ok, no problem but actually writing the test plan is the real and very important part i have this fancy dream of having a solid  test plan editor in which i begin writing the test plan for gnucash, ok no problem.
you could convince me easily Josh :)

The validator:

>> How much of this validator would be custom work vs. being provided by
>> dogtail?

It will be a custom work, yes dogtail dump the node data, in both plain and xml format but its dump method is very obvious and limited so it will be a custom work.

>> - filtering:  If you dump a whole dialog, but are only interested in
>> checking the validity of a few fields, you need to filter the others so
>>   that `diff` doesn't report extra differences.
That's why it will be a custom work, also there is other issues, like packing widget which in most cases will be filtered

>> canonicalizing: If you change the visual layout of the dialog, that
>>  (probably) shouldn't invalidate the test.  Something needs to bring the
>>  dumped/asserted state (and perhaps the "expected" state as well) into a
>>  "canonical" form for a good comparison.

really this a good point, first which is the benefit of keeping the validation reference in separate file to make an update easily and without code change just replacing the file, also i there is no way to avoid the need to update the test cases but you could try to minimize that need, because changing the GUI may lead to make the testcase logic itself not correct, for example renaming the button, dogtail till now i didn't find any way other than name, label which is name to uniquly identify the node (control, widget), i'm with you is diff my give false negative in case of changing the layout only, and we need to add more intellegent  to validate, but if I could succesfully separate the validator it will be easy to replace diff later one, another  issue diff will be very useful in validation widget like trees and tables.

In Conclusion:

I'm going to delay the testplan, for later on.
you are ok, with the validator but I will consider the issues you mentioned.


Best regards,
Ahmed Sayed




 
____________________________________________________________________________________
Food fight? Enjoy some healthy debate 
in the Yahoo! Answers Food & Drink Q&A.
http://answers.yahoo.com/dir/?link=list&sid=396545367