[GNC-dev] Problems with python 3 and the gnucash python bindings.

[John Ralls]

> On Jul 10, 2018, at 11:58 AM, David Osguthorpe <david.osguthorpe at gmail.com> wrote:
> 
> Hi All,
> 
> 
> In upgrading to gnucash 3.2 from 2.6.18 and updating my python scripts I
> have found an issue with the gnucash bindings and python 3.
> 
> 
> I saw this with query runs that failed to produce any results when they
> should have, and used to under 2.6.18.
> 
> 
> So far it appears the issue is with the wrapping of the
> qof_query_search_for call.
> 
> By inspection of the generated c code it appears by default swig wraps
> python 3 unicode strings by converting to a new python 3 byte string object
> (SWIG_AsCharPtrAndSize) and passes its char buffer pointer as the char *
> pointer to the wrapped function call.
> 
> This object and pointer is freed at the end of the wrapper function.
> 
> Unfortunately it seems qof_query_search_for just stores the char * pointer
> for the query which means by the time the query is run the pointer is
> invalid (and indeed junk was printed using gnucash debug logging as the
> Query Object Type: which is what lead me to this problem).
> 
> 
> This is going to be a general problem for any gnucash function that simply
> stores a char * pointer rather than copying the string.
> 
> 
> According to the Python 3 documentation the PyUnicode_AsUTF8 function
> creates a char * buffer which is cached in the unicode object so exists for
> the lifetime of the unicode object.
> 
> I dont understand why this isnt the default SWIG wrapping for char pointers
> but it does seem to solve the problem here.
> 
> (I havent been able to find any real discussion of this via Google. The
> only hint Ive seen is that its a memory issue as PyUnicode_AsUTF8 needs to
> store 2 versions of the string.)
> 
> 
> This requires setting up swig typemaps.
> 
> So far I have used a very specific typemap for QofIdType and QofIdTypeConst
> which solves the problem and python query runs produce results (and my
> scripts work as under 2.6.18).
> 
> 
> The question is where and how to implement this.
> 
> Currently Ive put the typemaps in gnucash_core.i - but they probably
> should be in base-typemaps.i.
> 
> Then the question is how general to make this - it could be a typemap for
> all char * pointers otherwise each explicit character type (eg gchar *)
> would need its own typemap.
> 
> I think its clear that any const char * should map to PyUnicode_AsUTF8.
> 
> 
> Also note that the GSList typemap in base-typemaps.i needs to be updated to
> use unicode strings for python 3 - otherwise it requires python 3 byte
> strings.
> 
> 
> David
> 
> 
> PS. Another solution would be to force byte string only arguments for
> python 3 using a SWIG define SWIG_PYTHON_STRICT_BYTE_CHAR.
> 
> This would require a major re-write of the gnucash_core.py to perform the
> unicode<->byte transformations.

ISTM it would be better to fix qof_query. It shouldn't be assuming that just because it has typedeffed const char* to "QofIdTypeConst" that it will necessarily get a statically allocated char* that is safe to keep. There's absolutely nothing preventing it from being a stack or heap object (stack is more likely in C, as in 
  QofIdTypeConst type = "book"; 
  qof_query_search_for(query, type);
) that will cause the same crash.

There are probably other cases where someone has made the same bad assumption and stored an unduplicated char*, though I hope none quite so egregious.

Please file a bug: https://wiki.gnucash.org/wiki/Bugzilla

Regards,
John Ralls