Text encoding issue when importing Japanese OFX files

[Christian Stimming]

Dear Erik,

thanks for the nice explanation of your OFX import with utf-8 characters. Some 
questions:

-  Which menu item did you use for the OFX import in gnucash - "File -> 
Import -> Import OFX", or any of the Online Banking items?
- Which libofx version is installed on your system?
- Can you provide a (small) example file with at least one line of Japanese 
characters which is currently not imported correctly?

Regards,

Christian

Am Dienstag, 27. Oktober 2009 07:09 schrieb z33b0b at gmail.com:
> Hello dev folks --
>
> Running GnuCash 2.3.7 on Ubuntu 9.04.
>
> I might be able to serve as a fun test case, as I have accounts in Japan. 
> I can download data in OFX format with the following header:
>
>
> OFXHEADER:100
> DATA:OFXSGML
> VERSION:102
> SECURITY:NONE
> ENCODING:UTF-8
> CHARSET:CSUNICODE
> COMPRESSION:NONE
> OLDFILEUID:NONE
> NEWFILEUID:NONE
>
>
> The transaction list includes Japanese text.  I assume the encoding is as
> marked, as GVim and other text editors (and even CLI less) seem to display
> it just fine.  Attempting to import into GnuCash, however, renders all
> Japanese text into ASCII gibberish, a phenomenon well-known among
> Japanese-savvy circles as "moji-bake" (that's pronounced "bah keh", not
> "bake").  I've attached a screenshot in .png format for reference.
>
> I cannot tell if the issue is simply a font problem, or if GnuCash isn't
> recognizing the UTF-8 text.  For that matter, I'm not even sure where to
> specify fonts.  I tried altering the fonts specified in the Default CSS
> setting, but to no apparent avail.  For what it's worth, most font display
> problems will show unrecognized glyphs as empty squares rather than random
> ASCII, which makes me think that GnuCash isn't respecting the UTF-8
> encoding marked in the header, and is instead trying to read the file as
> ASCII.  My system locale is set to use UTF-8 by default:
>
>
> eirikr at Boreas:~$ echo $LANG
> en_US.UTF-8
> eirikr at Boreas:~$
>
>
> Meanwhile, I can enter Japanese text directly into GnuCash just fine, but
> I'd really rather not have to manually enter all my Japanese-language
> transactions when I can just download them instead.  Any ideas what's going
> wrong with the import, and better yet, how to fix it?
>
> Cheers,
>
> -- Erik Anderson