locale issues with data format when upgrading 1.8 -> 2.0

[Derek Atkins]

Chris Shoemaker <c.shoemaker at cox.net> writes:

> Now, going _forward_, we can still choose either "forward-friendly",
> "SaveAs...", or even both.  This decision is quite independent of what
> to do about 1.8->2.0 and it probably needs to be made in the context
> of a rethink about our data-file versioning method anyway.

Can we get back to the matter at hand, which is how to deal with the
xml encoding issues in the upgrade from 1.8 to 2.0?  Let's relegate
the larger issue of data format compatibility across versions to
another thread, but unfortunately you've already hijacked this one.

I think it's a major issue that someone in an ascii-like but
non-latin1 locale will get garbage during the default upgrade path.
libxml doesn't really provide a way to do proper detection, and 1.8
doesn't include an encoding in the data file..  Unfortunately the XML
spec says that the lack of an encoding parameter means the data is in
utf-8, but that's not the case in 1.8 -- the data is in whatever
locale the user was using.

So, how do we solve this?

let's ignore the backwards compatibility issues for the moment --
let's worry only about forward compatibility...

-derek

-- 
       Derek Atkins, SB '93 MIT EE, SM '95 MIT Media Laboratory
       Member, MIT Student Information Processing Board  (SIPB)
       URL: http://web.mit.edu/warlord/    PP-ASEL-IA     N1NWH
       warlord at MIT.EDU                        PGP key available