[GNC] GnuCash 5.5 (macOS) can't get quotes, because it doesn't see JSON::Parse

[john]

> On Dec 24, 2023, at 11:30, Jim DeLaHunt <list+gnucash at jdlh.com> wrote:
> 
> 
> On 2023-12-24 10:25, john wrote:
>> 
>>> On Dec 23, 2023, at 20:03, Jim DeLaHunt <list+gnucash at jdlh.com> wrote:
>>> 
>>> ...Then I launched the GnuCash application. Now, the "Get Quotes" button in the Price Database dialogue was active, and I could retrieve quotes successfully.
>>> 
>>> Yay!
>>> 
>>> Conclusions I tentatively draw from this experience:
>>> 
>>> 1. GnuCash users on macOS with Apple Silicon must be careful to run
>>>   gnc-fq-update with the arch -arch x86_64 qualifier.
>>> 2. gnc-fq-update fails to report at least some problems with Perl
>>>   modules compiled for Apple Silicon, where the GnuCash application
>>>   rejects the configuration.
>>> 3. When the GnuCash application rejects the configuration due to Perl
>>>   modules compiled for Apple Silicon, neither the diagnostics printed
>>>   by gnucash-cli nor the telemetry in the --debug log written by the
>>>   GnuCash application mentions that Perl modules compiled for the
>>>   wrong architecture are the problem.
>>> 4. It would be nice if the GnuCash + Finance::Quote system did not
>>>   suffer from this fragility. But I understand that the root cause of
>>>   the fragility is that WebKitGtk doesn't work when built for Apple
>>>   Silicon, and that is not trivial to overcome.
>>> 5. The Online Quotes wiki page should get some information about the
>>>   need to get Perl modules compiled for x86 architecture, but I don't
>>>   think I know enough to write that information correctly.
>>> 
>>> Does that sound right?
>>> 
>>> Again, thank you for your help, John.
>> Jim,
>> 
>> ...the parts that were actually causing the problem are /Library/Perl/5.30/darwin-thread-multi-2level/auto/DateTime/DateTime.bundle and JSON/Parse.bundle, the compiled C resources that the perl parts depend on...
> Thank you for this information, John. It is way more than I know about how Perl modules work on multi-architecture aware macOS.
>> ...GnuCash's perl scripts, finance-quote-wrapper and gnc-fq-update, are of necessity quite generic: They have to run on both Unix and Microsoft operating systems. Moreover perl itself doesn't report many errors in machine-interpretable form. It would be difficult to get the information that modules installed for the wrong architecture into a tracefile....
> 
> I see the problem. Yet I suspect that the Finance::Quote infrastructure is getting complex and error-prone enough that it is worth more attention.  I also suspect that this problem seems big to me because I just experienced it. It may now work for me, so that I may be able to ignore it for the next five years — and then the Finance::Quote infrastructure will seem elegant and reliable to me. :-)
> 
> One of my confusions was that "gnc-fq-update" said that the necessary modules were installed, while "gnucash-cli --quotes info" reported that one of the options was missing. How about having gnc-fq-update and gnucash-cli --quotes info both check not just what modules Perl say it has installed, but attempting to initialise Finance::Quote and retrieve a quote as well?  I suspect that in my case, the attempt to initialise Finance::Quote would have failed.  Then those diagnostics perhaps would have given me better information.

gnc-fq-update is a perl script with a shebang, and that shebang says /usr/bin/perl. The same is true of finance-quote-wrapper. If you run
   file /usr/bin/perl
it will return
   /usr/bin/perl: Mach-O universal binary with 2 architectures: [x86_64:Mach-O 64-bit executable x86_64] [arm64e:Mach-O 64-bit executable arm64e]
   /usr/bin/perl (for architecture x86_64):	Mach-O 64-bit executable x86_64
   /usr/bin/perl (for architecture arm64e):	Mach-O 64-bit executable arm64e

Which of those executables gets run depends on the environment from which it's run. The same is true of Terminal and your shell (bash or zsh). So if you start Terminal by double-clicking from Finder it uses the arm64e executable; that in turn starts a shell that also uses the arm64e executable. GnuCash is an x86_64 binary, so running it--even from an arm64e shell--fires up Rosetta2 and starts it in x86_64 translation. Modules' bundles are loaded using the standard C function dlopen(), and it can load only bundles that are the same architecture as the the program that calls it.

All gnc-fq-update does is run cpan to install Test2 (which is pure perl and doesn't cause trouble), Finance::Quote (which pulls in its dependencies including DateTime) and JSON::Parse. If it followed that by trying to use Finance::Quote it would generally succeed because it would be using the same architecture that it just installed for.

As you observed in your own troubleshooting, cpan determines that a module is installed based only on the pure-perl module.pm (e.g. Parse.pm for JSON::Parse and Quote.pm for Finance::Quote). It doesn't query the corresponding bundle's architecture (if there is one; most modules are pure perl). The architecture error shows up only when perl tries to dlopen() the bundles and doesn't find a version matching the running executable. As you also observed there's no uninstall facility in cpan. There is a force-install (-f -i) facility that will reinstall an already-installed module, but it also turns off tests so it's not something you'd want to use routinely.

> 
>> As for the wiki's Online Quotes page, are the year-old directions to prefix gnc-fq-update with arch -arch x86_64 on Apple Silicon macs not clear enough? What more should it say? Perhaps something about diagnosing a failed installation and the steps needed to repair it?
> I did not see those instructions, because they were about 60% of the way through over 100kB of text, under a heading of "Installing Finance::Quote on macOS" <https://wiki.gnucash.org/wiki/Online_Quotes#Installing_Finance::Quote_on_macOS>,and my framing of the problem was that I was troubleshooting an installed and functional Finance::Quote deployment which stopped working when I upgraded the GnuCash application.

It seems unlikely that F::Q stopped working when you upgraded GnuCash. F::Q didn't work because it was installed without x86_64 versions of DateTime.bundle and Parse.bundle. GnuCash doesn't do that, nor does upgrading GnuCash change anything about the F::Q installation.

> 
> So, certainly making a reference to macOS architectures in the troubleshooting section would help. Refactoring that page into focussed parts much shorter than 100 kB would help also.

The whole troubleshooting section needs to be rewritten, it's for the pre-5.x implementation. No doubt that impeded your attempt to diagnose the problem. I'll do that, but not today.

> 
> Thank you again for your helpful answers, your expertise, and your careful building of all this mostly-elegant and mostly-reliable infrastructure.

You're welcome, though the credit for Finance::Quote integration belongs mostly to others. The original Scheme implementation was written long before I joined the project and Geert designed and mostly wrote the new C++ one; I just finished it up when he got distracted by real life.

Regards,
John Ralls