[GNC-dev] Robust error handling and respecting the Apha Vantage API limits in gnc-fq-helper.

[Edward d'Auvergne]

On Sun, 19 Jan 2020 at 01:51, Mike Alexander <mike at msalexander.com> wrote:
>
> I'm glad you're looking at this since it does need some work. However I think there are a couple of things you're not aware of. I, too, have quite a ffew currencies in my file (around 30) and by coincidence I was running a price fetch in the background when I first saw your message. It completed successfully after fetching all currency quotes as usual.

It does not surprise me that there are multiple ways to fix F::Q.


> You're right that GnuCash fetches one currency per call, but this is the only API for currencies that F::Q provides. It fetches all the other quotes for a given source in one call to F::Q. The F::Q currency quote code has a test for the rate limiting message that AlphaVantage returns when currency quote fetches come too fast.

With my debugging code, this internal F::Q rate limit is visibly not
being triggered.  I would assume that is because the current GnuCach
code is calling F::Q once per currency pair, rather than the usual one
F::Q call with a list of currency pairs.


> However at some time in the past AlphaVantage changed this message. I updated F::Q to look for the new message and sent a pull request upstream. This pul request has never been applied which means that the current F::Q does not recognize when AlphaVantage is trying to tell it to slow down. My copy of F::Q has this change which is probably why it works for me and not for you.

As I said, the F::Q developers do not seem to be responsive.  So I
chose to fix things on the GnuCash side instead.


> The patch is simple, just change it to look for a "Note" field in the returned data instead of an "Information" field.
>
> Try this and see if it helps.

I have my own F::Q fixes and patches, and it looks like your changes
would work quite well.  But again I am tackling this from the GnuCash
side.


> Your change to make GnuCash handle F::Q failures better seems like a good idea.

It's better than GnuCash crashing ;)  That change should really be
considered for 'master'.


> Putting the delay on GnuCash's side of the API might also be a good idea, but really shouldn't be necessary. It also would be a problem for someone who has actually paid for an AlphaVantage subscription and hence can get currency quotes faster. Some day I hope that F::Q will find a currency quote source that works better and faster at which time you wouldn't want the delay in GnuCash.

That's a good point.  In the future, F::Q might also be able to
respect the API fetching limits across multiple calls, and then this
would add an additional delay.  That could be fixed by properly timing
this while loop.  As for sleeping for different times in the loop for
those with an AlphaVantage subscription, how could that be done?  Do
you know how F::Q knows about this?  I can see that you can currently
buy 30, 120, 300, 600, or 1200 API requests per minute.  Should there
be a GnuCash user setting for this, that is then used by the while
loop in this perl script?

Regards,

Edward