[GNC-dev] Robust error handling and respecting the Apha Vantage API limits in gnc-fq-helper.

[Edward d'Auvergne]

On Fri, 7 Feb 2020 at 05:50, Mike Alexander <mike at msalexander.com> wrote:
>
> On 4 Feb 2020, at 6:21, Edward d'Auvergne wrote:
>
> That's a good point. In the future, F::Q might also be able to
> respect the API fetching limits across multiple calls, and then this
> would add an additional delay. That could be fixed by properly timing
> this while loop. As for sleeping for different times in the loop for
> those with an AlphaVantage subscription, how could that be done? Do
> you know how F::Q knows about this? I can see that you can currently
> buy 30, 120, 300, 600, or 1200 API requests per minute. Should there
> be a GnuCash user setting for this, that is then used by the while
> loop in this perl script?
>
> I don't think this should be a GNuCash configuration option. Right now AlphaVantage tells the caller if too many requests have been made in too short a time, but it doesn't return information about how many are too many. F::Q recognizes this error return and delays the next call. It is pretty stupid about how long to delay. If AlphaVantage returned more information about how fast is too fast it could be smarter about it. Unless this is changed in AlphaVantage I don't think thee is much that could usefully be done about this.

That sounds like it would work, however I see quite a different
behaviour with the debugging (in my first post) and from the code
(gnucash/price-quotes.scm and libgnucash/quotes/gnc-fq-helper.in).
What I see in both is that for each iteration of the while loop, F::Q
is called once per currency.  If the fetch fails, due to the
Alphavantage API limits, the first result is #<eof>.  This seems to
cause the guile 'fdes' file descriptor to close.  Hence all subsequent
currencies are looped through without calling F::Q.

There is no sleeping in this.  F::Q 1.49 does not recognise serial
calls.  F::Q's delay is only invoked when a list of currencies is feed
into it in a single call.  But GnuCash clearly does not do this.

In any case whether or not the pipe is closed, the loop runs at full
Perl execution speed, limited solely by the IO.  F::Q currently does
not insert a sleep period into this loop.  The result is that only the
first few currencies that are within the Alphavantage API limit will
be fetched, the rest will be skipped.  From the debugging output, the
currency skipping is very fast.  This is exactly the behaviour I have
always seen in GnuCash since the Yahoo API shutdown.

Regards,

Edward