[GNC-dev] Robust error handling and respecting the Apha Vantage API limits in gnc-fq-helper.

[Edward d'Auvergne]

On Wed, 12 Feb 2020 at 07:44, Mike Alexander <mta at umich.edu> wrote:
> On 11 Feb 2020, at 3:59, Edward d'Auvergne wrote:
> > F::Q uses this URL for all queries:
> >
> > https://www.alphavantage.co/query?function=GLOBAL_QUOTE&datatype=json&symbol=IDRUSD&apikey=$MY_KEY
> >
> > The result I see is:
> >
> > {
> >     "Global Quote": {
> >         "01. symbol": "IDRUSD",
> >         "02. open": "0.0001",
> >         "03. high": "0.0001",
> >         "04. low": "0.0001",
> >         "05. price": "0.0001",
> >         "06. volume": "0",
> >         "07. latest trading day": "2020-02-10",
> >         "08. previous close": "0.0001",
> >         "09. change": "0.0000",
> >         "10. change percent": "0.0000%"
> >     }
> > }
>
> It doesn't use that URL for currency exchange rates.  It uses
>
> https://www.alphavantage.co/query?function=CURRENCY_EXCHANGE_RATE&from_currency=IDR&to_currency=USD&apikey=xxx
>
> which returns
>
> {
>      "Realtime Currency Exchange Rate": {
>          "1. From_Currency Code": "IDR",
>          "2. From_Currency Name": "Indonesian Rupiah",
>          "3. To_Currency Code": "USD",
>          "4. To_Currency Name": "United States Dollar",
>          "5. Exchange Rate": "0.00007297",
>          "6. Last Refreshed": "2020-02-12 06:30:18",
>          "7. Time Zone": "UTC",
>          "8. Bid Price": "-",
>          "9. Ask Price": "-"
>      }
> }

I see.  The duplication of code in the 'currency' subroutine in
lib/Finance/Quote.pm and the 'alphavantage' sobroutine in
lib/Finance/Quote/AlphaVantage.pm is confusing.  The first uses
CURRENCY_EXCHANGE_RATE and the second uses GLOBAL_QUOTE.  I wonder why
the 'alphavantage' subroutine was not modified, or split to add a new
'alphavantage_currency' subroutine, and called from 'currency'?  The
'sleep_before_query' function which uses the variable:

"""
my $maxQueries = { quantity =>5 , seconds => 60}; # no more than x
                                                  # queries per y
                                                  # seconds, based on
                                                  #
https://www.alphavantage.co/support/#support
"""

seems to be more robust than how 'currency' handles things (this seems
to be the major algorithmic difference between the two).  This could
even shift into searching for an ALPHAVANTAGE_QUERIES_PER_MIN
environmental variable to allow those with an Alpha Vantage
subscription to fetch quotes faster.  In the 'alphavantage' function,
the sleep if the query returns a 'Note' logic seems to be a final
fallback mechanism if the 'sleep_before_query' function fails, whereas
in 'currency' this is the main mechanism for dealing with the API
fetching limits.

It looks like some but not all parts of the 'alphavantage' function
were copied into 'currency'.  But if multiple currency sources are
made available, it would make more sense to make the 'currency'
subrountine provider independent and defer the fetching to subroutines
in the lib/Finance/Quote/*.pm modules.


> > It looks like this code then kicks in, from the "currency" subroutine
> > in lib/Finance/Quote.pm:
> >
> > """
> > if ( $exchange_rate < 0.001 ) {
> >     # exchange_rate is too little. we'll get more accuracy by using
> >     # the inverse rate and inverse it
> >     my $inverse_rate = $this->currency( $to, $from );
> >     {
> >         local $^W = 0;
> >         return undef unless ( $exchange_rate + 0 );
> >     }
> >     $exchange_rate = int( 100000000 / $inverse_rate + .5 ) /
> > 100000000;
> > }
> > """
> >
>
> That code should be removed.  It actually gets the same accuracy and it
> gets a different answer.  The reverse quote is 13683.5.  This inverts to
> a rate of 0.00007308 which is not the same as 0.00007297.  That's a
> trivial difference, but F::Q should really return the quote it was asked
> for if possible.

Maybe it would be needed if other quote providers are added in the
future?  Maybe it should be converted into a function that checks the
number of significant figures and asks for the inverse if <= 2?


> I noticed that there was some action on my pull request after you
> rattled their cage.  Thanks for doing that.  That may make this
> discussion largely irrelevant.

No problems.  Hopefully this will be fixed and Finance:Quote v1.50
released soon.  Then GnuCash will finally be able to fetch quotes
correctly after 2 long years!

Regards,

Edward