[GNC] Strange prices update issue (possibly with TSP)
Kalpesh Patel
kalpesh.patel at usa.net
Mon Aug 14 16:54:55 EDT 2023
Some good find in debugging ... Strawberry Perl cannot resolve the FQDN name one out of three times for www.tsp.gov on my system (this might be obfuscating the client behavior in response to real backend issue). It might be that I am reaching servers that are in error state (rc = 500 which is Internal Server Error; my observation on F::Q could be cause of concern on amount of data being returned). Below is the output from the failed run:
C:\Users\kalpesh\OneDrive\QuickenStuff\HELPERS>set DEBUG=1
C:\Users\kalpesh\OneDrive\QuickenStuff\HELPERS>test2.pl
### [Mon Aug 14 16:26:56 2023] YIND_URL_HEAD : 'https://query1.finance.yahoo.com/v8/finance/chart/'
### [Mon Aug 14 16:26:56 2023] YIND_URL_TAIL : '?interval=1d&period1=1691440016&period2=1692044816'
### AlphaVantage->new args : {}
### COUNT_URL: 'http://www.panix.com/~hd-fxsts/finance-quote.html?tsp'
### Code: '200'
### [Mon Aug 14 16:26:56 2023] url : 'https://www.tsp.gov/data/fund-price-history.csv?startdate=2023-08-07&enddate=2023-08-14&Lfunds=1&InvFunds=1&download=1'
### [Mon Aug 14 16:26:56 2023] reply: bless( {
### _content => 'Can\'t connect to www.tsp.gov:443 (nodename nor servname provided, or not known)
nodename nor servname provided, or not known at C:/Strawberry/perl/site/lib/LWP/Protocol/http.pm line 50.
',
### _headers => bless( {
### '::std_case' => {
### 'client-date' => 'Client-Date',
### 'client-warning' => 'Client-Warning'
### },
### 'client-date' => 'Mon, 14 Aug 2023 20:26:56 GMT',
### 'client-warning' => 'Internal response',
### 'content-type' => 'text/plain'
### }, 'HTTP::Headers' ),
### _msg => 'Can\'t connect to www.tsp.gov:443 (nodename nor servname provided, or not known)',
### _rc => 500,
### _request => bless( {
### _content => '',
### _headers => bless( {
### 'user-agent' => 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/102.0.5005.61 Safari/537.36'
### }, 'HTTP::Headers' ),
### _method => 'GET',
### _uri => bless( do{\(my $o = 'https://www.tsp.gov/data/fund-price-history.csv?startdate=2023-08-07&enddate=2023-08-14&Lfunds=1&InvFunds=1&download=1')}, 'URI::https' )
### }, 'HTTP::Request' )
### }, 'HTTP::Response' )
C:\Users\kalpesh\OneDrive\QuickenStuff\HELPERS>
My test file (test2.pl) is as follows:
#test2.pl - start
use Finance::Quote;
use Data::Dumper;
$q = Finance::Quote->new;
%info = $q->fetch('tsp', "L2030");
print Dumper(%info);
# test2.pl - end
Interesting side notes for GNC (I am on Windows 11 with GNC v4.14 and now F::Q v1.58 so might have some subtle from it):
I was able to put the module in debug mode (set DEBUG=1 in cmd.exe; of course install the Smart::Comment Perl module first) and have them write out the smart comment messages to cmd console as you can see from above when I run a small test file that I've created. However, when I lunch gnc in normal fashion from the same console where debugging is enabled, the screen does not display anything, not even the smart comment; it is almost like it is ignoring all env settings or not being passed to sub-shell. Windows is notorious for this so there may not be a "clean" way of doing so. There is a new window that pops up and then disappears at the start up (this process seems to be part normal start up - check if F::Q is there or not) and does not display anything also. I attempted to start gnc with debug (with --log=gnc.scm=debug option) as well but that also does not emit anything either. On the note of specifying a file name for logging (--logto= parameter), it does not create nor write any file anywhere on the file system when only a fixed file name is given. I took away that option and it was happy to write a file to %LOCALAPPDATA% path with gnucash.trace.<random_six_alphanumeric_characters>.log.
Interesting side notes for F::Q TSP module:
When it does work properly, the return payload from HTTP seem to be hundreds of lines if not thousands ... Here is the snippet of successful run:
C:\Users\kalpesh\OneDrive\QuickenStuff\HELPERS>test2.pl
### [Mon Aug 14 15:42:00 2023] YIND_URL_HEAD : 'https://query1.finance.yahoo.com/v8/finance/chart/'
### [Mon Aug 14 15:42:00 2023] YIND_URL_TAIL : '?interval=1d&period1=1691437320&period2=1692042120'
### AlphaVantage->new args : {}
### COUNT_URL: 'http://www.panix.com/~hd-fxsts/finance-quote.html?tsp'
### Code: '200'
### [Mon Aug 14 15:42:01 2023] url : 'https://www.tsp.gov/data/fund-price-history.csv?startdate=2023-08-07&enddate=2023-08-14&Lfunds=1&InvFunds=1&download=1'
### [Mon Aug 14 15:42:01 2023] reply: bless( {
### _content => 'Date,L Income,L 2025,L 2030,L 2035,L 2040,L 2045,L 2050,L 2055,L 2060,L 2065,G Fund,F Fund,C Fund,S Fund,I Fund
2023-08-11,24.0849,12.2327,42.9435,12.8255,48.3009,13.1625,28.7245,14.0971,14.0954,14.0934,17.6469,18.3534,69.1565,70.2391,37.9419
2023-08-10,24.1082,12.2494,43.0380,12.8567,48.4288,13.1999,28.8108,14.1442,14.1425,14.1406,17.6449,18.4205,69.2207,70.3124,38.2391
2023-08-09,24.1016,12.2437,42.9976,12.8439,48.3758,13.1845,28.7742,14.1155,14.1139,14.1120,17.6429,18.5294,69.1951,70.4343,38.0083
2023-08-08,24.1272,12.2634,43.1181,12.8835,48.5391,13.2323,28.8859,14.1834,14.1818,14.1799,17.6409,18.5100,69.6795,71.0740,38.0127
2023-08-07,24.1541,12.2847,43.2515,12.9273,48.7198,13.2853,29.0100,14.2616,14.2599,14.2580,17.6389,18.4477,69.9736,71.7269,38.2463
2023-08-04,24.1053,12.2504,43.0581,12.8645,48.4638,13.2110,28.8379,14.1583,14.1566,14.1547,17.6328,18.4709,69.3457,71.4617,38.0032
2023-08-03,24.1034,12.2517,43.0826,12.8723,48.4974,13.2209,28.8626,14.1833,14.1815,14.1796,17.6308,18.3188,69.7102,71.6307,37.8756
2023-08-02,24.1277,12.2685,43.1728,12.9023,48.6200,13.2568,28.9449,14.2243,14.2226,14.2207,17.6289,18.4426,69.8850,71.8330,38.0070
2023-08-01,24.2277,12.3420,43.6036,13.0440,49.2020,13.4269,29.3407,14.4570,14.4553,14.4534,17.6269,18.4987,70.8648,73.1095,38.7558
2023-07-31,24.2660,12.3694,43.7577,13.0949,49.4108,13.4880,29.4819,14.5344,14.5328,14.5310,17.6249,18.6043,71.0523,73.4061,39.1413
2023-07-28,24.2467,12.3571,43.6964,13.0751,49.3307,13.4647,29.4283,14.5045,14.5029,14.5010,17.6191,18.5820,70.9459,72.6930,39.1484
2023-07-27,24.1833,12.3118,43.4387,12.9905,48.9842,13.3635,29.1936,14.3691,14.3675,14.3656,17.6172,18.5164,70.2457,71.6543,38.8820
2023-07-26,24.2258,12.3417,43.6042,13.0453,49.2086,13.4293,29.3454,14.4499,14.4484,14.4465,17.6152,18.6653,70.6948,72.5871,38.9458
2023-07-25,24.2163,12.3359,43.5775,13.0365,49.1733,13.4189,29.3220,14.4395,14.4379,14.4361,17.6133,18.6144,70.7047,72.2457,38.9313
2023-07-24,24.1974,12.3223,43.4989,13.0110,49.0687,13.3885,29.2512,14.3963,14.3947,14.3929,17.6114,18.6334,70.5044,72.1292,38.7806
2023-07-21,24.1841,12.3139,43.4569,12.9977,49.0157,13.3735,29.2167,14.3747,14.3732,14.3714,17.6055,18.6571,70.2201,72.1244,38.8425
2023-07-20,24.1808,12.3123,43.4515,12.9961,49.0096,13.3719,29.2133,14.3737,14.3722,14.3704,17.6036,18.6455,70.1974,72.2722,38.8218
2023-07-19,24.2266,12.3452,43.6382,13.0578,49.2628,13.4460,29.3849,14.4692,14.4677,14.4659,17.6017,18.7521,70.6685,73.1168,39.0026
2023-07-18,24.2119,12.3358,43.5911,13.0423,49.2001,13.4277,29.3433,14.4485,14.4469,14.4451,17.5998,18.6938,70.5015,72.8049,39.0442
2023-07-17,24.1639,12.3012,43.3926,12.9773,48.9337,13.3500,29.1627,14.3422,14.3407,14.3389,17.5978,18.6769,70.0036,72.0267,38.7854
2023-07-14,24.1430,12.2877,43.3244,12.9552,48.8445,13.3242,29.1032,14.3088,14.3072,14.3055,17.5920,18.6543,69.7345,71.3406,38.8917
2023-07-13,24.1672,12.3051,43.4224,12.9878,48.9782,13.3634,29.1941,14.3581,14.3566,14.3548,17.5901,18.7318,69.8035,72.0356,39.0693
2023-07-12,24.0841,12.2462,43.0912,12.8792,48.5343,13.2341,28.8948,14.1877,14.1862,14.1844,17.5881,18.6216,69.2107,71.2102,38.4014
2023-07-11,24.0007,12.1877,42.7678,12.7731,48.1017,13.1082,28.6041,14.0255,14.0239,14.0220,17.5862,18.4682,68.7006,70.6988,37.6648
2023-07-10,23.9468,12.1490,42.5478,12.7011,47.8070,13.0223,28.4048,13.9094,13.9078,13.9059,17.5843,18.4385,68.2399,69.6977,37.3488
2023-07-07,23.9111,12.1253,42.4239,12.6607,47.6429,12.9745,28.2947,13.8488,13.8471,13.8453,17.5785,18.3742,68.0764,68.6900,37.2301
2023-07-06,23.8994,12.1170,42.3776,12.6455,47.5804,12.9561,28.2516,13.8230,13.8214,13.8195,17.5765,18.3788,68.2614,68.0180,37.0282
[snip]
Three observation on F::Q:
1) This request downloads CSVs that goes back to 2003-05-31 (20+ years!) even though request is only for 7 days in the URL. It is like enddate= parameter is ignored or incorrect parameter passed. Possibly a bug in TSP.pm module?
2) Is the anonymous data collection of modules utilized still being collected? Code execution is not harming anything if not being collected but this is more curiosity on my part for v1.58 which was released few days back.
3) Looks like Smart::Comment when activated emits messages for all modules rather than the single one that is being instantiated for the source and called to action.
On to debugging the Perl LWP module...
-----Original Message-----
From: Bruce Schuck <bschuck at asgard-systems.com>
Sent: Friday, August 11, 2023 9:40 PM
To: gnucash-user at gnucash.org
Cc: Kalpesh Patel <kalpesh.patel at usa.net>; john <jralls at ceridwen.us>
Subject: Re: [GNC] Strange prices update issue (possibly with TSP)
On Aug 10, 2023, at 12:13, Kalpesh Patel <kalpesh.patel at usa.net <mailto:kalpesh.patel at usa.net> > wrote:
> The difficult part in debugging this is that when I do query three
> times from the command line, it succeeds without any issue for me as
> well, but when I update prices from GNC, it fails two times and then
> succeeds third time consistently. So this is not an issue with
> throttling from TSP source which you validated as well. Is there more
> in terms of debugging this by any chance? Can I use STDERR and STDOUT
> from Perl to emanate messages to console but not interfere with what
> JSON comes back or butcher the response back to gnc-fq-helper so it
> doesn't bomb to run a full cycles from GNC to test? Any help in debugging is appreciated.
Kalpesh,
Many, but not all, of the F::Q modules make use of the Smart::Comments module. TSP.pm is one of them. The environment variable DEBUG will need to be set, and with no changes to the Perl code you should see helpful information printed to STDERR. In the code, these are lines that begin with "###" (note, just 3 pound signs) in those modules using Smart::Comments.
A quick look at the TSP module and I see two where the "tsp" method will return *without* setting "success" and "errormsg" for any of the symbols passed to it. This will happen if the http get call fails, or whatever is downloaded is not 2 or more lines.
my $reply = $ua->get($url, @HEADERS);
### [<now>] url : $url
### [<now>] reply: $reply
return unless ($reply->is_success);
my @line = split(/\n/, $reply->content);
return unless (@line > 1);
In the code snippet, there are two examples Smart::Comments use which could be useful in your case. With DEBUG set the code would print the url to STDERR, and entire $reply object (status, headers, cookies, body).
Whether or not this helps you, the omission of not returning the proper failure data is an issue.
Hope this helps.
Bruce S
More information about the gnucash-user
mailing list