[GNC] OFX Import with Investment Transactions

Sun Oct 3 00:30:46 EDT 2021

Thanks Robin!
Good detective work here, and thanks for the examples, I think there 
must be a bug :( . I'll take a look.

> If -- after importing the OFX file -- you open one of the account
> registers, you'll see duplicate transactions! Although, if you close and
> re-open the file, those duplicated transactions do not appear when you
> re-open the account register. I don't know what to make of that.

That is super weird :(
Jean

On 10/2/2021 5:19 PM, Robin Chattopadhyay wrote:
> Hi Jean-
>
> I understand the logic for #1 and have no concerns about that.
>
> As for #2, I did continue past the first matching dialog and there were no
> additional dialogs, the focus returned to the account list.
>
> There are definitely FITIDs that are already in the database that are also
> in the most recent OFX file. This is because the provider doesn't allow you
> to specify a date range, only 'current quarter' or 'some other quarter'.
>
> I tried something else to research...
>
> Using 4.8, I created a blank file and then imported an OFX file from Sep.
> 25, 2021. This file had 66 transactions and imported as expected with one
> matcher screen per security/account as you indicated would happen. I then
> attempted to import the OFX file from Oct. 2. This file has 84 transactions
> -- the same 66 transactions from the Sep. 25 file plus 18 new transactions
> dated 9/24.
>
> The program then presents the dialog for the first account. There are two
> transactions in the most recent file for this account; however they're
> already in the database, thus triggering the message "OFX file
> /home/robin/Downloads.qfx' imported transactions for account 'xxx' 2
> transactions processed, no transactions to match". That makes sense as
> those are older transactions that are already in the database. I clicked
> 'Close' for that dialog and then the focus returns to the main application.
> It's almost as if the program -- having determined that there were no new
> transactions to import into the first account -- decided not to look at the
> subsequent accounts.
>
> In service to the greater good, I am attaching the test file I created and
> the Oct. 2 OFX file. I have removed all of the position data (there are
> limits to what I'm willing to share :-) ), but that will not negatively
> affect the import process.
>
> Something else that I just noticed...
>
> If -- after importing the OFX file -- you open one of the account
> registers, you'll see duplicate transactions! Although, if you close and
> re-open the file, those duplicated transactions do not appear when you
> re-open the account register. I don't know what to make of that.
>
> On Sat, Oct 2, 2021 at 1:52 PM Jean L <ripngo at gmail.com> wrote:
>> Yes, there's a new behavior with OFX import. The match dialog shows up
>> for each target account, so if you're importing securities the dialog
>> will show up for each security. This could be a real annoyance if there
>> are many different securities, so we may want to change that behavior.
>> The new behavior was introduced to fix an issue when a single OFX file
>> had transfers between accounts.
>>
>> I see two issues here:
>> 1) The fact that the match dialog runs for each target account (see
>> comment above)
>> 2) The fact that some of your imported transactions were simply ignored.
>>
>> About 2): Did you try to continue with the import after the first
>> matching dialog? Did it then go to the next securities? Or did you just
>> abort thinking something was wrong (not expecting the new behavior of
>> seeing the matching dialog for each security instead of all together as
>> before)
>>
>> If that was not the case (i.e., you tried continuing but nothing else
>> happened) one reason why this could happen is if the FITID used for the
>> imported transactions were already present in the target account. This
>> is new in the latest version: transactions that were previously imported
>> are no longer shown the matching dialog. In your case this seems
>> erroneous. So either it's a bug (i.e., the FITID of the imported
>> transactions are NOT found in any of the transactions of the account
>> you're imported into, so they should show) or it's a problem with the
>> OFX data that the transactions seem to reuse existing FITIDs.
>> The fact that things work normally when you import into a blank account
>> would indicate that indeed, the FITID of the imported transactions
>> already exist in your target account.
>>
>> SO, could you re-try importing with the latest version, but making sure
>> you continue clicking OK for each match dialog? Does this work or not?
>> If not, then we'll need to check whether the FITID of the new imported
>> transactions somehow have been used before in the previously imported
>> transactions. To see that, you'd need to pick one that was not imported
>> (looking at the OFX file), get its FITID, and see in your account
>> database (save it as an xml file so it can be looked at with a regular
>> editor) whether that FITID exists.
>>
>> Jean
>>
>> On 10/2/2021 8:58 AM, Robin Chattopadhyay wrote:
>>> Ubuntu 20.04 LTS (VMWare virtual machine in case that matters)
>>> Gnucash 4.8
>>> libofx 0.9.15
>>>
>>> I tried importing an OFX file from my 401K provider this morning and I
> got
>>> a message box that said it had imported 2 transactions for a single
>>> security in the file and there were no additional transactions to
> process.
>>> This was incorrect as there were 18 new transactions across nine
> securities
>>> (along with a number of other transactions that had already been
> imported
>>> previously).
>>>
>>> I tried a number of things to resolve (after taking a backup, of
> course):
>>> 1 - Tools > Import Map Editor and deleted all of the associations for
> this
>>> file. When I re-imported, I was prompted to map each security in the
> file
>>> to the correct account. No issues there. But when importing the OFX file
>>> again, I had the same experience as described above
>>> 2 - Created a blank file and attempted to import the file there. Through
>>> the import process, I created new securities and new accounts, nothing
>>> unexpected there. Then the generic transaction importer dialog popped
> up,
>>> but with just the two transactions for the same security cited in the
>>> original problem dialog. I imported those, clicked OK and then the
> generic
>>> transaction importer dialog came up *again* but only with transactions
> for
>>> a single, different security. I imported those and repeated the process
> for
>>> each security with transactions in the file. Each time the generic
>>> transaction importer dialog came up, it only had transactions for a
> single
>>> security. This is definitely new behavior that I didn't see in 4.6.
>>> Previously the import dialog had all the new transactions in the file
>>> 3 - Reverted to 4.7. Crashed when selecting Import from the File menu
>>> (Trace/breakpoint trap (core dumped). Not unexpected, but thought I
> would
>>> try anyway.
>>> 4 - Reverted to 4.6. This works as it used to with all the new
> transactions
>>> in a single dialog
>>>
>>> Finally, I don't know if this matters, but I scanned stdout from the
>>> build/make/install process to see if anything looked obviously wrong
> and I
>>> found this (I don't know if it's relevant):
>>>
>>> -- Performing Test HAVE_OFX_BUG_39
>>> -- Performing Test HAVE_OFX_BUG_39 - Failed
>>>
>>> Thanks,
>>> Robin
>>> _______________________________________________
>>> gnucash-user mailing list
>>> gnucash-user at gnucash.org
>>> To update your subscription preferences or to unsubscribe:
>>> https://lists.gnucash.org/mailman/listinfo/gnucash-user
>>> If you are using Nabble or Gmane, please see
> https://wiki.gnucash.org/wiki/Mailing_Lists for more information.
>>> -----
>>> Please remember to CC this list on all your replies.
>>> You can do this by using Reply-To-List or Reply-All.
>> _______________________________________________
>> gnucash-user mailing list
>> gnucash-user at gnucash.org
>> To update your subscription preferences or to unsubscribe:
>> https://lists.gnucash.org/mailman/listinfo/gnucash-user
>> If you are using Nabble or Gmane, please see
> https://wiki.gnucash.org/wiki/Mailing_Lists for more information.
>> -----
>> Please remember to CC this list on all your replies.
>> You can do this by using Reply-To-List or Reply-All.
>>
>>
>> _______________________________________________
>> gnucash-user mailing list
>> gnucash-user at gnucash.org
>> To update your subscription preferences or to unsubscribe:
>> https://lists.gnucash.org/mailman/listinfo/gnucash-user
>> If you are using Nabble or Gmane, please see https://wiki.gnucash.org/wiki/Mailing_Lists for more information.
>> -----
>> Please remember to CC this list on all your replies.
>> You can do this by using Reply-To-List or Reply-All.