[GNC] OFX Import with Investment Transactions

Sun Oct 3 10:10:16 EDT 2021

Regarding those duplicate transactions in the account register during the
import process,  there has been a long standing bug in the generic import
assistant  which allows the incoming transactions to be seen but if they
are selected they cannot be unselected without changing the final result
when the import is done.

If the duplicate s are those same incoming transactions,  they are probably
from that old bug number 721139.

On Sat, Oct 2, 2021, 11:31 PM Jean L <ripngo at gmail.com> wrote:

> Thanks Robin!
> Good detective work here, and thanks for the examples, I think there
> must be a bug :( . I'll take a look.
>
> > If -- after importing the OFX file -- you open one of the account
> > registers, you'll see duplicate transactions! Although, if you close and
> > re-open the file, those duplicated transactions do not appear when you
> > re-open the account register. I don't know what to make of that.
>
> That is super weird :(
> Jean
>
> On 10/2/2021 5:19 PM, Robin Chattopadhyay wrote:
> > Hi Jean-
> >
> > I understand the logic for #1 and have no concerns about that.
> >
> > As for #2, I did continue past the first matching dialog and there were
> no
> > additional dialogs, the focus returned to the account list.
> >
> > There are definitely FITIDs that are already in the database that are
> also
> > in the most recent OFX file. This is because the provider doesn't allow
> you
> > to specify a date range, only 'current quarter' or 'some other quarter'.
> >
> > I tried something else to research...
> >
> > Using 4.8, I created a blank file and then imported an OFX file from Sep.
> > 25, 2021. This file had 66 transactions and imported as expected with one
> > matcher screen per security/account as you indicated would happen. I then
> > attempted to import the OFX file from Oct. 2. This file has 84
> transactions
> > -- the same 66 transactions from the Sep. 25 file plus 18 new
> transactions
> > dated 9/24.
> >
> > The program then presents the dialog for the first account. There are two
> > transactions in the most recent file for this account; however they're
> > already in the database, thus triggering the message "OFX file
> > /home/robin/Downloads.qfx' imported transactions for account 'xxx' 2
> > transactions processed, no transactions to match". That makes sense as
> > those are older transactions that are already in the database. I clicked
> > 'Close' for that dialog and then the focus returns to the main
> application.
> > It's almost as if the program -- having determined that there were no new
> > transactions to import into the first account -- decided not to look at
> the
> > subsequent accounts.
> >
> > In service to the greater good, I am attaching the test file I created
> and
> > the Oct. 2 OFX file. I have removed all of the position data (there are
> > limits to what I'm willing to share :-) ), but that will not negatively
> > affect the import process.
> >
> > Something else that I just noticed...
> >
> > If -- after importing the OFX file -- you open one of the account
> > registers, you'll see duplicate transactions! Although, if you close and
> > re-open the file, those duplicated transactions do not appear when you
> > re-open the account register. I don't know what to make of that.
> >
> > On Sat, Oct 2, 2021 at 1:52 PM Jean L <ripngo at gmail.com> wrote:
> >> Yes, there's a new behavior with OFX import. The match dialog shows up
> >> for each target account, so if you're importing securities the dialog
> >> will show up for each security. This could be a real annoyance if there
> >> are many different securities, so we may want to change that behavior.
> >> The new behavior was introduced to fix an issue when a single OFX file
> >> had transfers between accounts.
> >>
> >> I see two issues here:
> >> 1) The fact that the match dialog runs for each target account (see
> >> comment above)
> >> 2) The fact that some of your imported transactions were simply ignored.
> >>
> >> About 2): Did you try to continue with the import after the first
> >> matching dialog? Did it then go to the next securities? Or did you just
> >> abort thinking something was wrong (not expecting the new behavior of
> >> seeing the matching dialog for each security instead of all together as
> >> before)
> >>
> >> If that was not the case (i.e., you tried continuing but nothing else
> >> happened) one reason why this could happen is if the FITID used for the
> >> imported transactions were already present in the target account. This
> >> is new in the latest version: transactions that were previously imported
> >> are no longer shown the matching dialog. In your case this seems
> >> erroneous. So either it's a bug (i.e., the FITID of the imported
> >> transactions are NOT found in any of the transactions of the account
> >> you're imported into, so they should show) or it's a problem with the
> >> OFX data that the transactions seem to reuse existing FITIDs.
> >> The fact that things work normally when you import into a blank account
> >> would indicate that indeed, the FITID of the imported transactions
> >> already exist in your target account.
> >>
> >> SO, could you re-try importing with the latest version, but making sure
> >> you continue clicking OK for each match dialog? Does this work or not?
> >> If not, then we'll need to check whether the FITID of the new imported
> >> transactions somehow have been used before in the previously imported
> >> transactions. To see that, you'd need to pick one that was not imported
> >> (looking at the OFX file), get its FITID, and see in your account
> >> database (save it as an xml file so it can be looked at with a regular
> >> editor) whether that FITID exists.
> >>
> >> Jean
> >>
> >> On 10/2/2021 8:58 AM, Robin Chattopadhyay wrote:
> >>> Ubuntu 20.04 LTS (VMWare virtual machine in case that matters)
> >>> Gnucash 4.8
> >>> libofx 0.9.15
> >>>
> >>> I tried importing an OFX file from my 401K provider this morning and I
> > got
> >>> a message box that said it had imported 2 transactions for a single
> >>> security in the file and there were no additional transactions to
> > process.
> >>> This was incorrect as there were 18 new transactions across nine
> > securities
> >>> (along with a number of other transactions that had already been
> > imported
> >>> previously).
> >>>
> >>> I tried a number of things to resolve (after taking a backup, of
> > course):
> >>> 1 - Tools > Import Map Editor and deleted all of the associations for
> > this
> >>> file. When I re-imported, I was prompted to map each security in the
> > file
> >>> to the correct account. No issues there. But when importing the OFX
> file
> >>> again, I had the same experience as described above
> >>> 2 - Created a blank file and attempted to import the file there.
> Through
> >>> the import process, I created new securities and new accounts, nothing
> >>> unexpected there. Then the generic transaction importer dialog popped
> > up,
> >>> but with just the two transactions for the same security cited in the
> >>> original problem dialog. I imported those, clicked OK and then the
> > generic
> >>> transaction importer dialog came up *again* but only with transactions
> > for
> >>> a single, different security. I imported those and repeated the process
> > for
> >>> each security with transactions in the file. Each time the generic
> >>> transaction importer dialog came up, it only had transactions for a
> > single
> >>> security. This is definitely new behavior that I didn't see in 4.6.
> >>> Previously the import dialog had all the new transactions in the file
> >>> 3 - Reverted to 4.7. Crashed when selecting Import from the File menu
> >>> (Trace/breakpoint trap (core dumped). Not unexpected, but thought I
> > would
> >>> try anyway.
> >>> 4 - Reverted to 4.6. This works as it used to with all the new
> > transactions
> >>> in a single dialog
> >>>
> >>> Finally, I don't know if this matters, but I scanned stdout from the
> >>> build/make/install process to see if anything looked obviously wrong
> > and I
> >>> found this (I don't know if it's relevant):
> >>>
> >>> -- Performing Test HAVE_OFX_BUG_39
> >>> -- Performing Test HAVE_OFX_BUG_39 - Failed
> >>>
> >>> Thanks,
> >>> Robin
> >>> _______________________________________________
> >>> gnucash-user mailing list
> >>> gnucash-user at gnucash.org
> >>> To update your subscription preferences or to unsubscribe:
> >>> https://lists.gnucash.org/mailman/listinfo/gnucash-user
> >>> If you are using Nabble or Gmane, please see
> > https://wiki.gnucash.org/wiki/Mailing_Lists for more information.
> >>> -----
> >>> Please remember to CC this list on all your replies.
> >>> You can do this by using Reply-To-List or Reply-All.
> >> _______________________________________________
> >> gnucash-user mailing list
> >> gnucash-user at gnucash.org
> >> To update your subscription preferences or to unsubscribe:
> >> https://lists.gnucash.org/mailman/listinfo/gnucash-user
> >> If you are using Nabble or Gmane, please see
> > https://wiki.gnucash.org/wiki/Mailing_Lists for more information.
> >> -----
> >> Please remember to CC this list on all your replies.
> >> You can do this by using Reply-To-List or Reply-All.
> >>
> >>
> >> _______________________________________________
> >> gnucash-user mailing list
> >> gnucash-user at gnucash.org
> >> To update your subscription preferences or to unsubscribe:
> >> https://lists.gnucash.org/mailman/listinfo/gnucash-user
> >> If you are using Nabble or Gmane, please see
> https://wiki.gnucash.org/wiki/Mailing_Lists for more information.
> >> -----
> >> Please remember to CC this list on all your replies.
> >> You can do this by using Reply-To-List or Reply-All.
>
> _______________________________________________
> gnucash-user mailing list
> gnucash-user at gnucash.org
> To update your subscription preferences or to unsubscribe:
> https://lists.gnucash.org/mailman/listinfo/gnucash-user
> If you are using Nabble or Gmane, please see
> https://wiki.gnucash.org/wiki/Mailing_Lists for more information.
> -----
> Please remember to CC this list on all your replies.
> You can do this by using Reply-To-List or Reply-All.
>