Improving the new duplicate detection tool [8.10.18 +onward]

@ag_maryam, @1P_Dave: Thank you. It's appreciated.
I have started a new thread for desired improvements to the tool at: https://1password.community/discussion/142887/improving-the-new-duplicate-detection-tool-8-10-18-onward/p1?new=1

ps: @Chris Messina, did the tool fix all those duplicates? I have as many as you. The tool fixed less than 10 of mine. If yours were not fixed, please continue the discussion in the new thread.

Hello @spinoza! 👋

Thank you for the feedback. I've merged your comments together into one thread so that we can keep the conversation about your specific issue in one place.

The new "Items with duplicates" feature is very conservative by design. If two items are even a tiny bit different (modified date, etc), 1Password will not remove them. This is done so that 1Password does not perform bulk actions that may result in data loss.

But there are still over a thousand items with the same essential data (username, PW, creation/mod date, etc.)

If the username, password, and creation/modified metadata are all the same then can you tell me what is different between these "duplicate" items? I look forward to hearing from you. 🙂

-Dave

@1P_Dave , I wish I knew what was different. The duplicates usually seem identical to me. My assumption that the metadata must differ stems from what tech support suggested in the past. In other words, since the duplicate detector didn't flag it, the metadata MUST be different. (This was back when we had the duplicate detector in OP7, before its removal during the transition to OP8.)

I repeatedly asked about the specific differences, or at least to have them described in general terms. I never received a direct response, only comments about the database format being proprietary, promises to pass it along to the development team, and vague hopes that it might be fixed in a future release. I never got the impression that the issue was thoroughly investigated or that there was much concern about it.

The origin of the issue was my early adoption of moving standalone databases to the cloud in OP6. I had standalone databases on various devices, similar, but none containing the latest version of every password. I assumed there would be a merge function or something to prevent duplicates, but there wasn't. Nor was there any warning against what I did, nor any way to undo it. (After my report, a safeguard to prevent this was added to the code, but it was too late for me and a few others.) My initial databases of about 2000 passwords ballooned to 8000. I've been trying to fix this for, what, over 10 years now?

@1P_Dave, I think the concern about data loss can be addressed by a prominent warning and suggestion to create a backup. (If you want a beta tester for a broader duplicate removal feature, I volunteer as tribute)

@ag_maryam fyi

@spinoza

Thanks for the reply. Are the items in the same vault or in different vaults? The new duplicate remover tool acts on a per-vault basis to avoid situations where users might have different permissions for items in different vaults that might prevent the tool from automatically removing duplicates. However this is something that the team is hoping to iterate on in the future.

If the duplicates are not in different vaults then would you be willing to send in the item structure (JSON) of both an item and its duplicate so that our support team could compare them to see what the difference is? If you're willing then please send an email to support+forum@1Password.com with a link to this thread and the team will provide instructions on how to generate the item JSON and redact any sensitive information such as passwords and other data before sending it to us. Then they'll provide a secure share link for you to upload the JSON files.

-Dave