MRC Converter suite: Enpass to 1Password - unexpectedly matched multiple outkey values for 'url'

Shamelessly paging @MrC , sorry in advance for bothering!

@charliesigma

No shame in the ping, thanks for bringing it to my attention. Let me recreate it in Windows to be sure I give you a proper fix.

This is a peculiar edge case, in that both Website and URL are allowed to match as the primary (1Password) url key for a Login record, and stock records would not have both labels.

Back shortly...

For sure!
The enpass parser does have something that I interpret as a treatment for such cases, under login:


line 405
[ '*additionalurls', 0, qr/^(?:URL|Website):::__MAIN:::(?!13$)/, ], # support additional URLs within the main section


I tried to mess with what I assume are regex statements there but I frankly don't have lots of experience with regex (not even sure it IS regex lol), especially when embedded in perl. Buut, I might be totally wrong in the previous assumptions.

If I comment line 404 [ 'url', 3, qr/^(?:URL|Website)(:::__MAIN:::13)?$/, ],, then it parses but both URLs get added to the 'notes' section.

If I comment line 405, it fails the same way described above.

@charliesigma

This converter has grown over time, to accommodate Enpass' growth of templates and export types. So there is a wide variety of labels and data in the wild. This is exacerbated by Enpass not canonicalizing it's imports from other password managers, leaving records as a bizarre mixture of old, new, and custom formats.

The issue is that my workaround to avoid duplicate keys is not catching this case, due to your labels being both URL and Website. The simplistic match of any given field label (e.g. Website) against all the other field labels in the record isn't working.

The field pattern matching template you show is where additional URLS should be placed. I have to change from simple label matching to using the field type and uid.

ahh I see. That makes sense! Something I figured is that the UID field might not be very useful because dumping the categories using --dumpcats show 'website' and URLs have different UIDs between different stored entries:


1 __MAIN::URL::13::4
1 __MAIN::URL::13::5
1 __MAIN::URL::14::5
1 __MAIN::URL::6949::5
1 __MAIN::URL::6957::6
1 __MAIN::URL::6978::6
1 __MAIN::URL::7856::4
1 __MAIN::URL::8022::5
1 __MAIN::URL::8618::6
1 __MAIN::URL::8883::5
1 __MAIN::URL::9553::8
1 __MAIN::URL::9736::5
1 __MAIN::URL::9957::5
1 __MAIN::Username::10::1
1 __MAIN::Username::10::2
1 __MAIN::Username::10::3
1 __MAIN::Website::13::4
1 __MAIN::Website::13::5
1 __MAIN::Website::13::7
1 __MAIN::Website::14::5
1 __MAIN::Website::15::5
1 __MAIN::Website::16::5
1 __MAIN::Website::5567::5
1 __MAIN::Website::5684::5
1 __MAIN::Website::5984::5
1 __MAIN::Website::6169::5
1 __MAIN::Website::6353::5
1 __MAIN::Website::6455::5
1 __MAIN::Website::7472::6
1 __MAIN::Website::7529::5
1 __MAIN::Website::7662::4
1 __MAIN::Website::7959::4
1 __MAIN::Website::8075::4
1 __MAIN::Website::8150::4
1 __MAIN::Website::8336::5
1 __MAIN::Website::8487::5
1 __MAIN::Website::8522::6
1 __MAIN::Website::8689::4
1 __MAIN::Website::8708::4
1 __MAIN::Website::8895::5
1 __MAIN::Website::9143::6
1 __MAIN::Website::9362::5
1 __MAIN::Website::9601::5
1 __MAIN::Website::9708::5
1 __MAIN::Website::9856::6
1 ADDITIONAL DETAILS::URL::5010::13
1 ADDITIONAL DETAILS::URL::5018::11
1 ADDITIONAL DETAILS::URL::5093::10
1 ADDITIONAL DETAILS::URL::5093::9
1 ADDITIONAL DETAILS::URL::5119::10
1 ADDITIONAL DETAILS::URL::5141::15
1 ADDITIONAL DETAILS::URL::5150::15
1 ADDITIONAL DETAILS::URL::5169::14
1 ADDITIONAL DETAILS::URL::5181::12
1 ADDITIONAL DETAILS::URL::5191::11
1 ADDITIONAL DETAILS::URL::5202::12
1 ADDITIONAL DETAILS::URL::5208::13
1 ADDITIONAL DETAILS::URL::5244::11
1 ADDITIONAL DETAILS::URL::5290::11
1 ADDITIONAL DETAILS::URL::5349::10
1 ADDITIONAL DETAILS::URL::5487::13
1 ADDITIONAL DETAILS::URL::5492::10
1 ADDITIONAL DETAILS::URL::5539::10
1 ADDITIONAL DETAILS::URL::5541::10
1 ADDITIONAL DETAILS::URL::5546::8
1 ADDITIONAL DETAILS::URL::5549::10
1 ADDITIONAL DETAILS::URL::5592::16
1 ADDITIONAL DETAILS::URL::5635::12
1 ADDITIONAL DETAILS::URL::5657::10
1 ADDITIONAL DETAILS::URL::5724::10
1 ADDITIONAL DETAILS::URL::5732::13
1 ADDITIONAL DETAILS::URL::5751::11
1 ADDITIONAL DETAILS::URL::5765::10
1 ADDITIONAL DETAILS::URL::5773::11
1 ADDITIONAL DETAILS::URL::5887::11
1 ADDITIONAL DETAILS::URL::5897::10
1 ADDITIONAL DETAILS::URL::5955::10
1 ADDITIONAL DETAILS::URL::5988::9
1 ADDITIONAL DETAILS::URL::5993::17
1 ADDITIONAL DETAILS::URL::6003::11
1 ADDITIONAL DETAILS::URL::6012::16
1 ADDITIONAL DETAILS::URL::6047::10
1 ADDITIONAL DETAILS::URL::6057::12
1 ADDITIONAL DETAILS::URL::6123::10
1 ADDITIONAL DETAILS::URL::6123::11
1 ADDITIONAL DETAILS::URL::6143::11
1 ADDITIONAL DETAILS::URL::6163::10
1 ADDITIONAL DETAILS::URL::6184::12
1 ADDITIONAL DETAILS::URL::6270::11
1 ADDITIONAL DETAILS::URL::6272::11
1 ADDITIONAL DETAILS::URL::6281::11
1 ADDITIONAL DETAILS::URL::6296::10
1 ADDITIONAL DETAILS::URL::6367::8
1 ADDITIONAL DETAILS::URL::6372::10
1 ADDITIONAL DETAILS::URL::6379::10
1 ADDITIONAL DETAILS::URL::6412::10
1 ADDITIONAL DETAILS::URL::6418::12
1 ADDITIONAL DETAILS::URL::6447::9
1 ADDITIONAL DETAILS::URL::6483::10
1 ADDITIONAL DETAILS::URL::6512::10
1 ADDITIONAL DETAILS::URL::6516::10
1 ADDITIONAL DETAILS::URL::6520::10
1 ADDITIONAL DETAILS::URL::6553::10
1 ADDITIONAL DETAILS::URL::6693::12
1 ADDITIONAL DETAILS::URL::6704::10
1 ADDITIONAL DETAILS::URL::6720::15
1 ADDITIONAL DETAILS::URL::6759::12
1 ADDITIONAL DETAILS::URL::6782::10
1 ADDITIONAL DETAILS::Website::13::5
1 ADDITIONAL DETAILS::Website::14::6
1 ADDITIONAL DETAILS::Website::5035::10
1 ADDITIONAL DETAILS::Website::5038::14
1 ADDITIONAL DETAILS::Website::5191::10
1 ADDITIONAL DETAILS::Website::5223::11
1 ADDITIONAL DETAILS::Website::5236::10
1 ADDITIONAL DETAILS::Website::5304::12
1 ADDITIONAL DETAILS::Website::5335::10
1 ADDITIONAL DETAILS::Website::5450::15
1 ADDITIONAL DETAILS::Website::5451::11
1 ADDITIONAL DETAILS::Website::5522::10
1 ADDITIONAL DETAILS::Website::5530::11
1 ADDITIONAL DETAILS::Website::5577::10
1 ADDITIONAL DETAILS::Website::5669::7
1 ADDITIONAL DETAILS::Website::5727::15
1 ADDITIONAL DETAILS::Website::5782::10
1 ADDITIONAL DETAILS::Website::5793::10
1 ADDITIONAL DETAILS::Website::5803::12
1 ADDITIONAL DETAILS::Website::5836::10
1 ADDITIONAL DETAILS::Website::5846::10
1 ADDITIONAL DETAILS::Website::5850::10
1 ADDITIONAL DETAILS::Website::5939::13
1 ADDITIONAL DETAILS::Website::5952::10
1 ADDITIONAL DETAILS::Website::5952::11
1 ADDITIONAL DETAILS::Website::5976::17
1 ADDITIONAL DETAILS::Website::5983::6
1 ADDITIONAL DETAILS::Website::5988::12
1 ADDITIONAL DETAILS::Website::6097::11
1 ADDITIONAL DETAILS::Website::6106::19
1 ADDITIONAL DETAILS::Website::6170::10
1 ADDITIONAL DETAILS::Website::6198::14
1 ADDITIONAL DETAILS::Website::6269::10
1 ADDITIONAL DETAILS::Website::6280::12
1 ADDITIONAL DETAILS::Website::6290::10
1 ADDITIONAL DETAILS::Website::6306::20
1 ADDITIONAL DETAILS::Website::6312::10
1 ADDITIONAL DETAILS::Website::6315::10
1 ADDITIONAL DETAILS::Website::6315::8
1 ADDITIONAL DETAILS::Website::6326::10
1 ADDITIONAL DETAILS::Website::6356::10
1 ADDITIONAL DETAILS::Website::6414::10
1 ADDITIONAL DETAILS::Website::6414::13
1 ADDITIONAL DETAILS::Website::6441::11
1 ADDITIONAL DETAILS::Website::6493::21
1 ADDITIONAL DETAILS::Website::6518::10
1 ADDITIONAL DETAILS::Website::6543::10
1 ADDITIONAL DETAILS::Website::6560::10
1 ADDITIONAL DETAILS::Website::6572::11
1 ADDITIONAL DETAILS::Website::6616::10
1 ADDITIONAL DETAILS::Website::6690::10
1 ADDITIONAL DETAILS::Website::6726::10
1 ADDITIONAL DETAILS::Website::6768::16
1 ADDITIONAL DETAILS::Website::6771::10
1 ADDITIONAL DETAILS::Website::6796::10
1 ADDITIONAL DETAILS::Website::6811::8
1 ADDITIONAL DETAILS::Website::6830::10
1 ADDITIONAL DETAILS::Website::6843::10
1 ADDITIONAL DETAILS::Website::6878::11
1 ADDITIONAL DETAILS::Website::6888::18
1 ADDITIONAL DETAILS::Website::6894::10
1 ADDITIONAL DETAILS::Website::7048::10
1 ADDITIONAL DETAILS::Website::7049::12
1 ADDITIONAL DETAILS::Website::7066::11
1 ADDITIONAL DETAILS::Website::7125::11
1 ADDITIONAL DETAILS::Website::7159::10
1 ADDITIONAL DETAILS::Website::7170::10
1 ADDITIONAL DETAILS::Website::7211::10
1 ADDITIONAL DETAILS::Website::7240::10
1 ADDITIONAL DETAILS::Website::7270::15
1 ADDITIONAL DETAILS::Website::7354::11
1 ADDITIONAL DETAILS::Website::7355::11
1 ADDITIONAL DETAILS::Website::7355::12


The type is always consistent as URL though.

@charliesigma

Thanks for the dump; it's always good to see the variety of data that exists.

Generally the converter suite uses labels, as this is the way it grew over time. Most password manager exports are simple label / value pair based exports (e.g. CSV files). Exports where data is typed have traditionally been rare, yet have become more common. Enpass is no exception here - its early export formats were awful. Since this particular converter still has to support older export formats, I can't just change it to assume typed data. At the stage of conversion where this needs to happen, the key / value pairs are simply being collected, to be generalized and record-split later. This is why you see the workaround to append UID values, and why some patterns in the field matching templates support those appended UIDs. But this clearly is grossly inadequate.

@MrC no problem at all, anything I can do to help. If there were other URLs that used the same login information, for example, I always added to my entries to allow me to log in faster. Even Enpass started asking if I wanted to add the new website URL to that specific login, and when I said yes, it probably added to the "Additional Details" area (very last entry). My 1password entry from today is one example of it (removed all identifiable information):


{
"archived": 0,
"auto_submit": 1,
"category": "login",
"createdAt": 1679150965,
"favorite": 0,
"fields": [
{
"deleted": 0,
"label": "Username",
"order": 1,
"sensitive": 0,
"type": "username",
"uid": 10,
"updated_at": 1679150965,
"value": "",
"value_updated_at": 1679150965
},
{
"deleted": 0,
"label": "E-mail",
"order": 2,
"sensitive": 0,
"type": "email",
"uid": 12,
"updated_at": 1679151127,
"value": "",
"value_updated_at": 1679151127
},
{
"deleted": 0,
"label": "Password",
"order": 3,
"sensitive": 1,
"type": "password",
"uid": 11,
"updated_at": 1679150965,
"value": "",
"value_updated_at": 1679150965
},
{
"deleted": 0,
"history": [
{
"updated_at": 1679153244,
"value": ""
}
],
"label": "Secret Key",
"order": 4,
"sensitive": 1,
"type": "password",
"uid": 102,
"updated_at": 1679153244,
"value": "",
"value_updated_at": 1679153244
},
{
"deleted": 0,
"label": "Website",
"order": 5,
"sensitive": 0,
"type": "url",
"uid": 13,
"updated_at": 1679152731,
"value": "https://my.1password.com",
"value_updated_at": 1679150965
},
{
"deleted": 0,
"label": "ADDITIONAL DETAILS",
"order": 6,
"sensitive": 0,
"type": "section",
"uid": 101,
"updated_at": 1679152731,
"value": "",
"value_updated_at": 1679150965
},
{
"deleted": 0,
"label": "Phone number",
"order": 7,
"sensitive": 0,
"type": "phone",
"uid": 14,
"updated_at": 1679152731,
"value": "",
"value_updated_at": 1679150965
},
{
"deleted": 0,
"label": "Security question",
"order": 8,
"sensitive": 0,
"type": "text",
"uid": 15,
"updated_at": 1679150965,
"value": "",
"value_updated_at": 1679150965
},
{
"deleted": 0,
"label": "Security answer",
"order": 9,
"sensitive": 1,
"type": "text",
"uid": 16,
"updated_at": 1679150965,
"value": "",
"value_updated_at": 1679150965
},
{
"deleted": 0,
"label": "URL",
"order": 10,
"sensitive": 0,
"type": "url",
"uid": 7323,
"updated_at": 1679152076,
"value": "my.1password.ca",
"value_updated_at": 1679152076
}
],
"icon": {
"fav": "my.1password.com",
"image": {
"file": "misc/login"
},
"type": 1,
"uuid": ""
},
"note": "",
"subtitle": "",
"template_type": "login.default",
"title": "1password",
"trashed": 0,
"updated_at": 1679154120,
"uuid": ""
},


@charliesigma

I want to fix this issue in a more robust way, but don't want to keep you waiting too long. Have you customized the URL labels to be other than the two variants I've mentioned, URL or Website?

Looking at the dumpcats results it doesn't seem like it, as I tried to keep it as organized as possible. The one I posted is a summarized version as there were too many characters to post.

@charliesigma

Ok, no matter. I decided to update the converter suite to handle labels for the additional URLs. The enpass converter is the first to require this (others have only needed support for a list of URLs, without labels).

I've coded this to only support these extra URLs if they are in the main section of the record. If they are in the Additional Details area, they are treated as unmapped fields, so go into the notes area. Is that how you'd prefer it, or would you like them mapped as additional URLs in the 1Password login record?