Forum Discussion

ondrej-at-scrive's avatar
ondrej-at-scrive
New Contributor
3 months ago

'op read' mistreats binary content

I wanted to write a command for git crypt unlock <FILE>, but since the command requires file as input and I was figuring out how to get content of attachment, I first tried it with op read 'op://<my-vault>//git-crypt.key' > git-crypt.key. Then, trying to unlock with now stored key, I was met with error "not a valid git-crypt key file".

After some investigation, I found out that the stored key is slightly modified. This seem to happen when:

  • The content is at least partially binary
  • The content contains some invalid Unicode sequences or certain control characters
  • The content is directly redirected into a file using > operator

It seems that ascii-only content isn't affected. The binary content is also not affected when it's being piped into another process (e.g. op read 'op://<my-vault>/<my-item>/git-crypt.key' | cat > git-crypt.key - extra cat in the pipeline helps op store the contents correctly). What also works correctly is git crypt unlock <(op read -n ...) as it also creates a inter-process pipe.

Since there's quite glaring occurrence of ef bf bd, which is a Unicode replacement character (), and sequence 594f 7f63 is transformed to just 594f 63.. (7f being a DELETE control character), it seems that the content undergoes some UTF-8 decoding/processing.

This is bit confusing as it's neither documented, nor is there any -b | --binary option to control this behavior.

# Create a binary file and upload it to 1Password
> dd if=/dev/urandom of=binary-data bs=1 count=32

# Fetch the attachment from 1Password using CLI
> op read -n 'op://<my-vault>/Test/binary-data' > binary-data-redirected-to-file
> op read -n 'op://<my-vault>/Test/binary-data' | cat > binary-data-piped-through-cat

# Print content
> hexxy -n binary-data
0000000: 00c6 773b 1963 95f1 6dc5 1bb6 bdde 4946  ..w;.c..m.....IF
0000010: 9f0e 594f 7f63 b6ed 2392 f9e1 91b3 abfc  ..YO.c..#.......

> hexxy -n binary-data-redirected-to-file
0000000: efbf bd77 3b63 efbf bdef bfbd 6def bfbd  ...w;c......m...
0000010: efbf bdef bfbd efbf bd49 46ef bfbd 594f  .........IF...YO
0000020: 63ef bfbd efbf bd23 efbf bdef bfbd e191  c......#........
0000030: b3ef bfbd efbf bd                            .......

> hexxy -n binary-data-piped-through-cat
0000000: 00c6 773b 1963 95f1 6dc5 1bb6 bdde 4946  ..w;.c..m.....IF
0000010: 9f0e 594f 7f63 b6ed 2392 f9e1 91b3 abfc  ..YO.c..#.......

Rant on the side: Not being able to use <code> tag on forum is bit dumb.

1 Reply

  • Hi ondrej-at-scrive​ ,

    Thanks for sharing this.

    I am passing this along to become part of the documentation in the near term and as a feature request for a better solution longterm.

    Also thanks for the note about the <code> tag, I'll provide that back to the right team as well.

    Thanks again,
    Phil