SSH Agent: agent refused operation

UPDATE: I had to restart my computer
It would be helpful to mention this in the official instructions.

I'm facing the same problem on macOS 13.3.1 (a).
No matter what I try to do I always get this error when doing git fetch from a github repo


sign_and_send_pubkey: signing failed for ED25519 "/Users/jumar/Downloads/id_ed25519.pub" from agent: agent refused operation


1Password log contains these messages:


INFO 2023-05-18T09:47:11.828 tokio-runtime-worker(ThreadId(8)) [1P:foundation/op-sys-info/src/process_information/macos/non_app_store.rs:86] failed to find NSApplication related to pid 3960
INFO 2023-05-18T09:47:11.837 tokio-runtime-worker(ThreadId(1176)) [1P:foundation/op-apple/src/biometry_service.rs:308] System biometry info: BiometricStatus { current_policy: BiometricsOnly, current_method: TouchId, current_availability: NotEnrolled }
INFO 2023-05-18T09:47:15.074 tokio-runtime-worker(ThreadId(3)) [1P:ssh/op-ssh-agent/src/lib.rs:541] Session was not authorized


Even if my 1Password app is opened and unlocked it's still the same error.

More SSH logs:


debug1: Next authentication method: publickey
debug1: Offering public key: /Users/jumar/Downloads/id_ed25519.pub ED25519 SHA256:As31qZ4Rff4WbHnS6nikN84c+FRxbMERDnvYIgexE8c explicit agent
debug3: send packet: type 50
debug2: we sent a publickey packet, wait for reply
debug3: receive packet: type 60
debug1: Server accepts key: /Users/jumar/Downloads/id_ed25519.pub ED25519 SHA256:As31qZ4Rff4WbHnS6nikN84c+FRxbMERDnvYIgexE8c explicit agent
debug3: sign_and_send_pubkey: using publickey with ED25519 SHA256:As31qZ4Rff4WbHnS6nikN84c+FRxbMERDnvYIgexE8c
debug3: sign_and_send_pubkey: signing using ssh-ed25519 SHA256:As31qZ4Rff4WbHnS6nikN84c+FRxbMERDnvYIgexE8c
sign_and_send_pubkey: signing failed for ED25519 "/Users/jumar/Downloads/id_ed25519.pub" from agent: agent refused operation