SSH error: signing failed for RSA "" from agent: agent refused operation

Hey, I'm having a problem with the ssh agent. Any time I'm trying to use ssh and authenticate (for multiple servers), I'm getting the following error: sign_and_send_pubkey: signing failed for RSA "...