Authenticate using CLI (non-gui)

I have enabled the 1Password SSH Agent which works fine, apart from one usecase: accessing my computer remotely and then trying to access other SSH hosts. It then prompts for a password using the GUI...