SSH Agent not triggering for remote host with only IPv6 address

When I SSH into a host that has an IPv6 address but no IPv4 address, the SSH Agent is not triggered and I receive a password prompt. I receive the same behavior when I connect via IP or hostname ...