I have a similar issue with my Synology NAS attempting to use Safari and 1Password to login. I setup the key after picking the USB Key/Passkey option. Logged out, then at the prompt picked "use a different method of login" and was presented with the "use your iPhone to login" and the QR code. Scanned QR code and was able to login by authorizing on the iPhone. Same process using Chrome browser. Passkey login does work, but apparently Synology won't use a Passkey that isn't in the iCloud Keychain. I am able to use other passkeys to login to other sites on both Safari and Chrome, so I know that Passkeys can work with my setup. I have iCloud sync off for "passwords and Keychain" with all passwords and passkeys only in 1Password. I restarted my computer with no change noted.