TrustedUserCAKeys not working on a domain computer

[dwatley]

"OpenSSH for Windows" version
v1.0.0.0-Beta

Server OperatingSystem
Windows 10 Enterprise

Client OperatingSystem
Windows 10 Enterprise

What is failing
TrustedUserCAKeys doesn't appear to be working for a domain computer running Windows. Identical setup (identical CA Key, identical signed user key) on non-domain computer running Linux works as expected. Should also add this is using a Yubikey device, though I don't believe that matters.

Actual output
Here is the DEBUG3 log of the failure.

debug1: userauth-request for user contoso\\\\dwatley service ssh-connection method publickey [preauth]
debug1: attempt 1 failures 0 [preauth]
debug2: input_userauth_request: try method publickey [preauth]
debug1: userauth_pubkey: test whether pkalg/pkblob are acceptable for ECDSA SHA256:BzwUAn8oLRT54zs5e70vsJDeQ7aQnc+9/5n2RhtQImQ [preauth]
debug3: mm_key_allowed entering [preauth]
debug3: mm_request_send entering: type 22 [preauth]
debug3: mm_key_allowed: waiting for MONITOR_ANS_KEYALLOWED [preauth]
debug3: mm_request_receive_expect entering: type 23 [preauth]
debug3: mm_request_receive entering [preauth]
debug3: mm_request_receive entering
debug3: monitor_read: checking request 22
debug3: mm_answer_keyallowed entering
debug3: mm_answer_keyallowed: key_from_blob: 0000025871AD1D50
debug3: mm_answer_keyallowed: key is not allowed
debug3: mm_request_send entering: type 23
debug2: userauth_pubkey: authenticated 0 pkalg ecdsa-sha2-nistp256 [preauth]
debug3: userauth_finish: failure partial=0 next methods="publickey,keyboard-interactive" [preauth]
debug3: send packet: type 51 [preauth]
debug3: receive packet: type 50 [preauth]
debug1: userauth-request for user contoso\\\\dwatley service ssh-connection method publickey [preauth]
debug1: attempt 2 failures 1 [preauth]
debug2: input_userauth_request: try method publickey [preauth]
debug1: userauth_pubkey: test whether pkalg/pkblob are acceptable for ECDSA-CERT SHA256:+iRhr5z28xnKndebv4JV+sXl2agXRZ22cXwO6nZKokI [preauth]
debug3: mm_key_allowed entering [preauth]
debug3: mm_request_send entering: type 22 [preauth]
debug3: mm_key_allowed: waiting for MONITOR_ANS_KEYALLOWED [preauth]
debug3: mm_request_receive_expect entering: type 23 [preauth]
debug3: mm_request_receive entering [preauth]
debug3: mm_request_receive entering
debug3: monitor_read: checking request 22
debug3: mm_answer_keyallowed entering
debug3: mm_answer_keyallowed: key_from_blob: 0000025871AD1170
Certificate invalid: name is not a listed principal
debug3: mm_answer_keyallowed: key is not allowed

For what it's worth, using authorized_keys works as expected with no issue, but being able to use the TrustedUserCAKeys feature would be much preferred. I'm not sure that the fact that this is a domain joined computer matters, but did want to specify that. My initial thought was maybe the way the user cert was signed might result in a different identity between the two platforms (Windows domain joined, Linux not). What other options are there for debugging this further beyond enabling DEBUG3 as I have already?

Relevant sshd_config params:

debug2: parse_server_config: config __PROGRAMDATA__\\ssh/sshd_config len 376
debug3: __PROGRAMDATA__\\ssh/sshd_config:24 setting LogLevel DEBUG3
debug3: __PROGRAMDATA__\\ssh/sshd_config:40 setting PubkeyAuthentication yes
debug3: __PROGRAMDATA__\\ssh/sshd_config:51 setting PasswordAuthentication no
debug3: __PROGRAMDATA__\\ssh/sshd_config:52 setting ChallengeResponseAuthentication yes
debug3: __PROGRAMDATA__\\ssh/sshd_config:53 setting PermitEmptyPasswords no
debug3: __PROGRAMDATA__\\ssh/sshd_config:59 setting AuthorizedKeysFile none
debug3: __PROGRAMDATA__\\ssh/sshd_config:63 setting TrustedUserCAKeys __PROGRAMDATA__/ssh/ca_userkeys.pub
debug3: __PROGRAMDATA__\\ssh/sshd_config:69 setting HostbasedAuthentication no
debug3: __PROGRAMDATA__\\ssh/sshd_config:103 setting Subsystem sftp     sftp-server.exe
debug1: sshd version OpenSSH_7.6, LibreSSL 2.5.3

[dwatley]

With a bit more testing I'm able to get the cert accepted, but can't complete the login due to the identity on the system not matching:

The principal on the cert is dwatley, if I attempt to login with dwatley@host the cert is accepted, but there is no user on the system with that identity due to it being a domain account (contoso\dwatley@host) rather than a local account (dwatley@host) so it fails as expected:

debug3: mm_answer_keyverify: publickey 000001FB4382F5A0 signature verified
debug3: mm_request_send entering: type 25
debug1: monitor_child_preauth: dwatley has been authenticated by privileged process
debug3: mm_get_keystate: Waiting for new keys
debug3: mm_request_receive_expect entering: type 26
debug3: mm_request_receive entering
debug3: mm_get_keystate: GOT new keys
debug2: userauth_pubkey: authenticated 1 pkalg ecdsa-sha2-nistp256-cert-v01@openssh.com [preauth]
debug3: send packet: type 52 [preauth]
debug3: mm_request_send entering: type 26 [preauth]
debug3: mm_send_keystate: Finished sending state [preauth]
debug1: monitor_read_log: child log fd closed
debug1: LsaLogonUser failed NTSTATUS: 1
unable to generate token for user dwatley
debug1: LsaLogonUser failed NTSTATUS: 1
unable to generate token on 2nd attempt for user dwatley
unable to get security token for user dwatley
posix_spawn failed

The username and cert principal matching appears to be handled by this function in sshkey.c

I don't believe adjusting the principal will be an option because then of course this would break logins for Linux systems with identical user names, and it seems reasonable that matching the user portion of the domain\user combination on Windows would make this work.

As an exercise I tried creating a custom ca_cert and signing a user cert with a principal as various domain\user combinations (contoso\dwatley, contoso\\dwatley) and could not get it to work, so I'm guessing there's some Windows specific work to be done here to make this a reality. Thoughts? This certainly isn't my expertise, but based on my findings my hope is that it would be relatively simple to implement.

Again the end goal would be having an identical setup between Windows and Linux using an identical TrustedUserCAKey and signed user cert with the username (minus domain) as the principal on the cert.

[dwatley]

Now I'm able to login if I leverage the AuthorizedPrincipalsFile and place the username in the file.

Auth user: contoso\dwatley
TrustedUserCAKeys: enabled
User Cert: principle is dwatley
AuthorizedPrincipalsFile: C:/ProgramData/ssh/authorized_principles (contents dwatley)

I'm not sure this is how it's intended to work, but it works. I should also mention that environment variable expansion seems to work differently depending on the sshd_config option. For instance TrustedUserCAKeys allows the use of __PROGRAMDATA__ in the file path, but AuthorizedPrincipalsFile seems to require an absolute path, else when using __PROGRAMDATA__ it attempts to search relative to the user's profile with __PROGRAMDATA__ inserted:

debug3: __PROGRAMDATA__\\ssh/sshd_config:68 setting AuthorizedPrincipalsFile __PROGRAMDATA__/ssh/authorized_principles
...
...
debug1: trying authorized principals file C:\\Users\\dwatley\\__PROGRAMDATA__/ssh/authorized_principles
debug3: Failed to open file:C:\\Users\\dwatley\\__PROGRAMDATA__/ssh/authorized_principles error:2
debug1: Could not open authorized principals 'C:\\Users\\dwatley\\__PROGRAMDATA__/ssh/authorized_principles': No such file or directory

I got around this with the sshd_config option implemented this way: AuthorizedPrincipalsFile C:/ProgramData/ssh/authorized_principles

[bagajjal]

FYI, We have a strict check that the PROGRAMDATA should be in the start of the file name.. We suggest full file paths in sshd config

[dwatley]

Right, in the sshd_config, __PROGRAMDATA__ is used for both TrustedUserCAKeys and AuthorizedPrincipalsFile at the very beginning of the line, but only works for TrustedUserCAKeys, and requires the full path. This is just a note that the two configuration settings don't behave the same way. I have no issue using full paths as a best practice if necessary. The primary issue I'm raising here regarding domain credentials used for ssh auth.

[manojampalam]

@dwatley thanks for the detailed report. On Windows, in a domain environment, its necessary to fully qualify the user name to avoid resolution confusion - user with the same name can exist locally and in different reachable domains.

Can you try signing your user cert with dwatley@contoso as principal?

[dwatley]

Thanks for the response. Signing with dwatley@contoso in the principal works. So the format definitely matters here, good to know. I also tried signing the cert with multiple principals comma separated, so dwatley, dwatley@contoso and that worked. So it seems there's some options here.

1. Use the AuthorizedPrincipalsFile containing usernames that will match. Ultimately the domain user is authenticated as usual, and the principal listed in the cert is presumably checked against a list of principals in the file rather than the username@domain that authenticated. This works, I suppose with caveats in a multiple domain environment depending on the intended result?
2. Sign the user cert with more than one principal. This also works.

My earlier comment regarding checking the username separately was strictly regarding the principal in the cert as I understand the domain is necessary to tie it to the correct user on a domain joined computer. Thinking about it more, considering it's possible to attach multiple principals to a cert that's probably the correct thing to do for a single user cert intended to be used cross platform?

[manojampalam]

Yes. Your inferences seem right. I've added a reference to this thread in Certificate Usage Samples

[bagajjal]

@dwatley - Thanks for pointing the path resolution issue of PROGRAMDATA in AuthorizedPrincipalsFile. There is a minor bug, I have fixed it. Here is the private binary.

please take backup of sshd.exe, use private sshd.exe.

[dwatley]

@bagajjal - Confirming the __PROGRAMDATA__ fix for AuthorizedPrincipalsFile. Works as expected.

[dwatley]

@manojampalam / @NoMoreFood - Just re-iterating that the above all works with the TrustedUserCAKeys config option. Using the principals file, and/or signing with multiple principals. The format has changed to NetBios\SamAccountName based on the referenced issue. Thanks for all the work!