SSH certificates setup

Following an explanatory post on how to use SSH within Public-key Infrastructure (PKI), here is an example how to deploy it within almost any environment. Primary candidates are virtual guests, but of course also hosts, including e.g. Proxmox VE cluster nodes as those appear as if completely regular hosts from SSH perspective out-of-the-box (without obscure command-line options added) even when clustered - ever since the SSH host key bugfix.

Roles and Parties

There will be 3 roles mentioned going forward, the terms as universally understood:

• Certification Authority (CA) which will distribute its public key (for verification of its signatures) and sign other public keys (of connecting users and/or hosts being connected to);
• Control host from which connections are meant to be initiated by the SSH client or the respective user - which will have their public key signed by a CA;
• Target host on which incoming connections are handled by the SSH server and presenting itself with public host key equally signed by a CA.

Combined roles and parties

Combining roles (of a party) is possible, but generally always decreases the security level of such system.

One such controversial is combining a Control and Target into one - an architecture under which Proxmox VE falls under with its very philosophy of being able to control any host of the cluster (and guests therein), i.e. a Target, from any other node, i.e. an architecture without a designated Control host.

Example scenario

For the sake of simplicity, we assume one external Control party which doubles as a sole CA and multitude of Targets. This means performing signing of all the keys in the same environment as from which the control connections are made. A separate setup would only be more practical in an automated environment, which is beyond scope here.

Ramp-up

Further, we assume a non-PKI starting environment, as that is the situation most readers will begin with. We will intentionally - more on that below - make use of the previously described setup of strict SSH approach,   but with a lenient alias. In fact, let’s make two, one for secure shell ssh   and another for secure copy scp   (which uses ssh):

cat >> ~/.ssh/config <<< "StrictHostKeyChecking yes"

alias blind-ssh='ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no'
alias blind-scp='scp -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no'

Blind connections

Ideally, blind connections should NOT be used, not even for the initial setup. It is explicitly mentioned here as an instrumental approach to cover two concepts:

• blind-ssh as a pre-PKI setup way of executing a command on a target, i.e. could be instead done securely by performing the command on the host’s console, either physical or with an out-of-band access, or should be part of installation and/or deployment of such host to begin with;

• blind-scp as an independent mechanism of distributing files across, i.e. shared storage or manual transfer could be utilised instead.

If you already have a secure environment, regular ssh and scp should be simply used instead. For virtual hosts, execution of commands or distribution of files should be considered upon image creation already.

Root connections

We abstract from privilege considerations by assuming any connection to a Target is under the root user. This may appear (and actually is) ill-advised, but is unfortunately a standard Proxmox VE setup and CANNOT be disabled without loss of feature set. Should one be considering connecting with non-privileged users, further e.g. sudo setup needs to be in place, which is out of scope here.

Setup

Certification Authority key

We will first generate CA’s key pair in a new staging directory. This directory can later be completely dismantled, but of course the CA key should be retained elsewhere then.

(umask 077; mkdir ~/stage)
cd ~/stage

ssh-keygen -t ed25519 -f ssh_ca_key -C "SSH CA Key"

Control key

As our CA resides on the Control host, we will right away create a user key and sign it:

ssh-keygen -f ssh_control_key -t ed25519 -C "Control User Key"
ssh-keygen -s ssh_ca_key -I control -n root -V +14d ssh_control_key.pub

We have just created user’s private key ssh_control_key, respective public key ssh_control_key.pub and in turn signed it by the CA creating a user certificate ssh_control_key-cert.pub.

ssh-keygen -L -f ssh_control_key-cert.pub

Target keys

We will demonstrate setting up a single Target host for connections from our Control host/user. This has to be repeated (automated) for as many targets as we wish to deploy. For the sake of convenience, consider the following script (interleaved with explanations), which assumes setting Target’s hostname or IP address into the TARGET variable:

TARGET=<host or address>

Sign host key for target

First, we will generate identity and principals (concepts explained previously) for our certificate that we will be issuing for the Target host, we can also do this manually, but running e.g. hostname   command remotely and concatenating its comma-delimited outputs for -s, -f and -I switches allow us to list the hostname, the FQDN and the IP address all as principals without any risk of typos.

IDENT=`blind-ssh root@$TARGET "hostname"`
PRINC=`blind-ssh root@$TARGET "(hostname -s; hostname -f; hostname -I) | xargs -n1 | paste -sd,"`

We will now let the remote Target itself generate its new host key (in addition to whichever it already had prior, so as not to disrupt any other parties) and copy over its public key to the control for signing by the CA.

Note that we are NOT setting any validity period on the host key, but we are free to do so as well - if we are ready to consider rotations further down the road.

blind-ssh root@$TARGET "ssh-keygen -t ed25519 -f /etc/ssh/ssh_managed_host_key"
blind-scp root@$TARGET:/etc/ssh/ssh_managed_host_key.pub .

Now with the Target’s public host key on the Control/CA host, we sign it with the affixed identity and principals as previously populated and simply copy it back over to the Target host.

ssh-keygen -s ssh_ca_key -h -I $IDENT -n $PRINC ssh_managed_host_key.pub
blind-scp ssh_managed_host_key-cert.pub root@$TARGET:/etc/ssh/

Configure target

The only thing left is to configure Target host to trust users that had their keys signed by our CA.

We will append our CA’s public key to the remote Target host’s list of (supposedly all pre-existing) trusted CAs that can sign user keys.

blind-ssh root@$TARGET "cat >> /etc/ssh/ssh_trusted_user_ca" < ssh_ca_key.pub

Still on the Target host, we create a new (single) partial configuration file which will simply point to the new host key, the corresponding certificate and the trusted user CA’s key record:

blind-ssh root@$TARGET "cat > /etc/ssh/sshd_config.d/pki.conf" << EOF
HostKey /etc/ssh/ssh_managed_host_key
HostCertificate /etc/ssh/ssh_managed_host_key-cert.pub
TrustedUserCAKeys /etc/ssh/ssh_trusted_user_ca
EOF

All that is left to do is to apply the new setup by reloading the SSH daemon:

blind-ssh root@$TARGET "systemctl reload-or-restart sshd"

First connection

There is a one-off setup of Control configuration needed first (and only once) - we set our Control user to recognise Target host keys when signed by our CA:

cat >> ~/.ssh/known_hosts <<< "@cert-authority * `cat ssh_ca_key.pub`"

We could now test our first connection with the previously signed user key, without being in the blind:

ssh -i ssh_control_key -v root@$TARGET

And we should be right in, no prompts about unknown hosts, no passwords. But for some more convenience, we should really make use of client configuration.

First, let’s move the user key and certificate into the usual directory - as we are still in the staging one:

mv ssh_control_key* ~/.ssh/

Now the full configuration for host which we will simply alias as h1:

cat >> ~/.ssh/config << EOF
Host t1
    HostName $TARGET
    User root
    Port 22
    IdentityFile ~/.ssh/ssh_control_key
    CertificateFile ~/.ssh/ssh_control_key-cert.pub
EOF

From now on, our connections are as simple as:

ssh t1

Rotation

If you paid attention, we used an example of generating user key signed only for a specified period, after which it would be failing. It is very straightforward to simply generate a new one any time and sign it without having to change anything further on the targets anymore - especially on our model setup where CA is on the Control host.

If you wish to also rotate Target host key, while more elaborate, this is now trivial - the above steps for the Target setup specifically (combined into a single script) will serve just that purpose.

Troubleshooting

If troubleshooting, the client ssh from the Control host can be invoked with multiple -v, e.g. -vvv for more detailed output which will produce additional debug lines prepended with debug and numberical designation of the level. On a successful certificate based connection, both user and host, we would want to see some of the following:

debug3: record_hostkey: found ca key type ED25519 in file /root/.ssh/known_hosts:1
debug3: load_hostkeys_file: loaded 1 keys from 10.10.10.10
debug1: Server host certificate: ssh-ed25519-cert-v01@openssh.com SHA256:JfMaLJE0AziLPRGnfC75EiL4pxwFNmDWpWT6KiDikQw, serial 0 ID "pve" CA ssh-ed25519 SHA256:sJvDprmv3JQ2n+9OeqnvIdQayrFFlxX8/RtzKhBKXe0 valid forever
debug2: Server host certificate hostname: pve
debug2: Server host certificate hostname: pve.lab.internal
debug2: Server host certificate hostname: 10.10.10.10
debug1: Host '10.10.10.10' is known and matches the ED25519-CERT host certificate.

debug1: Will attempt key: ssh_control_key ED25519-CERT SHA256:mDucgr+IrmNYIT/4eEIVjVNnN0lApBVdDgYrVDqyrKY explicit
debug1: Offering public key: ssh_control_key ED25519-CERT SHA256:mDucgr+IrmNYIT/4eEIVjVNnN0lApBVdDgYrVDqyrKY explicit
debug1: Server accepts key: ssh_control_key ED25519-CERT SHA256:mDucgr+IrmNYIT/4eEIVjVNnN0lApBVdDgYrVDqyrKY explicit

In case of need, the Target (server-side) log can be checked with journalctl -u ssh, or alternatively journalctl -t sshd.

Final touch

One of the last pieces of advice for any well set up system would be to eventually prevent root SSH connections altogether, even with key, even with a signed one - there is the PermitRootLogin   that can be set to no. This would, however cause Proxmox VE to fail. The second best option is to prevent root connections with a password, i.e. only allowing a key. This is covered by the value prohibit-password that comes with stock Debian (but NOT Proxmox VE) install, however - be aware of the remaining bug that could cause you getting cut off with passwordless root before doing so.