If you have a SSH server, you’ve probably heard that you should be using SSH key authentication instead of passwords. There’s a lot of benefits to key authentication: better experience, smooth login, ability to log in via scripts, service accounts become possible, and keys are significantly harder to crack than a password. But if you deal with a lot of servers, or you manage a lot of users, or both, there’s a better way using SSH certificates instead of simple keys.
When using simple key authentication, any user can generate their own SSH keys
and add the public portion to their authorized_keys
file if you haven’t taken
care to restrict access to it (or if they have broad sudo
/doas
privileges).
Depending on what other protections you have in place, this can mean users could
access servers from devices they shouldn’t be allowed to use. In an enterprise
environment, for example, you may have Internet-exposed SSH servers you want to
restrict to access from managed devices only. Or you may want to allow some
users to log in as either themselves or as other users on the system, which
requires copying the public portion of a key to multiple authorized_keys
files. These keys also don’t expire, and need to be manually removed when a user
should no longer have access.
You may also want to give users some way to trust that they’re connecting to a trusted server without making them compare fingerprints. Normally, you would need to distribute a list of valid server fingerprints to everyone, and make sure they get an updated list every time a new server is added, or an existing server is reinstalled.
The solution to both problems is an SSH certificate. Functionally similar to a TLS certificate, SSH certificates can be used for both identifying a server to a user and identifying a user to a server. In this post, we’ll explore how to set up your SSH Certificate Authority (CA), how to issue certificates for both users and hosts, and how to revoke SSH certificates when needed.
Creating your SSH CA Link to heading
The first thing you need to do is create your Certificate Authority keys. Unlike TLS CAs, SSH CAs don’t have the concept of intermediate issuers because you don’t actually have a CA certificat, you only have SSH keys trusted as signing keys. This means your SSH CA is your root CA and can’t be kept offline. If possible, the private key should always be kept in secure hardware of some kind, such as a Hardware Security Module (HSM) or local TPM. If you do, you’ll need to first create a private key on the secure storage using the PKCS#11 interface.
Create a key in Secure Hardware Link to heading
First you need to know the PKCS#11 URI of the secure hardware module you want to
use. You can find this with the p11tool
command, part of the gnutls-utils
package on CentOS and similar distributions:
$ p11tool --list-tokens
Token 0:
URL: pkcs11:model=SuperHSM;manufacturer=SuperSoft;serial=883197678b507ca7;token=puffy-hsm
Label: puffy-hsm
Type: Super Secure HSM
Flags: RNG, Requires login
Manufacturer: SuperSoft
Model: SuperHSM
Serial: 883197678b507ca7
Module: /usr/lib64/pkcs11/libsuperhsm2.so
Make note of the URL and the Module, you will need both of these later.
Now generate a private key for your SSH CA. Since you’re storing the private key in secure hardware you could use one key for all your CAs, or you can use one key per CA.
$ p11tool --generate-privkey=ecdsa --sec-param=high --login --label ssh-user-ca 'pkcs11:model=SuperHSM;manufacturer=SuperSoft;serial=883197678b507ca7;token=puffy-hsm'
warning: no --outfile was specified and the generated public key will be printed on screen.
Generating an EC/ECDSA key...
Token 'puffy-hsm' with URL 'pkcs11:model=SuperHSM;manufacturer=SuperSoft;serial=883197678b507ca7;token=puffy-hsm' requires user PIN
Enter PIN:
-----BEGIN PUBLIC KEY-----
MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEmbX149h8eCyhg3Sx1DBPVjdW0jP+
gCeB33mr/I2vv9EOgr1Ec/lEablY2D/PR+AbZ7wII5aMAp9+9vdiaTVTxw==
-----END PUBLIC KEY-----
This is not the public key you want to save! You need the public key in the
standard SSH format. To get that, run ssh-keygen -D libsuperhsm2.so -e >ssh-ca.pub
.
You will use this later when issuing SSH certificates.
Create a key on disk Link to heading
If you don’t have secure hardware available, you will need to store your private keys on disk. These will allow anyone with access to create new certificates trusted by all your devices, so be sure to appropriately protect them!
Creating a private key on disk is a matter of a single command:
$ ssh-keygen -t ecdsa -b 521 -f ssh-ca -C 'SSH CA Certificate'
Generating public/private ecdsa key pair.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in ssh-ca.
Your public key has been saved in ssh-ca.pub.
The key fingerprint is:
SHA256:B6Sms0weOZLIOz/TRV3xZj/vxznRxfC7eVyGY9y+5/c SSH CA Certificate
The key's randomart image is:
+---[ECDSA 521]---+
| . .. |
| o .. . |
| o o . + + |
|.. . + . o o . +|
|..o B . S . . =+|
| .= = . . =oB|
| o = . . *B|
| oo . *O|
| .o oE|
+----[SHA256]-----+
This will create two files: ssh-ca
(the private key) and ssh-ca.pub
(the
public key). You should create one set of keys for each SSH CA you intend to use;
this usually means creating two sets of keys, one for host certificates and one
for user certificates.
Signing a Host Certificate Link to heading
Now that you have your CA keys created, you’re ready to issue SSH certificates.
The first type we’ll look at is a host certificate, which identifies a SSH
server to a connecting client. All you need is your SSH server’s host public key,
stored in /etc/ssh/ssh_host_*_key.pub
, and the keys generated earlier.
Signing with PKCS#11 Link to heading
To sign with a private key stored in secure hardware, use the -D
option to
ssh-keygen
:
$ ssh-keygen -s ssh-ca.pub -D libsuperhsm2.so -I dev1234.example.com -h -n dev1234.example.com,dev1234,10.20.0.134 -V +52w -z $(date +%s) ./ssh_host_ecdsa_key.pub
Enter PIN for 'puffy-hsm':
Signed host key ./ssh_host_ecdsa_key-cert.pub: id "dev1234.example.com" serial 1670459268 for dev1234.example.com,dev1234,10.20.0.134 valid from 2022-12-05T22:45:00 to 2023-12-04T22:46:02
Let’s break that command down a bit. Each option means:
-s ssh-ca.pub
: Usessh-ca.pub
as the SSH public key-D libsuperhsm2.so
: Look for the private key using the PKCS#11 librarylibsuperhsm2.so
-I dev1234.example.com
: The certificate’s identity. This can be any alphanumeric string, but it’s typically best to use the server’s hostname.-h
: Create a host certificate.-n dev1234.example.com,dev1234,10.20.0.134
: A comma-separated list of principals the certificate is valid for. For a host certificate, these are all the hostnames or IP addresses valid for connecting to the server.-V +52w
: The certificte is valid for the specified time, in this case for the next 52 weeks.-z $(date +%s)
: Set the certificate serial number.date +%s
prints the current time as seconds since the UNIX epoch, which causes the certificate serial number to be set to the current time. You don’t need this option, but it can sometimes be helpful to have a serial number defined. If you don’t set a serial number, 0 will be used../ssh_host_ecdsa_key.pub
: The path to the host public key to sign.
Signing with on-disk keys Link to heading
To sign with a private key stored on disk, leave out the -D
option and pass
the private key to -s
:
$ ssh-keygen -s ssh-ca -I dev1234.example.com -h -n dev1234.example.com,dev1234,10.20.0.134 -V +52w -z $(date +%s) ./ssh_host_ecdsa_key.pub
Signed host key ./ssh_host_ecdsa_key-cert.pub: id "dev1234.example.com" serial 1670459268 for dev1234.example.com,dev1234,10.20.0.134 valid from 2022-12-05T22:45:00 to 2023-12-04T22:46:02
Using the host certificate Link to heading
Once you’ve generated a host SSH certificate, you need to tell sshd
to use it.
Add this line to /etc/ssh/sshd_config
and restart sshd
:
HostCertificate /etc/ssh/ssh_host_ecdsa_key-cert.pub
Now your SSH server is configured to present a host certificate, so the last
step is to tell clients to trust the certificate. Start by generating the
known_hosts
format file:
$ printf '@cert-authority *.example.com %s' "$(cat ssh-ca.pub)" >ssh_known_hosts
Copy ssh_known_hosts
to clients who should trust the SSH host certificates as
/etc/ssh/ssh_known_hosts
. After that, you can remove any entries in the user’s
known_hosts
file for the server; they will automatically trust it based on the
SSH host certificate. And when you eventually reinstall a host, as long as you
sign a new SSH host certificate for the host users won’t see the dreaded
“WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!” message.
Signing a User Certificate Link to heading
Once you have SSH host certificates deployed, the next step is to use SSH user certificates to identify users to servers. The commands to do so are similar to host certificates. You will also need the public portion of the user’s SSH key.
Signing with PKCS#11 Link to heading
To sign a user certificate with a key stored in secure hardware, we once again
use the -D
option to ssh-keygen
:
$ ssh-keygen -s ssh-ca.pub -D libsuperhsm2.so -I jgoguen@example.com -n jgoguen,deploybot -V +2w -z $(date +%s) ./user.pub
Enter PIN for 'puffy-hsm':
Signed user key ./user-cert.pub: id "jgoguen@example.com" serial 1670459268 for jgoguen,deploybot valid from 2022-12-06T23:06:00 to 2022-12-20T23:07:00
Let’s break that command down. Each option means:
-s ssh-ca.pub
: Usessh-ca.pub
as the SSH public key-D libsuperhsm2.so
: Look for the private key using the PKCS#11 librarylibsuperhsm2.so
-I jgoguen@example.com
: The certificate’s identity. This can be any alphanumeric string, but it’s typically best to use some unique identifier for the user. Their email address, username, or userPrincipalName are all good choices.-n jgoguen,deploybot
: A comma-separated list of principals the certificate is valid for. This allows the user to log in to any host accepting the certificate as eitherjgoguen
ordeploybot
.-V +2w
: The certificte is valid for the specified time, in this case for the next 2 weeks.-z $(date +%s)
: Set the certificate serial number.date +%s
prints the current time as seconds since the UNIX epoch, which causes the certificate serial number to be set to the current time. You don’t need this option, but it can sometimes be helpful to have a serial number defined. If you don’t set a serial number, 0 will be used../jgoguen.pub
: The path to the user public key to sign.
Signing with on-disk keys Link to heading
Again, this is similar to how a host certificate is signed.
$ ssh-keygen -s ssh-ca -I jgoguen@example.com -n jgoguen,deploybot -V +2w -z $(date +%s) ./user.pub
Signed user key ./user-cert.pub: id "jgoguen@example.com" serial 1670459268 for jgoguen,deploybot valid from 2022-12-06T23:06:00 to 2022-12-20T23:07:00
Once again, the options are the same as for signing with a PKCS#11 key except
that the -s
option points to the private key to use for signing.
Using the user certificate Link to heading
To use the user certificate, first all servers must be updated to trust the user
CA signing key. Copy the contents of the CA public key (in this example,
ssh-ca.pub
) to all servers as /etc/ssh/user-ca.pub
. Then add this line to
/etc/sshd/sshd_config
on all servers that should accept SSH certificate
authentication and restart sshd
:
TrustedUserCAKeys /etc/ssh/user-ca.pub
Next, deliver the file user-cert.pub
to the user’s client device. This can be
stored in their ~/.ssh
directory. The ssh
command will expect to find three
files: user
(unless the user’s private key is stored in a PKCS#11 store),
user.pub
, and user-cert.pub
. In either /etc/ssh/ssh_config
or
~/.ssh/config
, set the IdentityFile
directive to point to the private key
file or set the PKCS11Provider
directive to point to the PKCS#11 library used
to store the private key.
Trust But Verify Link to heading
Verifying that SSH certificates are in use is as simple as watching the logs of
your SSH server. On a modern Linux distribution with systemd
, that can be done
with journalctl -f -u sshd.service
. For a successful login, you’ll see
something like this in the SSH server logs:
sshd[52436]: Accepted publickey for deploybot from 10.20.30.40 port 9923 ssh2: ECDSA-CERT ID jgoguen@example.com (serial 1670459268) CA ECDSA SHA256:tltbnMalWg+skhm+VlGLd2xHiVPozyuOPl34WypdEO0
If the user tries to log in with a different username, you’ll instead see something like this:
sshd[52436]: error: key_cert_check_authority: invalid certificate
sshd[52436]: error: Certificate invalid: name is not a listed principal
And if the certificate is expired:
sshd[52436]: error: key_cert_check_authority: invalid certificate
sshd[52436]: error: Certificate invalid: expired
Beyond The Basics Link to heading
Now that you have SSH cetificates deployed and in use, there’s a lot more you can do. Here’s a few ideas you might want to consider.
Restrict SSH options Link to heading
Maybe you don’t want to allow users to forward ports through the SSH server, or
you don’t want to allow X11 forwarding. If you check man ssh-keygen
and look
for the -O
flag, you’ll see a number of different things you can enforce in
the certificate itself. For example, if you wanted to deny port forwarding and
X11 forwarding, you would add -O no-x11-forwarding -O no-port-forwarding
to
the ssh-keygen
command used to issue a user certificate:
$ ssh-keygen -s ssh-ca.pub -D libsuperhsm2.so -I jgoguen@example.com -n jgoguen,deploybot -V +2w -z $(date +%s) -O no-x11-forwarding -O no-port-forwarding ./user.pub
You can inspect the newly-issued certificate to verify these options are now set:
$ ssh-keygen -f ./user-cert.pub -L
./user-cert.pub:
Type: ecdsa-sha2-nistp256-cert-v01@openssh.com user certificate
Public key: ECDSA-CERT SHA256:yeYf7VNJMtYm1Dr+ZqqZHV3DlPpndqFgeOfNfYWXLGk
Signing CA: ECDSA SHA256:B6Sms0weOZLIOz/TRV3xZj/vxznRxfC7eVyGY9y+5/c (using ecdsa-sha2-nistp521)
Key ID: "jgoguen@example.com"
Serial: 1670460599
Valid: from 2022-12-07T19:49:00 to 2022-12-21T19:49:59
Principals:
jgoguen
deploybot
Critical Options: (none)
Extensions:
permit-agent-forwarding
permit-pty
permit-user-rc
The permit-X11-forwarding
and permit-port-forwarding
options, which are
enabled by default, are not present, so users will not be able to forward ports
or X11 connections.
Enforce a Bastion Host Link to heading
A good way to improve the security of your SSH servers is to simply not expose
them to the Internet in the first place and require all connections to pass
through a bastion host. A bastion host is an otherwise normal SSH server, but
it’s the only SSH server exposed to the public Internet. All connections to
other SSH servers are required to originate from the bastion host, and a bastion
host should have a much more locked down configuration. Forcing all connections
through a bastion host requires only two simple Host
stanzas in
/etc/ssh/ssh_config
or ~/.ssh/config
. If using private keys stored on disk,
use these stanzas:
Host *.example.com
ProxyJump ssh-gw.example.com
IdentityFile ~/.ssh/user
Host ssh-gw.example.com
ProxyJump none
IdentityFile ~/.ssh/user
If you’re using private keys stored in secure hardware, use these stanzas instead:
Host *.example.com
ProxyJump ssh-gw.example.com
CertificateFile ~/.ssh/user-cert.pub
# For a PKCS#11 URI, only a subset of path arguments are supported by OpenSSH
IdentityFile pkcs11:object=ssh-user
Host ssh-gw.example.com
ProxyJump none
CertificateFile ~/.ssh/user-cert.pub
# For a PKCS#11 URI, only a subset of path arguments are supported by OpenSSH
IdentityFile pkcs11:object=ssh-user
If you add the SSH certificate and key to your local SSH agent, you can skip the
CertificateFile
and IdentityFile
parameters entirely.
Enforce certificate authentication Link to heading
If you previously had public key authentication enabled, using certificates
doesn’t change that. Users can still log in with their SSH private keys, and
they can add their own SSH keys to their authorized_keys
file to allow them to
log in without using their SSH certificate. Fortunately, putting a stop to this
is a one-line change. In /etc/ssh/sshd_config
make sure this line is present
then restart sshd
:
AuthorizedKeysFile none
This will prevent sshd
from checking for any authorized_keys
file anywhere,
effectively removing the ability to log in with standard SSH keys and enforcing
SSH certificates. If you want to go one step further and completely remove the
ability to log in with anything but certificates, also add this line to
/etc/ssh/sshd_config
and restart sshd
:
AuthenticationMethods publickey
If you do decide to completely remove any other authentication method, you should keep a “break glass” user account active that can log in with a password (and 2-factor authentication if possible).
Allow alternate user login on specific hosts Link to heading
The examples given so far will allow logging in as the deploybot
user on any
host where that user exists. You may want to restrict where a user can log in as
deploybot
though, such as only allowing users to log in as deploybot
on
build test hosts. To achieve this, start by removing deploybot
as a principal
on the issued certificate. Then, on every host you want to allow deploybot
logins for, add this line to /etc/ssh/sshd_config
:
AuthorizedPrincipalsFile /etc/ssh/auth_principals/auth_principals_%u
Then create the directory /etc/ssh/auth_principals
and create the file
/etc/ssh/auth_principals/auth_principals_deploybot
. In that file, add the
principals of users allowed to log in as deploybot
, one per line. Restart
sshd
when finished. Now a user with a certificate that has a principal in this
file can simply log in with ssh deploybot@buildtest1234.example.com
but ssh deploybot@dev1234.example.com
would fail.
Conclusion Link to heading
If you manage large fleets of machines, or you manage many users, SSH certificates give you a fairly easy way to improve the overall security of your servers with easier and better access control compared to plain SSH keys or passwords. One thing that isn’t covered here is how to make this scale (doing this manualy for more than just a few hosts and users will be painful), but even a simple web service properly secured and authenticated will go a long way to allowing you to scale at least to a small organization.