Manage several SSH keys for different servers
Keep separate SSH keys for different servers, clients or devices, tell SSH which key to use per host with IdentityFile and IdentitiesOnly, name keys clearly, rotate them and remove old keys from authorized_keys.
- Intermediate
- Linux VPS
- 5 min read
One SSH key for everything is simple, but it means one lost laptop exposes every server you manage. Separate keys per client, project or device limit the damage and make it easy to revoke access precisely. The cost is a little organisation. This guide shows how to name keys, tell SSH which one to use for each server, and rotate or remove keys without locking yourself out.
Before you start
- The OpenSSH client on Linux, macOS or Windows.
- Working SSH access to your servers with at least one key. See Set up SSH keys if you are starting from scratch.
- An SSH config file. Use the SSH config file explains the basics.
Decide how to split your keys
Common approaches:
- One key per device: a laptop key, a desktop key, a phone key. If a device is lost, you remove only its key.
- One key per client or project: keeps access to different customers separate, which is often a contractual requirement.
- Separate keys for automation: CI pipelines and backup jobs get their own keys, ideally restricted on the server side.
Many people combine the first two. Avoid one key per server; it adds work without much extra safety.
Name keys clearly
Use the file name and the comment to say what a key is for. The comment appears at the end of the public key and in authorized_keys, which makes cleanup easy later:
ssh-keygen -t ed25519 -f ~/.ssh/clientA_ed25519 -C "deploy@laptop clientA 2026-11"
ssh-keygen -t ed25519 -f ~/.ssh/personal_ed25519 -C "me@laptop personal 2026-11"-f sets the file name and -C the comment. Set a passphrase for each one.
Tell SSH which key to use per host
Map each server to its key in ~/.ssh/config:
Host clientA-web
HostName 203.0.113.10
User deploy
IdentityFile ~/.ssh/clientA_ed25519
IdentitiesOnly yes
Host personal-vps
HostName 198.51.100.20
User admin
IdentityFile ~/.ssh/personal_ed25519
IdentitiesOnly yesIdentitiesOnly yes matters here. Without it, SSH also offers every key loaded in your agent, in no particular order. A server that sees too many wrong keys disconnects with Too many authentication failures, and you may also reveal to one client's server which other keys you hold.
For a one-off connection without a config entry:
ssh -i ~/.ssh/clientA_ed25519 -o IdentitiesOnly=yes [email protected]Install a key on a server
Copy the matching public key with ssh-copy-id (Linux and macOS):
ssh-copy-id -i ~/.ssh/clientA_ed25519.pub [email protected]Or append it by hand on the server, one key per line, in ~/.ssh/authorized_keys of the user you log in as. The file should be mode 600 and the ~/.ssh folder mode 700.
See which keys a server accepts
On the server, list the authorized keys with their fingerprints and comments:
ssh-keygen -lf ~/.ssh/authorized_keys256 SHA256:Xk3... deploy@laptop clientA 2026-11 (ED25519)
256 SHA256:Pq9... old-desktop 2023 (ED25519)On your computer, print the fingerprint of a local key to match it:
ssh-keygen -lf ~/.ssh/clientA_ed25519.pubRepeat for every user that has an authorized_keys file, including root:
sudo ls -l /root/.ssh/authorized_keys /home/*/.ssh/authorized_keysRemove an old key
- Make sure you have a different working key, and keep a session open.
- Back up the file:
cp ~/.ssh/authorized_keys ~/.ssh/authorized_keys.bak- Open it in an editor and delete the line for the old key. The comment at the end of the line tells you which one it is:
nano ~/.ssh/authorized_keys- From a new terminal, confirm you can still log in.
Deleting the only key you can log in with locks you out of SSH. Keep your current session open until a new login succeeds. If it does happen, use the browser console from the control panel in your Dashboard to restore the backup.
Rotate keys
Rotation means replacing a key with a new one, for example yearly, when someone leaves a team, or when a device may be compromised:
- Generate a new key with a dated comment.
- Add the new public key to every server that trusted the old one.
- Update
IdentityFilein your SSH config and test each host. - Remove the old public key from every server.
- Delete the old private key from your computer and from any agent with
ssh-add -d ~/.ssh/old_key.
Keep a simple list of which servers each key is installed on; it makes steps 2 and 4 quick.
Check it works
For each host, confirm the expected key is offered and accepted:
ssh -v clientA-web exit 2>&1 | grep -E "Offering|Authenticated"You should see only the intended key offered, followed by an Authenticated line.
Troubleshooting
- `Too many authentication failures`: add
IdentitiesOnly yesto the host block, or remove unused keys from your agent withssh-add -D. See Fix SSH too many authentication failures. - `Permission denied (publickey)` after rotation: the new public key is missing on that server, or your config still points to the old file.
- SSH ignores your IdentityFile: a
Host *block earlier in the config set a different key first. Move general blocks to the end. - `UNPROTECTED PRIVATE KEY FILE!`: run
chmod 600on the private key.
Next steps
- Compare plans for your next server on Linux VPS.
Was this guide helpful?
Still stuck?
Ask RunRacks AI for a quick answer, or open a ticket and our team will take a look at your server.
Related guides
Linux VPS
Convert keys between PuTTY (.ppk) and OpenSSH formats
Use PuTTYgen to convert a PuTTY .ppk private key to OpenSSH format and back, understand PPK version 2 and 3, generate a new Ed25519 key and load it in PuTTY.
- Intermediate
- Linux VPS
- 5 min read
Linux VPS
Deploy a Node.js app with PM2 behind Nginx
Install a Node.js LTS release, keep your app running with PM2 across crashes and reboots, and serve it through Nginx with HTTPS.
- Intermediate
- Linux VPS
- 3 min read
Linux VPS
Create an SFTP-only user limited to one folder
Give someone file access over SFTP without a shell, locked inside one folder. Configure internal-sftp with a Match block, set the chroot ownership right and test with WinSCP.
- Advanced
- Linux VPS
- 5 min read