Manage several SSH keys for different servers

Keep separate SSH keys for different servers, clients or devices, tell SSH which key to use per host with IdentityFile and IdentitiesOnly, name keys clearly, rotate them and remove old keys from authorized_keys.

  • Intermediate
  • Linux VPS
  • 5 min read

By RunRacks Team · Published

One SSH key for everything is simple, but it means one lost laptop exposes every server you manage. Separate keys per client, project or device limit the damage and make it easy to revoke access precisely. The cost is a little organisation. This guide shows how to name keys, tell SSH which one to use for each server, and rotate or remove keys without locking yourself out.

Before you start

  • The OpenSSH client on Linux, macOS or Windows.
  • Working SSH access to your servers with at least one key. See Set up SSH keys if you are starting from scratch.
  • An SSH config file. Use the SSH config file explains the basics.

Decide how to split your keys

Common approaches:

  • One key per device: a laptop key, a desktop key, a phone key. If a device is lost, you remove only its key.
  • One key per client or project: keeps access to different customers separate, which is often a contractual requirement.
  • Separate keys for automation: CI pipelines and backup jobs get their own keys, ideally restricted on the server side.

Many people combine the first two. Avoid one key per server; it adds work without much extra safety.

Name keys clearly

Use the file name and the comment to say what a key is for. The comment appears at the end of the public key and in authorized_keys, which makes cleanup easy later:

Terminal
ssh-keygen -t ed25519 -f ~/.ssh/clientA_ed25519 -C "deploy@laptop clientA 2026-11"
ssh-keygen -t ed25519 -f ~/.ssh/personal_ed25519 -C "me@laptop personal 2026-11"

-f sets the file name and -C the comment. Set a passphrase for each one.

Tell SSH which key to use per host

Map each server to its key in ~/.ssh/config:

Text
Host clientA-web
    HostName 203.0.113.10
    User deploy
    IdentityFile ~/.ssh/clientA_ed25519
    IdentitiesOnly yes

Host personal-vps
    HostName 198.51.100.20
    User admin
    IdentityFile ~/.ssh/personal_ed25519
    IdentitiesOnly yes

IdentitiesOnly yes matters here. Without it, SSH also offers every key loaded in your agent, in no particular order. A server that sees too many wrong keys disconnects with Too many authentication failures, and you may also reveal to one client's server which other keys you hold.

For a one-off connection without a config entry:

Terminal
ssh -i ~/.ssh/clientA_ed25519 -o IdentitiesOnly=yes [email protected]

Install a key on a server

Copy the matching public key with ssh-copy-id (Linux and macOS):

Terminal
ssh-copy-id -i ~/.ssh/clientA_ed25519.pub [email protected]

Or append it by hand on the server, one key per line, in ~/.ssh/authorized_keys of the user you log in as. The file should be mode 600 and the ~/.ssh folder mode 700.

See which keys a server accepts

On the server, list the authorized keys with their fingerprints and comments:

Terminal
ssh-keygen -lf ~/.ssh/authorized_keys
Text
256 SHA256:Xk3... deploy@laptop clientA 2026-11 (ED25519)
256 SHA256:Pq9... old-desktop 2023 (ED25519)

On your computer, print the fingerprint of a local key to match it:

Terminal
ssh-keygen -lf ~/.ssh/clientA_ed25519.pub

Repeat for every user that has an authorized_keys file, including root:

Terminal
sudo ls -l /root/.ssh/authorized_keys /home/*/.ssh/authorized_keys

Remove an old key

  1. Make sure you have a different working key, and keep a session open.
  2. Back up the file:
Terminal
   cp ~/.ssh/authorized_keys ~/.ssh/authorized_keys.bak
  1. Open it in an editor and delete the line for the old key. The comment at the end of the line tells you which one it is:
Terminal
   nano ~/.ssh/authorized_keys
  1. From a new terminal, confirm you can still log in.

Rotate keys

Rotation means replacing a key with a new one, for example yearly, when someone leaves a team, or when a device may be compromised:

  1. Generate a new key with a dated comment.
  2. Add the new public key to every server that trusted the old one.
  3. Update IdentityFile in your SSH config and test each host.
  4. Remove the old public key from every server.
  5. Delete the old private key from your computer and from any agent with ssh-add -d ~/.ssh/old_key.

Keep a simple list of which servers each key is installed on; it makes steps 2 and 4 quick.

Check it works

For each host, confirm the expected key is offered and accepted:

Terminal
ssh -v clientA-web exit 2>&1 | grep -E "Offering|Authenticated"

You should see only the intended key offered, followed by an Authenticated line.

Troubleshooting

  • `Too many authentication failures`: add IdentitiesOnly yes to the host block, or remove unused keys from your agent with ssh-add -D. See Fix SSH too many authentication failures.
  • `Permission denied (publickey)` after rotation: the new public key is missing on that server, or your config still points to the old file.
  • SSH ignores your IdentityFile: a Host * block earlier in the config set a different key first. Move general blocks to the end.
  • `UNPROTECTED PRIVATE KEY FILE!`: run chmod 600 on the private key.

Next steps

  • Compare plans for your next server on Linux VPS.

Was this guide helpful?

Still stuck?

Ask RunRacks AI for a quick answer, or open a ticket and our team will take a look at your server.

Linux VPS

Deploy a Node.js app with PM2 behind Nginx

Install a Node.js LTS release, keep your app running with PM2 across crashes and reboots, and serve it through Nginx with HTTPS.

  • Intermediate
  • Linux VPS
  • 3 min read

Linux VPS

Create an SFTP-only user limited to one folder

Give someone file access over SFTP without a shell, locked inside one folder. Configure internal-sftp with a Match block, set the chroot ownership right and test with WinSCP.

  • Advanced
  • Linux VPS
  • 5 min read