Manage several SSH keys for different servers

One SSH key for everything is simple, but it means one lost laptop exposes every server you manage. Separate keys per client, project or device limit the damage and make it easy to revoke access precisely. The cost is a little organisation. This guide shows how to name keys, tell SSH which one to use for each server, and rotate or remove keys without locking yourself out.

Before you start

• The OpenSSH client on Linux, macOS or Windows.
• Working SSH access to your servers with at least one key. See Set up SSH keys if you are starting from scratch.
• An SSH config file. Use the SSH config file explains the basics.

Decide how to split your keys

Common approaches:

• One key per device: a laptop key, a desktop key, a phone key. If a device is lost, you remove only its key.
• One key per client or project: keeps access to different customers separate, which is often a contractual requirement.
• Separate keys for automation: CI pipelines and backup jobs get their own keys, ideally restricted on the server side.

Many people combine the first two. Avoid one key per server; it adds work without much extra safety.

Name keys clearly

Use the file name and the comment to say what a key is for. The comment appears at the end of the public key and in authorized_keys, which makes cleanup easy later:

ssh-keygen -t ed25519 -f ~/.ssh/clientA_ed25519 -C "deploy@laptop clientA 2026-11"
ssh-keygen -t ed25519 -f ~/.ssh/personal_ed25519 -C "me@laptop personal 2026-11"

-f sets the file name and -C the comment. Set a passphrase for each one.

Tell SSH which key to use per host

Map each server to its key in ~/.ssh/config:

Host clientA-web
    HostName 203.0.113.10
    User deploy
    IdentityFile ~/.ssh/clientA_ed25519
    IdentitiesOnly yes

Host personal-vps
    HostName 198.51.100.20
    User admin
    IdentityFile ~/.ssh/personal_ed25519
    IdentitiesOnly yes

IdentitiesOnly yes matters here. Without it, SSH also offers every key loaded in your agent, in no particular order. A server that sees too many wrong keys disconnects with Too many authentication failures, and you may also reveal to one client's server which other keys you hold.

For a one-off connection without a config entry:

ssh -i ~/.ssh/clientA_ed25519 -o IdentitiesOnly=yes deploy@203.0.113.10

Install a key on a server

Copy the matching public key with ssh-copy-id (Linux and macOS):

ssh-copy-id -i ~/.ssh/clientA_ed25519.pub deploy@203.0.113.10

Or append it by hand on the server, one key per line, in ~/.ssh/authorized_keys of the user you log in as. The file should be mode 600 and the ~/.ssh folder mode 700.

See which keys a server accepts

On the server, list the authorized keys with their fingerprints and comments:

ssh-keygen -lf ~/.ssh/authorized_keys

256 SHA256:Xk3... deploy@laptop clientA 2026-11 (ED25519)
256 SHA256:Pq9... old-desktop 2023 (ED25519)

On your computer, print the fingerprint of a local key to match it:

ssh-keygen -lf ~/.ssh/clientA_ed25519.pub

Repeat for every user that has an authorized_keys file, including root:

sudo ls -l /root/.ssh/authorized_keys /home/*/.ssh/authorized_keys

Remove an old key

1. Make sure you have a different working key, and keep a session open.
2. Back up the file:

   cp ~/.ssh/authorized_keys ~/.ssh/authorized_keys.bak

1. Open it in an editor and delete the line for the old key. The comment at the end of the line tells you which one it is:

   nano ~/.ssh/authorized_keys

1. From a new terminal, confirm you can still log in.

Rotate keys

Rotation means replacing a key with a new one, for example yearly, when someone leaves a team, or when a device may be compromised:

1. Generate a new key with a dated comment.
2. Add the new public key to every server that trusted the old one.
3. Update IdentityFile in your SSH config and test each host.
4. Remove the old public key from every server.
5. Delete the old private key from your computer and from any agent with ssh-add -d ~/.ssh/old_key.

Keep a simple list of which servers each key is installed on; it makes steps 2 and 4 quick.

Check it works

For each host, confirm the expected key is offered and accepted:

ssh -v clientA-web exit 2>&1 | grep -E "Offering|Authenticated"

You should see only the intended key offered, followed by an Authenticated line.

Troubleshooting

• Too many authentication failures: add IdentitiesOnly yes to the host block, or remove unused keys from your agent with ssh-add -D. See Fix SSH too many authentication failures.
• Permission denied (publickey) after rotation: the new public key is missing on that server, or your config still points to the old file.
• SSH ignores your IdentityFile: a Host * block earlier in the config set a different key first. Move general blocks to the end.
• UNPROTECTED PRIVATE KEY FILE!: run chmod 600 on the private key.