Advanced SSH key management for Linux power users

Introduction

Secure Shell (SSH) keys form the backbone of secure remote system access for Linux power users. While the basic concepts of SSH key pairs are widely understood, implementing robust key management strategies often receives less attention than it deserves. This guide explores comprehensive approaches to SSH key management that go beyond the basics, helping you maintain security while enhancing productivity.

Whether you're managing dozens of servers or collaborating on multiple projects, proper SSH key management is crucial for maintaining security without sacrificing convenience. We'll examine various techniques that balance security requirements with practical usability.

Key generation strategies

Modern key types and specifications

The security of your SSH connections begins with proper key generation. While RSA has been the traditional standard, more modern options offer better security profiles:

# Ed25519 - Modern, secure and recommended
ssh-keygen -t ed25519 -a 100 -C "work_laptop_$(date +%Y-%m)"

# RSA with 4096 bits - Compatible with older systems
ssh-keygen -t rsa -b 4096 -a 100 -C "personal_server_$(date +%Y-%m)"

# ECDSA - Elliptic curve option
ssh-keygen -t ecdsa -b 521 -C "project_x_access"

The -a flag increases the key derivation function rounds, making brute-force attacks more difficult. The comment flag (-C) helps identify keys - I've found that including both purpose and date in comments is invaluable when reviewing keys months or years later.

Purpose-specific keys

Rather than using a single key pair for all connections, consider generating purpose-specific keys:

• Per-host keys (highest security)
• Per-project keys (good balance)
• Per-client/workplace keys (practical compromise)

This isolation strategy limits the blast radius of potential compromises while making key rotation more manageable.

Organized storage and structure

Directory organization

As your collection of keys grows, organization becomes essential. Consider this structured approach:

~/.ssh/
├── config               # Main configuration file
├── keys/                # Directory for private keys
│   ├── personal/        # Personal project keys
│   ├── work/            # Work-related keys
│   └── clients/         # Client-specific keys
├── authorized_keys      # Public keys for incoming connections
├── known_hosts          # Verified host fingerprints
└── config.d/            # Modular configuration
    ├── personal.conf
    ├── work.conf
    └── clients.conf

This structure separates keys by context, making them easier to manage, back up, and rotate.

Modular SSH configuration

The SSH config file becomes unwieldy as you add more hosts and special configurations. Break it into manageable chunks with the Include directive:

# ~/.ssh/config
Include config.d/*.conf

# Global defaults
Host *
    ServerAliveInterval 60
    IdentitiesOnly yes
    AddKeysToAgent yes

Each included file can then contain context-specific configurations:

# ~/.ssh/config.d/work.conf
Host workserver
    HostName work-server.example.com
    User devops
    Port 2222
    IdentityFile ~/.ssh/keys/work/primary
    ForwardAgent no

This approach makes configuration more maintainable and enables quickly toggling entire sets of configurations by renaming files.

Security enhancements

Key passphrase management

Unprotected private keys pose a significant security risk. Always use strong passphrases, and consider these tools to reduce the friction:

1. SSH Agent: Cache keys temporarily during your session

   eval "$(ssh-agent)"
   ssh-add ~/.ssh/keys/work/primary
   

2. Keychain: Persistent agent across logins

   keychain --quiet ~/.ssh/keys/work/primary
   source ~/.keychain/$HOSTNAME-sh
   

3. GNOME Keyring / KDE Wallet: Desktop integration

For servers where you need to automate connections without passphrases (like in scripts), use restricted keys with forced commands:

command="backup-script.sh",no-port-forwarding,no-X11-forwarding,no-agent-forwarding ssh-ed25519 AAAA...

Key rotation policies

Implementing regular key rotation significantly reduces risk. Develop a consistent routine:

1. Generate new key pairs
2. Distribute new public keys to servers
3. Test access with new keys
4. Remove old authorized keys
5. Archive or destroy old private keys

Automation tools can streamline this process significantly, as we'll discuss later.

SSH certificates

For environments with many hosts and users, SSH certificates provide substantial management benefits over traditional authorized_keys files.

Setting up a certificate authority

# Create a CA key
ssh-keygen -t ed25519 -a 100 -f ssh_ca -C "SSH Certificate Authority"

# Sign a user key
ssh-keygen -s ssh_ca -I "user@example.com" -n "user,admin" -V +52w user_key.pub

Benefits of certificate-based authentication include:

• Centralized management
• Built-in expiration
• Specific principal (username) restrictions
• No need to distribute authorized_keys to every host

The main configuration needed on servers is:

TrustedUserCAKeys /etc/ssh/ca.pub

This approach scales remarkably well in large environments and provides significantly better security properties than traditional key management.

Advanced tools and automation

SSH key management tools

Several specialized tools can enhance your key management workflow:

• ssh-audit: Analyze server SSH configurations
• ssh-copy-id: Simplify adding keys to remote hosts
• ssh-keyscan: Gather host public keys
• Vault by HashiCorp: Securely store and dynamically issue SSH credentials

Ansible for key distribution

Ansible excels at managing SSH keys across multiple systems:

- name: Set up authorized keys
  ansible.posix.authorized_key:
    user: "{{ item.user }}"
    key: "{{ lookup('file', item.key_file) }}"
    state: present
    exclusive: "{{ item.exclusive | default(false) }}"
  loop: "{{ ssh_keys }}"

Combined with Ansible Vault for storing sensitive information, this provides a secure and auditable way to manage keys across your infrastructure.

Git hooks for key verification

Implement client-side Git hooks to ensure you're not accidentally committing private keys:

#!/bin/bash
# .git/hooks/pre-commit

if git diff --cached --name-only | grep -E "id_(rsa|ed25519|ecdsa)|\.pem$"; then
  echo "WARNING: Possible private key being committed!"
  exit 1
fi

I've found this simple precaution has saved me more than once from potentially disastrous key exposure.

SSH agent forwarding and alternatives

Agent forwarding allows you to use local keys when connecting from one remote host to another. While convenient, it poses security risks if the intermediate host is compromised.

Safer alternatives

1. ProxyJump: Connect through jump hosts without exposing keys

   Host target-server
       HostName 10.0.1.2
       ProxyJump jumphost.example.com
   

2. SSH Socket Control: Reuse connections for performance

   Host *
       ControlMaster auto
       ControlPath ~/.ssh/control/%r@%h:%p
       ControlPersist 10m
   

3. sshuttle: Transparent proxy that works like a VPN

Each approach has different security implications; choose based on your specific requirements and threat model.

Troubleshooting and maintenance

Regular maintenance prevents future headaches. Implement these practices:

1. Audit key access regularly with grep or journalctl
2. Check for outdated or unused keys
3. Verify file permissions (chmod 600 for private keys)
4. Test remote access after changes

Common issues and solutions:

# Debug connection problems
ssh -vvv hostname

# Check key permissions
find ~/.ssh -type f -name "id_*" ! -perm 0600

# List fingerprints of all keys
for key in ~/.ssh/keys/**/id_*; do ssh-keygen -l -f "$key"; done

Conclusion

Effective SSH key management is an ongoing process that balances security with usability. By implementing purpose-specific keys, organized storage, regular rotation, and appropriate automation, you can maintain robust security while streamlining your workflow.

Consider starting with one aspect of the strategies outlined here and gradually incorporating others as they fit your workflow. Even small improvements to your SSH key management can yield significant security and productivity benefits over time.

For environments with many users and hosts, seriously consider implementing SSH certificates, as they provide the most scalable and secure approach to key management in complex environments.

Remember that the ultimate goal is to maintain the security of your systems without creating so much friction that you're tempted to work around your own safeguards.