Structured SSH key management: beyond basic organisation

The foundation of efficient remote access

As technical professionals, we often accumulate dozens of SSH keys over time - for different servers, projects, and environments. Without a deliberate organisation strategy, this collection quickly becomes unmanageable, leading to confusion, wasted time, and potential security risks. This article explores practical approaches to SSH key organisation that scale with your growing infrastructure needs.

SSH key organisation isn't just about tidiness - it's about creating systems that enhance security, improve troubleshooting, and enable automation. By implementing thoughtful directory structures, clear naming conventions, and supporting scripts, you can transform a chaotic collection of keys into a manageable, secure authentication system.

Beyond basic folders

Hierarchical directory structure

A well-designed directory structure forms the foundation of SSH key organisation. Instead of storing all keys in a flat structure within ~/.ssh/, consider this more sophisticated approach:

~/.ssh/
├── config                # Main configuration file
├── keys/                 # All private keys
│   ├── personal/         # Personal server keys
│   │   ├── homelab/
│   │   └── vps/
│   ├── work/             # Work-related keys
│   │   ├── production/
│   │   ├── staging/
│   │   └── ci/
│   └── clients/          # Client-specific keys
│       ├── client1/
│       └── client2/
├── pubkeys/              # Corresponding public keys (same structure)
├── revoked/              # Archive of retired keys
├── authorized_keys       # Public keys for incoming connections
├── known_hosts           # Verified host fingerprints
└── config.d/             # Modular configuration
    ├── personal.conf
    ├── work.conf
    └── clients.conf

This structure provides several advantages:

• Context separation: Keys are grouped by their purpose and trust domain
• Scalability: New categories can be added without restructuring
• Backup precision: Sensitive keys can be backed up separately
• Rotation management: Makes it easier to identify keys due for rotation

File naming conventions

While directories provide high-level organisation, thoughtful file naming makes individual key identification much easier:

[purpose]_[environment]_[type]_[creation-date]

For example:

• gitlab_prod_ed25519_2024-03
• bastion_staging_rsa_2024-01
• deploy_ci_ed25519_2024-02

This naming pattern encodes critical information directly in the filename, making it immediately apparent what each key is for, when it was created, and what type of key it is.

Modular SSH configuration

As your collection of keys grows, your SSH configuration file becomes increasingly complex. Break it into manageable chunks using the Include directive:

# ~/.ssh/config
Include config.d/*.conf

# Global defaults applied to all hosts
Host *
    ServerAliveInterval 60
    IdentitiesOnly yes
    AddKeysToAgent yes

Each included file can then contain context-specific configurations:

# ~/.ssh/config.d/work.conf
Host prod-*
    User devops
    IdentityFile ~/.ssh/keys/work/production/prod_ed25519_2024-03
    ForwardAgent no

Host staging-*
    User developer
    IdentityFile ~/.ssh/keys/work/staging/staging_ed25519_2024-03
    ForwardAgent no

This approach offers several benefits:

1. Logical grouping: Settings for related hosts stay together
2. Selective enabling: Comment out Include lines to disable entire groups
3. Easier sharing: Share specific config modules without exposing everything
4. Version control: Track changes to specific configs independently

Automated organisation with helper scripts

Manual organisation inevitably breaks down over time. These helper scripts can maintain your structure:

Key generation script

#!/bin/bash
# gen-ssh-key.sh - Generate a new SSH key with proper naming and placement

# Required parameters
if [ $# -lt 3 ]; then
    echo "Usage: $0 <purpose> <environment> <type>"
    echo "Example: $0 gitlab prod ed25519"
    exit 1
fi

PURPOSE=$1
ENV=$2
TYPE=$3
DATE=$(date +%Y-%m)
FILENAME="${PURPOSE}_${ENV}_${TYPE}_${DATE}"
DIRECTORY="$HOME/.ssh/keys/work/$ENV"

# Create directory if it doesn't exist
mkdir -p "$DIRECTORY"
mkdir -p "$HOME/.ssh/pubkeys/work/$ENV"

# Generate the key
ssh-keygen -t "$TYPE" -a 100 -f "$DIRECTORY/$FILENAME" -C "$USER@$HOSTNAME $PURPOSE $ENV key ($DATE)"

# Copy the public key to the pubkeys directory for easier management
cp "$DIRECTORY/$FILENAME.pub" "$HOME/.ssh/pubkeys/work/$ENV/"

echo "Key generated at: $DIRECTORY/$FILENAME"
echo "Public key copied to: $HOME/.ssh/pubkeys/work/$ENV/$FILENAME.pub"

Key auditing script

Regular audits help identify unused or outdated keys:

#!/bin/bash
# audit-ssh-keys.sh - Check SSH key age and usage

echo "SSH Key Audit Report - $(date)"
echo "========================================"

find ~/.ssh/keys -type f -name "*_*_*_*" ! -name "*.pub" | while read -r keyfile; do
    # Extract data from filename
    filename=$(basename "$keyfile")
    IFS='_' read -r purpose env type date <<< "$filename"

    # Get creation date
    created=$(stat -c %y "$keyfile" | cut -d ' ' -f1)

    # Calculate age in days
    age=$(( ( $(date +%s) - $(date -d "$created" +%s) ) / 86400 ))

    # Check last usage from access time
    last_access=$(stat -c %x "$keyfile" | cut -d ' ' -f1)
    days_since_use=$(( ( $(date +%s) - $(date -d "$last_access" +%s) ) / 86400 ))

    # Get key fingerprint
    fingerprint=$(ssh-keygen -l -f "$keyfile" | awk '{print $2}')

    echo "Key: $filename"
    echo "  Purpose: $purpose ($env)"
    echo "  Type: $type"
    echo "  Created: $created ($age days ago)"
    echo "  Last used: $last_access ($days_since_use days ago)"
    echo "  Fingerprint: $fingerprint"

    # Flag keys based on age and usage
    if [ $age -gt 365 ]; then
        echo "  WARNING: Key is over 1 year old - consider rotation"
    fi

    if [ $days_since_use -gt 180 ]; then
        echo "  WARNING: Key unused for over 6 months - consider archiving"
    fi

    echo ""
done

Integration with password managers

For enhanced security and easier recovery, consider storing SSH key passphrases in a password manager. Most modern password managers allow storing structured data, which can mirror your SSH key organisation:

1. Create a section for SSH keys
2. Use folders matching your directory structure
3. For each key, store:
   • Key location
   • Passphrase
   • Purpose and access details
   • Expiration/rotation date

This approach ensures you:

• Never lose access due to forgotten passphrases
• Can recover from disasters with both keys and their passphrases
• Maintain an independent record of what systems each key accesses

Making organisation sustainable

Thoughtful SSH key organisation is an investment that pays significant dividends in improved security and efficiency. By implementing hierarchical directories, clear naming conventions, modular configurations, and supporting scripts, you transform a potential point of chaos into a structured system.

Start with the directory structure and naming conventions, which provide immediate benefits. Then gradually incorporate modular configurations and automation scripts as your key collection grows. Regular audits will help maintain the system and identify opportunities for improvement.

Remember that the perfect organisation system is the one you'll actually use consistently. Adapt these suggestions to match your workflow and security requirements, focusing on creating a sustainable system rather than a perfect one.