Skip to main content

Docker housekeeping: managing resources like a pro

devops
A comprehensive guide to maintaining a clean Docker environment through effective resource management, pruning strategies, and monitoring techniques.

Introduction

Docker has transformed how we develop, test, and deploy applications, but this convenience comes with a cost. Over time, Docker environments tend to accumulate unused images, containers, volumes, and networks that consume valuable disk space and potentially impact performance. Implementing proper housekeeping practices becomes essential for maintaining an efficient and trouble-free Docker environment.

This guide addresses the challenge of Docker resource management, focusing on practical techniques to identify, clean, and maintain your Docker environment. Whether you're running Docker on a development machine with limited resources or managing production servers, these practices will help you reclaim disk space, improve performance, and maintain better visibility into your Docker resources.

Prerequisites:

  • Basic familiarity with Docker concepts and commands
  • Docker installed on your system
  • Administrative access to manage Docker resources

Understanding Docker's resource usage

Before diving into cleanup strategies, it's important to understand how Docker consumes resources and where these resources accumulate over time.

Resource types in Docker

Docker's ecosystem consists of several resource types, each with different lifecycle considerations:

Images
The base building blocks containing the application code, binaries, and dependencies. Images can consume significant disk space, especially when multiple versions accumulate.
Containers
Running or stopped instances of images. Even stopped containers retain their file system changes and configuration.
Volumes
Persistent data storage areas that exist independently of containers. Volumes persist even after their associated containers are removed.
Networks
Virtual networking interfaces connecting containers. These consume fewer resources but can accumulate over time.
Build cache
Intermediate layers created during image building that speed up subsequent builds but consume disk space.

The first step in effective housekeeping is assessing your current Docker resource usage.

Assessing Docker resource utilisation

Docker provides built-in tools to analyse resource consumption across your environment.

Docker system information commands

The docker system command group provides essential information about resource usage:

# Display Docker disk usage information
docker system df

# Display detailed usage information with content breakdown
docker system df -v

# Show overall Docker information including storage driver details
docker system info

Let's examine the output of docker system df -v to understand what we're looking at:

Images space usage:
REPOSITORY                TAG      IMAGE ID       CREATED        SIZE      SHARED SIZE   UNIQUE SIZE   CONTAINERS
postgres                  13       ebd5a5681d90   4 weeks ago    373.6MB   0B            373.6MB       1
python                    3.9      a7830c6cd3c0   5 weeks ago    915.4MB   0B            915.4MB       0
redis                     latest   7614ae9453d1   2 months ago   113.2MB   0B            113.2MB       0

Containers space usage:
CONTAINER ID   IMAGE          COMMAND                  LOCAL VOLUMES   SIZE       CREATED       STATUS       NAMES
f99373433a0f   postgres:13    "docker-entrypoint.s…"   1               25.6MB     3 days ago    Up 3 days    db

Local Volumes space usage:
VOLUME NAME                                                        LINKS     SIZE
db_data                                                            1         428.2MB

Build cache usage: 0B

This detailed output provides:

  • Images and their sizes, showing which ones are actively being used
  • Containers and their writable layer sizes
  • Volumes and their disk usage
  • Build cache consumption

With this information, you can identify areas where cleanup will be most beneficial.

Essential Docker cleanup commands

Docker provides several commands to remove unused resources. These can be executed individually or combined for more comprehensive cleanup.

Pruning individual resource types

# Remove all stopped containers
docker container prune

# Remove all unused images (not referenced by any container)
docker image prune

# Remove all unused volumes (not referenced by any container)
docker volume prune

# Remove all unused networks (not connected to any container)
docker network prune

# Remove unused build cache
docker builder prune

System-wide pruning

For a more comprehensive cleanup:

# Remove stopped containers, unused networks, dangling images
docker system prune

# Also remove unused volumes (use with caution)
docker system prune --volumes

# Remove all unused images, not just dangling ones
docker system prune -a

Caution

The `--volumes` flag will remove all volumes not used by at least one container. This can lead to data loss if you have volumes containing important data that aren't currently attached to a running container.

Strategic housekeeping approaches

Rather than indiscriminately removing all unused resources, a strategic approach ensures you clean only what's truly unnecessary while preserving what's needed.

Age-based pruning

One of the most effective strategies is to remove resources based on age. Docker pruning commands support filtering by creation time:

# Remove images created more than 7 days ago
docker image prune -a --filter "until=168h"

# Remove containers that stopped more than 24 hours ago
docker container prune --filter "until=24h"

This approach ensures that only truly outdated resources are removed, maintaining a balance between cleanup and convenience.

Label-based management

Implementing a labelling strategy provides more control over resource lifecycle management:

# Create containers with environment labels
docker run --label environment=dev --name dev-app1 nginx

# Create containers with project labels
docker run --label project=website --name website-db postgres

# Prune resources by label
docker container prune --filter "label=environment=dev"
docker image prune --filter "label!=environment=production"

Labels enable selective pruning based on logical groupings rather than just age or usage status.

Image tagging strategy

Implementing a consistent tagging strategy helps maintain control over which images to keep and which to remove:

# Tag images with date information
docker tag myapp:latest myapp:$(date +%Y%m%d)

# Keep only the N most recent tags
docker image ls --format "{{.Repository}}:{{.Tag}}" | grep "myapp:" | sort -r | tail -n +6 | xargs docker rmi

By maintaining a consistent tagging convention and retention policy, you can automate the removal of outdated images while preserving important versions.

Automating Docker housekeeping

Manual cleanup is effective but inconsistent. Automating housekeeping tasks ensures regular maintenance without administrative overhead.

Scheduled cleanup with cron

Create a cleanup script and schedule it with cron:

#!/bin/bash
# docker-cleanup.sh

# Log start time
echo "Starting Docker cleanup at $(date)" >> /var/log/docker-cleanup.log

# Remove stopped containers older than 7 days
docker container prune --force --filter "until=168h"

# Remove unused images older than 30 days
docker image prune --all --force --filter "until=720h"

# Remove unused networks
docker network prune --force

# Remove build cache older than 7 days
docker builder prune --all --filter "until=168h" --force

# Log completion
echo "Docker cleanup completed at $(date)" >> /var/log/docker-cleanup.log

Schedule this script to run weekly:

# Edit crontab
crontab -e

# Add this line to run at 1am every Sunday
0 1 * * 0 /path/to/docker-cleanup.sh

Using Docker's built-in pruning options

When running Docker in daemon mode, you can configure automatic pruning:

# Edit Docker daemon configuration
sudo nano /etc/docker/daemon.json

# Add pruning configuration
{
  "storage-driver": "overlay2",
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "10m",
    "max-file": "3"
  },
  "builder": {
    "gc": {
      "enabled": true,
      "defaultKeepStorage": "20GB"
    }
  }
}

This configuration enables automatic garbage collection of the build cache when it exceeds 20GB.

Advanced resource management techniques

Beyond basic pruning, several advanced techniques provide more granular control over Docker resources.

Monitoring Docker disk usage with tools

Several external tools can help monitor Docker disk usage more comprehensively:

# Install docker-compose-viz (requires Python)
pip install docker-compose-viz

# Generate visualization of your docker-compose setup
docker-compose-viz -f docker-compose.yml -o docker-compose.png

# Use container-diff to analyze differences between images
container-diff analyze --type=history image1 image2

These tools provide insights beyond what Docker's built-in commands offer, helping identify resource usage patterns.

Optimising container storage with custom cleanup

For containers that generate substantial internal waste, consider implementing application-specific cleanup:

# Add temporary cleanup container using the same volumes
docker run --rm --volumes-from your-container-name alpine sh -c "rm -rf /path/to/logs/*"

# Or create a maintenance script inside the container
docker exec your-container-name sh -c "find /app/temp -type f -mtime +7 -delete"

This approach addresses application-specific resource consumption that Docker pruning commands can't manage.

Implementing resource limits

Prevent excessive resource consumption by implementing limits:

# Limit container memory and CPU
docker run -d --name limited-app --memory=512m --cpus=1.0 your-image

# Configure log rotation to prevent log file growth
docker run -d --log-opt max-size=10m --log-opt max-file=3 your-image

Setting appropriate limits prevents containers from consuming excessive resources in the first place.

Ensuring safe cleanup operations

Aggressive cleanup can lead to data loss or service disruptions. Implementing safeguards ensures efficient yet safe housekeeping.

Pre-cleanup verification

Before any major cleanup operation, verify what will be removed:

# Dry run for container pruning
docker container prune --filter "until=168h" --force=false

# List images that would be removed
docker image ls --filter "dangling=true"
docker image ls --format "{{.Repository}}:{{.Tag}}" --filter "before=nginx:1.19"

This verification step helps prevent accidental removal of important resources.

Backup essential resources

Create backups of critical data before cleanup operations:

# Export important container configurations
docker inspect container-name > container-config.json

# Save important images
docker save -o my-images.tar image1:tag image2:tag

# Backup volumes
docker run --rm -v volume-name:/source -v $(pwd):/backup alpine tar -czf /backup/volume-backup.tar.gz -C /source .

These backups provide a safety net in case cleanup operations remove resources that are later needed.

Tip

Consider implementing a backup rotation strategy for critical Docker resources, similar to database backup rotation. This ensures you always have recent backups available without consuming excessive disk space.

Real-world housekeeping scenarios

I've worked with Docker in various environments, from development laptops to large-scale production systems. Here are some practical scenarios and solutions based on that experience:

Development environment cleanup

On developer machines where Docker is used for local testing, disk space often becomes an issue. I've found this maintenance routine to be effective without being disruptive:

# Daily cleanup script for development environments
#!/bin/bash

# Stop containers not used in the last 3 days
docker container ls -a --filter "status=exited" --filter "until=72h" -q | xargs -r docker container rm

# Remove images not used in the last week
docker image prune -a --filter "until=168h" --force

# Remove build cache
docker builder prune --force

This approach preserves recently used resources while aggressively cleaning older ones, which works well in development contexts where older resources are rarely needed again.

Production environment strategy

In production environments, I've found that resource preservation is typically more important than space savings. This approach focuses on targeted cleaning:

# Weekly production cleanup script
#!/bin/bash

# Remove only dangling images
docker image prune --force

# Remove exited containers older than 30 days
docker container prune --filter "until=720h" --force

# Log pruning activity
echo "Cleanup performed on $(date)" >> /var/log/docker-maintenance.log

This conservative approach prevents removal of potentially important resources while still maintaining some level of housekeeping.

Conclusion

Effective Docker housekeeping is a balance between resource optimisation and operational needs. The strategies outlined in this guide provide a framework for developing a housekeeping approach tailored to your specific environment:

  • For development machines, more aggressive cleaning with age-based filters works well
  • For production systems, conservative pruning with careful verification is preferable
  • Automated schedules ensure consistent maintenance without manual intervention
  • Proper labelling and tagging strategies make selective cleanup more precise

Implementing these practices not only recovers disk space but also improves system performance and simplifies resource management. By making Docker housekeeping a regular part of your operational routine, you'll ensure your Docker environment remains efficient, organised, and trouble-free.

Further learning

To continue building your Docker management skills, consider exploring:

  • Docker Compose management strategies
  • Container logging management
  • Monitoring Docker performance with Prometheus
  • Implementing automated scaling and resource allocation