Preface

During the daily use of Linux systems, especially for cloud host users with limited disk space, many people will encounter the situation where disk space is gradually filled up.

There are many reasons for tight disk space, such as a large number of system-generated log files, Docker images occupying space, and uncleaned package manager caches. Facing these different reasons, the clean-up methods also vary. To help everyone manage system space more efficiently, I have summarized several common space usage issues and their corresponding clean-up methods in this tutorial, hoping to enable you to quickly troubleshoot the specific cause and solve the problem efficiently when encountering insufficient space.

Clean-up steps

View file system disk usage

Before starting the clean-up, we need to have an idea of the overall disk space usage. This step is achieved by using thedf -hcommand, where the ”-h” parameter uses a human-readable format, using units like KB, MB, and GB when displaying space usage. The following is a typicaldf -hcommand output:

$ df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda1        50G   20G   28G  42% /
tmpfs           7.8G     0  7.8G   0% /dev/shm
tmpfs           7.8G  1.5G  6.3G  20% /run
/dev/sdb1       200G   50G  150G  25% /mnt/data

Filesystem Size Used Avail Use% Mounted on

Identify the causes of high disk space usage and perform targeted clean-up

/dev/sda1 50G 20G 28G 42% /

Disk space usage caused by log files

System logs

Search targets

tmpfs 7.8G 0 7.8G 0% /dev/shmtmpfs 7.8G 1.5G 6.3G 20% /run/dev/sdb1 200G 50G 150G 25% /mnt/dataExplanation of output fields:According to general experience, high disk space usage is usually related to the following aspects.df -hUsually, the path of system log files is

/var/logExplanation of output fields:, so if the tight disk space is caused by log files, thenExplanation of output fields:/varwill show abnormalities. At this time, thecommand display may look like this:

du -ah /var/log | sort -n -r | head -n 10

, whereas for cloud hosts with generally small hard drive space, it is more likely that all content is in a single partition, so the display is more likely to be as follows:tmpfs 7.8G 1.5G 6.3G 20% /runSince at this time the

directory is part of the main partition, further judgment is needed to determine whether thedirectory is occupying space. You can use the du command: 和 du -ah /var/log | sort -n -r | head -n 10 This command will list the

Clear system logs

top 10 files and directories occupying the most space under the directory, and the possible output is as follows:

truncate -s 0 /var/log/messages 

Detailed explanation:
• truncate: This is a command used to adjust file size. It can be used to extend or shrink files.
• -s 0: This parameter sets the file size to 0 bytes. That is, it clears the file content but retains the file's metadata (such as permissions, owner, etc.).

The effect of this command is toclear /var/log/messages the content of the file, without deleting the file itself. It is a common operation when managing log files, especially when log files are too large, clearing the file to free up space while keeping the log file structure and permissions unchanged.

Application logs

Search targets

In addition to system log files, different applications may store logs in other locations. Taking docker as an example, by default, docker records logs in JSON file format under Linux systems and stores the logs in the /var/lib/docker/containers/[container-id]/[container-id]-json.log path, which can still be viewed using theducommand:

du -h /var/lib/docker/containers/*/*-json.log | sort -hr

The command is explained as follows:
• du -h: Recursively displays the size of each log file, using a human-readable format (such as KB, MB).
• /var/lib/docker/containers/*/*-json.log: This is the default log file path for Docker, matching the log files of all containers.
• sort -hr: Sorts by size in descending order.
• -h: Sorts in human-readable format.
• -r: Sorts in descending order.

This command may display results like the following:

According to the previous logic, it should also be processed in a top-down priority manner.

Clear application logs

Then, you can use the truncate command to clear the log file in the same way as before:

truncate -s 0 /var/lib/docker/containers//-json.log

This will set the log file size to 0 bytes, but the file itself still exists, and Docker will continue to write new logs to this file.

Things to note:

• Will not affect container operation: Clearing log files will not affect running containers; they will continue to record new logs.

• Regularly clean up logs: Docker does not automatically clean up logs by default, so log files can become very large. You can use this method to clean up logs regularly, or configure Docker to limit the size of log files.

There are 2 ways to limit docker logs:

Log rotation mechanism: You can configure Docker's log rotation mechanism to limit the size of log files and the number of logs retained.

To prevent log files from becoming too large, you can configure log rotation when Docker starts a container, as shown below:

docker run --log-opt max-size=10m --log-opt max-file=3 ...

This means each log file is at most 10MB, and a maximum of 3 log files are retained.

Modify docker configuration: You can limit container log size by modifying the Docker configuration file

1. Find or create the Docker configuration file:

Docker's configuration file is usually /etc/docker/daemon.json. If the file does not exist, you can create it manually.

2. Edit the daemon.json file:

Open the /etc/docker/daemon.json file, and add or modify the log limit configuration. The specific log options are as follows:

{
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "10m",   // 每个日志文件最大为10MB
    "max-file": "3"      // 最多保留3个日志文件
  }
}

"log-driver": "json-file",

"log-opts": {
"max-size": "10m", // Maximum size of each log file is 10MB
"max-file": "3" // Retain a maximum of 3 log files

3. Restart the Docker service:

After modifying the configuration file, you need to restart the Docker service for it to take effect:

sudo systemctl restart docker

4. Verify if the configuration has taken effect：

You can view Docker's current logging options using the following command to confirm if the settings have taken effect:

docker info | grep 'Logging Driver'

If everything is normal, all newly started containers will manage log files according to the set log limits.

Note: This configuration will apply to all newly started containers; already running containers will not be automatically affected. If you need existing containers to apply the log limits, you can stop and delete these containers first, and then restart them based on the new configuration.

Disk space usage caused by applications

This scenario needs to be analyzed separately based on different applications and cannot be generalized, so I will analyze it using the most common situation of space occupancy caused by docker image files.

Search for unused images

The most common occupancy of docker itself is caused by image files. For example, when installing WordPress, when using the commanddocker pull wordpress, the default is to pull the latest version of the image, which means adding the:latesttag. If you use thecommand will also automatically pull the Llama 3.2 model file when Ollama finds it is not available locally, which is actually similar tocommand to create a container at this time, it will use the latest version, and the old version of the image will be useless; if there are many containers and you prefer to upgrade, the leftover unused images will occupy a lot of space.

Use the following command to view unused docker images (dangling images):

docker images -f "dangling=true"

The so-called dangling images are those without tags. Usually, dangling images are old images left behind when rebuilding or updating images. However, this command cannot display thosetaggedbut unused by containers images.

If you want to view ”all images not used by any container”, not just dangling images, it is a bit troublesome because docker has no direct command to list ”all images not used by containers”. Therefore, there are generally the following two ways to achieve this:

• Script Method

To achieve the above function, you need to run the following script directly:

docker images --format "{{.Repository}}:{{.Tag}} {{.ID}}" | while read line; do(echoline | awk '{print 2}');img" --format "{{.Image}}" | grep -q img; thenline;

The principle of this script is:

1. List all images
2. Check if these images are used by any container
3. If not used, output the image

• Human Eye Observation Method

Use the following commands:
• docker images -a: List all images, including used and unused ones.
• docker ps -a: List all containers, including running and stopped containers.

Then use the advanced skill ”Human Eye Observation Method”: manually compare the output of these two commands to find all images not used by any container yourself.

Additional knowledge: overlay and overlay2

When we are using thedh -fcommand, we may encounter output similar to the following:

So, what is the overlay in this red box?

When using Docker, overlay and overlay2 are two common storage drivers that determine how Docker manages container files and image layers in the file system. Docker uses these storage drivers to provide Union Filesystem functionality to efficiently manage container images and layers.

1. Overlay (OverlayFS)

overlay is an early storage driver, implemented using the OverlayFS in the Linux kernel. It “overlays” multiple file system layers together, enabling containers to share the same base image layer without needing to copy the entire file system for each container. This mechanism allows containers to be more lightweight and saves disk space.

Features:

• Image layer overlaying: Each Docker image consists of multiple read-only layers. When starting a container, Docker overlays a writable layer on top of these read-only layers. Any write operations will occur in this writable layer, avoiding modification of the base layer.

• Lower performance: Compared to overlay2, the performance of overlay is slightly worse, especially when managing a large number of layers.

Disadvantages:

• Fewer supported layers: overlay supports fewer image layers, which can cause performance issues when the number of layers exceeds a certain limit.

• Gradually phased out: Due to the limitations of overlay, Docker later introduced overlay2 and gradually recommended using the latter.

2. Overlay2

overlay2 is an improved version of overlay, also based on Linux's OverlayFS, but it resolves some of overlay's performance bottlenecks and limitations. Since Docker version 1.13, overlay2 has become the recommended default storage driver, suitable for a wider range of scenarios, especially when dealing with more image layers.

Features:

• Support for multi-layer stacking: overlay2 allows Docker to use a single filesystem layer to represent multiple layers of an image, thereby improving performance and efficiency. Even if a Docker image has many layers, it can handle them better without the performance degradation caused by many layers as seen in overlay.

• Better performance: Compared to overlay, overlay2 has better write and read performance, especially when dealing with a large number of layers or files.

Improvements:

• Reduced space waste: In scenarios where multiple containers share the same image layer, overlay2 can use storage space more efficiently, avoiding unnecessary file copying.

• Reduced system calls: By simplifying file operations, overlay2 improves the performance of filesystem operations.

Although overlay2 seems better, we usually do not manually specify its use. When installing Docker, the choice of storage driver is typically determined automatically by Docker based on the operating system's kernel, and users usually do not need to manually specify the storage driver. For example, when you use apt to install Docker, Docker will detect your system environment and choose the most appropriate storage driver, which should usually be overlay2, as it is currently the recommended default driver.

So do not be surprised when you see this output; just focus on the corresponding specific path on the right.

Clean up images

If you are sure you want to clean up unused images, you can directly use the following command:

docker image prune -a

This will delete all images not used by any container, including tagged images. Use with caution, as you will need to re-pull or rebuild these images after deletion.

In addition, if you have deployed a Docker graphical management UI like Portainer, it is actually very convenient to operate. Just check ”Unused” in the Filter under Images, and you can directly see all unused images, as shown in the figure below:

Note: There are other applications, such as space occupied by mysql (mariadb), where cleanup requires certain professional knowledge to support, so I won't mention much about it here.

Disk space usage caused by package manager cache files

Uncleared package manager cache (such as apt in Debian systems) is usually due to the long-term accumulation of software package caches downloaded during system installation and upgrade processes, resulting in a large amount of disk space being occupied. Below are several common causes and solutions:

APT cache accumulation

When using apt or apt-get to install and update software, the system saves the downloaded .deb installation packages in the cache directory (usually /var/cache/apt/archives). If not cleaned regularly, these cache files will occupy space for a long time.

Cleanup solutions:

• Manually clean APT cache

Use the following command to clean the cache:

sudo apt clean

This command will delete all downloaded .deb package files.

• Automatically clean unused software packages

Use the following command to clean up package files no longer needed on the system, including dependencies and older versions:

sudo apt autoremove

• Clean only obsolete cache:

apt-get autoclean will delete obsolete package cache files (i.e., caches that are no longer available or have been replaced):

sudo apt-get autoclean

Cache not automatically cleaned up

By default, the system does not automatically clean downloaded cache files, especially when the system is frequently updated and installed, making cache files easy to accumulate.

Cleanup solutions:

• Enable regular automatic cleanup:

You can avoid the problem of not cleaning for a long time by setting the system to regularly clean cache files. You can create regular cleanup tasks:

1. Use cron to run the apt clean command regularly.
2. Ensure regular automatic cleanup is executed via /etc/cron.daily/apt or a custom cron job.

Configure cron tasks (e.g., clean once a month)

1. Open the cron editor:

sudo crontab -e

2. Add the following content to clean once a month:

@monthly apt clean

Accumulation of unused packages and old versions

After a system upgrade, older versions of some software packages and dependencies that are no longer needed will remain in the system, and these uncleared packages will occupy disk space.

Cleanup solutions:

• Use apt autoremove Clean up software packages that are no longer needed：

sudo apt autoremove

The command will clean up all dependencies and software packages that are no longer used, freeing up disk space.

Disk space usage caused by other situations

This situation is mainly because files downloaded by users themselves or temporary files in the system (such as the /tmp directory) may occupy a large amount of disk space if not cleaned for a long time. In this case, generally follow the previous troubleshooting approach and usedf -h和du -h --max-depth=1commands alternately to troubleshoot level by level, finding cases where temporary files or directories occupy too much space, and then usermcommand to perform the deletion, so I won't repeat myself.

Afterword

Although I have summarized several common disk space usage issues and their corresponding cleanup methods in this article, there will inevitably be omissions, especially space usage issues caused by certain applications themselves (such as Docker images, etc.). However, even if this article does not cover certain specific scenarios in detail, through the troubleshooting ideas mentioned in the text, usingdf -hto check the overall disk usage, and then combining it withdu -h --max-depth=1to investigate the source of usage directory by directory, which will ultimately help you find the “culprit” causing the space shortage and take corresponding cleanup measures.

I hope this tutorial can provide you with ideas and help you better manage the disk space of your Linux system.