Disk Space Detective Work With du and ncdu
A Linux system can appear perfectly healthy until it suddenly reports that a filesystem is full. Updates fail, applications stop writing data, databases behave unpredictably and users may be unable to save files.
The obvious question is: what is using all the space?
Linux provides several ways to investigate storage usage, but two of the most useful are:
- du, a standard command-line utility for measuring directory and file sizes
- ncdu, an interactive disk-usage browser that makes large directories easier to explore
Used together, these tools can help you trace storage problems from an entire filesystem down to a specific log, backup, cache or forgotten archive.
This guide explains how to investigate disk usage safely, interpret the results and avoid deleting files that the operating system still needs.
Start With
df
, Not
du
Before investigating individual directories, confirm which filesystem is actually full.
Run:
df -h
The -h option displays sizes in a human-readable format.
Example:
Filesystem Size Used Avail Use% Mounted on
/dev/sda2 80G 74G 2.1G 98% /
/dev/sdb1 500G 210G 265G 45% /srv/data
tmpfs 3.9G 2.0M 3.9G 1% /run
In this example, the root filesystem mounted at / is 98% full, while the separate data drive still has plenty of free space.
That distinction matters. Investigating /srv/data would not explain why / is running out of capacity if it is mounted from a different filesystem.
To display filesystem types as well:
df -hT
You can check one specific path:
df -h /
or:
df -h /var
df tells you which filesystem is full. du helps explain what is occupying it.
What Does
du
Do?
du stands for disk usage.
It calculates the storage consumed by files and directories.
The simplest command is:
du
However, this can produce a large amount of output in raw block counts. A more useful starting point is:
du -h
The -h option uses readable units such as kilobytes, megabytes and gigabytes.
To show only the total size of the current directory:
du -sh .
The options mean:
- -s — display a summary rather than every subdirectory
- -h — display human-readable sizes
- . — inspect the current directory
For example:
18G .
Find the Largest Top-Level Directories
When the root filesystem is full, start by measuring its immediate subdirectories:
sudo du -xhd1 /
The options mean:
- -x — remain on the current filesystem
- -h — use readable sizes
- -d1 — show only one directory level
- / — begin at the filesystem root
Example output:
12G /usr
38G /var
8.1G /home
2.4G /opt
62G /
This immediately suggests that /var deserves further investigation.
Continue one level deeper:
sudo du -xhd1 /var
You might see:
1.2G /var/cache
4.8G /var/lib
29G /var/log
36G /var
Now inspect /var/log:
sudo du -xhd1 /var/log
This process is the command-line equivalent of following a trail. Start broad, identify the largest directory and continue narrowing the search.
Why the
-x
Option Matters
Linux often has several filesystems mounted beneath /.
These may include:
- External drives
- Network shares
- Backup volumes
- Container mounts
- Temporary filesystems
- Separate home or data partitions
Without -x, a command starting at / may descend into all of them. That can make the scan slower and produce misleading results.
Use:
sudo du -xhd1 /
when you want to investigate only the filesystem containing /.
You can inspect another filesystem separately:
sudo du -xhd1 /srv/data
Sort the Results by Size
du output becomes much more useful when passed through sort.
For example:
sudo du -xhd1 /var | sort -h
The smallest entries appear first and the largest appear last.
To reverse the order:
sudo du -xhd1 /var | sort -hr
Example:
36G /var
29G /var/log
4.8G /var/lib
1.2G /var/cache
This is one of the most useful commands for routine disk-space investigations:
sudo du -xhd1 / | sort -hr
Show the Largest Directories Anywhere Below a Path
To list large directories at several levels, omit the depth limit:
sudo du -xh /var | sort -hr | head -30
This displays the 30 largest results below /var.
However, remember that directory totals overlap.
For example:
29G /var/log
18G /var/log/application
15G /var/log/application/archive
These are not three separate amounts that should be added together. The size of /var/log includes its subdirectories.
Find the Largest Files
Directory totals tell you where to look, but sometimes one individual file is responsible for most of the problem.
Use find to locate files and combine it with du or sort.
A practical command is:
sudo find /var -xdev -type f -printf '%s %p\n' 2>/dev/null |
sort -nr |
head -20
This prints the 20 largest files below /var, sorted by their size in bytes.
For more readable output:
sudo find /var -xdev -type f -exec du -h {} + 2>/dev/null |
sort -hr |
head -20
This may reveal:
- Multi-gigabyte log files
- Database dumps
- Backup archives
- Virtual disk images
- Core dumps
- ISO files
- Forgotten compressed archives
- Application caches
For an entire filesystem:
sudo find / -xdev -type f -exec du -h {} + 2>/dev/null |
sort -hr |
head -30
This can take some time on systems with many files.
Find Files Larger Than a Certain Size
To locate files larger than 1 GB:
sudo find / -xdev -type f -size +1G -print
To include readable sizes:
sudo find / -xdev -type f -size +1G -exec ls -lh {} \;
For files larger than 500 MB beneath /var:
sudo find /var -xdev -type f -size +500M -exec ls -lh {} \;
Large files are not automatically unnecessary. A database, virtual machine or mail store may legitimately require substantial space.
Always identify the file’s purpose before removing it.
Ignore Permission Errors
When scanning system directories as an ordinary user, you may see messages such as:
Permission denied
Use sudo when appropriate:
sudo du -xhd1 /
You can also hide error messages:
sudo du -xhd1 / 2>/dev/null
The 2>/dev/null part redirects standard error output away from the screen.
Do not use this automatically during every investigation. Error messages can reveal inaccessible mounts, broken links or filesystems with problems.
What Is
ncdu
?
ncdu stands for NCurses Disk Usage.
It scans a directory and presents the results in an interactive terminal interface. You can move through directories, sort entries and identify large files without repeatedly typing du commands.
It is particularly useful when:
- You want to explore storage interactively
- The directory structure is complicated
- You need to move quickly between levels
- You want an easier view than raw terminal output
- You are working over SSH without a graphical desktop
Installing
ncdu
On Ubuntu or Debian:
sudo apt update
sudo apt install ncdu
On Fedora:
sudo dnf install ncdu
On Rocky Linux, AlmaLinux or Red Hat-based systems, it may be available through the distribution’s configured repositories:
sudo dnf install ncdu
On Arch Linux:
sudo pacman -S ncdu
On openSUSE:
sudo zypper install ncdu
Scan a Directory With
ncdu
To inspect your home directory:
ncdu ~
To inspect /var:
sudo ncdu /var
To examine the root filesystem without crossing into other mounted filesystems:
sudo ncdu -x /
The -x option has the same general purpose as it does with du: it prevents the scan from crossing filesystem boundaries.
After scanning, ncdu displays directories and files ordered by size.
Typical controls include:
- Arrow keys — move through the list
- Enter — open a directory
- Left arrow — return to the parent directory
- d — delete the selected item
- i — display information
- r — refresh or recalculate
- q — quit
Controls can vary slightly between versions. Press:
?
inside ncdu to display its help screen.
Be Careful With Deletion Inside
ncdu
ncdu may allow you to delete files directly.
That is convenient, but it also makes accidental deletion easier.
Before deleting anything, ask:
- Is this file part of an installed package?
- Is it being used by an application?
- Is it a database or virtual disk?
- Is it a current backup?
- Is it required for recovery?
- Could it be open by a running process?
- Does the application have its own supported cleanup process?
For production systems, it is often safer to use ncdu for investigation and perform the actual cleanup through the appropriate service or package-management command.
Scan Once and Review Later
On large servers, repeatedly scanning the filesystem can take time and generate significant disk activity.
Some versions of ncdu support exporting scan results.
A typical export command is:
sudo ncdu -x -o /tmp/root-scan.ncdu /
You can then review the saved scan:
ncdu -f /tmp/root-scan.ncdu
This is useful when:
- The live scan takes a long time
- You want to analyse the results later
- You need to avoid repeatedly scanning slow storage
- You are collecting evidence during an incident
The exact options may vary by ncdu version, so check:
ncdu --help
Common Causes of Unexpected Disk Usage
Once you have found the largest directories, the next task is deciding what created them.
Log Files
Logs commonly live under:
/var/log
Check their sizes:
sudo du -h /var/log/* | sort -hr | head -20
Large logs can result from:
- Repeated application errors
- Debug logging left enabled
- Failed authentication attempts
- Hardware errors
- A broken service restarting continuously
- Missing log rotation
- Excessive web-server traffic
Do not simply delete an active log file without checking the application.
View recent entries:
sudo tail -100 /var/log/example.log
Follow new entries:
sudo tail -f /var/log/example.log
If the file grows rapidly while you watch it, investigate the repeated error rather than treating the log itself as the root cause.
systemd Journal Usage
On systems using systemd, journal files can consume considerable space.
Check current usage:
sudo journalctl --disk-usage
Remove archived journal data older than 14 days:
sudo journalctl --vacuum-time=14d
Reduce total journal usage to approximately 500 MB:
sudo journalctl --vacuum-size=500M
Review the logs before deleting them, especially after a system failure. They may contain the information needed to identify what happened.
Persistent journal limits can be configured in:
/etc/systemd/journald.conf
After changing the configuration:
sudo systemctl restart systemd-journald
Package Caches
Package managers retain downloaded files so they can be reused.
Ubuntu and Debian
Check the cache:
sudo du -sh /var/cache/apt
Clean downloaded packages:
sudo apt clean
Remove package files that are no longer downloadable or useful:
sudo apt autoclean
Remove packages installed automatically that are no longer required:
sudo apt autoremove
Review the removal list carefully before confirming.
Fedora and Red Hat-Based Systems
Check the package-manager cache:
sudo du -sh /var/cache/dnf
Clean it:
sudo dnf clean all
Arch Linux
Check:
du -sh /var/cache/pacman/pkg
Remove older cached packages carefully with the appropriate cache-cleaning tool rather than deleting everything blindly.
Old Kernels
Linux updates may leave several older kernels installed.
Check the running kernel:
uname -r
On Ubuntu or Debian:
dpkg --list | grep linux-image
On Fedora or Red Hat-based systems:
rpm -q kernel
Keep at least one known-good fallback kernel. Never remove the kernel currently in use.
Where possible, let the package manager remove obsolete kernels rather than deleting files directly from /boot.
On Ubuntu:
sudo apt autoremove
Review the proposed changes before approving them.
Docker and Container Storage
Container platforms can consume large amounts of space in:
/var/lib/docker
/var/lib/containers
Check Docker usage:
docker system df
Show detailed information:
docker system df -v
Unused containers, images, build caches and volumes may accumulate over time.
List them before deleting anything:
docker ps -a
docker images
docker volume ls
Docker provides cleanup commands such as:
docker system prune
This can remove unused data, so read the confirmation prompt carefully.
Do not manually delete random files from /var/lib/docker. Doing so can corrupt Docker’s internal state.
Snap Packages
On Ubuntu systems, older Snap package revisions may remain stored under:
/var/lib/snapd/snaps
Check usage:
sudo du -sh /var/lib/snapd/snaps
List installed revisions:
snap list --all
Disabled older revisions may be removable, but use Snap’s own management tools rather than deleting the files directly.
Flatpak Data
Flatpak applications and runtimes may occupy space under locations such as:
/var/lib/flatpak
~/.local/share/flatpak
Check usage:
du -sh ~/.local/share/flatpak 2>/dev/null
sudo du -sh /var/lib/flatpak 2>/dev/null
Remove unused runtimes:
flatpak uninstall --unused
Review the proposed changes before confirming.
User Caches
User caches often live in:
~/.cache
Inspect them:
du -h -d1 ~/.cache | sort -hr
Or use:
ncdu ~/.cache
Browsers, development tools, package managers and media applications can build substantial caches.
Closing the application before clearing its cache is usually safer. Some applications may recreate the data immediately.
Downloads and Forgotten Archives
Common storage offenders include:
- ISO images
- ZIP and TAR archives
- Old installers
- Database exports
- Video recordings
- Virtual-machine images
- Duplicate backup files
Check a user’s home directory:
du -h -d1 ~ | sort -hr
Then inspect likely locations:
du -h -d1 ~/Downloads | sort -hr
ncdu is particularly helpful here:
ncdu ~
Backups Stored on the Wrong Filesystem
A backup is not useful if it fills the system it is meant to protect.
Check common backup paths such as:
/backup
/var/backups
/srv/backups
/opt/backups
A scheduled job may be writing backups to the root filesystem because:
- The intended external drive failed to mount
- A network share became unavailable
- The backup path was configured incorrectly
- Old backups are not being expired
- The mount point existed as an ordinary directory
This is especially dangerous when a backup destination is normally mounted over an empty directory. If the mount fails, the backup software may continue writing into that directory on the root filesystem.
Confirm the mount before trusting the path:
findmnt /backup
Databases
Database files can grow because of:
- Expected business data
- Unexpired transaction logs
- Binary logs
- Temporary tables
- Failed maintenance
- Large audit records
- Old database dumps
Common locations include:
/var/lib/mysql
/var/lib/postgresql
/var/lib/mongodb
Do not delete database files directly.
Use the database’s supported tools to inspect tables, logs and retention policies.
For example, finding that /var/lib/mysql is large is only the beginning. The database administrator must determine whether the size is expected and which supported cleanup process is appropriate.
Core Dumps
When an application crashes, it may create a core dump containing its memory state.
Check systemd-managed core dumps:
coredumpctl list
Check their storage usage:
sudo du -sh /var/lib/systemd/coredump 2>/dev/null
Core dumps can be important for troubleshooting, but old dumps may no longer be required.
Review them before removal.
Temporary Files
Temporary data commonly appears under:
/tmp
/var/tmp
Inspect it:
sudo du -xhd1 /tmp | sort -hr
sudo du -xhd1 /var/tmp | sort -hr
Do not assume everything in /tmp is safe to delete while the system is running. Applications may have active temporary files or sockets there.
A reboot may clear some temporary content, depending on the distribution’s configuration.
Why
df
and
du
Sometimes Disagree
A common puzzle is that df reports a full filesystem while du cannot find enough files to explain it.
For example:
df: 70 GB used
du: 45 GB found
Several situations can cause this.
Deleted Files Still Open by a Process
Linux allows a process to continue using an open file after its directory entry has been deleted.
The filename disappears, so du no longer counts it. However, the filesystem space is not released until the process closes the file.
Find deleted but open files with:
sudo lsof +L1
You may see:
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NLINK NAME
java 4821 app 12w REG 8,2 15G 0 /var/log/app.log (deleted)
In this example, a Java process still holds a deleted 15 GB log file open.
The correct fix may be to restart or reload the relevant service:
sudo systemctl restart example-service
Do not restart production services without considering the impact.
Once the process closes the file, the space becomes available again.
Files Hidden Beneath a Mount Point
Suppose files exist inside /mnt/data, and another filesystem is mounted over that directory.
The original files still consume space on the underlying filesystem, but they are hidden while the mount is active.
Check the mount:
findmnt /mnt/data
To inspect what is underneath, you may need to stop relevant services and unmount the filesystem:
sudo umount /mnt/data
Then inspect the directory:
sudo du -sh /mnt/data
This operation can disrupt applications, so it should be planned carefully.
Filesystem Reserved Space
Filesystems such as ext4 may reserve a percentage of their blocks for the root user and for maintaining system stability.
Check an ext filesystem:
sudo tune2fs -l /dev/sda2 | grep -i reserved
Reserved blocks can make the available space shown by df lower than expected.
Changing the reserved percentage is an administrative decision and should not be used as a substitute for proper cleanup or capacity planning.
Sparse Files
A sparse file can have a large apparent size but consume much less physical storage.
Compare:
ls -lh large-file
with:
du -h large-file
For example:
ls: 100G
du: 12G
The file appears to be 100 GB long but currently occupies only 12 GB of actual blocks.
Virtual disks and database files may use sparse allocation.
Use:
du -h --apparent-size large-file
to display the apparent size.
Inodes Can Run Out Before Disk Space
A filesystem can have free gigabytes but still be unable to create new files because it has exhausted its inodes.
Check inode usage:
df -ih
Example:
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/sda2 5.0M 5.0M 0 100% /
This usually indicates millions of small files rather than a few large ones.
Find directories containing many entries:
sudo find /var -xdev -printf '%h\n' 2>/dev/null |
sort |
uniq -c |
sort -nr |
head -30
Possible causes include:
- Mail queues
- Session files
- Cache directories
- Temporary application files
- Monitoring data
- Broken cleanup jobs
Deleting one large file will not solve an inode shortage. You need to identify and safely remove large numbers of unnecessary small files.
Apparent Size Versus Allocated Size
By default, du reports the filesystem blocks allocated to a file.
To report apparent file sizes instead:
du -sh --apparent-size directory
These figures can differ because of:
- Sparse files
- Compression
- Filesystem block sizes
- Deduplication
- Copy-on-write behaviour
- Hard links
For capacity investigations, allocated size is usually the more important number because it reflects actual filesystem consumption.
Hard Links and Double Counting
Multiple filenames can point to the same underlying data through hard links.
du normally tries not to count the same inode repeatedly during one scan.
However, comparing separate scans or using unusual options can make totals confusing.
Inspect link counts with:
ls -li filename
Files with the same inode number on the same filesystem refer to the same underlying data.
Check Files Modified Recently
When disk usage rises suddenly, identify recently changed large files.
Files changed during the last 24 hours:
sudo find /var -xdev -type f -mtime -1 -exec ls -lh {} \; 2>/dev/null
Files larger than 100 MB modified within the last day:
sudo find /var -xdev -type f -mtime -1 -size +100M \
-exec ls -lh {} \; 2>/dev/null
Files changed during the last 60 minutes:
sudo find /var -xdev -type f -mmin -60 -exec ls -lh {} \; 2>/dev/null
This can reveal rapidly expanding logs, backups or temporary files.
Watch Disk Usage Change in Real Time
To monitor filesystem usage:
watch -n 5 'df -h /'
This refreshes the result every five seconds.
To watch a particular file:
watch -n 2 'ls -lh /var/log/example.log'
To monitor the size of a directory:
watch -n 10 'du -sh /var/lib/example'
Repeated du scans can create additional I/O load on large directories, so choose a sensible refresh interval.
A Safe Investigation Workflow
When a Linux filesystem is nearly full, use this order.
1. Confirm the affected filesystem
df -hT
2. Check inode usage
df -ih
3. Scan the top-level directories
sudo du -xhd1 / | sort -hr
4. Follow the largest path
For example:
sudo du -xhd1 /var | sort -hr
sudo du -xhd1 /var/log | sort -hr
5. Use
ncdu
for interactive exploration
sudo ncdu -x /
6. Find unusually large files
sudo find / -xdev -type f -size +1G -exec ls -lh {} \;
7. Check deleted files still held open
sudo lsof +L1
8. Check system logs and package caches
sudo journalctl --disk-usage
sudo du -sh /var/cache/*
9. Confirm important mount points
findmnt
10. Clean up through supported application tools
Use the application, database, package manager or container platform’s own cleanup commands whenever possible.
11. Recheck free space
df -h
12. Fix the cause
Adjust retention, rotation, monitoring or capacity so that the same problem does not return.
What Not to Delete
Avoid manually deleting files from the following locations unless you understand their purpose:
/boot
/etc
/usr
/bin
/sbin
/lib
/lib64
/var/lib
These areas may contain:
- Kernels
- Bootloader files
- System configuration
- Shared libraries
- Installed packages
- Databases
- Package-manager state
- Application data
A file being large does not mean it is unnecessary.
Deleting a database file or shared library may free space immediately while leaving the system unusable.
Never Use Wildcard Deletion Without Checking
Commands such as:
sudo rm -rf /var/log/*
are dangerous.
They may remove files, subdirectories, permissions and state that applications expect to exist.
A better approach is to:
- Identify the specific log.
- Check what creates it.
- Fix the repeated error.
- Configure log rotation.
- Truncate or remove it only through an appropriate method.
To empty a known active text log while preserving the file:
sudo truncate -s 0 /var/log/example.log
Even this should be done only after confirming that the log can safely be cleared.
Configure Log Rotation
Most Linux systems use logrotate to compress, archive and remove old logs.
Configuration is commonly stored in:
/etc/logrotate.conf
/etc/logrotate.d/
Check the configuration for an application:
cat /etc/logrotate.d/example
Test log rotation in debug mode:
sudo logrotate -d /etc/logrotate.conf
Force a rotation for testing:
sudo logrotate -f /etc/logrotate.conf
Forcing rotation can affect services and logs, so use it carefully.
A healthy rotation policy should define:
- How often logs rotate
- How many copies are retained
- Whether old logs are compressed
- Whether the application must reload after rotation
- Maximum-size thresholds where appropriate
Preventing Future Disk-Space Emergencies
Finding the large file solves today’s problem. Preventing it from returning requires better controls.
Useful measures include:
- Configure log retention.
- Monitor filesystem usage.
- Alert before filesystems reach critical levels.
- Expire old backups automatically.
- Review container images and volumes.
- Clean package caches during maintenance.
- Monitor database growth.
- Validate backup mount points.
- Keep temporary directories under control.
- Set sensible journal limits.
- Review disk capacity before deploying new workloads.
A system should ideally alert well before reaching 100%.
Common warning thresholds might include:
- 75% — early capacity warning
- 85% — investigation required
- 90% — urgent action
- 95% — critical condition
The correct levels depend on the size and purpose of the filesystem. Five per cent free on a multi-terabyte volume is very different from five per cent free on a 20 GB root partition.
Useful
du
Commands at a Glance
Check the current directory’s total size:
du -sh .
Show immediate subdirectories:
du -h -d1
Investigate the root filesystem without crossing mounts:
sudo du -xhd1 /
Sort directories from largest to smallest:
sudo du -xhd1 /var | sort -hr
Show the 20 largest results below a directory:
sudo du -xh /var | sort -hr | head -20
Check apparent size:
du -sh --apparent-size directory
Useful
ncdu
Commands at a Glance
Scan a home directory:
ncdu ~
Scan /var:
sudo ncdu /var
Scan only the root filesystem:
sudo ncdu -x /
Export a scan:
sudo ncdu -x -o /tmp/disk-scan.ncdu /
Open a saved scan:
ncdu -f /tmp/disk-scan.ncdu
Final Thoughts
Linux disk-space troubleshooting works best when it is treated as detective work rather than a deletion exercise.
Start with df to identify the affected filesystem. Use du to narrow the problem from large directories to individual files, and use ncdu when an interactive view makes the structure easier to understand.
Then investigate why the data grew.
A large log may indicate a broken application. A full backup directory may indicate a failed mounted drive. A large container directory may contain unused images—or critical active volumes. Deleted files may still be consuming space because a process has them open.
The goal is not simply to find something large enough to delete. It is to reclaim space safely while fixing the condition that caused the filesystem to fill.
Need Help With a Full Linux Server?
Hamilton Group can help diagnose Linux storage problems, oversized logs, runaway applications, container storage, backup failures and filesystems that repeatedly run out of capacity.
We can safely identify what is consuming the space, protect important data and put monitoring and retention controls in place to prevent the problem from returning.
Call 0330 043 0069 or visit hgmssp.com to speak with one of our IT specialists.