How to create sparse files in Linux using ‘dd’ command

What are sparse files

Sparse files are files that have large amounts of space preallocated to them, without occupying the entire amount from the filesystem. They are useful for reducing the amount of time and disk space involved in creating loop filesystems or large disk images for virtualized guests, among other things. Sparse files are commonly used for disk images, database snapshots, log files, etc.

Advantage of sparse files
The advantage of sparse files is that storage is only allocated when actually needed: disk space is saved, and large files can be created even if there is insufficient free space on the file system.

Disadvantage of sparse files
Disadvantages are that sparse files may become fragmented. The file system free space reports may be misleading and copying a sparse file with a program that does not explicitly support them may copy the entire, uncompressed size of the file, including the sparse, mostly zero sections which are not on disk – losing the benefits of the sparse property in the file.

We can see this behavior with /var/log/lastlog file.

# ls -lh /var/log/lastlog
-rw-r--r--. 1 root root 286K Dec  3 04:50 /var/log/lastlog
# du -sh /var/log/lastlog
12K     /var/log/lastlog

creating sparse files

1. Sparse files can be created using the ‘dd‘ command’s ‘seek‘ option.

# dd if=/dev/zero of=sparse_file bs=1 count=0 seek=512M
0+0 records in
0+0 records out
0 bytes (0 B) copied, 0.000213705 s, 0.0 kB/s

# ls -hl sparse_file
-rw-r--r--. 1 root root 512M Dec  3 05:51 sparse_file

# du -sh sparse_file
0       sparse_file

2. To see disk usage of file with “ls” command we can use “-s” option:

# ls -lhs sparse_file
0 -rw-r--r--. 1 root root 512M Dec  3 05:51 sparse_file

3. To see apparent size of the file using “du” we can use “–apparent-size” option:

# du -h --apparent-size sparse_file
512M    sparse_file

How to copy a sparse file

Copying a sparse file with a program that does not explicitly support them may copy the entire, uncompressed size of the file, including the sparse. So, when you copy a sparse file using cp command, the destination file gets changed to a fully allocated file. To copy a sparse file by keeping the destination copy as a sparse file, use any of the below commands.

# cp --sparse=always source_file new_file
# rsync --sparse source_file new_file
# cpio --sparse
# tar --sparse