Create 16GB file, has it grow only on use

[anon80458984]

I am on Linux, ext4, x86_64.

Is there a way in Rust to:

1. create a "16GB" file (but have it take much less space)
2. mmap the file
3. start using pages of the file
4. have the on disk file size not be 16GB, but only the "used" pages (i.e. the "suffix pages" that are all 0 take no space)

[aticu]

Linux supports this natively with a feature called "sparse files". There is no extra handling needed to support this.

The following program creates such a file:

use mmap::{MapOption, MemoryMap};
use std::{
    error::Error,
    fs::OpenOptions,
    io::{Seek, SeekFrom, Write},
    os::unix::io::AsRawFd,
};

const SIZE: u64 = 16 * 1024 * 1024 * 1024;

fn main() -> Result<(), Box<Error>> {
    let mut open_options = OpenOptions::new();
    open_options.read(true).write(true).create(true);

    let mut file = open_options.open("testfile")?;

    // Make sure the file is actually 16GB in size
    file.seek(SeekFrom::Start(SIZE - 1))?;
    file.write(&[0x00])?;

    // mmap the file
    let fd = file.as_raw_fd();
    let mmap = MemoryMap::new(
        SIZE as usize,
        &[
            MapOption::MapFd(fd),
            MapOption::MapReadable,
            MapOption::MapWritable,
        ],
    )?;

    unsafe {
        // Write somewhere in the middle
        *mmap.data().add(SIZE as usize / 2) = 0x01;
    }

    Ok(())
}

It uses the mmap crate to mmap the file.

Running this program produces a file that stat reports a size of 17179869184 for, without using that much space on the filesystem.

This was tested on an ext4 file system with Linux kernel version 5.7.12.