Rust create sparse file?

#1

Is there a way to create sparse files from Rust? Here, by Sparse file I mean files where if an entire Page is 0, the OS optimizes it away.

Googling, there are various ways to create such files via truncate or dd. However, I can't find anyway to create one from Rust (googling returns sparse linear algebra libraries).

[Context: for mmapped on-disk b-trees, if we drop an entire page, we don't want it to waste disk space.]

#2

According to this article, you can create a sparse file by using lseek() to move the file's cursor to a particular location and the OS will automatically treat the space in between as empty...

int create_sparse_file(char *path, uint64_t size)
{
    int fd = 0; 
    fd = open(path, O_RDWR|O_CREAT, 0666);
    if (fd == -1) {
        return -1;
    }    
    if (lseek(fd, size - 1, SEEK_CUR) == -1) {
        return -1;
    }    
    write(fd, "\0", 1);
    close(fd);
    return 0;
}

... and looking at the source code for std's filesystem APIs on *nix, it seems like we use lseek() (imported as lseek64()) when implementing Seek for std::fs::File.

I haven't tried it, but in theory you should be able to create a sparse file by just seeking to the end location.

3 Likes
#3

The full patch was declined (issue 58635), but you can see a method of copying sparse files in rust in this PR. There's also the hole-punch crate, which seems unmaintained, but can likely be used as a reference.

2 Likes