i'm looking for something like golang's fs.FS interface, but for rust.
I've seen a few crates for this, such as filesystem and vfs, but nothing as flexible or generic.
am i missing something?
I might be missing something obvious, but isn't std::fs the equivalent? std::fs - Rust
1 Like
std::fs is a direct interface to the os filesystem.
fs.FS is a virtual filesystem interface. it allows you to write code that can manipulate anything filesystem-shaped, from a zip file, NFS mount, or just a sandboxed directory.
3 Likes
This is not something that should be solved at the language level. If you need to treat something as a filesystem, use the OS' capabilities or a userspace FS (e.g. FUSE on Linux).
1 Like
The way most projects in rust would approach this is to define traits for the operations they’re interested in, which others can then implement, and to provide implementations for those traits for common cases. There’s less pressure in rust to have “the” virtual filesystem interface, versus projects defining interfaces for their idiosyncrasies.
Sometimes those traits grow to become de facto standards (as has largely happened with the futures and rand crates. There hasn’t, to my knowledge, been a uniquely successful file system-style library yet, though.
Can I ask in return why you want a single authoritative interface for file system-shaped things? For example, are you hoping to be able to consume that interface while relying on the community to write support for specific implementations?
- FUSE has numerous problems, most notably how processes can get stuck in a syscall for an indeterminate amount of time, not even able to respond to signals!
- using FUSE for intra-process communication is extremely silly, as it takes 0-syscall operation (eg. reading a chunk from a zip file) into a 3-syscall operation (read, then 2 syscalls to receive and respond to the FUSE request)
- FUSE is not cross platform
- rust is frequently used in embedded environments, in which case your only filesystem might be a virtual filesystem, such as putting assets in an embedded zip file.
2 Likes
common, well-supported interfaces let you easily compose complex programs with simple combinators. packing game assets into the executible becomes nearly trivial with the smallest bit of forethought. converting a zip file to a tarball in-memory becomes easy.
It sounds like you're mixing use-cases here. Do you have an actual, real thing you want to implement?
Because afaik games only need read-only access, they don't need the intra-process communication (which would involve writing) you mention.
Plus writing can be a very expensive operation in a tar or zip, so it's a terrible idea to do this via either FUSE or an inprocess-VFS that's backed by a ZIP.
Maybe what you need is sqlite? That's portable and would be supported by more languages than rust. Oh right, a language-specific VFS wouldn't work with FFI libs.
1 Like
An important use case for reifying the concept of "filesystem" is capabilities-based FS access where you can only access whatever is reachable from the FS instance you hold – whether it’s the "baseline reality", a read-only view thereof, a chroot, a sandbox, a container, a RAM-backed temp FS, whatever. I think this would be valuable to have in the language eventually, but also something that should first exist in the ecosystem, and should evolve along with Rust’s general "contexts&caps" story, which is very much a work in progress.
2 Likes
You mean GitHub - bytecodealliance/cap-std: Capability-oriented version of the Rust standard library?
1 Like
But then so is using an artificial filesystem. If you don't need files, don't try to shove some other functionality into a file-shaped hole. If you need channels, or shared memory, or queues, then use those.
2 Likes
Arguably, what's often needed is something that simply impl Read or impl AsyncRead, with its own, non-filesystem, lookup structure that reflects whatever the underlying file or device offers you. For NFS, you'd probably end up with something that looks very filesystem-like, but for tar, it should look like a tape with metadata (for example), allowing for the fact that in tar, it's legitimate to have the same file path twice in the same tar file, and to only distinguish them based on position.
1 Like
Thanks for explaining!
I think this may be a combination of expecting to find things in this community in the same shape you found them in another community - entirely naturally - and also a lack of familiarity with the Rust library ecosystem, leaving you with a "where is this tool I expect"-shaped question where the direct answer is less useful than an indirect one.
You've mentioned embedding files into programs twice, which is a need I have some familiarity with. I would highly recommend you take a look at the rust-embed crate to do so! It does use a lightly-filesystem-shaped set of traits to do what it does, and it generates implementations of those traits at compile time for you. The only issue I expect you may run into is if you need to run your program in no_std environments, as I'm not sure if the developers of that crate have ever looked into supporting it.
Having written a fair bit of code in both Go and Rust, I've never felt a need for the kinds of "universal" interfaces that are common in Go. The way Rust's traits system works makes it easier to work with foreign traits, going one direction, and easier to ask developers to accommodate traits idiosyncratic to your needs, going in the other, than it is to negotiate a community-wide consensus on these kinds of interfaces. There are outliers (futures being the flag example, I'd say), where the community has settled onto a shared standard, but they're rare.
4 Likes
List/read/write are mostly very simple to paper over, though there are quirks like recursive walk over a network being far too inefficient to not make explicit, but even simply for abstracting real file systems, locks/exclusive access and watches are hard to abstract well, since they have such different capabilities, limits and performance.
In practice, in Rust it's often just easier to make whatever abstraction you need on the spot, especially if you can make it at a higher level with more information available (eg "i want to watch all *.png files in this directory recursively")
That said, this type of "pure interface" does exist in Rust occasionally, the math library mint is just a set of "vocabulary" types and traits different implementation libraries can opt to switch to so you can use these implementations together without them having knowledge of each other. A VFS vocabulary library is possible in theory, even without std support, though it would need buy-in from many libraries to become useful due to the orphan rule (since it's probably just adding a wrapping trait impl to an optional dep, this doesn't sound too scary as a PR though)
3 Likes
I'm actually quite familiar with the rust ecosystem, some of my code is in the standard library lmao.
my actually usecase is translating an api from Go, it implements fs.FS, and i want to provide equivalent functionality.
1 Like
This topic was automatically closed 90 days after the last reply. We invite you to open a new topic if you have further questions or comments.