File system reading

help
1

I have a file structure like this:

pack.000 (directory)
    section.000.level.000.json
    // more...
    section.005.level.040.json
pack.001 (directory)
    section.000.level.000.json
    // more...
    section.003.level.030.json

Would there be a smart way (using the glob crate) to determine the minimum and maximum 'section' from each 'pack' directory - reading the filenames - without having to process all files from the directory?
I don't know if this is possible anyway with a filestystem search mechanism...
(If not I could contemplate creating section directories).
And is the glob crate the 'recommended' crate for searching files?

2

It looks like you could greatly benefit from storing the data in a database and indexing some metadata about it.

File systems aren't, in general, guaranteed to return glob results in sorted order, so this doesn't seem possible to me without scanning all file names.

3

True, I contemplated a database.
However, I decided to make the metadata and data completely 'transparently human readable and accessible' by jsons and png's and sound files etc.
It is not a big problem. I will create a 'cached' version anyway during startup of the game.

I think glob returns names garantueed in alphabetical order.

4

You can just store the file names/paths in the database and keep the actual files on the filesystem.

That's not what I'm saying. I'm saying that the file system doesn't. Which means that glob very much has to iterate through all the files and sort them after it got them from the actual file system.

2 Likes
5

You can just store the file names/paths in the database and keep the actual files on the filesystem.

That is kind of what I am going to do (caching).
I do not store the filenames but the keys, from which it is - by logic - easy to generate the filename.
The 'resourcekey' is fixed in size and can be easily stored in a file.

pub enum ResourceKey {
    Pack { key: PackKey },
    Section { key: SectionKey },
    Level { key: LevelKey },
}

pub struct PackKey {
    pub pack_id: u16,
}

pub struct SectionKey {
    pub pack_id: u16,
    pub section_id: u16,
}

pub struct LevelKey {
    pub pack_id: u16,
    pub section_id: u16,
    pub level_id: u16,
}
6

I would be tempted to have whatever process creates these resources also create a metadata file with the information you're looking for.

That "database" would want updating if the resources are added/removed dynamically or it would get out of sync.

7

That is about my plan. The source data is transparent. The cached data is binary, compressed and very fast.
An airtight 'out of sync' plan is not possible I think.

8

This topic was automatically closed 90 days after the last reply. We invite you to open a new topic if you have further questions or comments.