Dealing with too many open files

I'm working on a program that needs to send files as fast as possible to a server, but I'm running into the limit of open file descriptors, resulting in a "too many open files" error.

I can work around this by reducing the concurrency rate manually, but that isn't really flexible enough since I don't want to hardcode something like that.

I've also thought about using retry logic for this, but that also seems kind of messy to just mindlessly keep slamming the filesystem with retries until it works again.

Is there anything that is more ideal for this? A kind of "file descriptor backpressure" crate or pattern?

Is it possible to have some kind of async system that creates a queue and moves on to the next file to open whenever it is able to without having to deal with these errors?

How often are the files changed? Do those files have any interdependencies, i.e. if one file changes and another file changes, sending an outdated version of one file and an up-to-date version of the other may cause errors on the receiving end?

I'm not super concerned about file changes right now. It's really just whenever the program can open the file then send it. The main problem is just optimizing for speed since it is potentially a very large number of files.

I'm fairly sure there is, unfortunately, no general mechanism to wait for a new file descriptor to become available.

You could request the maximum number of open files from the operating system at start, then limit concurrency to that (minus a few fds needed for other things).

If you can't limit concurrency like that, something like a semaphore might also work, e.g. reset the semaphore initially the max number of fds, then decrease the value when you take one, and increase it again on close.

You could try setting up a file manager that acts a little like a Rc, that gives out a file FileRef object that implements DerefMut<Target=File>, and decrements the count and keeps a count of how many have been handed out. Once it starts getting errors, it knows the cap (and can back off 5-10% to be safe, if you like), and can be responsible for throttling further requests. Check out jon Gjengset's YouTube stream on smart pointers for details as to how that would work.

1 Like

If this is a Linux server, and you have root access, you can raise the open file limit. I think the exact configuration file to modify for a permanent increase depends on your distribution, but you should be able to use ulimit for a temporary increase to test.

If you're running on a thin VPS and can't raise it high enough, or otherwise the hard max is still too high, then I'd recommend rate limiting with a Semaphore. You should be able to find the maximum number of open file descriptors from the OS, maybe subtract a few for leeway (I think stdin/stdout count towards your limit?), and then use that number as the number of permits available in the Semaphore. It should act as an artificial resource limiter without having to mess with the actual resources or change any other code.

Semaphores are one of those synchronization primitives I completely forget about until the single time it's the right thing to use, like here. Like, you'd think limiting access to N concurrent users is a common problem, and early computer scientists certainly did, but I almost never use them in real life.

3 Likes

This topic was automatically closed 90 days after the last reply. We invite you to open a new topic if you have further questions or comments.