How to cheaply read non-blocking from a File?

I know this question is asked all the time, but I still can't find an answer: How to (cheaply) read non-blocking from a file? I found three ways, but all of them don't work (well) in my case:

  • Let a blocking read run in a separate thread and communicate over channels and just leave the thread and file descriptor be if it hangs. This doesn't work for since my program runs for a long time (it's a daemon, so potentially days, months, years ...) and there is a limit on the open file descriptors allowed.
  • Use tokio or another async library. I don't want to do this because at least tokio is - I think - pretty big and I don't want to have so much overhead for so little.
  • Extract the file descriptor and then use linux syscalls to put the file descriptor in non-blocking mode. I don't want to do that because I would need to use unsafe for this.

Is there any other possibility or do I have to use one of the last two options?

Thanks!

Assuming that by "file" you mean special files like pipes, sockets and the like you can use mio::Poll to have a cross-platform abstraction for readiness notifications. Once a fd is ready you can keep reading until you get a wouldblock error.

For regular (on-disk) files cross-platform support for nonblocking IO is spotty because they can't be handled with a readiness-model, you need a completion-based model such as io_uring or iocp which aren't available on all operating systems. That's why async runtimes have a fallback to use threads instead.

1 Like

By "file" I mean all files which find / would list. How does this work for that?

Well, that's mostly regular files. In that case io_uring is the only sane way to get non-blocking io on linux. There are also half-baked solutions like aio or rwf_nowait. But none of them are portable.

If you need a portable solution then use threads, that's what the async runtimes do under the hood anyway on most platforms.

3 Likes

This topic was automatically closed 90 days after the last reply. We invite you to open a new topic if you have further questions or comments.