Non-linebuffered stdout

In my neovim rpc library nvim-rs I'm using tokio's stdout to communicate with the neovim parent instance. Turns out, the linebuffering of the wrapped std::io::Stdout is harmfull, so I need to get rid of it.

One option is to use unsafe { tokio::fs::File::from_raw_fd(1) } instead, but I'm having trouble with the safety of std::os::fd:FromRawFd. Filedescriptor 1 surely is open, but I don't understand the ownership issues described in std::io - Rust. It says "no code is allowed to access...", but how could I ever ensure this with the fd 1? Every user of my lib could use the same unsafe function... Or is " Types like File own their file descriptor" authoritative here, is that true for tokio::fs::File as well? So the above is sound no matter what? I tried reading the OS specific docs, but I don't really understand, because there's no mention of File... and I also want to support mac OS, and that does not have specific docs...

So any enlightenment would be appreciated :slight_smile: Of if you know of a different way to get a non-linebuffered stdout, that would be helpfull, too.

1 Like

How about wrapping it in a BufWriter? That way you can flush it when you have enough data.

Hmm, not sure why I'd need that, I know exactly when I have enough data and what I want to write out... on the contrary, I need to remove the linebuffering to increase my control. If I wrap Stdout in a BufWriter, I'd still be linebuffered though, right? I mean, to be precise, that's what I have right now and it's not working :slight_smile:

I was thinking it would send the entire message in one go instead of just the first line. I don't really understand the bug though. Do you have a way of observing this without neovim?

This will only ever possibly work if you're fully okay with the consequences of closing file descriptor 1 after the File is dropped. "IO safety" is sort of nebulous, but it's unquestionably about who has permission to close the file descriptor.

1 Like

(Much reading later...) Yeah, "I/O Safety" is a mess, isn't it.

Anyway, as @CAD97 just noted, the unambiguous part about I/O Safety is that when an OwnedFd drops, the file descriptor gets closed. From this we can conclude that StdOut and friends cannot hold OwnedFds,[1] since the file descriptors don't close when you drop them.[2] Long story short, file descriptors 0/1/2 are special, and the exact role they play in I/O Safety is not decided; this is the best issue I found on the topic.[3]

What to do in the meanwhile? If you read all the way to the bottom of the I/O Safety docs, you'll find that the exclusivity portions apply to file descriptors specifically, and not any (potentially shared) open file description. And duping file descriptors is also supported. So if working on a duped file descriptor is adequate for your needs, you can (safely!)...

    use std::os::fd::AsFd;
    let owned = std::io::stdout().as_fd().try_clone_to_owned().unwrap();
    let file = std::fs::File::from(owned);
    let tokio_file = tokio::fs::File::from_std(file);

...and that should bypass the buffer (but won't necessarily play nice with other users of StdOut, e.g. you'll almost surely get line-tearing and other intermixing if anything happens to use StdOut).

(I didn't look into whether or not it's adequate for your needs.)


  1. or at least, not ones that correspond to the "root" file descriptors 0, 1, 2 ↩︎

  2. Or they leak the OwnedFd, maybe, but (a) this is like not having one from a user perspective and (b) this is incompatible with the stricter view of I/O Safety as it is currently documented. ↩︎

  3. IMNSHO anything stronger than "you can't close them yourself" is impractical for these file descriptors, but I'm not on any teams. ↩︎

5 Likes

switchable output buffering: #60673
ability to take ownership of stdio descriptors: ACP #148

Thanks for all those helpfull answers! Using stdout while using my lib will be a logic error anyways, so I don't mind using a non-owned FD. @quinedot's solution does not use unsafe, so that's proably my way forward :slight_smile:

This topic was automatically closed 90 days after the last reply. We invite you to open a new topic if you have further questions or comments.