The buffer I/O causes the system cache to be uncontrolled. In scenarios where resources are insufficient, directIO is a good choice. However, I encountered problems when using directIO.
Through libc's code, I can use it normally.
fn test_libc() {
let path = CString::new("/tmp/1.txt").unwrap();
let flags = libc::O_DIRECT | libc::O_WRONLY | libc::O_CREAT;
let mode: c_int = 0o644;
let fd = unsafe { libc::open(path.as_ptr(), flags, mode) };
use std::os::fd::FromRawFd;
let mut file = unsafe {File::from_raw_fd(fd)};
let buffer = vec![b'2'; 4096];
file.write_all(&buffer).unwrap();
}
Using the standard library code, I ran into problems.
fn test_std() {
use std::os::unix::fs::OpenOptionsExt;
let mut file = OpenOptions::new()
.write(true)
.create(true)
.custom_flags(libc::O_DIRECT)
.open("/tmp/1.txt").unwrap();
let buffer = vec![b'1'; 4096];
file.write_all(&buffer).unwrap();
}
EINVAL The filesystem does not support the O_DIRECT flag. See
NOTES for more information.
And in NOTES there's a whole bunch of paragraphs on O_DIRECT, but the plausible ones seem like
Under Linux 2.4, transfer sizes, the alignment of the user
buffer, and the file offset must all be multiples of the logical
block size of the filesystem. Since Linux 2.6.0, alignment to
the logical block size of the underlying storage (typically 512
bytes) suffices. The logical block size can be determined using
the ioctl(2) BLKSSZGET operation or from the shell using the
command:
blockdev --getss
And
O_DIRECT support was added under Linux in kernel version 2.4.10.
Older Linux kernels simply ignore this flag. Some filesystems
may not implement the flag, in which case open() fails with the
error EINVAL if it is used.
If that panic line number is on the open(), I would guess you're hitting the latter, but then you should see the same thing earlier.
If it's the former paragraph, it would be on the later read() call itself, and it probably only works by accident on the first program. The reliable solution needs you to do an aligned allocation. There's no way to do this dynamically with a Vec in stable yet (the allocator_api nightly feature might provide a way in the future?), but for now you can either directly use alloc() (hopefully wrapped for safety) or create a buffer type with an #[align()] attribute
This is a simple DirectIO I wrote, and I expected it to work, but it was somewhat unexpected. Case 1 works properly, but case 2 fails in the 512 scenario.
fn case1() {
let path = "/tmp/1.txt";
test(path, 512);
println!("case1 success");
}
fn case2() {
let path = "/tmp/1.txt";
for i in 1..10240 {
test(path, i);
}
}
fn test(path: &str, len: usize) {
let mut file = DirectIO::open(path).unwrap();
let expect = vec![b'1'; len];
file.write_all(&expect)
.map_err(|e| {
eprintln!("err len: {}", len);
e
})
.unwrap();
file.flush().unwrap();
let mut file = File::open(path).unwrap();
let mut actual = Vec::with_capacity(len);
file.read_to_end(&mut actual).unwrap();
assert_eq!(actual, expect);
std::fs::remove_file(path).unwrap();
}
use libc::c_int;
use std::ffi::CString;
use std::io;
use std::io::Write;
const SIZE_OF_BLOCK: usize = 512;
const ALIGN_SIZE_OF_BLOCK: usize = !(SIZE_OF_BLOCK - 1);
pub struct DirectIO {
fd: c_int,
buf: [u8; SIZE_OF_BLOCK],
n: usize,
}
impl Drop for DirectIO {
fn drop(&mut self) {
unsafe {
let i = libc::close(self.fd);
if i != 0 {
panic!("{}", io::Error::last_os_error())
}
}
}
}
impl DirectIO {
pub fn open(path: &str) -> io::Result<Self> {
let path = CString::new(path).unwrap();
let flags = libc::O_DIRECT | libc::O_WRONLY | libc::O_CREAT | libc::O_TRUNC;
let mode: c_int = 0o644;
let fd = unsafe { libc::open(path.as_ptr(), flags, mode) };
if fd == -1 {
Err(io::Error::last_os_error())
} else {
Ok(Self {
fd,
buf: [0u8; SIZE_OF_BLOCK],
n: 0,
})
}
}
fn write_direct(&self, buf: &[u8]) -> io::Result<usize> {
if buf.is_empty() {
return Ok(0);
}
let rt = unsafe { libc::write(self.fd, buf.as_ptr().cast(), buf.len()) };
if rt >= 0 {
Ok(rt as usize)
} else {
Err(io::Error::last_os_error())
}
}
}
impl Write for DirectIO {
fn write(&mut self, mut buf: &[u8]) -> io::Result<usize> {
let data_len = buf.len();
if self.n != 0 {
let end = self.n + buf.len();
if end < self.buf.len() {
self.buf[self.n..end].copy_from_slice(buf);
self.n = end;
return Ok(buf.len());
} else {
let r = self.buf.len() - self.n;
self.buf[self.n..].copy_from_slice(&buf[..r]);
let n = self.write_direct(&self.buf)?;
assert_eq!(n, self.buf.len());
self.n = 0;
buf = &buf[r..];
}
}
while buf.len() >= SIZE_OF_BLOCK {
let r = buf.len() & ALIGN_SIZE_OF_BLOCK;
let n = self.write_direct(&buf[..r])?;
buf = &buf[n..];
}
if !buf.is_empty() {
self.buf[0..buf.len()].copy_from_slice(buf);
self.n = buf.len();
}
Ok(data_len)
}
fn flush(&mut self) -> io::Result<()> {
let fd = self.fd;
let flags: c_int = unsafe { libc::fcntl(fd, libc::F_GETFL) };
if flags == -1 {
return Err(std::io::Error::last_os_error());
}
let new_flags: c_int = flags & !libc::O_DIRECT;
if unsafe { libc::fcntl(fd, libc::F_SETFL, new_flags) } == -1 {
return Err(std::io::Error::last_os_error());
}
self.write_direct(&self.buf[..self.n]).unwrap();
self.n = 0;
if unsafe { libc::fcntl(fd, libc::F_SETFL, flags) } == -1 {
return Err(std::io::Error::last_os_error());
}
Ok(())
}
}
that buf will be very unlikely to be aligned. You would need to do:
#[repr(C)] // lay out the items in order like C does
#[align(512)] // or whatever
pub struct DirectIO {
buf: [u8; SIZE_OF_BLOCK], // must be first so it has the alignment of the type as a whole
fd: c_int,
n: usize,
}
Also, you probably don't want to unwrap() and therefore panic if you're trying to check each alignment...
Thanks, unwrap is just written in the demo. I tried the writing you gave, but still the same problem.
#[repr(align(512))] // lay out the items in order like C does
pub struct DirectIO {
buf: [u8; SIZE_OF_BLOCK], // must be first so it has the alignment of the type as a whole
fd: c_int,
n: usize,
}
