How does one detect an interactive shell?

I have an executable that can be run from an interactive shell (Windows/Mac/Linux) or from a startup script. How can I determine whether the executable is being run from an interactive shell? Is this possible using native Rust without using the libc crate or other FFI? (One constraint I have is to minimize the size of the executable.)

On UNIX, you can use atty.

On Windows I guess if you're looking for a no additional ffi way then you can test to see if stdout is redirected to an actual file. If so, getting the file metadata should succeed. Something like this would probably work:

use std::{mem, fs, io};
// Windows only trait imports
use std::os::windows::io::{AsRawHandle, FromRawHandle};

fn is_stdin_file() -> bool {
	let stdout = io::stdout();
	let stdout = stdout.lock();
	unsafe {
		let rawhandle = stdout.as_raw_handle();
		let f = mem::ManuallyDrop::new(fs::File::from_raw_handle(rawhandle));
		f.metadata().is_ok()
	}
}

If you're willing to use some FFI then you can test if you're running in a console by seeing if you're sharing with any other process (i.e. an interactive shell process):

fn is_console() -> bool {
	unsafe {
		let mut buffer = [0u32; 1];
		let count = GetConsoleProcessList(buffer.as_mut_ptr(), 1);
		count != 1
	}
}

#[link(name="Kernel32")]
extern "system" {
	fn GetConsoleProcessList(processList: *mut u32, count: u32) -> u32;
}

If the process list returns exactly one entry then it's being run all by itself, otherwise it's likely running in an interactive shell.

1 Like

atty does have Windows support, so it's probably the best solution.

Unfortunately, atty is not suitable in this case. It is a wrapper around libc and winapi. If libc and winapi were the best/only option, then it would be better for me to inline those calls to minimize the executable size.

You will need libc or winapi, because Windows and macOS do not expose stable system call APIs. I don't think the crates themselves should have much of an impact on binary size.

Indeed. On Windows you will always have the kernel32 library dynamically linked into your process no matter what.

Even if you use LTO?

That you for the replies - they have been helpful. I work on low-level security code that has unusual requirements. C and hand-coding are normally the tools of choice. Every byte counts, but some tradeoffs are acceptable for the sake of provable correctness. The typical atty techniques are not suitable and using external crates is the least desirable option. However, the code @chrisd posted has given me some ideas about inspecting the environment. Thanks again for everyone's input.