64-bit calling 32-bit DLL causes a crash

This function is compiled into a DLL and used by the script. It creates a rust thread to monitor the channel, and callback function to the script after receiving the data. I tested that it is normal in 32-bit and 64-bit, but when I use a 64-bit program to send data to 32-bit, pipe functions such as receiving and reading are normal, but the callback function call_back will cause the 32-bit receiver to crash , It seems that because of receiving 64-bit data, the function is destroyed.
I can't find out the cause of the crash, what went wrong?

pub extern "C" fn pipe_new(pipe_name: *const i8, ptr: usize, sz: usize) {
	unsafe {
		let name: Vec<u16> = format!(r"\\.\pipe\{}", CStr::from_ptr(pipe_name).to_str().unwrap()).as_str().encode_utf16().chain(Some(0)).collect();
		let call_back = transmute::<_, extern "C" fn()>(ptr); 
		// call_back(); // Calling here is normal.
		std::thread::spawn(move || {
			let mut len = 0u32;
			loop
			{
				let pipe = CreateNamedPipeW(name.as_ptr(), 3, 0, 255, 0, 0, 0, null_mut()); 
				ConnectNamedPipe(pipe, null_mut());
				let mut buf = vec![0u8; sz];
				ReadFile(pipe, buf.as_mut_ptr() as *mut c_void, sz as u32, &mut len, null_mut());
				call_back(); //If a 64-bit program sends data, it will cause a crash.
				CloseHandle(pipe);
			}
			
		});
	}
}

I'm not sure if linking 32-bit and 64-bit binaries can ever work. If it's possible at all, you'll need to avoid usize in the interface because it changes width based on the architecture. Instead, you'll need to specify a fixed-width type like u32 or u64.

Also, *const i8 will be the wrong size; You'll need to figure out a way to avoid that type as well.

1 Like

There are fundamental differences between the ABIs of functions compiled for 32-bit and 64-bit targets. Even if you only ever use fixed-size types in all of your interfaces, there will be differences in e.g. what registers or stack locations the compiler will put your function arguments and return values in. These differences make it borderline impossible to correctly interoperate between two such pieces of code.

If you are compiling your main executable for a 64-bit target, then use a 64-bit DLL to extend it. Conversely, if your main executable is 32-bit, then only call into 32-bit DLLs. Nothing else can reasonably be expected to work.

3 Likes

It doesn't seem so. 64-bit programs can call 32-bit normally. If you don't use a new thread and let dll block the thread, it will run normally. e.g.

pub extern "C" fn pipe_new(pipe_name: *const i8, ptr: usize, sz: usize) {
	unsafe {
		let name: Vec<u16> = format!(r"\\.\pipe\{}", CStr::from_ptr(pipe_name).to_str().unwrap()).as_str().encode_utf16().chain(Some(0)).collect();
		let call_back = transmute::<_, extern "C" fn()>(ptr); 
		let mut len = 0u32;
		let pipe = CreateNamedPipeW(name.as_ptr(), 3, 0, 255, 0, 0, 0, null_mut()); 
		ConnectNamedPipe(pipe, null_mut());
		let mut buf = vec![0u8; sz];
		ReadFile(pipe, buf.as_mut_ptr() as *mut c_void, sz as u32, &mut len, null_mut());
		call_back(); //If a 64-bit program sends data, it will cause a crash.
		CloseHandle(pipe);	
	}
}

I tried all the possible problems caused by the data structure, and it still crashed.

Microsoft said you can't.

4 Likes

The fact that a piece of code happens to not crash at this time does not constitute proof that it is indeed legal. The fact that its "correctness" depends on whether or not you start a new thread already shows that what you are trying to do is at least flaky, but certainly not robust and memory-safe by any standard.

Thanks. But I'm still confused. If there is no "callback function", it is completely normal.

Thanks. This means that a 64.exe communicates with 32.exe, 32.exe must wrap its receiving function with COM?

1 Like

That article seems to recommends so.