Idiomatic way to pass a callback (WNDPROC) to the Win32 API?

1

I'm trying to avoid using global state if possible, but I suspect this is impossible.

When creating a window, I want to pass the Win32 API a callback function (known as a window procedure) which will be called whenever a window event occurs (window moved, mouse clicked, window resized etc). I have my own Window struct which is a wrapper around the Windows HWND type. I want to avoid using global state if possible and have the window procedure call one of my struct's methods.

Is the only way to do this to have my Window struct stored globally, as in statically in an Arc<Mutex<Option<Window>>>, and have the window procedure check to see if it's been instantiated yet and then call the struct's method?

2

Naw. You can attach data to Windows. In your case you'll be attaching a pointer to your structure. GetWindowLong is used to read the data (as a pointer sized value). SetWindowLong to write data. When you create the Window class just request enough space for whatever you plan to store with each Window. Look for the cbWndExtra field in the WNDCLASSEXA struct.

Bear in mind that the ANSI functions (those ending in A) use the current locale which may or may not be UTF-8.

3

Thanks!

Something like this (excuse the pseudo-code)? Will there be an issue with the pointer becoming invalidated once I return from new()?
EDIT: not if I wrap it in an Rc!

struct Window {
    inner: HWND,
}
impl Window {
    fn new() -> Window {

        ... //Create window class etc. Pass in window_proc() as the window procedure

        let my_window = Window { CreateWindowW(...) };
        SetWindowLongPtrW(my_window.inner, GWLP_USERDATA, my_window.as_ptr() as LONG_PTR);
        my_window //Won't the memory address of my_window change when I return from this function, causing Win32 to be storing an invalid pointer address for it?
    }

    fn window_proc(&self, msg: UINT, w_param: WPARAM, l_param: LPARAM) -> LRESULT {
        // Actual window proc logic here
    }
}

unsafe extern "system" fn window_proc(hwnd: HWND, msg: UINT, w_param: WPARAM, l_param: LPARAM) -> LRESULT {
    //Call GetWindowLongPtrW to get ptr to Window, and dereference it
    //Then call that window's window proc and return LRESULT through
}
5

Yes. You have to wrap your Rust struct in something to ensure it's stored on the heap, the address is stable, and ownership is under your control. You will have to take care that the pointer you stored with the Window handle also holds a strong reference. Were I in your shoes I'd place a canary in the struct to help ensure I got the details correct.

But, your pseudo-code is pointing you in the right direction.

GWLP_USERDATA. Sweet. The last time I had to do this sort of thing GWLP_USERDATA did not exist. It's cool the Microsoft now includes that.

6

It's been there since at least 2005, so it's sure been a while :slightly_smiling_face:

Be careful here -- depending on what definition of HWND you're using, your Window struct may be Send/Sync, so you may inadvertently be exposing the Rc cross-thread, which is unsound. I'm also not completely confident that the winproc is 100% guaranteed to only be called on the same thread it's initialized from; I think it is, but I'd need to double-check the documentation before I'm fully comfortable relying on that to be true. If my memory serves me, giving a NULL HWND to the message pumping API only processes/dispatches messages owned on the current thread, so it's probably fully sound to say that if you don't expose Send/Sync access to the HWND that messages will only be dispatched on the single thread with access to the HWND. It's probably even an error to use a window HWND from a thread other than the creating one; I just barely know enough to know what to be wary of.

(winapi's HWND is not Send/Sync, but the windows crate's is, so if you're using the windows crate you probably want to put a PhantomData<*const Window> in your Window, representing by-example the *const Window stored in the GWLP_USERDATA region.)

I'm assuming you've written something with the shape of fn new() -> Rc<Window>. There's now a three-way choice to be made: do you store Rc::as_ptr, Rc::into_raw, or rc::Weak::into_raw on the window? Rc::into_raw is sound but wrong, as now it's impossible for the Window to drop, since it's holding a strong reference to itself. Storing Rc::as_ptr is unsound; Rc::get_mut allows you to get &mut Window access and cause problems[1] with that. You can make it sound, but you'd need to make Window not Unpin and use Pin<Rc<Window>> instead. Storing Weak::into_raw is thus the safest option, so long as you upgrade the weak reference from the window procedure instead of just assuming the pointer is valid; Rc::make_mut exists and will invalidate your weak references[2]. Rc::new_cyclic exists specifically to enable this Weak<Self> pattern in pure safe Rust code.

There's actually also one other alternative: if your Window is only the HWND and no other state, you don't actually need to put it behind another allocation. You can instead just create a temporary imitation Window and call the method on that, e.g.

let my_window = ManuallyDrop::new(Window { inner: hwnd });
my_window.window_proc()

and "all" you have to do to ensure this is okay is make sure the window procedure never has the opportunity to drop your imitation Window (thus ManuallyDrop) and never assume &mut Window means exclusive access to the HWND.

... though all of these are implicitly assuming that the window procedure is only called for messages for instances of Window, and you don't get other bonus messages. Cross-referencing my C++ WinProc I'm using (which does use a global), I'd probably write this as roughly

struct Window {
    hwnd: HWND,
    // other state/fields
    marker: PhantomData<Weak<Self>>,
}

impl Window {
    pub fn new() -> Rc<Self> {
        let hwnd = /* ... */;
        Rc::new_cyclic(|weak| {
            SetWindowLongPtrW(hwnd, GWLP_USERDATA, weak.into_raw() as LONG_PTR);
            Window { hwnd, ..., marker: PhantomData }
        })
    }

    fn proc(&self, msg: UINT, w: WPARAM, l: LPARAM) -> LRESULT {
        // ...
    }
}

unsafe extern "system" fn win_msg_proc(hwnd: HWND, msg: UINT, w: WPARAM, l: LPARAM) -> LRESULT {
    if hwnd.is_null() {
        return DefWindowProcW(hwnd, msg, w, l); // bonus thread message, probably
    }

    let raw = GetWindowLongPtrW(hwnd, GWLP_USERDATA) as *const Window;
    if raw.is_null() {
        return DefWindowProcW(hwnd, msg, w, l); // weird, different window source?
    }

    let weak = ManuallyDrop::new(Weak::from_raw(raw));
    let Some(window) = weak.upgrade()
    else {
        return DefWindowProcW(hwnd, msg, w, l); // lost window
    };

    window.proc(msg, w, l)
}

(completely untested)

  1. The problem is similar to reentrancy: if you grab &mut Window and then run the message pump while that borrow is still active, the message dispatch will create &Window, aliasing your &mut, resulting in UB. ↩︎

  2. When storing a raw as_ptr on the window, we're essentially storing a weak pointer without actually increasing the weak count. The unsoundness of not upgrading the weak reference is functionally the same both times, it's just a difference of what Rc method we use to realize the unsoundness. ↩︎

1 Like
7

I'd increment the reference count before the SetWindowLongPtrW then decrement in a WM_DESTROY handler. That guarantees the Rust struct has the same lifetime as the Window handle and eliminates having to deal with a weak reference.

Yeah. Delphi mostly put an end to my having to deal with the raw Windows API.

8

The traditional approach is a WindowData that you Box::new().into_raw() and put in GWLP_USERDATA, and drop(Box::from_raw()) on WM_NCDESTROY, and the Window you return from new is a thin HWND wrapper. I've been considering creating this as a crate wrapping this up specifically for windows since I've needed it several times now.

Note that window procedures are highly likely to reenter and can be invoked across threads completely out of your processes control (by global window hooks at least, iirc), so ensure your raw procedure only hands out shared references.

2 Likes
9

FWIW The create_window sample in the windows-rs source shows a basic window setup with wndproc. Other samples there are helpful, too.

1 Like
10

This topic was automatically closed 90 days after the last reply. We invite you to open a new topic if you have further questions or comments.