How to allocate memory for an FFI call

I would like to call the Windows function GetAdaptersAddresses, which expects a pointer to a buffer passed as adapteraddresses argument to the function (see Windows documentation and example code in C).

I assume that the buffer needs to be properly aligned, just as malloc in C also cares about alignment:

The malloc() and calloc() functions return a pointer to the allocated memory, which is suitably aligned for any built-in type.

I have seen that the default-net crate uses the memalloc::allocate function. I don't want to do that for the following reasons:

How do I generally allocate buffers to be used for FFI calls? Which function should I use, and how do I select the proper alignment? Ideally, I would like a solution which just relies on std.

Moreover: Is default-net unsound due to this non-aligned allocation, or does it not matter?

Why don't you just declare the out parameter on the stack? (Also, memory allocators usually return pointers that have large enough alignment for any type, so you shouldn't need to worry about it. Rust can violate this assumption using the align(_) attribute, but we are talking about a C struct here, so even if you want to heap-allocate, libc::malloc() should be fine.)

On my machine, I need about 16 kB. Maybe a bit big for the stack?

Note that the allocation needs to be bigger than IP_ADAPTER_ADDRESSES_LH because it's actually an array being written.

Yes, like malloc(). But what is the Rust equivalent of that?

Edit: Note that memalloc::allocate explicitly uses u8 alignment.

Yes, that should work, but is that idiomatic? Can't I use a Rust-based allocator? Do I need the libc crate?

Hm, alright. Not a Windows user here; only looking at the documentation, I thought it was the head of a linked list (since the docs say that it's a linked list). But indeed, if it's an array, then it can be too big to stack-allocate.

Well, the direct equivalent is libc::malloc(). (Does that exist/work on Windows?) But I wouldn't bother heap-allocating raw pointers manually. You could just create a Vec::with_capacity() and then obtain a pointer to it using spare_capacity_mut(). (Or even zero-initialize the whole array of structs and avoid any sort of unsafe related to MaybeUninit.)

You totally can, that's what I described above. If you still absolutely positively want to deal with raw pointers for some reason, then use <std::alloc::System as GlobalAlloc>::alloc().

Yeah, it is, but the elements don't seem to have an own allocation (they all use the memory of the passed buffer, I believe, because according to the example code you only need to free that one buffer to clean up).

I'm not really a Windows user either, except for gaming and when I need to. I think libc does also exist on Windows, but I don't know much about it (yet) and/or where are the differences.

I will know the length in bytes that I need. AFAIK Vec<u8> won't ensure any alignment, right? If I used Vec<T> with some type T (which one?) which does care about alignment, then I have to do an extra division to calculate the element lengthcount. It all seems a bit messy.

This needs a Layout. How do I figure out the alignment that malloc guarantees? I guess I could do something like std::mem::align_of::<u128>() to get an alignment "which is suitably aligned for any built-in type" (quoted from the malloc manpage)? Is that idiomatic?

I guess in the particular case, I might also do std::mem::align_of::<IP_ADAPTER_ADDRESSES_LH>(), though I'm not entirely sure if that's correct because I won't really know how this weird Windows API function uses the memory and what it expects in regard to alignment.


Extra question: Should I use std::alloc::System to ensure alignment? Can I pass a "fake" alignment of 1 in that case? Or are there other reasons to use the System allocator over the global allocator using std::alloc::alloc?

Vec::<T>::with_capacity(n_bytes / size_of::<T>) is certainly not what I would consider "messy", especially compared to going all manual with raw pointers and all that. (I mean, Vec is the type for dynamic arrays.) As to "which one?": isn't it the struct IP_ADAPTER_ADDRESSES_LH?

You can get a Layout for any (sized) type usig Layout::new().

#include <stddef.h>

size_t max_align = _Alignof(max_align_t);

I believe no. I was suggesting System because I misread the documentation and thought that GlobalAlloc::alloc() was still unstable, but it isn't. I don't think the distinction matters a lot, and you are probably not going to get away with a fake alignment of 1 either way (Miri can potentially find that it's UB, if for no other reason).

Hmm, thanks. Though I'm still not sure what to use as T:

Also, I wonder what's the general solution for this problem, i.e. if some FFI requires some (malloc aligned) scratch space with n bytes size. Would I need to use bindgen to get max_align_t? Is there some "Rusty" way to get the maximum alignment?


Edit: Also note that IP_ADAPTER_ADDRESSES_LH might be much bigger than the actual alignment needed. If for some reason, a random amount of bytes is requested (and not a multiple of that structure, e.g. due to extra data needed for pointed-to sub-elements), I will have some rounding loss if I use the Vec approach.

No function taking an out argument of type *mut T should expect higher alignment than align_of::<T>(). If it does, that's a spectacularly bad API, and I don't think you can in general guard against it, except for extreme approaches such as reverse engineering the function. If Vec::<IP_ADAPTER_ADDRESSES_LH>::with_capacity(N).as_mut_ptr() is not good enough for that function, then you've got much bigger problems.

I'm feeling you are still approaching this from the wrong angle. If a function needs to write an array of T, then passing a pointer to the buffer of a Vec<T> should suffice. If it doesn't, then that's pretty much a special case that needs special handling. In particular, no conforming C program will expect types with alignment > _Alignof(max_align_t), since that'd be impossible to serve by malloc() (which doesn't take an alignment parameter).

So to sum up: the best approach is to just allocate a Vec, and the second best approach is to call malloc() if you need to pass it to something that expects the return value of malloc.

1 Like

See my edit above: The memory needed might not be a multiple of IP_ADAPTER_ADDRESSES_LH.

I'm afraid I don't follow. Why would you request a "random amount of bytes"? The usual usage pattern of array out parameters in C is

void fill_buf(T *buf, size_t max_elems) { … }

where buf is expected to be a pointer to max_elems items, each of size T. Is this not what the aforementioned function does? If not, then what is it doing?

I think I was wrong that it's actually an array. It's basically a chunk of memory which holds both the linked list and also possibly other elements. On my system, I have the following sizes:

  • std::mem::size_of::<windows::Win32::NetworkManagement::IpHelper::IP_ADAPTER_ADDRESSES_LH>() == 448
  • *sizepointer == 16383 (after calling GetAdaptersAddresses with a too-small buffer length)

I.e. the function requests 16383 bytes (which is an odd number of bytes) on my computer with my setup.

So my problem is:

  • I need a buffer of n bytes (where n may be any number, including odd numbers).
  • I want the buffer to be aligned for any data type (just like malloc would do, to not run into bad surprises).
  • I would like to use std.

I also expect this problem to generally occur often when dealing with C APIs because C functions often need some sort of buffers, and these are not always allowed to be unaligned, I guess.

Then you can construct a Layout with that particular size and the alignment of the struct type, I believe.

I wouldn't expect that exactly for the reason I explained previously. This particular API you are dealing with is weird. A well-behaved C API should expect an output buffer which is either a pointer to a single element or an array, with alignment not exceeding that of max_align_t.

It depends on the application. In this case, the docs for GetAdaptersAddresses () say that it'll fill the buffer with an intrusive linked-list of IP_ADAPTER_ADDRESSES_LH's.

Addresses are returned as a linked list of IP_ADAPTER_ADDRESSES structures in the buffer pointed to by the AdapterAddresses parameter.

If it's storing a bunch of IP_ADAPTER_ADDRESSES_LH values then you just need to make sure the allocation has the same alignment as IP_ADAPTER_ADDRESSES_LH.

Don't over-think things.

The only point that you might need to be wary of is that each node in the linked list points to a different part of the same buffer. That means you'll need to make sure you don't accidentally re-allocate.

I'd probably do the allocations manually with std::alloc::alloc(), possibly wrapping the unsafe code in a type that can manage the buffer and has an fn iter(&self) -> impl Iterator<Item=&IP_ADAPTER_ADDRESSES_LH> method for traversing the linked list.

// call it once to find out the required buffer size
let mut size = 0;

GetAdaptersAddresses(..., None, &mut size);

let align =  std::mem::align_of::<IP_ADAPTER_ADDRESSES_LH>();
let layout = Layout::from_size_align(size, align).unwrap();
let buffer: *mut IP_ADAPTER_ADDRESSES_LH = std::alloc::alloc(layout).cast();

// Call it for real this time
let mut size = 0;
let ret = GetAdaptersAddresses(..., Some(buffer), &mut size);
handle_errors(ret);

let addresses = std::slice::from_raw_parts(buffer, size as usize);

do_something_with_addresses(addresses);

std::alloc::dealloc(layout, buffer.cast());
2 Likes

Just watch out when using this solution: you'll have to be careful to handle the ERROR_NO_DATA case and not attempt to allocate a buffer if you get it from the first GetAdaptersAddresses() call, since std::alloc::alloc() does not accept a 0-size layout. Also, you'll have to check if alloc() returns ptr::null(), and not attempt to use the pointer. Way too many crates have been bitten by not properly checking these things.

Meanwhile, the documentation for GetAdaptersAddresses() discourages this particular method:

One method that can be used to determine the memory needed to return the IP_ADAPTER_ADDRESSES structures pointed to by the AdapterAddresses parameter is to pass too small a buffer size as indicated in the SizePointer parameter in the first call to the GetAdaptersAddresses function, so the function will fail with ERROR_BUFFER_OVERFLOW. When the return value is ERROR_BUFFER_OVERFLOW, the SizePointer parameter returned points to the required size of the buffer to hold the adapter information. Note that it is possible for the buffer size required for the IP_ADAPTER_ADDRESSES structures pointed to by the AdapterAddresses parameter to change between subsequent calls to the GetAdaptersAddresses function if an adapter address is added or removed. However, this method of using the GetAdaptersAddresses function is strongly discouraged. This method requires calling the GetAdaptersAddresses function multiple times.

Instead, it recommends starting with size 15000, and using a larger allocation if you get ERROR_BUFFER_OVERFLOW.

One problem might be that if you look at the fields of IP_ADAPTER_ADDRESSES_LH, it contains pointers to other types such as IP_ADAPTER_UNICAST_ADDRESS_LH, IP_ADAPTER_ANYCAST_ADDRESS_XP, IP_ADAPTER_MULTICAST_ADDRESS_XP, IP_ADAPTER_DNS_SERVER_ADDRESS_XP, IP_ADAPTER_PREFIX_XP, IP_ADAPTER_WINS_SERVER_ADDRESS_LH, IP_ADAPTER_GATEWAY_ADDRESS_LH, and IP_ADAPTER_DNS_SUFFIX. So even though the buffer starts with an IP_ADAPTER_ADDRESSES_LH, that may not be the only thing it contains.

1 Like

I might as well give my own attempt at a solution. The example in GetAdaptersAddress() uses HeapAlloc()/HeapFree() to allocate memory, so clearly the alignment of that API must be sufficient. If we look at the documentation for HeapAlloc(), it says:

The alignment of memory returned by HeapAlloc is MEMORY_ALLOCATION_ALIGNMENT in WinNT.h:

#if defined(_WIN64) || defined(_M_ALPHA)
#define MEMORY_ALLOCATION_ALIGNMENT 16
#else
#define MEMORY_ALLOCATION_ALIGNMENT 8
#endif

And this value is exposed by windows-rs. So we can use it as our alignment when creating a Layout for alloc::alloc(), and otherwise follow the method of the example:

/*
[dependencies.windows]
version = "0.48.0"
features = [
    "Win32_Foundation",
    "Win32_NetworkManagement_IpHelper",
    "Win32_NetworkManagement_Ndis",
    "Win32_Networking_WinSock",
    "Win32_System_SystemServices",
]
*/

use std::alloc::{self, Layout};
use windows::core::Result as WinResult;
use windows::Win32::{
    Foundation::{ERROR_BUFFER_OVERFLOW, ERROR_NOT_ENOUGH_MEMORY, ERROR_SUCCESS, WIN32_ERROR},
    NetworkManagement::IpHelper::{
        GetAdaptersAddresses, GET_ADAPTERS_ADDRESSES_FLAGS, IP_ADAPTER_ADDRESSES_LH,
    },
    System::SystemServices::MEMORY_ALLOCATION_ALIGNMENT,
};

pub fn get_adapters_addresses(
    family: u32,
    flags: GET_ADAPTERS_ADDRESSES_FLAGS,
) -> WinResult<(*mut IP_ADAPTER_ADDRESSES_LH, usize)> {
    unsafe {
        let mut size_u32 = 15000;
        let align: usize = MEMORY_ALLOCATION_ALIGNMENT.try_into().unwrap();
        loop {
            let size: usize = size_u32.try_into().unwrap();
            let Ok(layout) = Layout::from_size_align(size, align) else {
                return Err(ERROR_NOT_ENOUGH_MEMORY.into());
            };

            let ptr = alloc::alloc(layout);
            if ptr.is_null() {
                return Err(ERROR_NOT_ENOUGH_MEMORY.into());
            }

            let result = GetAdaptersAddresses(family, flags, None, Some(ptr.cast()), &mut size_u32);
            match WIN32_ERROR(result) {
                ERROR_SUCCESS => return Ok((ptr.cast(), size)),
                ERROR_BUFFER_OVERFLOW => {
                    alloc::dealloc(ptr, layout);
                    continue;
                }
                error => {
                    alloc::dealloc(ptr, layout);
                    return Err(error.into());
                }
            }
        }
    }
}

pub unsafe fn free_adapters_addresses(ptr: *mut IP_ADAPTER_ADDRESSES_LH, size: usize) {
    let align: usize = MEMORY_ALLOCATION_ALIGNMENT.try_into().unwrap();
    let layout = Layout::from_size_align(size, align).unwrap();
    alloc::dealloc(ptr.cast(), layout);
}

fn main() {
    let (head, size) = get_adapters_addresses(0, Default::default()).unwrap();
    unsafe {
        let mut ptr = head;
        while !ptr.is_null() {
            println!("{}", (*ptr).FriendlyName.display());
            ptr = (*ptr).Next;
        }
        free_adapters_addresses(head, size);
    }
}

Alternatively, to avoid lugging around the size, we can call HeapAlloc()/HeapFree() directly, as std already does internally.

Alternative method
/*
[dependencies.windows]
version = "0.48.0"
features = [
    "Win32_Foundation",
    "Win32_NetworkManagement_IpHelper",
    "Win32_NetworkManagement_Ndis",
    "Win32_Networking_WinSock",
    "Win32_System_Memory",
]
*/

use windows::core::Result as WinResult;
use windows::Win32::{
    Foundation::{ERROR_BUFFER_OVERFLOW, ERROR_NOT_ENOUGH_MEMORY, ERROR_SUCCESS, WIN32_ERROR},
    NetworkManagement::IpHelper::{
        GetAdaptersAddresses, GET_ADAPTERS_ADDRESSES_FLAGS, IP_ADAPTER_ADDRESSES_LH,
    },
    System::Memory::{GetProcessHeap, HeapAlloc, HeapFree},
};

pub fn get_adapters_addresses(
    family: u32,
    flags: GET_ADAPTERS_ADDRESSES_FLAGS,
) -> WinResult<*mut IP_ADAPTER_ADDRESSES_LH> {
    unsafe {
        let heap = GetProcessHeap()?;
        let mut size_u32 = 15000;
        loop {
            let size: usize = size_u32.try_into().unwrap();
            let ptr = HeapAlloc(heap, Default::default(), size);
            if ptr.is_null() {
                return Err(ERROR_NOT_ENOUGH_MEMORY.into());
            }

            let result = GetAdaptersAddresses(family, flags, None, Some(ptr.cast()), &mut size_u32);
            match WIN32_ERROR(result) {
                ERROR_SUCCESS => return Ok(ptr.cast()),
                ERROR_BUFFER_OVERFLOW => {
                    HeapFree(heap, Default::default(), Some(ptr)).ok()?;
                    continue;
                }
                error => {
                    // ignore secondary error
                    HeapFree(heap, Default::default(), Some(ptr));
                    return Err(error.into());
                }
            }
        }
    }
}

pub unsafe fn free_adapters_addresses(ptr: *mut IP_ADAPTER_ADDRESSES_LH) -> WinResult<()> {
    let heap = GetProcessHeap()?;
    HeapFree(heap, Default::default(), Some(ptr.cast())).ok()
}

fn main() {
    let head = get_adapters_addresses(0, Default::default()).unwrap();
    unsafe {
        let mut ptr = head;
        while !ptr.is_null() {
            println!("{}", (*ptr).FriendlyName.display());
            ptr = (*ptr).Next;
        }
        free_adapters_addresses(head).unwrap();
    }
}

(Also, depending on the desired semantics, we could make get_adapters_address() successfully return a null pointer on ERROR_NO_DATA, to indicate an empty linked list. If free_adapters_address() is implemented with alloc::dealloc(), it would need to be modified to do nothing when it receives a null pointer.)

4 Likes

Not sure, see what @LegionMammal978 wrote:

Basically I tried to say the same here:

I wouldn't be surprised if other C APIs use similar techniques. The type of the pointer can (and should) still be IP_ADAPTER_ADDRESSES_LH because that is the first element it points to. And from there, you can get the next element or other pointed-to contents (which may be of different types that have another alignment requirement).

To give an example:

struct LinkedList {
    uint64_t *numbers;
    uint32_t number_count;
    struct LinkedList *next;
}

Say we're on a system where pointers are 32 bit wide, then the alignment requirement for LinkedList could be 4 bytes while, depending on the platform, the numbers array (which could use the same allocation) might require an 8 byte alignment. Even if the alignment must be 8, it doesn't seem too unreasonable to me to return a pointer type struct LinkedList * instead of void * when returning memory which contains the linked list and pointed-to numbers arrays. I mean… what other pointer type should such an API return? uint64_t * would be a worse choice in my opinion.


The docs say that it's a "pointer to a buffer that contains a linked list of IP_ADAPTER_ADDRESSES structures". It doesn't say that the buffer only contains these structs. In fact it doesn't.

Well, I'm using unsafe, and I certainly want to avoid UB (even if only on future platforms).


Oh, good to keep that in mind.

But ironically, this may still require re-trying. And if you only retry once, the needed buffer sized might have changed again, and you need to retry once more. So I don't really understand which method is discouraged. The proposed example code still does this try-three-times approach. (Really ugly API.)

Is the point to "hope" that 15000 will be enough in the first place? For me, it wasn't enough.

Yeah, for this particular problem (when dealing with the Windows API) I may use that windows::Win32::System::SystemServices::MEMORY_ALLOCATION_ALIGNMENT constant. But… what if I have a general C API similar to that one? How do I allocate memory in a way that it is aligned for any kind of value? (Which may be necessary for some APIs, as explained above.) Would std::mem::align_of::<u128>() be a good choice? (which evaluates to 8 on Playground, but can't provide a link as it's malfunctioning right now)

I think that's a good idea. I currently panic. While it's unprobable (I would expect a system to have a loopback interface), returning an empty list may be the better behavior.

You can get the alignment that malloc uses via libc::max_align_t on unix-like platforms. It looks like it's part of C11, I'm not clear on why it isn't present on Windows if that's the case.

Using it would looks something like std::mem::align_of::<libc::max_align_t>()

1 Like

So I need the libc crate? Ideally, I was looking for a way doing this just using std.

I mean you don't need it, but if you want a constant that represents the alignment C uses libc is going to be the simplest way to do that.

cppreference says that max_align_t usually has an alignment of either 8 or 16, so if you just use 16 you'll probably be fine on the major platforms.

1 Like