U16 slice to u8 iterator

Hello all!

Lets say I have a slice of u16. I want to iterate over each of its bytes but can't find any solution.

My current situtation is:

for byte in my_u16_slice.iter().map(|v| v.to_ne_bytes().iter().take(2)).into_iter() {

Of course, it doesn't work.

My only solution for now is:

for v in &my_u16_slice {
  for b in v.to_ne_bytes().iter() {

But this doesn't use iterator.

Surprisingly, I can't find anything similar on internet.

Any idea?

Thanks in advance.

my_u16_slice.iter().copied().flat_map(|x| x.to_ne_bytes().iter().copied())

or

my_u16_slice.iter().flat_map(|&x| x.to_ne_bytes().iter().copied())

This could be reduced to just my_u16_slice.iter().copied().flat_map(u16::to_ne_bytes) if IntoIterator were implemented for arrays, but that's not available yet unfortunately.

1 Like

Thanks, it still complains about ownership:

641 | for b in my_u16_slice.iter().copied().flat_map(|v| v.to_ne_bytes().iter().copied()) {
    |                                                    ---------------^^^^^^^^^^^^^^^^
    |                                                    |
    |                                                    returns a value referencing data owned by the current function
    |                                                    temporary value created here

I rarely suggest using an unsafe function, but it seems <[u16]>::align_to::<u8>() would be appropriate here, as u8 and u16 both are copy without any niche/invalid values, and u16 has alignment larger than that of 8.

So my_u16_slice.align_to::<u8>().1 could work, like this.

Ah, whoops, sorry about that—I should've checked my intuition on the playground first!

Try my_u16_slice.iter().flat_map(|&v| std::array::IntoIter::new(v.to_ne_bytes())). This lets you use IntoIterator for arrays in a slightly clunky indirect form while the actual impl is still unstable.

4 Likes

I agree that the nicest solution would be something that casts it to &[u8] and iterates over that. Though, I prefer a method like this one over align_to to be honest.

fn as_u8_slice(slice: &[u16]) -> &[u8] {
    let len = 2*slice.len();
    let ptr = slice.as_ptr().cast::<u8>();
    unsafe {
        std::slice::from_raw_parts(ptr, len)
    }
}

The zerocopy crate can also do it without unsafe code. Here you can just import the AsBytes trait and call .as_bytes() on your u16 slice.

3 Likes

You can also decompose @cole-miller solution futher, to remove the closures entirely.

my_u16_slice.iter()
    .copied()
    .map(u16::to_ne_bytes)
    .flat_map(std::array::IntoIter::new)
2 Likes

Thanks for your precious answers all. I realize it's actually not an easy task.

I tried to implement few of your answers and the resulting assembly code is still messy:

I'm not sure you can have a fast assembly with iterators in such case without relying on unsafe.

The safe_transmute crate is another option, similar to what zerocopy provides.

What's wrong with align_to in this case?

This is the problem with align_to

The method may make the middle slice the greatest length possible for a given type and input slice, but only your algorithm's performance should depend on that, not its correctness. It is permissible for all of the input data to be returned as the prefix or suffix slice.

emphasis mine

3 Likes

If we’re listing crates here that offer the &[u16] to &[u8] conversion without unsafe, we could also list bytemuck.

1 Like

Consider looking at it from a slightly different direction, and writing the u16s into pairs of bytes instead of splitting the u16s into individual bytes:

let (my_bytes, _) = my_bytes.as_chunks_mut();
bytes.iter().copied()
    .zip(my_bytes)
    .for_each(|(a, b)| *b = a.to_ne_bytes());

Codegen: https://godbolt.org/z/311bETG6E

Note that, despite using iterators, that's compiled down to just a single memcpy.

Already works on nightly, though, so won't be long! :tada:
https://play.rust-lang.org/?version=nightly&mode=debug&edition=2018&gist=74f4924a5737202731c306ec7d73bc93

3 Likes

Oh, I see, thanks.

let (my_bytes, _) = my_bytes.as_chunks_mut();
bytes.iter().copied()
    .zip(my_bytes)
    .for_each(|(a, b)| *b = a.to_ne_bytes());

Cool! But only work in nightly. :frowning:

Make sure you don’t confuse something working on nightly with #![feature(…)] flags with something working on nightly without #![feature(…)] flags. This is working without any feature flags. The IntoIterator implementation for arrays was recently implemented and also stabilized in #84147 and is thus (most likely) going to arrive in stable with Rust 1.53 on June 17th 2021.

Edit: Oh, wait you weren’t even talking about the array IntoIterator? :see_no_evil: I should’ve probably read more carefully what you wrote there.

Yeah, as_chunks_mut isn't stable yet.

You can do this on stable, but it's obviously less elegant:

pub fn five(bytes: &[u16], my_bytes: &mut [u8]) {
    bytes.iter().copied()
        .zip(my_bytes.chunks_mut(2).map(|x| -> &mut [u8;2] { x.try_into().unwrap() }))
        .for_each(|(a, b)| *b = a.to_ne_bytes());
}

https://play.rust-lang.org/?version=stable&mode=debug&edition=2018&gist=48548e371c984ae1535421759631ee83

Thanks, it lost the memcpy though:

In this case I would use copy_from_slice to make it a bit more elegant:

pub fn five(bytes: &[u16], my_bytes: &mut [u8]) {
    bytes.iter().copied()
        .zip(my_bytes.chunks_exact_mut(2))
        .for_each(|(a, b)| b.copy_from_slice(&a.to_ne_bytes()));
}

That's because with chunks_mut the last slice may have less elements than the others, thus that try_into may actually fail. chunks_exact_mut however solves this problem:

3 Likes

I have to think more about how you interpreted the statement. This said, when I used align_to to build simd vectors, the stride was aligned as “requested”, in this case u8. AFAIK having read how the algorithm “came to be”, that’s guaranteed. What’s up in the air is the length of the three blocks (prefix, middle and suffix). Unsafe, performance may vary, but no UB.