How do I make a function accept Nested Slices and vectors?

help
1

Hi!

How do I make a function accept nested slices and vectors? From old posts here, I found few approachs like,

fn function<T: AsRef<[i32]>(v: T) {}

and

fn function(v: impl Into<Vec<Vec<i32>>) {}

and as input, I have couple of arrays with different sizes and both of these approaches fail with a error like,

the trait `std::convert::From<[[{integer}; 2]; 6916]>` is not implemented for `std::vec::Vec<std::vec::Vec<i32>>

For reasons, I can not change the function definition a lot. i.e. It must be able to accept Vec<Vec<i32>> because I don't have access to the caller code and the caller code calls my function by transferring ownership of nested vector to my function.

2

Because the layour of Vec<T> is different from the layout of [T] and from that of [T; N], and the Vec can contain more or less elements than N, it is not possible to simply coerce &Vec<T> to &[T] or &[[T; N]].

You could, however, iterate over the outer vector and convert each of its items to &[T] (or &[T; N] using the provided TryFrom impl, which only works for arrays of at most 32 elements for the time being), then collect those pointers into another vector. And only if you are willing to copy all the elements themselves can you achieve a completely contiguous memory layout.

1 Like
3

The 32 elements limit will be a issue for me so I can not use this.

Right now, I use a macro that converts, nvec![[i32]] to vec![vec![i32]].

macro_rules! nvec {
    [$($elements:expr), *] => {
        {
            let mut out = vec![];
            $(
              out.push($elements.to_vec());
            )*
            out
        }
    };
}

For a large input (7000 items of type [i32;2]), This macro ends up allocating about 2-2.5gb of ram. Complete program here.

Is there some way I can optimize this macro? I am using the repetition syntax to accept items I checked the docs here but there is no way for me to get the number of items accepted by the macro.

Using so many values of type vec![] also increases the compile times a lot but I guess there is not much I can do there since I have to convert all of them to Vec.

Please let me know if I should make this a separate thread and discuss it there. :slight_smile:

4

One optimization you could do is to use with_capacity to create the outer vector.

1 Like
5

Yes! But how do I know what size to allocate? I can't get the number of items that were passed to the macro if i am using repetition syntax.
I guess, I could accept the number of items passed as input as a separate argument but I would like to avoid it if there are nicer alternatives.

6

Something sounds suspiciously off to me. [[i32; 2]; 7000 is not a large array (unless we are talking µCs, etc.), it's 7000 * 4 * 2 = 56 kB. That's 35 000 times smaller than 2 GB.

Looking at the playground, the compilation process itself is being killed. Is the problem that the macro is taking too long to compile? In that case, optimizing the runtime properties of the generated code doesn't help.

7

You can count it like this:

macro_rules! into_unit {
    ( $( $token:tt )* ) => { () };
}

macro_rules! nvec {
    [$($elements:expr), *] => {
        {
            let len = [ $( into_unit!($elements) ),* ].len();
            let mut out = Vec::with_capacity(len);
            $(
              out.push($elements.to_vec());
            )*
            out
        }
    };
}
1 Like
8

Yes, Something definitely isn't right. :frowning:

I later copy pasted that code from the playground in a new project and tried to build it and let it run for about 3 minutes and in 3 minutes, It took 5.5GB of ram and kept taking up more ram.

I don't now if it's the compile time or memory consumption that is killed the process on playground.

9

Well, does it behave the same if you only try to compile it (cargo build --release)?

10

It behaves the same with or without --release flag.

11

Yes but what I meant, does it behave the same if you run vs. only compile it? i.e. cargo run --release vs cargo build --release. I checked and it gets killed even if I only build it. So clearly the macro expansion is blowing up and it doesn't matter whether you preallocate or not – the compilation itself doesn't finish, it's not a runtime inefficiency.

12

Oh okay. Well, It is behaving exactly the same if I run cargo run --release or cargo build --release.
And, Yes, The compilation process basically never finishes. I had a similar issue happen in the past where a lot of Vec in a project(about 10k at that time, just 10k lines of vec!) increase compile times a lot.
Also, Since I am allocating in the macro memory usage balloons at compile time.

And, There isn't much to do at run time. I am calling test::black_box() at the end of the program and nothing else.

13

I tried reducing the number of arrays in the macro, and if I do that, it compiles just fine. It looks like the compiler's representation is not efficient enough for accomodating this invocation. I don't think there's anything you can do about it.

However, I don't think you need a macro, either. Why don't you just build a primitive array and make it into a vector, since you're already cloning every element? At least this compiles in debug mode. Not in release mode though.

I think as a workaround, you could create a data file and read it, instead of trying to store this information in the source code itself.

14

Okay. Thank you sooo much for taking the time to respond. :slight_smile:

I'll do this for now and also ask the people writing caller code to use more optimized function signatures. I don't know if they'll actually do it but worth a try.

I was doing this in the past but it results in a lot of frustration. Sometimes, When I copy the input, a space or newline may also get copied and pasted in the file. Then, I get parser related errors which take at least a few minutes if not seconds to fix. So, I dropped this idea.
Also, All of my code is not structured to handle this change. So, I'll have to refactor everything and that'll take maybe few hours to do. So, I don't reallly want to do that.

15

Instead of counting the elements, you can do

macro_rules! nvec {
    [$($elements:expr), *] => { vec![$($elements.to_vec()),*] };
}

Which will allocate the outer vec in one go, and write into it in place.

1 Like
17

Yepp, From @alice's code I got this exact idea and replaced the macro with this.

18

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.