New crate: ndarray-parallel


#1

ndarray-parallel is simply the start of rayon integration for ndarray. It seems this might be one of the first outside data structure crates that integrate with rayon and implement ParallelIterator.


#2

Interesting. I’d have expected rayon integration to be an optional feature of ndarray.


#3

Hey, so recently we started this: https://github.com/Metadiff/gir and was wondering to how much low level is the ndarray going? Would it be possible to allocate ndarrays from a common shared buffer (e.g. memory manager) and is it possible to zip several arrays in the same iterator. Its quite nice to see the integration with rayon. Also is there any broadcasting support. It would be really nice to have a CPU backend based on something like this.


#4

It is the goal that it’s an optional feature in the crate itself. As it is now, on the rayon side they haven’t declared which API is versioned and which is internal, so ndarray itself can’t depend on rayon properly yet. Secondly, even with a versioned implementation API, the coupling of versions is hard to handle.

(Three ways to handle ndarray + rayon dependency)


#5

There is simple broadcasting support. In for example a + b, b will be broadcast to the shape of a if possible. What’s not implemented is “co-broadcasting” where a + b broadcasts both into a different common shape. (Example: shape [3, 1] + shape [1, 3] => shape [3, 3] in numpy).

There will probably not be a zip iterator for arrays. The reason is that there is already a zip method (.zip_mut_with); in general the methods allow us to handle the various memory layout cases more efficiently. Trying to stuff all that logic into an Iterator’s next method runs into the problem in this blog post.

I expect we will use rayon to parallelize array zips as well.


#6

Oh about allocators, well, ndarray doesn’t support different allocators in the same Array type or anything like that, but writing one would be possible. Would be most interesting if there was an existing allocator API in Rust for this.


#7

https://doc.rust-lang.org/book/custom-allocators.html


#8

Ah that’s a different thing (ndarray uses the same allocator as Vec<T> does). I mean akin to Vec<T, A = DefaultAllocator> which is an per-type allocator.


#9

RFC for user-defined allocators: https://github.com/rust-lang/rfcs/pull/1398 (not yet implemented, much less stable)