Memory use issues with threads

I recently created this thread in Crystal forum:

about running identical implementations of the app in discussion in Crystal, Rust, and Nim (add C++ too, which has the same problem).

The Crystal version consistently uses 2.5 - 3.5 GB less memory for the large input numbers in the discussion, versus every other language.

Someone in the Nim community responded in the Crystal forum why, and cited the response in the Nim forum (go to end of thread)
https://forum.nim-lang.org/t/4950#56891

and stated the issue with Nim was it was copying the primes array to each thread, while Crystal shares it, as it's a read only (immutable) array.

Is this the reason that the Rust version also uses so much more memory?
If so, what can be done to mitigate it, and not affect performance (the Rust version is fastest by a good measure).

The Crystal thread has gist links to the source code for all versions.

It does clone the Vec called primes. It doesn't need to.

   let (lastwins, cnts): (Vec<_>, Vec<_>) = { // store outputs in these arrays
     let counter = RelaxedCounter::new();
     restwins.par_iter().map( |r_hi| {
       let out = twins_sieve(*r_hi, kmin, kmax, kb, start_num, end_num, modpg,
-                            &primes.to_vec(), &resinvrs);
+                            &primes, &resinvrs);
       print!("\r{} of {} twinpairs done", counter.increment(), pairscnt);
       out
     }).unzip()
   };

(I've assumed this is the file in question, but note that it doesn't compile due to a syntax error...)

1 Like

Incidentally, if that's a hot loop, the print should be removed or output handled some other way. It involves a lock and some system calls.

1 Like

it's just a single missing closed paranthesis (between lines 205/206); I've independently come up with and made the same change and it compiles successfully. I won't be doing any performance tests on my machine here to confirm it resolves the problem.

Yep, that was it, changing &primes.to_vec() --> &primes fixed it.

When I wrote this two years ago, it wouldn't compile without the.to_vec() for some reason.

Rust now uses a smidgen less memory than Crystal for these inputs: ~11GB vs ~11.7 GB max.

Thanks for the quick response. I've made the change in the gist.

Happy Holidays! :evergreen_tree:

4 Likes

This topic was automatically closed 90 days after the last reply. We invite you to open a new topic if you have further questions or comments.