Any dynamic storage allocator(s) written in Rust?

Hey everyone,

I'm quite new to Rust (2 chapters in the Book) so I apologize in advance for the potential Captain Obvious in the room, and also for asking in the wrong place if this is not the right one.

I'm thinking about writing a malloc() replacement--and upgrade--in Rust as part of my PhD in CS. Do you have any related existing projects in mind?* Thanks in advance!

Cheers,
X

* Edit: it is also possible that writing a Rusty malloc() is by definition a bad idea. But given Rust's systems nature and so much loved performance/safety features, I'd bet against that possibility. Right?

Hey, this is a great time to write allocators in Rust! One of the historical stumbling blocks for writing pure-rust allocators was that thread-locals were slow in Rust (due to the need of lazy init and destructors).

There's recent progress on fast thread-locals:

That's still nightly-only, but looks like something stabilizable.

I think the most feature-full existing Rust allocator is elfmalloc: https://github.com/ezrosent/allocators-rs/tree/master/elfmalloc. I'd love to see a direct port of GitHub - microsoft/mimalloc: mimalloc is a compact general purpose allocator with excellent performance. to Rust and, with fast thread-locals, that should be feasible I think.

2 Likes

See also https://github.com/rust-lang/wg-allocators

Well, that's (in-a-good-sense) much more info than what I was hoping to mine. Thank you sir!

1 Like

Other allocator-like things: bumpalo - Rust, slab - Rust, heapless - Rust

I am working on a memory profiler for Python written in Rust, (Redirecting…), which has some similarities to a memory allocator (I override malloc()/free()/etc.). One relevant lesson, as mentioned above, is that Rust's thread-locals aren't good enough, I'm doing LTO with C thread locals for now until the newer stuff is stable.

Redox OS's system allocator is written in Rust:

@cappadokes I've just realized I have a great exercise for the reader about rust allocators: Tracking GlobalAllocator · Issue #9309 · rust-lang/rust-analyzer · GitHub (totally not trying to nerd-snipe you :wink: :wink: :wink: )

*braces oneself*

Just a quick dumb question here, in order not to pollute the GitHub issue:

"Specifically, this allocator will maintain an intrusive doubly linked list of currently allocated heap blocks, and will provide an API to iterate the blocks."

From the little that I know, allocators typically keep an indexing structure for free block-bookkeeping. I understand that your suggestion is to extend this feature to allocated blocks too. A naive and rough heuristic would say that such an extension would double the allocator's response latency compared to a standard mechanism (if "meta" header fields were picked smart enough, memory could be kept about the same).

I can see why live heap feedback would be sweet to have, but is the performance trade-off acceptable? (that is, if I'm not missing something in the first place)

1 Like

That's a great question, answered Tracking GlobalAllocator · Issue #9309 · rust-lang/rust-analyzer · GitHub.

The TL;DR is that this is specifically for profiling (you'd cfg-out the thing in production), so 2x time/space overhead would be totally acceptable. The current state of the art is using massif, which I think gives way higher overhead.

Yup, makes sense. Thanks again for the feedback! :mag_right:

Cheers,
X

P.S. Since you brought up massif, you might also wanna check out heaptrack. I've found it to be faster and more practical.

This topic was automatically closed 90 days after the last reply. We invite you to open a new topic if you have further questions or comments.