My gamedever wishlist for Rust

[tomaka]

I have been developping a game in Rust on my free time for almost a year now, and I have written several libraries. During this time, I have encountered lots of problems with Rust and its ecosystem.

Usually when I have a problem I try to fix it myself (or at least open an issue), but after this post I thought that I’d share everything I encountered. Some things are more related to the needs of my own projects while some others apply to gamedev as a whole, but I didn’t include anything really specific to my game.

Two things to note:

• I didn’t mention obvious things such as HKTs, plugins or cargo install. I think that these features would be awesome to have (especially plugins, I don’t think people realize how plugins are a total game-changer), but I don’t want to overload the list.

• None of these are critical. They are all only missed optimisations, things that are annoying to use, things that can be bypass by hacks or with C libraries, etc.

So here we go.

Language/stdlib

• No way to store a borrower and a borrowed in the same struct. See this link for more infos.

• There is no easy way to cast between (f32, f32, f32) and [f32; 3] (for vectors, matrices, etc.). This causes problems of interoperability between functions/libraries that use the former and functions/libraries that use the latter.

• Fixed-size arrays are missing traits. For example you can’t call .clone() on a [u8; 50]. Fixed-size arrays are just too annoying to use right now because of the traits that aren’t implemented on them. This means that I have to use a Vec and lose some data locality.

• Unsized types do not implement Copy. I want to create my own Box-like type but that stores the data in video memory. Since this is video memory, I need to restrict the content to Copy types. But if I write struct Buffer<T: ?Sized + Copy>, then I can’t use Buffer<[u8]> because [u8] isn’t Copy.

• There’s no way to build unsized structs. If you take for example struct Foo { val1: f32, rest: [u8] }, there’s just no way to build it. Not even with mem::uninitialized.

• Even if you want to allocate space for an unsized struct, it is too hard to build a pointer to it. Even if you manually allocate some memory to store an unsized struct (like Foo in the previous point), the only way to build a *mut Foo or a &Foo is to transmute from a (usize, usize). Oh and if you do it with an unsized enum or a trait object, you will get a segfault. Right now there’s no way to combine unsized structs and safety.

• No write-only references. Some low-level APIs forbid you from reading from pointers they give to you, and if you do your program can get killed by the O/S. This can happen for example when mapping video memory. Currently the only way to provide a safe wrapper around such a pointer is to provide only setters and no getters. But this is a crappy solution in terms of usability and it comes with a performance cost.

• Loading symbols at runtime from a non-Rust shared library is too hard. Some time ago glutin switched from linking to xlib at compile-time to using dlopen to load it (in order to handle systems that don’t have xlib). The current solution works using some hacky macros.

• There’s no min! and max! macro. You need to write max(max(max(max(max(max, a), b), c), d), e) for example. I have a code with 15 max like this.

• No way to detect at compile-time whether we are in the main thread. This one is a bit weird, but on OS/X some GUI operations can only be done in the main thread. It would be nicer if this was detectable at compile-time.

• No way to get the backtrace of a panic. You can recover from a panic by spawning a thread, but there’s no way to intercept what is printed to stdout and write it in logs (or show a message box to the user). Games are usually not run from the CLI (especially on Windows), so if there’s a panic and the program closes you will lose the logs.

Tooling

• Missing an option in Cargo to optimize just dependencies. Right now I have to choose between a 40 seconds compile-time followed by a 30 seconds loading screen (as the loading is carried on by unoptimized Rust code), or a 1mn30 compile-time with a <1s loading screen. Having this option would give me a 40 seconds compile-time with a <1s loading screen.

• There’s no way to build a Windows application that doesn’t show a console. This requires passing an option to the linker, and can only be done in hacky ways.

  • Rust 1.18 stabilized #![windows_subsystem = "windows"]

• If you use a build script, everything is recompiled if you modify any file in the repository. I’m using a build script in my game, and even if I modify a file that is loaded at runtime the whole game is recompiled if I do cargo run. The whole point of loading files at runtime is that you can modify them without recompiling your program.

Missing libraries

• A good maths library. There are several maths libraries, but each one has its drawbacks. nalgebra is missing computer graphics-related functionnalities because it’s a pure maths library, and cgmath has a very bad API.

  • https://github.com/rustsim/
  • https://www.nalgebra.org/nalgebra_glm/

• Audio formats decoders. There is currently a good WAV decoder, but WAV is not really the audio format you want. There are FLAC decoders, but all of them are either under GPL, don’t compile, or are just drafts. FLAC too isn’t that great. There are bindings for vorbis and libsndfile but they both misteriously crash on my Windows machine with the MSVC nightly (I suspect a bug in rustc or gcc-rs). Even if they worked, some pure-Rust decoders would be awesome.

  • https://github.com/tomaka/rodio
  • https://github.com/RustAudio

• A UI library that handles the layout of the UI but doesn’t do any rendering. My rendering process is very optimized, so I want some kind of library that returns a list of things to draw instead of drawing itself.

• A quadtree implementation.

  • ntree - (970 downloads)
  • aabb-quadtree - (273 downloads)
  • quadtree - (version 0.0.0, 82 downloads, no repo / docs links)

• A joystick/raw-input library. Initially the idea was that this would be integrated inside glutin, but since it’s possible to write a stand-alone library I think it would be better this way.

  • https://gitlab.com/gilrs-project/gilrs

• An alternative to freetype on Windows. The Windows API provides a way to load fonts, so this could be used as an alternative to building and linking to freetype.

  • https://gitlab.redox-os.org/redox-os/rusttype

• A better library for the spine format. The spine software is a 2D skeletal animation software, and to decode the exported files I wrote my own library (here is an old gif). However this library is incomplete, slow, and tied for my needs. It has already received some hacks to ignore errors.

• A cohesive collection of AI libraries. There are a couple of disparate AI libraries around but the ecosystem would greatly benefit from a larger, coordinated effort in this space, similar to rustsim or rustaudio.

  • https://github.com/orhanbalci/rust-steering-behaviors/
  • https://github.com/PistonDevelopers/ai_behavior
  • https://gitlab.com/neachdainn/aspen

Problems that are being solved

• It is too annoying to build for Android. Most notably the lack of pre-compiled binaries means that we can’t do continuous integration on Travis for example.

  • Announcing cargo-apk though the user still needs to install Android SDK/NDK (or use a docker image by @tomaka)

• Android pre-compiled binaries were promised.

  • rustup now supports rustup target add arm-linux-androideabi - https://blog.rust-lang.org/2016/05/13/rustup.html#example-running-rust-on-android

• Backtraces don’t work on Windows. Some debuggers, graphics debuggers and profilers crash or don’t work when being shown a Rust program.

  • Solved with the MSVC nightly. However the MSVC nightly can trigger linking errors which are solved by tweaking the source code of your dependencies, so it’s not that great either.

• Custom hashing function in HashMaps.

• Compilation for emscripten (or WebAssembly), while maybe not critical, would be a big plus.

  • See https://github.com/rustwasm

• PNaCl/JS LLVM targets

[huon]

There’s no min! and max! macro. You need to write max(max(max(max(max(max, a), b), c), d), e) for example. I have a code with 15 max like this.

This seems like a macro you can write yourself easily enough. Is there something I’m missing?

macro_rules! max {
     ($e: expr) => { $e };
     ($e: expr, $($rest: tt)*) => { max($e, max!($($rest)*)) }
}

[tomaka]

The reason why I have 15 “max” in my code is because it takes less time to copy-paste max( 15 times than it is to write a macro and think about where I should put it in my code structure!

A max! macro would still be a good addition in my opinion, but I admit that this is probably the weakest point in the list.

[llogiq]

• Perhaps we should add From impls for same-typed tuple of some sizes to arrays of those size and types and vice versa.
• The missing traits will come once we have type-level numbers down. Otherwise we should open up and document the macros that define the missing traits so that someone who wishes to have them can just call the respective macro.
• The unsized type (building, copying, pointing, etc.) thing could also be remedied with type-level numbers, at least once we have them; however, this would require some further magic.
• Write-only references could be done with wrapper types and a lint (as I’ve hinted before). This way we could at least check for all reading operations within a crate, and perhaps also add a blacklist of known reading operations to check. Apart from that perhaps use conditional compilation to allow for one secure and one fast code path?
• Perhaps we may be able to statically check which parts of the code may run inside non-main threads? This would of course require checking the whole call tree.

[vasya]

Alternatively, we may have a function like Vec.maxBy, Vec.minBy.
Given an array and a comparator this function would produce the lowes/highest element of that array, Option<T>.

At least that’s what you can find in Scala language.
List("Reuben", "and", "Rachel").minBy(_.length) == Some("and")

[brson]

Nice writeup. Thanks @tomaka.

[notriddle]

No way to detect at compile-time whether we are in the main thread. This one is a bit weird, but on OS/X some GUI operations can only be done in the main thread. It would be nicer if this was detectable at compile-time.

This can be implemented using a data type that is only constructed at the start of the main thread and is required for any GUI operations that must occur in the main thread. Something like this:

#[derive(Copy, Clone)]
struct MainThread {
    _prevent_construction: ()
}
impl !Send for MainThread {}
impl !Sync for MainThread {}

fn main() {
    let token = MainThread{ _prevent_construction: () }
    do_stuff_that_must_be_in_main_thread(token);
}

This way, if a function must be run in the main thread (and any function that calls such a function is) it can just accept a token. The token cannot exist outside the main thread, because it is only constructed in the main thread and cannot be sent to another.

[tafia]

First, I have no experience in spine, nor in game development in general.
Spine library looks like a good start!

However this library is incomplete, slow, and tied for my needs

Can you describe

1. what would a “complete” library look like. I’ve looked at the esoteric apis, is this something you’d like or a more rusty one ?
2. slow: I saw 2 benchmarks in your project. What would be a reasonable target in your opinion to consider it as decent compared to other languages ?

I’m not sure I can really help but I’d like to try at least this game devs look fun!

[tomaka]

1. what would a “complete” library look like. I’ve looked at the esoteric apis, is this something you’d like or a more rusty one ?

Unfortunately I picked spine not long before switching to Rust, so I haven’t tried the official APIs a lot. I don’t know if they are good in terms of usability. However I’d expect that the official APIs are the most appropriate to get the most performances.

2. slow: I saw 2 benchmarks in your project. What would be a reasonable target in your opinion to consider it as decent compared to other languages ?

I haven’t compared my library to others. I say it’s slow because it’s missing a lot of optimizations.

For example:

• Everything is done using strings and hashmaps of strings, instead of just numeric identifiers.
• Everything is recalculated at every frame even though it’s technically not needed.
• The algorithm for bezier curves is really naive. I’m not an expert in this but I’m almost certain that there’s a better way.

The calculate function of the library is currently the place that eats up the most CPU of my game behind the OpenGL driver, and it’s not surprising to me.

[tomaka]

The library is also buggy.

For example I’ve had a problem where an element would rotate 340° counter-clockwise instead of 20° clockwise like expected.
I ended up modifying the model instead of fixing the bug in the library.

[tafia]

I’ll try to see what I can do.

For the moment, I’ll focus on “your” api as you’re already using it. Then if I manage to get something, I’ll try to implement the other apis as well.

Thanks a lot!

[tomaka]

Don’t write a library for me!

If you want to learn how 2D vectorial animations work, then fine, write a library and I’ll happy to use it if it’s good. But don’t write a library just for someone on the Internet that you don’t know!

Also if you don’t know how it works I’d suggest starting from scratch. The existing code is messy and doesn’t have comments. You can of course look at it and copy-paste things, but it’s not worth keeping the whole thing in my opinion.

[mbrubeck]

The missing traits will come once we have type-level numbers down. Otherwise we should open up and document the macros that define the missing traits so that someone who wishes to have them can just call the respective macro.

Doesn’t coherence prevent this from working outside of libstd?

[llogiq]

No, actually it’s the orphan rule. Still, you’re right, it’s impossible to define outside of libstd. Probably the best thing we can do is to declare a macro that defines a bare function to do the conversations.

[tafia]

But don’t write a library just for someone on the Internet that you don’t know!

You just look so sympathetic

Yes it is all for fun and learning anyway, might probably/certainly never be used.
And as I don’t really know how to test it without rebuilding an entire game (which I have no experience at), it is simpler for me to work based on yours, at least for the moment.

[glaebhoerl]

Quite a few of these implicate major additions to the type system (which I would like to see!), rather than just minor niceties. (You’re quite likely aware of many of these, but for everyone else reading.)

No way to store a borrower and a borrowed in the same struct. See this link for more infos.

This needs something like unmoveable types or existential lifetimes (as @Ericson2314 also commented). I haven’t begun to think about what the latter would mean.

There is no easy way to cast between (f32, f32, f32) and [f32; 3]

This needs (ideally) a safe Transmute trait, which has been proposed several times before, under different names (originally Coercible, also Cast, …). Also type-level constants, like the next one, for the ; 3 part.

Fixed-size arrays are missing traits.

This would need something like the ability to parameterize generics over type-level constants, which has also been proposed a few times before. This (along with the functionality around const fn) needs a lot of care and hard thinking to really get it “right”, rather than just doing it the “obvious” way and getting a lot of duplication and verbosity, like C++ does.

Unsized types do not implement Copy.

There’s no way to build unsized structs.

Even if you want to allocate space for an unsized struct, it is too hard to build a pointer to it.

I’m actually not sure what features these would need… part of me thinks this kind of thing would’ve been easier if we had gone down the “SST” road rather than “DST5” (see the Thoughts on DST series of posts), though I’m not really certain.

No write-only references.

These have been proposed as &out or &uninit references, and are probably my favorite pet proposed feature. The interesting thing about them is that they need to be truly linear, rather than affine (like all existing Rust types), which puts them into tension with panics and unwinding.

There’s no min! and max! macro.

You could write this as a fold: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15].iter().fold(0, |a, b| max(a, *b)), though I’m not sure what the threshold is where it starts being a win. (Almost surely by 15.)

[ruudva]

There are FLAC decoders, but all of them are either under GPL, don’t compile, or are just drafts.

I intend to license Claxon under a more permissive licence once it becomes stable.

[bvssvni]

Some things that have been discussed, but currently lacking brain cycles to implement. Cross platform library for joystick/controller input: https://github.com/PistonDevelopers/piston/issues/884. Port stb_truetype.h to Rust: https://github.com/PistonDevelopers/piston/issues/901. Sorry for the layout but there is a 2 line restriction.

[excaliburHisSheath]

For custom hashers. Hashing integer types (u64, mostly) is a very hot path for my code, and its currently showing up much higher in my profiles than it probably ought to be. I’d also benefit from a stable, robust plugin system. There are a lot of places in game-dev where complex, boilerplate-heavy tasks could be made simple and stable through compile-time code generation and manipulation.

[bombless]

You need a proxy type to get something similar to write only references; as for building DST, you need to make a structure to carry a pointer, then make DST in its method with transmute.
I know these might be obvious and of course hacky, I’m just saying…