tl;dr; see "The Question" part.
Good day, rustaceans!
I with my friends are working on the bwapi-rs library that would eventually allow us to write bots for the legendary Starcraft game series using the Rust language (instead of C++ and Java which are the current options).
For over a decade there is a bot developing community around that game. Enthusiasts write bots and compete in various championships. Many of them research AI and Machine Learning. BWAPI is used by universities to teach their students. There is even a twitch channel that broadcasts games online.
As a first step towards our goal we have almost finished the BWAPI-C library which wraps existing C++ BWAPI and exposes bare C interface to be used by any other language capable of FFI. We hope, that our C library would eventually be used by others to bring BWAPI support to their language of choice. So, Rust is only one of such languages.
Right now I'm trying to figure out how to design Rust bindings in a most effective and idiomatic way. Of course, I would like to take advantage of Rust's immense abilities and design an API that's easy to use correctly, but at the same time that prevents you from doing something wrong.
Ok, to explain what means "wrong" I need to give a brief insight to the original BWAPI.
The API
In essence, BWAPI is a wrapper layer around the game engine that exposes an interface to:
- Read the game state (map contents, available resources, unit positions, …)
- Issue a command to a unit (move, attack, build, cast spell, …)
- Extend UI (on screen text, graphic primitives, …)
- And many more…
All of this is done by hacking into the game's memory and a lot of bit twiddling. Of course, all of this sound wildly unsafe or even dangerous. Yet, current implementation is quite stable. Blizzard even shared some info on the game internals, so it is OK.
From bot programmer's point of view, each in-game entity is represented by some object. Within a single game match each object has unique pointer and unique integer id associated with it. Basically, that means that objects are never disposed, so their memory location remains valid, much like Values in LLVM.
However, when match ends, all in-game objects are released and their ids may be reused. And that's where the problem arise, actually.
In order to write sound API in Rust, we need to restrict access to in-game objects and bound their lifetimes correctly. However, bots tend to store a lot of temporary information that is tied to in-game objects, like lists of seen hostile units and their expected position. We should provide a way to refer in-game objects whilst the match is active.
Before I'll write about the problems, let's recall the whole chain. So, we have BWAPI that is a C++ library. Next goes the BWAPI-C library which wraps C++ and yields bare-C API. Then we take the BWAPI-C's headers and bindgen them to generate contents of the bwapi-sys library.
Finally, we write idiomatic bindings around the bwapi-sys to make it actually useful. So, instead of painfully verbose C version:
Iterator* const minerals = (Iterator*) Game_getMinerals(Broodwar);
assert(minerals);
Unit* closest_mineral = NULL;
for (; Iterator_valid(minerals); Iterator_next(minerals)) {
Unit* const mineral = (Unit*) Iterator_get(minerals);
assert(mineral);
if (!closest_mineral || Unit_getDistance_Unit(unit, mineral) < Unit_getDistance_Unit(unit, closest_mineral))
closest_mineral = mineral;
}
if (closest_mineral)
Unit_rightClick_Unit(unit, closest_mineral, false);
Iterator_release(minerals);
…we'll have something like this:
if let Some(mineral) = game
.minerals()
.min_by_key(|m| unit.distance_to(m))
{
unit.right_click(&mineral, false);
}
All sys objects are wrapped with corresponding trait impls like Iterator and Drop. Also, conversion traits are provided to make distance_to() or right_click() intuitive. So far so good.
As I mentioned before, all in-game entities must be bound to some session object to be sure, that no reference outlives the session. And that is a problem, because we have no control over object lifetimes. All objects are created and released by the game engine, so there is a fixed time window in which we may access the objects.
In order to design sound API we need to express that contract as a lifetime, let's call it 'g. In that case Unit may be defined like this:
#[derive(Clone)]
pub struct Unit<'g> {
raw: *mut sys::Unit,
phantom: PhantomData<&'g Game>,
}
Essentially, it is a newtype that wraps a pointer to an opaque struct sys::Unit and ties it's lifetime to 'g. Because we do not control the actual lifetime of a raw object, we may clone Unit, unless clone outlives the game match.
All operations on Unit are delegated to library functions like this:
pub fn loaded_units(&self) -> Box<Iterator<Item = Unit<'g>>> {
unsafe {
let iter = sys::Unit_getLoadedUnits(self.raw);
Box::new(BwIterator::from(iter))
}
}
The loaded_units() method may be used to inspect contents of container units like bunkers, transport and so on. Notice, that we return an iterator that yields Units whose lifetime is also bounded by 'g.
Game Events
BWAPI provides a set of callbacks that fire when various game events occur. I wrapped them in a trait which should be implemented for some user's type later in the actual bot code. It is then registered as an event handler.
pub trait EventHandler<'g> {
fn on_start(&mut self, game: Game);
fn on_end(&mut self, is_winner: bool) -> Game;
fn on_frame(&mut self);
fn on_send_text(&mut self, text: &str);
fn on_receive_text(&mut self, player: &mut Player, text: &str);
fn on_player_left(&mut self, player: &mut Player);
fn on_nuke_detect(&mut self, target: Position);
fn on_unit_discover(&mut self, unit: &mut Unit);
fn on_unit_evade(&mut self, unit: &mut Unit);
fn on_unit_show(&mut self, unit: &mut Unit);
fn on_unit_hide(&mut self, unit: &mut Unit);
fn on_unit_create(&mut self, unit: &mut Unit);
fn on_unit_destroy(&mut self, unit: &mut Unit);
fn on_unit_morph(&mut self, unit: &mut Unit);
fn on_unit_renegade(&mut self, unit: &mut Unit);
fn on_save_game(&mut self, game_name: &str);
fn on_unit_complete(&mut self, unit: &mut Unit);
}
Particularly interesting are the first two functions:
fn on_start(&mut self, game: Game);
fn on_end(&mut self, is_winner: bool) -> Game;
Please note, that on_start receives the Game by value and on_end hands it back. Because the game session is a purely dynamic thing, I need a way to tie static lifetimes to some dynamic state.
Game is a singleton object that is a root of the whole API. Every other thing is done by calling it's methods. Every other object is acquired within the lifetime of the Game.
By using ownership transfer technique I'm trying to express the fact, that user code gets access to the full game API only after on_start is fired. At the same point, user must release the whole thing and give up all possible references upon on_end.
By design, user is not able to construct Game directly, so the only option left is to hand back the very same instance of Game that was received at on_start. If we tie all in-game state and all references to that Game instance all will be fine… hopefully.
The Problem
Well… I simply don't know how to express that lifetime 'g correctly. Depending on the implementation attempt, I'm facing either the hell of self referential structs, or a lifetime that is not bounded correctly. In the latter case, I found a way to store Unit outside of game session, hence, break the limitations (see the P.S. below).
There is a lot of unsafety in the implementation details anyway, so the question is how to represent that lifetime at all (unsafe way is OK too).
If it's impossible then it's interesting nevertheless, because it means that BWAPI is an example of a library (of may be even a whole class of libraries) that simply cannot be made safe in Rust terms. So, even safe Rust API would actually be unsound and that's definitely not what we want.
The Question
OK, let's summarize all of the above in a compact form.
The question is how to design sound API that will wrap all of the following:
- Underlying library controls lifetime of all in-game objects
- All objects are acquired by opaque raw pointers and never created or released directly
- There is a fixed amount of time when the whole API may be used, limited by callbacks
- User references may not outlive that time
- User code should be able to hold references to in-game objects in containers
- All API contracts must be expressed in code, it must be impossible to do the wrong thing
Is there a way to express all of the above in an API that would be safe Rust from the user's perspective?
P.S.
In order to avoid XY problem, I tried to skip as much details of my "solution" as possible. Just in case, current state of research may be found here.
Example of API misuse may be found here.
Please note the following struct:
struct Ai<'g> {
session: Option<Session<'g>>,
bad: Option<&'g Unit<'g>>
}
The idea was that Session struct would encapsulate all of the game's and AI's state, so it should be impossible to store Unit outside of that session. However, bad is an example of an external reference that potentially outlives the session. Due to incorrect lifetime specification the whole code compiles just fine.
Thank you very much for your attention! We're really looking forward to any suggestions and/or help!
