Announcing nitric - the successor of Specs

Don't get too excited yet, I'm just sharing the plan. The implementation has yet to happen :wink:

First of all, let me provide some context:

Specs is a library implementing the Entity-Component-System pattern. I've been working on it for nearly two years and in the last few months, I've thought a lot about its problems, how to fix them and generally how to improve the whole library. Going into detail about the problems of Specs is out of scope here.

I've decided that it makes a lot more sense to start a new library - nitric - instead of reworking Specs (read below for the exact plan). For one, there are many people currently using Specs and experimenting in the same repo would cause both confusion and a lot of breakage. Secondly, if the experiments fail we still have Specs. And thirdly I want to adjust the scope of the new library.

I think nitric's README explains everything pretty well, so I'm just gonna paste it here:

nitric

General-purpose data processing library.

Status notes: highly experimental, unfinished, work in progress, not recommended for use

This library is meant as a successor for Specs, a parallel ECS library. However, nitric aims to be more general and composable than Specs. In fact, Specs can be implemented as a frontend for nitric once this library is complete.

Motivation

Specs has many problems, both big and small. The library grew big without much planning and as it is now this will continue and make it very hard to maintain. That's what made me think about what I want to create. This is what I ended up with:

Vision

The vision for nitric is to provide a set of crates that evolve as the standard way to solve data processing problems. There were already very interesting use cases for Specs, including using it for a compiler and performing larger simulations, both outside of Specs' original scope (ECS for game development). This is what I intend to make nitric suitable for. So to list a few examples of what nitric could be used for in the future:

  • game development
  • game physics
  • simulations
  • compilers
  • data validation
  • Graphical User Interfaces

The question of how to structure your library/application is a very common one, everywhere in programming. The plan for nitric is not to force any of them, but to provide useful and modular facilities that allow for specific patterns (e.g. Entity Component System (1)), and to provide "recipes", similar to the Rust cookbook that show how common tasks can be solved. Nice side effects of that are that we can work on one implementation, that is efficient and can allow for neat extra functionality (debugging facilities, profiling, easy multi-threading, etc.).

(1) for ECS, also see this great presentation by Catherine West

Philosophy

nitric will be a collection of crates all about processing data. All of its crates follow this philosophy:

  • Only solve a single problem, in a reasonably composable way
  • Expose a general, composable and robust API
    • APIs should either be designed to not produce any error cases or return a Result with only the possible error conditions
    • Do not assume how the API is being used (-> composability)
    • Expose internals in a *-internals crate for stability by default, with the option to opt into more unstable facilities
  • Impose minimal friction to use nitric
    • The goal is for nitric to be cheap and easy to use in one place of your project for solving a particular problem
    • nitric is meant to be compatible with other data structure / ECS / CGS libraries, e.g. Specs, froggy, etc. instead of competing with them

Using nitric as ECS

How will this allow you to use nitric instead of Specs? Here's the tentative design for ECS:

  • nitric-entity: Provides entity allocators, storages, and with that mappings between entities and components

This would already be enough to have an ECS. Systems can simply be functions that accept references to the storages and eventually the allocator. In fact, that is the recommended way for libraries to expose their API; libraries should not assume how the code is executed. For example:

(pseudo code for now)

pub fn process_local_transforms(
    local: &Storage<LocalTransform>,
    global: &mut Storage<GlobalTransform>,
    parents: &Storage<Parent>)
{
    // compute global transforms    
}

If those component storages come from a dynamically typed, string mapped HashMap, fine. If they are stored in a struct - works, too. How systems are run also doesn't matter.

Now, there surely are other things Specs users would miss, so the next crate will be...

  • nitric-world

As you might have guessed, this provides a map that can store arbitrary component storages. In contrast to Specs, I also plan to include support for multiple values of the same type by allowing an additional key (e.g. a string).

  • nitric-graph

This will be a re-worked version of shred's Dispatcher, allowing to parallelize data processing (execution of systems). A "system" will simply be a FnMut(T) -> R, which means it's up to the user how the data is fetched (nitric will provide solutions for this, but it doesn't force you to use any of them).

Structure

The main crate, nitric will simply serve as a top-level crate for re-exporting all nitric-* crates. However, since everything is optional, nitric is controlled with Cargo features, only exporting the crates for which you enable the flag.

FAQ

What does this mean for Specs?

For the immediate future, this has no effect on Specs. It will not be deprecated. The biggest change for now is that I won't spend much time on it (just merge PRs and fix critical bugs).

As for when nitric is in a usable state, that has yet to be seen. In any case, it should be possible to make Specs a thin wrapper over nitric crates (if that's necessary). All that depends on how well nitric will be adopted.

What does this mean for Amethyst?

Amethyst (a game engine that makes heavy use of Specs) will continue to use Specs. Whether it will use nitric in the future will be decided by all members, through the usual RFC process.


That's the plan so far! If you're interested, feel free to check out the repository:

Also, I'd love to hear about your thoughts & additional requirements. I've started writing up the requirements here, but the list is far from complete.

Lastly, I want to say thank you.

Thank you to @kvark and @csherrat for starting Specs in April 2016 and building the foundation of it.

Thank you to all the people who contributed ideas and code to Specs.

Thank you for all the feedback on Specs which gave me the courage to further work on it.

The project wouldn't exist like this without you.

34 Likes

This work sounds very interesting! As somebody in the scientific computing space, Specs as a programming model (composition of Systems) seemed promising. It isn't quite the right fit because, as I understand it, specs really only have a single index space (the "Entity" index space). In scientific computing, you may have many index spaces. For example, a simulation application that uses a mesh data structure to track simulation state will have an index space of cells, the index space of faces between cells, etc. Meaning we don't really have a single entity, but many different types of entities.

I'm looking forward to hearing more about this project as it's fleshed out!

6 Likes

Since you didn't go into any details about the problems with specs (aside from maintenance difficulty), is there a concise list of the gotchas and pitfalls that one usually runs into when using it? Apart from trying to dig through the issues list, of course...

3 Likes

I'm really interested in following the development of nitric, it seems like a promising idea. Also thank you for that link to the CGS explanation, this seems like a great way to solve the awkwardness in ECS when you need a reference to an entity.

Oh, that's interesting. nitric will allow multiple id spaces (I even have that locally already, will push next week and post a link to the MR here).

I intend to create some demo applications as soon as there's actual code. Do you have a simple example from scientific computing in mind that could demonstrate the usage of multiple ID spaces?

Okay, I can do that. I don't have such a document anywhere online, so I'll just comment here:

First, let me clarify: Specs can be used just fine. It's used in Amethyst, several games and even runs in production and if it works for you, great!

I can't possibly write up all the problems there are, but let me list a few pain points that come to my mind:

  • flexibility
    • centered around World and requires everything to be stored inside
    • only one global Entity allocator can be used; no way to have multiple ID spaces, custom allocators, higher Entity limit, less bit set layers, etc
    • cannot store thread-unsafe data
    • no way to modify system graph once built
    • scripting is hard & rather hacky
    • no support for CGS / archetype ECS
  • understandability
    • API is often rather deep and requires going through multiple layers to understand it
  • scheduling
    • rayon is used to run systems, which performs well for one-time jobs, but not for frame-based models as we have them in games; this means CPU usage can be higher than necessary
  • scope
    • Specs is an ECS for game development

Many of those issues could be solved, but:

  • that would break the API multiple times and
  • we had to experiment with widely used code & API, which I don't consider a good idea

EDIT: Something that could be clarified in the README is that "problems of Specs" is not the sole reason for nitric. That's not very clear right now.

Overall, I decided it's better to start a new crate, with the following goals / intention:

  • create a general-purpose ECS / data processing library (extended scope)
    • bring ECS to more areas &
    • learn + document + eventually implement useful patterns in those other areas
  • start fresh, design modular crates; limit crates to performing one job, but make them rock solid for that

I hope that answers your question :slight_smile:


Little status update:

The first crate will be nitric-component, implementing component storages. I'm currently working on that crate and I'll post the link to the Merge Request here once its ready.

6 Likes

This is cool. The minimal nitric-entity crate sounds like what I was really after when someone tried to convince me to use ECS for the 2018 summer roguelike tutorial.

2 Likes

Coolness. Sounds like you're down a great path with Nitric!

As I am just now getting into Rust, (my last programming exp was in 2yr college learning Cpp2003). I'm excited to learn Rust, Specs, and Amethyst.. and with your new Nitric possibly giving Specs an upgrade, and thus Amethyst. Makes me feel I am on good ground.

Thanks, Tork!

Nitric development is on hold. Attention was turned back to specs as there are plenty of incremental improvements to be made there before a radical rewrite really makes sense.

Most recently there’s been a discussion about what Amethyst/specs can learn from the Legion ECS: