Questions from a Scala Tooling developer


#1

Hi all,

I went on holiday and read “The Book” and I am extremely impressed by Rust!

I am primarily a Scala developer, and an active contributor to libre software. I am one of the maintainers of http://ensime.org which brings IDE-like features to text editors (Emacs, Atom, Vim, etc) for Scala and Java. It’s fair to say that I am deeply involved with the Scala tooling community so naturally I have a lot of questions around Rust tooling :slight_smile:

I also have a few language / library questions that I’ll tag on at the end.

It would be great if you could help me out by reading my questions around Rust tooling and answering, if you can!

Tooling

ENSIME equivalent for Rust?

I understand that there is no interactive (aka presentation) compiler interface yet for Rust. Are there any plans to create one? Can I help you?

I would be very happy to discuss the architecture of a Rust language server further, leaning on my experiences in Scala, so please reach out to me about this if you’re a Rust compiler developer. I think the ENSIME model would work very well for you and you might want to consider being vscode compliant from the beginning to make it easier for editors to support Rust.

Actually I would love to write the Rust equivalent of ENSIME, but unfortunately my spare time is spread too thinly already so this could only realistically happen if somebody were to fund me to do it :frowning:

IMHO it’s tooling like this that makes or breaks a language, so hint hint :wink:

Emacs setup

Is there a tutorial for setting up Emacs to write Rust applications? e.g. for basic major mode, build tool support and a .ctags setup. I mena in depth, not just “install rust-mode and flymake-rust” (a flycheck plugin would be pretty useful).

Rustaceans in London?

A monthly / quarterly meetup sounds like the sort of thing skillsmatter.com would be willing to host… I’d love to meet some Rust hackers in real life! Is anybody interested in organising anything like this? A bunch of Rustaceans could just agree to come to the next (monthly) Church of Emacs meetup and maybe somebody could briefly talk about setting up a Rust dev environment, then we decanter to the pub and talk Rust…

Docker images and easy windows / osx installs?

Is there an official docker image available for continuous integration, e.g. with all the matrix of Rust versions and some common 3rd party libs in it? I use drone for GNU/Linux builds.

For Windows I use appveyor and for OSX I use travis… does anybody have a recipe for doing rust CI on those platforms?

Code formatting tool?

In Scala we have a tool called Scalariform which can be setup by the build tool to automatically format all the sources in a project according to a style guide. This is amazing because it just kills needless discussions about formatting. Is there such a thing in Rust?

Libraries / Ecosystem

User interfaces?

If I wanted to write a GUI for a cross-platform Rust application, what are my choices?

Even if there is a native GUI toolkit, I would still consider writing a webapp (e.g. purescript or racket) that talks to a local Rust server over Websockets.

Websockets?

Having said that, I assume there is a websockets library?

Distributing with licenses?

I noticed that a lot of the ecosystem is using MIT or BSD licensing that requires that the license be included with all binary distributions. This can get tedious to manage, has anybody written a library that will include the various licenses as part of the binary? e.g. automatically adding a --license-info flag to a CLI that prints out all the licenses that are used.

CSP / streams?

One thing I felt is missing is support for concurrent sequential programming or stream processing. Both Go and Clojure have excellent support for CSP which effectively rewrites seemingly procedural code to be asyncronous (never blocks a thread) but I didn’t see anything similar for Rust (the word “channel” is used in a similar but different way). Has anybody written anything like this? Which would presumably have to be implemented as a macro?

In addition, Scala (and Java) are now seeing a surge of streaming (or so called “reactive”) libraries. One of the most impressive is sirthias/swave (the ScalaDays talk is well worth watching). Is there anything similar for rust?

A standard benchmark I like is to imagine that you have an infinite supply of table data that you need to read one row at a time, you can process each row independently (parallelise the hell out of it) then fan in to do some slow persistence step. How hard is it to set up a pipeline that uses backpressure to keep memory usage to a minimum?

High Performance Linear Algebra?

One of my more popular projects is netlib-java which is used by Apache Spark and similar technologies. Would there be any interest in creating something similar for Rust? Perhaps something higher level, maybe more a Breeze for Rust (which tries to be a MATLAB like syntax in Scala). Although sadly I think this is another area where somebody would need to be willing to pay for it.

Language

recursion and call stack optimisation?

There was an example in The Book for is_symmetric which used recursion, but not mention of call stack optimisation. Does rust do tail call stack optimisation (or any other kind of recursion)? Without this I don’t think I’d want to write any recursive code.

ADTs?

In Scala it is becoming very idiomatic to define domain information as Abstract Data Types. This has two important parts:

  • products (case classes)
  • coproducts (sealed traits) - like an enum of products

where a coproduct and all its implementations are defined in a single file. Does Rust have anything similar and is there a way to access the implementing classes or a coproduct at a) compile time b) runtime?

You could say that List[T] is a coproduct with two class implementations: Cons[T] and the empty list. As is common, this is a recursive type because Cons[T] refers back to List[T].

Autoderiving

In Scala it is possible to do something equivalent to auto deriving for any interface (we would call it “type class derivation”). But in Rust it seems that this is limited to just a few interfaces that are defined in the standard library. Are there any plans to support userland interfaces? It would be really amazing if ADTs could be supported. The canonical example in Scala is to write a custom wire format marshaller, which is all derived at compile time for any ADT.

Error boilerplate

When I was following The Book’s error handling example I couldn’t help but think that there was a lot of boilerplate involved with creating the impls for the user’s custom error type. Is this really necessary? Could autoderiving perhaps help here?

Refined types

Is it possible to do anything like http://fommil.github.io/scalax15/#/6/3 in Rust? i.e. have something more specific than Int or String as a compile time type that is completely indistinguishable (no cost) from the base type at runtime.


#2

Looks like the closest thing to ENSIME is racer, but it’s very much a work in progress. Even if it wasn’t a work in progress I think its goals are a bit different than the project you mentioned. There is working going on at the compiler level to facilitate better tooling, but it isn’t here yet. Someone else could comment more on that.

There has been talk about an offical docker image but AFAIK there isn’t one. For easy installs and managing versions I liked to use rustup.

For code formatting there is cargo fmt.

For websockets I’ve been using ws-rs and I’ve been really impressed with it.


#3

Hi! I am one of the developers of the IntelliJ plugin for Rust ( https://intellij-rust.github.io/ ), and I think I can say something about tooling support for the Rust language.

First of all, there is a huge desire to produce a “presentation compiler” of sorts, here is the relevant RFC: https://github.com/rust-lang/rfcs/blob/master/text/1317-ide.md.

Unfortunately there is no RLS implementation yet. The rustc compiler is not quite suitable now for live analysis. As far as I know, there is no way to tell it “keep all the source code in memory, and I’ll send you incremental edits, so please analyse the sources incrementally” (there is an alpha version of incremental compilation, but it starts with parsing and macro expanding the whole crate), and there is no way to tell “hey, I have only these two files opened in the editor, so can you please produce errors for them as fast as possible, but you can ignore all other files”.

Another (huge imo) problem is that there is no library for analyzing Rust source code. Even the parser is not available outside of the compiler. There is syntex crate which is basically rustc parser without convenient interface (please correct me if I am wrong, I have had only a cursory look at syntex). So it will try to expand macros, print errors to stderr, read files directly from your file system, and erase some information from the source code (foo::<> and foo are indistinguishable). If there were a library which could produce a list of (AST node type, code span) pairs for a str, it would be possible to start building some nice tools outside the compiler. Alas there isn’t one (and you need lexer/parser generator as well to build one properly, I think).

The main tool outside of the compiler is Racer. I think @jwilm will be able to tell more because I, again, have had only a cursory look at it. Looks like it analyses Rust code mainly by looking at the text of the program, invoking syntex to parse only some fragments of the code. As far as I know, Racer does not have indices, so you can’t find symbol by name (is this correct?).

In IntelliJ Rust, we build our own parser, lexer, name resolution, indices e.t.c. Basically we want to build a separate compiler frontend, but, as you can imagine, it is a ton of work. It would be nice to avoid it, and use RLS, but there is no RLS =(

And a couple of questions :slight_smile:

  1. Have you seen the Dart server API ( https://htmlpreview.github.io/?https://github.com/dart-lang/sdk/blob/master/pkg/analysis_server/doc/api.html#notification_analysis.navigation )? I think it is a very good model for implementing a language server, because it is a bit more reactive (and mature) than the Microsoft API. It has some nice features for subscriptions and notifications (go to definition info is notification based for example), instead of request/resopnse.

  2. Am I correct that Ensime maintains indices of names, and not the Presentation Compiler? This looks a bit strange to me because it is natural to place indices inside the language analyser (like Dart does) …

cc @nrc


#4

The back-end of the compiler sometimes performs that optimization, but it’s not a good idea to rely on it, so writing recursive code meant to work on unbounded inputs is a bad idea. Use for/while iteration or use iterators.

Do you mean something like this?

enum Term {
    Var { name: String },
    Fun { arg: String, body: Box<Term> },
}

I see. In D recursive types need to be boxed and referenced, like in the Fun above. And in Rust Lists are rare, you usually use Vec.

To handle different errors this helps:
https://crates.io/crates/error-chain

In Rust you have structs with single fields, with no run-time overhead:

struct First(&'static str);
struct Last(&'static str);

fn hello(First(first): First, Last(last): Last) {
    println!("hello {} {}", first, last);
}

fn main() {
    let first = First("Bruce");
    let last = Last("Lee");
    hello(first, last);
}

You can add a private constructor, but a more refined handling of this stuff (like the static invariants of Ada language, or refinement typing of Haskell and F*) is still missing in Rust, but I’d like it a lot.


#5

We’ve been working on an RLS, there was a demo at Rustconf and we intend to announce the project in the next few weeks (though this will be an announcement that the code exists, rather than that it will be suitable for use with your editor).

Code formatting - Rustfmt (as linked above).

Rust does not promise TCO, it might sometimes happen if LLVM does it for us. There has been some discussion about adding ways to ensure the optimisation.

For ADTs, you would use structs and enums

There is the derive attribute for auto-deriving. There is functionality for implementing your own (‘custom derive’) but it has only recently landed in the compiler. There will be an announcement in the next few days (waiting for some renaming to land).


#6

Thanks for the details! It’s a shame that the PC doesn’t exist yet. Unfortunately this could set IDEA off on the same track as the scala plugin, which means that by the time there is a PC you won’t be able to use it.

To answer your questions:

  1. ENSIME has its own protocol with both sync and async parts, which I believe predates dart :stuck_out_tongue:

  2. ENSIME has a persistent searchable and indexed database of all symbols (resolved to their sources) for the entire project, but the presentation compiler only loads the symbols that are in the sources that are opened by the user (and their dependencies).


#7

thanks! So I see a problem with this use of enums: what if I have a product that needs to belong to two coproducts? Here is a real world example of an ADT from https://github.com/fommil/spray-json-shapeless/blob/master/src/test/scala/fommil/sjs/ExampleAst.scala note that the QualifierToken extends two traits

  sealed trait Token

  sealed trait RawToken extends Token
  case class Split(text: String) extends RawToken
  case class And(text: String) extends RawToken
  case class Or(text: String) extends RawToken

  sealed trait ContextualMarker extends RawToken
  case class Like(text: String) extends ContextualMarker
  case class Prefer(text: String) extends ContextualMarker
  case class Negate(text: String) extends ContextualMarker

  sealed trait TokenTree extends Token
  sealed trait ContextualToken extends TokenTree
  sealed trait CompressedToken extends TokenTree
  case class Unparsed(text: String) extends TokenTree
  case class AndCondition(left: TokenTree, right: TokenTree, text: String) extends TokenTree
  case class OrCondition(left: TokenTree, right: TokenTree, text: String) extends TokenTree

  case class Ignored(text: String = "") extends TokenTree
  case class Unclear(text: String = "") extends TokenTree

  object SpecialToken extends TokenTree

  sealed trait Term extends TokenTree

  case class DatabaseField(column: String)
  case class FieldTerm(text: String, field: DatabaseField, value: String) extends Term
  case class BoundedTerm(
    text: String,
    field: DatabaseField,
    low: Option[String] = None,
    high: Option[String] = None,
    inclusive: Boolean = false
  ) extends Term
  case class LikeTerm(term: FieldTerm, like: Option[Like]) extends Term  case class PreferToken(tree: TokenTree, before: Option[Prefer], after: Option[Prefer]) extends TokenTree
  case class InTerm(field: DatabaseField, value: List[String], text: String = "") extends CompressedToken

  case class QualifierToken(text: String, field: DatabaseField) extends ContextualToken with Term

#8

custom derive sounds very promising. Will it be possible to custom derive something outside of the definition of the struct/enum? e.g. say I want to derive a JSON marshaller for a struct that is in a third party library.


#9

No plans for that as yet - you always have to use derive as an attribute on a data type. The best solution is to create a newtype and add derive to that, but that is far from perfect because we don’t yet have a nice solution for ‘newtype deriving’.


#10

Already been happening: https://www.meetup.com/Rust-London-User-Group/