Cargo test internal packages

help
1

I have refactored my small learning-project at github (GitHub - Robbepop/cionc: The toy-compiler for the Cion programming language.) to have a main.rs in src/ that has its entire working code in internal crates, namely "cionc_parser" (in src/parser/) and "cionc_utils" (in src/utils/).
At least to my current noobish understanding, this is how you "should" structure a rust project.

However, since I did this refactoring I ran into problems with cargo and travisCI as well as coveralls.
For example, to test all my unit tests the command "cargo test" no longer suffices since I now need to specify the internal crates with "cargo test --package cionc_parser --package cionc_utils" which is kind of annoying to me.
Is there a better way to specify this behaviour in Cargo.toml or am I doing something entirely wrong?

Besides that I have problems getting coveralls to work with the travis-cargo tool.
In my "travis.yml" file I do "- travis-cargo --only stable coveralls --no-sudo --verify -- --package cionc_parser --package cionc_utils" in order to run all internal tests for stable builds and send the result to coveralls.
As documented in TravisCI's log files this successfully tests all my internal tests, however, tests are no longer uploaded to coveralls.

And yet I begin to feel clueless about why all my different attempts do fail.
Are there Cargo manifest features I don't know about?
Help! :frowning:

2

Interesting because I cloned your project and:

  • cargo build worked
  • cargo test worked(hmm, although it runs 0 tests :slight_smile:) - Ok, I see, running --package indeed tests them

Those 2 projects are now separate. To run their tests you have to go into their folder and run cargo test or use --package. But I'll think about it(to run from parent all of them)...

3

Ok, here is how I did it(don't know if there is a nicer solution):

[[test]]
name = "test_cionc_parser"
path = "src/parser/lib.rs"

[[test]]
name = "test_cionc_utils"
path = "src/utils/lib.rs"

That's in the root Cargo.toml. You basically tell it to create tests for the other crates.

Edit, removed:

[[test]]
name = "test_cionc"
path = "src/main.rs"

since it already creates and runs a test for this.

3 Likes
4

That's actually quite clever if it works.

5

Thank you very much! Worked for me. =)

6

What a beautiful hack!

7

If you call it a "hack" I am asking myself if my project layout and organisation is still reasonable according to the "rust"-way of organizing projects.
This is based on the thinking that you shouldn't require "hacks" in order to do something that is idiomatic.
So am I still organizing my project in an idiomatic way for Rust or am I doing something wrong here?

8

I would say that your project organization is reasonable, and that the hack is required just because tool support for multi crate projects is a bit lacking. Let me elaborate on this a bit.

Rust is a very interesting language when it comes to modularity. Almost all languages have a module (namespace) concept. However, none of the languages I know besides Rust has a concept of library (would really love to hear about such a language!). That is, in Java, C++, Python, Clojure, Ocaml, Haskell, etc. there is a single global shared namespace for modules. Modules of all the libraries are just dumped into a single search path (PYTHONPATH, GOPATH, classpath) and curious things happen if you have a name collision between libraries. However unlikely it may seems, it actually happens quite often in practice between different versions of the same library. That is, if you depend on A and B, and A depends on C 1.0.0 and B depends on C 2.0.0 then you have a name conflict between two version of C.

And Rust is quite different in this regard. First of all, it has modules which are analogous to similar concepts in other languages. But is also has crates, which are like libraries in other languages, except that in Rust this concept is reified. So, a single crate is a tree of modules, and the crucial thing is that the root module of a crate is anonymous. You declare dependencies between crates by using an extern crate foo; syntax. And <hand-waving> when you compile your project, it is the job of cargo to pass flags to the compiler for "look the crate named foo in the directory ~/.cargo/lib/bla-bla-bla" </hand-waving>. This allows to have several versions of a same library in a single binary without any problem (well, if the library itself does not assume that there can be only single copy of it).

To say the same with less words, lets quote the reference

Note: Unlike in many languages, use declarations in Rust do not declare linkage dependency with external crates. Rather, extern crate declarations declare linkage dependencies.

So, module is a unit of abstraction/encapsulation, and if you want logical modularity, split your code in different modules.

A crate is a unit of compilation and linking, and if you want physical modularity, split your code in different crates.

Corollary: you can have cyclic dependencies between modules, but crates form a direct acyclic graph

Corollary: coherence actually work.

And now the messy part about tools support for this beautiful modularity =)

Currently, the unit of incremental compilation is a crate, so you may want to separate code into several crates just to speed up incremental compilation (In my minim I segregate parsing into a separate crate because LALRPOP (which is wonderful, btw), generates massive amounts of code and I don't really enjoy recompiling it while I mess with other parts of implementation).

And there is also a push in the opposite direction. While cargo makes it easy to split a project into several crates with [dependencies.cionc_parser] path = "src/parser", it currently lacks convenient shortcuts for execute commands on all crates simultaneously. And the hack by @LilianMoraru is exactly for working around this problem. It makes tests from the root cargo project to point inside the children projects.

There is an RFC about similar topics: RFC: Add workspaces to Cargo by alexcrichton · Pull Request #1525 · rust-lang/rfcs · GitHub.

And in you particular case, maybe you should just use separate modules instead of separate crates?

9

Thank you very much for your very detailed answer!
Now I can also feel well about my current project's structure.^^

Rust certainly seems to be doing things better than other languages and I really like the module system.
However, due to its flexible design it is sometimes a bit hard to understand. :wink:

I think I now kind of understood the design rationals behind the module and crate system of Rust.
The division into crates for incremental compilation is a neat idea.

Thank you for pointing me to the RFC #1525, it is always good to hear that problems are already recognized as such!

To your last question:
Everything in my project was in a single crate before I started to factor parser and utils out into two different internal crates ... then my problems began ... However, I think this new project structure is better suited for future development - crates won't get smaller. :wink:

10

Assemblies in C# / the CLR are the closest thing I know of. An assembly is a container for modules (aka classes); assemblies are the unit of compilation when using the C# compiler; modules are uniquely identified by their fully qualified name and the assembly they are contained in; assemblies can be anonymous, but can also be identified by their true (qualified) name, which consists of a short name, a version number, and an optional public key. Two Foo.dll assemblies are different if they have different version numbers or different signers.

1 Like
11

This was the main use case I envisioned for my cargo local-pkgs subcommand: https://github.com/jonas-schievink/cargo-local-pkgs

Running cargo local-pkgs test will run the tests of all path-dependencies of the root crate (and of the root crate itself).

12