Prototyping methodologies

community
1

Hi Everyone,

Let me introduce this question by saying that I love Rust for many reasons. I feel like code written in Rust is "Crystalline", that is to say rigid, but also reliable. It's huge for my productively not debugging typos that become silent runtime errors, or memory smashers, or any other bugs caused by any number of sloppy things that plague code written in other languages. And it just feels good having that confidence and peace of mind.

The problems come up when I'm not sure what I'm trying to do yet. For example, I recently did a refactor where I wanted to take a Vec of structs and convert it into a struct of Vecs. Anyway, without exaggeration, it took me 3 full days to get my project compiling again. Granted there were hundreds of places that touched these objects, and I had specialized types to hold the indices, and the whole mechanics of borrowing an object and accessing a field was now flipped inside out. On the plus side, I love the way, when it did compile, it totally worked 100%.

The trouble is that I feel like Rust wants me to always make well defined interfaces, and strictly adhere to them. Which is wonderful for code trustworthiness when the "shape" of the solution is clear, but a real time sink when the program I'm writing isn't clear in my mind yet.

The discussion I'm trying to start is along the same lines as this thread by @botandrose

Anyway, I'm wondering if anyone has found some ways to get the best of both worlds.

Like, for example, a way to write big chunks of your program in a quick-and-dirty language like Python or JavaScript, and seamlessly link them together with the parts in Rust. And when it appears to work, go back and rewrite things in Rust module-by-module. I admit I haven't fully thought this through, and obviously if you had to precisely define which Rust types the JS objects would convert to, it might defeat the whole purpose. Anyway that's just one idea. I'm sure there are better ideas.

Any thoughts and opinions are appreciated.

Thank you.

1 Like
2

It's a great feeling isn't it. Don't loose sight of what you said there: "It's huge for my productively..."

That has bugged me also over the years. Not being sure what I'm trying to do or how I might do it is my natural state of being :slight_smile: But it's not unique to Rust. I have often found myself writing quick and dirty prototypes of some idea in Javascript which end up in C or C++. Basically I want to quickly find out if the idea even works at all, do I actually understand how the algorithm operates? do I get the results I expect.... My mind can be fully occupied with tinkering with the "idea" itself and I don't want to be distracted by worrying about all those petty details compilers complain about all the time.

My question is... Assuming this is not a trivial little program, are you sure that such a major rearrangement of your code would have been completed in significantly less time? Perhaps in C++ it would have taken only 2 days or 1. Well, as I said, don't loose sight of that productivity you opened with. And as you say, "it totally worked", there it is again, you have that confidence in reliability, you have saved all that time checking that your refactored code is not broken in some difficult to fathom way.

That is an interesting idea. Just now I'm having trouble with the "seamless" part. My gut tells me that the difficulty of bolting a Python or JS module, or whatever, to a Rust program would outweigh the convenience of being able to do the quick and dirty experiments in Python or JS.

In a way that has kind of happened with a system I am now working on. Chunks of Javascript and node.js became oxidized, actually I don't think we have any JS left except in the browser. But that was an easy progression as the chunks were free standing processes. I think the kids might call them "micro services" now a days.

7 Likes
3

Ironically, my best way of getting out of this situation is to creating very, very well defined interfaces. If there's a module I want to prototype, I try to reduce its external interface to the absolute minimum, and specify that extremely well, so that when I change the inside, minimal external code changes. I feel like if I do that, then I'm free to change anything inside the module and it'll still work.

Of course then there's the problem of coming up with a good general interface to the module, and that interface often ends up reflecting the first prototype. But regardless, it's my current best method for avoiding entire-codebase-shifting refactors. If the outside of something is well specified enough (and reduced enough), changing the inside should be proportionally easier.

2 Likes
4

Further to daboross comment above I was thinking about your situation there as well.

It seemed to me that you have pile of data there, in whatever arrays and structs, presumably all related to some task. Then your code has evolved over time using those arrays and structs directly. As such all the details of the data structure have permeated every nook and cranny of your code. When it comes time to change the data layout the whole code base needs modifying.

One could imagine a more modular scheme where the data is all wrapped up in some module(s) and is only manipulated through some API. One can then rearrange the data to ones hearts content without breaking the application code.

Of course, as you say, we often don't know where we are going when we start so it's easy for things to grow into a tangled mess as we explore here and there.

Perhaps the answer to your original question is to have a good long think about the problem before starting out. Less "agile" hacking and more good old fashioned design up front.

2 Likes
5

That looks like a leaky abstraction to me, then. You should probably encapsulate the representation of the data stucture, and only expose a thin public API for accessing it, independent of its representation.

I could imagine a library for calling Rust from Python and vice versa, and since we're talking about quick'n'dirty prototyping here, I think just slamming on a std::process::Command or a subprocess.run() on either side would be acceptable. Maybe hook up std{in,out,err} to pipes and communicate types and data structures transparently through serde_json? (This is to avoid having to write FFI code.)

I would certainly appreciate if it was easier to communicate between Rust and scripting languages which can be invoked from the command line like that – as long as it is solved without clobbering the core of either language with additional features.

6

Not to be defensive, but it's bloody hard (impossible?) to design good abstractions that don't leak implementation details when your internal data is stored with RefCells and you use the interior mutability pattern.

Not to derail the topic away from ideas to make quick'n'dirty prototyping easier, but if I could change one thing about Rust, it would be to allow an object to take ownership (for lifetime purposes) of any other object. A function like "drop_this_after_that" (there's definitely a better name). For example, the function would let an object that was borrowed from a Ref<>, then take ownership of its Ref<> (and in turn the object would inherit the Ref<>'s borrow from the original RefCell). That would mean the Ref<> wouldn't be accessible in code anymore, but would be hanging out in memory and wouldn't be dropped until right after the target object was. This feature would solve all the "cannot return object borrowed from temporary value" problems, and make creating sensible interfaces so much easier. But that's a tangent.

Does anyone have any experience with deno.land and rusty_v8? I haven't tried it yet, but I thought it might be a good base to start building a quick-prototyping environment.

2 Likes
7

IME, this is about equally true in every language. My professional career has been spent entirely in C++ and Javascript, but I have run into situations like this where touching one thing means changing a million other things almost exclusively in our Javascript codebase... because our Javascript codebase is where all the essential complexity and interdependencies happen to live. It's neither language's fault, I just happen to be working on a product where the client code is far more "interesting" than than the server code.

Why is that? Obviously interfaces constrain possible implementations, so it's easy to have an interface that cannot be implemented without some interior mutability, but isn't it always an option to Box or Arc things so you don't need to flood the code with Refs? And if you need the performance of not allocating that badly, aren't you already way past the prototyping stage and any possible solution would fundamentally require calling code to be involved in lifetime enforcement no matter what the language? (e.g. intrusive data structures)

IIUC this is basically asking for Go-like "escape analysis" where anything that doesn't have neatly tree-like stack lifetimes gets implicitly heap allocated instead.

People sometimes talk about making a "Rustscript" that's just slightly less hardcore than regular Rust, and I've always felt that if such a thing was made by far its biggest difference from regular Rust would be to implicitly Arc a bunch of things. In fact there's a language called Lobster that apparently intends to do precisely that: escape analysis + Arc by default with an "opt-in to a more Rust-like model".

Of course, whether this actually helps with the case you describe is unclear. If this was viable for your code, then you should just be using Box or Arc to begin with and then the viral nature of Refs goes away.

In general, I personally find Rust to be far more friendly to exploratory/prototyping code than most statically or dynamically typed languages.

This whole thread seems to me like a textbook case of discussing a claim ("Rust makes prototyping harder") at such an overly abstract level that it becomes at best essentially meaningless, and at worst simply false to everyone with different experiences. We're clearly not going to get very far without bringing in some very concrete use case.

So, what is the case you ran into where you think you need RefCells everywhere and you still consider yourself in a "prototyping" phase and you have so many callers it takes days to update them all and introducing Box, Arc or any other layer of indirection between the RefCells and the callers is not an option, all at the same time?

5 Likes
8

…which are far fewer in number than the kinds of memory management bugs the current ownership system prevents. It also places the importance of programmer annoyance above that of preventing bugs, which I find misguided in the context of a systems language.

Day by day, new posts are created on the URLO forum, where people new to the language want to change one of the two principal memory safety mechanisms, ownership and borrowing, being upset that code they are used to writing in other languages doesn't work in Rust as-is. Recent examples are: one, two, three, four.

That is not a good direction to go in. Rust is different from other languages, and if one didn't (have to) consider the problems it solves before, one will need to learn new patterns, and probably also un-learn old, bad habits that happen to be dangerous but have been largely ignored in the past.

6 Likes
9

I'm certainly not talking about completely replacing Rust's ownership and borrowing system. I agree, they are beautiful and prevent a whole bunch of problems inherent in other models.

I think what I described, (perhaps I communicated it badly), is a minor tweak that doesn't change any of the principles of how Rust's memory management works. Although I have no idea what would be involved implementing it in the compiler. As @Ixrec said, it would require temporary objects retained in this way to be allocated on, or moved to the heap as their stack frame might be deallocated.

I completely agree that one of the beautiful things about Rust, and its memory management in particular is the fact that the language won't let you shoot yourself in the foot unless you're really determined to do it. But Rust is a young language and there are certainly places where improvement is still possible without compromising the things that make it great.

10

I can't come up with an interpretation of what you said that would qualify as a "minor tweak" without also being blatantly unsound. Which, as H2CO3 was getting at, is pretty typical of suggested changes to ownership/lifetime stuff.

If you still think you have a novel idea, that is 100% worth posting a new thread. But be sure you spell it out in great detail in the OP post with complete code examples that won't compile today but will with the change and precisely what the proposed rules are. The fact that almost no one ever does all that is probably why most of these suggestions appear to be (and after much discussion, usually turn out to genuinely be) either deeply confused or a mere rephrasing of what Rust or some other language already does.

2 Likes
11

Hi @LukeTPeterson.

When it comes to prototyping, I have different strategies, depending on what it is that I want to quickly check. E.g. when I want to know if a numeric algorithm works correctly, I use GNU octave to find that out. In order to check if the UX of a user interface is any good, I would use something like Glade. In a corporate context, I wousd use Excel or MsAccess to validate a new business process. For other things, I would sometimes hack together a shell script.

As you can see, for me, prototpying always involves using a different technology to quickly validate an assumption and iterate over one aspect of the application without building a full-fledged application.
Now when I say "quickly", you have to take that with a grain of salt: finding a good numeric algorithm can take weeks, months or even years and involves doing mathematics. Optimizing a business process can take long as well and involves discussing with the business. The point is that it allows you to focus on one aspect, ignoring the others.

So I think it depends on what you want to quickly check. Reading your post and comments, I guess that you want to quickly check the design of your software. Now, unfortunately, you cannot use another technology for that: software design in other languages doesn't translate well to Rust. So how to approach this?

One could argue that Rust is the ideal language for validating software design because, as you may have experienced, the design is validated at a very early stage, at compile time, using various restrictions. Unfortunately, it doesn't really allow to validate different aspects of the design in parallel: most problems you have to fix one by one.

So then how to iterate and find a good design? I think it's an art, just like numerical mathematics, UX or optimizing business processes. Coming up with a good design is very hard.

I can tell you my personal approach. Now, reading again what you've written, I see that my personal approach is anything but quick and dirty programming. In fact, I've learned the hard way that for me, it works best if I try not to go against the language, unless I really know what I am doing and why I need an exception. For me, things that "go against the direction of the language" are precisely Rc, Arc, Cell, RefCell, Mutex, marking fields as pub, unsafe, macros and procedural macros. Higher-ranked trait bounds (for<'a>) are a red flag. I try to avoid these. If I think I need Rc or Arc, this means that maybe I need to think about ownership. If I think I need Cell, RefCell or Mutex, maybe I need to rethink ownership and mutability. unsafe is for rare cases only. Macros and procedural macros: maybe I'm over-engineering.

Now I have to put a big disclaimer here: this is what works for me, for the particular type of software I work on and it doesn't necessarily apply to your situation. For instance if you're building a user interface with GTK, you will probably need some Rc or Arc.

For me the caveat is that, yes, sometimes I need one of these things. The art is knowing when it's justified and when not. For instance, for one of my projects, I reached for an unsafe block because I knew for sure there was no other way around it. It works flawless (at least until now). In another occasion, I believed I needed specialization and I used macros and procedural macros to hack that into the language. I've regretted that ever since.

I hope this helps. I now realize that it doesn't really answer your question for quick and dirty programming in Rust (maybe to the contrary). Maybe the conclusion is that Rust isn't well suited for quick and dirty programming. Anyway, in an effort to help you, here are some things that I tend to do a little "dirtier" in Rust that I wouldn't in other languages:

  • Module and crate hierarchy. Can be fixed later and is usually not clear from the start, so no need to think too much about that upfront.
  • Accessing private fields from other structs in the same module. I do give this a second thought and I sometimes write an abstraction, but often these structs "belong together", so it's not a problem.
  • Borrowing in a method instead of in the struct. I learned that this makes the design much easier many times and sometimes it even makes sense. So instead of holding on to a reference of something in the struct, I just pass it when calling the method.

I hope this helps.

1 Like
12

For comparison, what kind of statically typed language is better than Rust at prototyping? Except for PLs with global type inference like Haskell, I don't really see what gold standard Rust would be comparable to, when it comes to prototyping.

If the crux of the problem is static typing itself, I suppose std::any could help. Maybe a library could build prototype-friendly features around it.

As for my approach: I start out identifying the basic building blocks that I'm certain to require, implement them and write unit tests for them. Then, I go over all the features I want and try to order them from least-effort (given my already-made building blocks) to most-effort (likely to become easier as the least-effort stuff gets implemented). I also allow the program to panic freely, until I figure out an overall design, and intersperse refactoring sessions on an as-needed basis.

13

Liberal use of clone() has been suggested as a way of avoiding some the compiler nag that makes quick and dirty prototyping difficult.

This and other techniques are discussed elsewhere in this forum, see Difficult/Long refactoring, suggestions? Also, a search for "difficult" returns many other issues around this topic.

While purist or advanced Rust developers might dismiss liberal use of clone() and similar techniques as sloppy code, I think there is considerable value in accommodating those new to Rust. It certainly would contribute to increased Rust adoption and, at a minimum they are working in Rust and therefore have an opportunity to improve their Rust skills as the need and/or time become available to optimize.

To address the original question, using a more forgiving language for prototyping, then converting to Rust, slowtec has a series on Porting a JavaScript App to WebAssembly with Rust. In a three-part blog series the experimental conversion of a JS/React web app to Rust/wasm is described. It makes for interesting reading with original JS and some Rust code provided.

14

Bryan Cantrill describes his first non-trivial project with Rust here: "Statemaps in Rust": Statemaps in Rust - YouTube

There he tells how he was having a horrible time fighting with the borrow checker and such as he tried to get familiar with Rust. In desperation he hacked the thing together just to get anything to work at all, convinced that cloning and the like would make it horribly slow compared to the existing C version of the program.

I guess we have all been there.

The punch line is that the finished program performed better than the original C code!

Coincidentally, Bryan has now joined a start up that is building it's infrastructure on Rust.

I conclude : Do not fear the clone.

1 Like
15

Oh not at all. It's very common for expert Rustaceans to recommend "cloning all the things" on your first draft of the code, then revisiting them after you actually know what the overall structure of the program looks like. Especially when we're taking about cloning to construct a struct with an owned type instead of a reference. In any language, it's often impossible to know which clones/deep copies/etc are necessary before your basic architecture has solidified, no matter how experienced you are with the language (as opposed to the specific program).

Rust doesn't force you to write .clone() to make you feel bad, but to ensure you're aware that your program will be doing something potentially expensive.

5 Likes
16

Now with a few years of Rust experience I don't find rigidity of ownership to be a problem, even for prototyping:

  • I have enough experience to predict what will be borrow-checker-friendly and what won't, so I can sprinkle Arc or clones where needed.
  • I have learned to use architectural patterns that fit Rust's tree-like ownership.

Things that slow down my prototyping are mainly types in general:

  • In quick'n'dirty prototypes I overuse tuples, and then quickly regret it, because code with x.0 + z.1 is unreadable. Defining a struct is not difficult at all, but it feels like an extra step, too pedantic for a prototype.
  • Numeric casts are noisy and a chore.
  • Type conversions in Rust are tuned for correctness, not for being quick'n'dirty. Sometimes it's .into(), sometimes it's verboser try_into().unwrap(), sometimes it's x.to_str().unwrap_or_default().

Plus compile times, even incremental in debug mode, are on the slow side for experimentation.

What would help me for prototyping:

  • Auto-cloning Arc into closures and async blocks, so that I don't have to do {let foo = Arc::clone(&foo); spawn(move || foo.x())}.
  • Tuples with named fields.
  • Implicit numeric conversions.
  • Faster compilation times.
5 Likes
17

Aren't these called structs? FYI, I concur on the other three items in your list; they do slow rapid prototyping.

18

There's a pretty broad spectrum of options between "struct" and "tuple" depending on exactly how you define the two. I believe what kornel's suggesting here is a type with named fields that doesn't require an upfront definition the way struct does. The concrete proposal that's probably most relevant here is RFC #2584: Structural Records:

Introduce structural records of the form { foo: 1u8, bar: true } of type { foo: u8, bar: bool } into the language. Another way to understand these sorts of objects is to think of them as "tuples with named fields", "unnamed structs", or "anonymous structs".

(terminology is a bit of a mess in this design space :slight_smile: )

IMO, the most interesting bullet point there was:

because the others are all fairly well-explored in other threads, and it's easy to guess what Rust will do about them in the future when it finally has the bandwidth to push them through, but this one I'm not aware of any plausible solutions for.

(someone did suggest clone || ... closures, but that's not getting much support for a bunch of reasons that I mostly agree with)

2 Likes
19

Way to go. Let's see how many different ways we can do the same thing in one language. Push the complexity to 11.

Hey, "class" is a kind of struct we could have that as well. Javascript got one, Rust should have one too. I'll be getting on with my RFC for it.

No seriously, how do we tell when Rust is complicated enough?

20

This is certainly the strongest point against adding structural records, and why I am extremely supportive of simply not doing this anytime soon because we have a huge backlog of far more important things to implement (which I take to be the de facto implicit response here, considering that RFC has not even been FCP'd to postpone despite being open since 2018).

1 Like