Great posts... nicely framed question.
I process gigabytes of data. Not tera... yet :). I’ve used sql (all of them :)) excel, python (Numpy and friends)... Notebooks etc. The analytics can be very heavy.
I agree Rust is not the means to experiment/explore. I hope that will change.
Where I have had success is in what I would coin as “byte-level”, as framed above, small domain (1) program types. Rust can’t be beat compared to python here. I’m talking 10-50 fold differences.
A quick addendum:
I’m a big fan of compile-time type checking. By definition, most valuable when dealing with a complex process or analysis. I consider the type-checker a “thought-partner” of sorts; helping to keep my thinking straight and consistent AND helping/promoting pattern finding (reusing types). Again inherently valuable and more so with added complexity.
This capacity is under-utilized in the domain(1). Inherently not a good thing because I'm still "paying the price", one way or another (I'm not talking in terms of computation, but design, engineering and other "meta").
The hurdle for me
A brief case in point, I built a SIMD csv parser and report generator using a recently published "super fast" algorithm (their words, but I won't refute it). The app/tool was a thin exe that wrapped/leveraged two libs/apis that I built using the std lib. One for running the SIMD tape-generator and the other to run a multi-threaded analysis.
I had it up-and-running within 2-3 weeks with thoughtful and poignant/useful input from contributors in this forum.
That said, the hurdle came quickly thereafter. I chalked the two or so years of "on-again, off-again" experimentation with Rust as inherently useful to improve as a software engineer (and "just for fun"). So I was willing to learn/do more. However, my past experience with refactoring, was not going to cut-it in Rust. Despite having unit tests and docs built-along the way, I was quickly "losing sight"; "too many moving parts" is how I described it to myself. So, the project was put aside, for now.
A counter point. I'm versed and have experience with another strongly typed language. To get to production-quality code required two revisions. The first revision was to clean-up/streamline the logic. A good part of the answer involved using some amount of type-level computation; that's when the app started to "pop" (a good thing). The second revision was to switch graphql libs. The enhanced "captured clarity" following the first revision also enabled me to augment the functionality of the app during the second revision without introducing new complexity (i.e., no new forks, nor abstractions required).
The punch-line
I have not yet accomplished the required familiarity/skill to be sufficiently "good at" overcoming the above hurdle in Rust. Rust has the added complexity of having to deal with three versions of a variable (owned, borrowed and exclusive/mut borrow). The borrow-checker is a unique feature indeed - it makes me better (I keep saying to myself :)), but it takes time to build those muscles/intuition. Furthermore, Rust not being either FP nor OO, but is both :)), translated into more to think about. In my case, I failed to leverage the power to reduce complexity. If "software is the wrong way to express an abstraction", Rust may require that much more imagination.
Furthermore, where I was able to find productivity gains elsewhere by leveraging the type system "that much more", I haven't in Rust, yet.
The punch-line within the punch-line
Generics and more generally abstract-data-types are "tough-enough" to reason about. In Rust, there's that and more to learn in order to productively navigate generics over lifetimes, constants and T where T is one of owned, borrowed or exclusive/mut-borrowed.
The path forward
The path to the level of familiarity with the tool ("generics") that increases productivity in a strongly typed language, may be as follows:
-
Design patterns; reduce complexity by "here is the answer 9/10" to solve "x". Idioms and conventions are our friends! A complete inventory here using generics is likely going to be useful.
-
Type quantification: The concepts are well articulated in RFCs and elsewhere, but what seems to be missing is the synthesis: A linear, streamlined mental model for quantifying the abstract data type in Rust. That means
- Inventory of syntax with the same semantic
- The types generated (or not) when implementing a trait (e.g., or i.e., trait objects create a new type)
- The types generated with lifetimes (variance yes, but context is likely the more challenging piece)
- Likely includes a "top-ten" need to know for the async and multi-threaded contexts
So, two prongs from different angles: (1) "here is the typical answer", pattern match to apply elsewhere, and (2) "this is how type-matching works when using generics", that starts with knowing the range of types "admitted" by a given generic expression.
Perhaps another way to capture the essence here: A resource that starts with what abstract data types enable married to the physics/truth of working with memory (a truth enforced by the type and borrow-checker).
Open invitation
My intended "brief addendum" turned into more ending with a plan-of-action. I really like building smart/useful things. Clarity of thought and expression is a critical step. Speed can matter. Rust helps me accordingly. If anyone is interested in further articulating what their path to taking their Rust skills to the next level might look like, please share. Perhaps by formalizing/articulating the right set of questions, we will be that much closer to productive answers.
=> at minimum, create options for using the domain(1) apps as a starting point for domain(2)?