Abstractions are beating me up and I'm starting to hate Rust

[dingoegret]

It all started with trying to extend the behavior of Vectors to have a into_audio_iter() so that I can include "meta data" that can be accessed when using iterator function continuations

raw_vec.into_audio_iter(sample_rate, bit_rate, channels).chunk(hundred_milliseconds).detect_silence().map(|x|{})

That way an iterator adapter like detect_silence() can access the sample rate to use. I didn't want to pass down the meta data along with the real data in Iterator::Item. I've been trying to do this for one week now!

This isn't a help me do this thing post. I've honestly been sapped of all enthusiasm to accomplish that goal. I have Python and C++ code that accomplishes it. I wanted to use it as a learning experience for Rust and so far its been absolute hell. The more and more I use Rust, the more and more I realize that maybe I'm just not smart enough to use it.

The borrow system is always touted as "the difficult" part of Rust. But for me that has been the easiest part! The rules are clear, the application of rules consistent, so far as my early experience goes. The Trait system is the scariest part. Its not always clear to me what is going on when I use traits. Particularly associated types, default generic type params, external traits and some other features. That when isolated, maybe I could really understand what these features do. But when combined together and used in funny ways that as far as I can see in the Iterator crates, is complete magic to me. Ambiguation, lifetime etc issues abound in my attempts.

If I define a AudioIntoIterExt: IntoIterator trait and define a fn into_audio_iter(self) -> Self::IntoIter its not clear to me why Self::IntoIter is an ambiguous association. In my small brain, the trait is just abstract and its up to the implementation to narrow down the concrete of what Self is and therefor Self::IntoIter. Of course I'm wrong but this is the kind of intuitive assumptions I make that get me into trouble with the language's traits all the time. Its also not clear if only the "static" methods defined in the AudioIntoIterExt trait are shared with IntoIterator or all future methods of impl AudioIntoIterExt as well. If they are shared how does that play with associated types I invent inside my trait? What about the interplay of generics and default generic params with all said above? The complexity quickly blows up.

I always hear disparaging opinions on Inheritance and Java and how it is so inferior. Say what you want, but that model of thinking is natural for people. The data proves it, the history of usage proves it. I even get people telling me that Functional is the way the people naturally think. I'm not a cognitive scientist or anything, but I am pretty sure that people actually tend to think about the world in hierarchies and symmetries of THINGS. The kind of model best fit by OOP and Java style inheritance systems. It is clear where the abstractions begin and end and where the concrete takes over. My biggest pain point with Rust right now is that it is not clear where the lines for abstractions are and how the MANY interconnected rules for traits apply to each other.

I see there are more features rolling out for Traits and I'm honestly sapped. I really really want to love this language because for me Rust contributes to that "programmer happiness" buzzword. I genuinely like it for some inexplicable reason. But since I hear no one else really expressing the abstraction features of the language as a pain point and only the borrow system, a seed of doubt in me is saying this is not for me. I want to be told I'm still a noob and I don't know what I'm talking about and your small OOP brain just needs to relearn concepts. I do, because otherwise if I have to implement one more Iterator and in a week still not understand where that lego piece fits into the hierarchy of other iterators I'm just done.

[dingoegret]

Some rules are also so inconsistently applied that it makes me question everything else about the language. Like turbofish using ::<Vec<_>>. Does that mean I can replace generic type params with _? And the for<> syntax. There's always these little items of incongruence that make me question everything. Then there's reading the docs for Iterators. Start at Vec, then Map then Iterator but don't forget FromIterator/IntoIterator. IntoIterator uses its IntoIter associated type to define what it is and what it is turns into Iterator which has the map function but its the Map type that's returned from Iterator which in turn has the next() function to return its Item associated type which is defined in its impl but its actually a generic type which gets constrained based on the return type of the closure you pass it but how is that even defined because the Item associated type uses a generic itself I::Item where I: Iterator and not only Iterator but I: Iterator<Item = and....

At some point in just trying to understand what's going I can't even get myself to figure out where in this chain of impls and traits and structs and Item over generic over constrained over here constrained over there, loopy de loop here and look at this super trait which is actually where this function is being called and my brain is just so overloaded that I don't have enough room left to figure out what even was it that I wanted to do

[matklad]

I totally agree with you that trait system is one of the more complex aspects of Rust, for both reading and writing Rust. I am horrified each time when I see x.foo() and foo is not an inherent method, because it's really complicated to figure out what this foo refers to.

That's why I personally always try to write the simplest Rust code possible, and I describe this as "C style Rust": use non-generic structs, mostly non-generic functions with an occasional impl Iterator<Item=Foo>. I definitely avoid using traits solely for the purpose of making code read nicer, so I almost never define extension traits.

Perhaps you face so many problems with traits because you use them too much? Could you use a generic function instead of AudioIntoIterExt?

Rust traits certainly are an extremely useful tool for building powerful abstractions, like iterators. However, I don't think they should be used liberally in "application-level" coding. IIRC, the whole of Cargo has about three traits.

Traits also can't really be compared to OOP, because OOP is dynamically dispatched and not zero-cost. They are more akin to C++ templates (which are also hellishly complicated, with SFINAE, ADL and other acronyms).

And finally, if you have specific problems with traits, don't hesitate to ask for help here, on IRC, or on Discrord: asking question is usually more effective and fun than hitting the same wall repeatedly

[dingoegret]

Okay well its heartening to hear that I'm not alone.

When I'm writing a program, I write the simplest possible code. Actually, the reason traits took me by surprise is because I have never really used them that much until now big time.

Rust's traits are the most flexible I've seen for abstraction. C++ Templates are also extremely flexible, more so than Rust's Generic system, but as you know templates are also a hellish landscape of misery.

With all that said, I haven't encountered too many other problems with Rust. Its only now that I'm delving into other people's crates and Rust's std, particularly Iterators that I'm realizing just how inadequate my Rust is. The complexity really blew up on me.

Rust's appeal to me is that I feel empowered writing low level code. Being made to feel stupid because of something simple like language features is the opposite of that. A feeling of competency is important to feeling good writing code. But some of the Rust in crates I've seen gives me this sinking realization 1) You're going to be struggling with this language for 2-3 years before competency, not even mastery. 2) You will have to always use Rust because if you miss a month to go work on other language X you lose that hard earned competency.

#2 is what's scary to me because its an all too common story with C++. You've either been on the language for a decade and haven't missed a beat or you hop back and forth relearning over and over and over again in a never ending cycle. And it does not get easier each time, that's C++.

[dingoegret]

Also want to mention I eventually kinda figured out how to move forward with the Iterators. Return type specified by the one of the traits is Self::SomeAssociatedType. In the impl it cant be the same definition because Self is the concrete type impl'ing, but it also needs to be the same definition because it implements the trait..

Paradoxically was stuck until my IDE replaced the impl def with <Self as TraitType>::SomeAssociatedType .. Yeah

[trentj]

@dingoegret

If I define a AudioIntoIterExt: IntoIterator trait and define a fn into_audio_iter(self) -> Self::IntoIter its not clear to me why Self::IntoIter is an ambiguous association.

(It looks like you worked this out, but) I think this kind of confusion often arises from cargo culting. This compiles fine by itself:

trait AudioIntoIterExt: IntoIterator {
    fn into_audio_iter(self) -> Self::IntoIter;
}

I assume you get the "ambiguous associated type" error because you tried to add another IntoIter on AudioIntoIterExt. Was that necessary though? Or were you just copying something you saw in someone else's code? Keeping things simple, as matklad suggests, helps avoid these situations because you start with the stuff you understand intuitively, and add complexity only when you already know that you need it and why.

In fact, the : IntoIterator itself seems suspicious to me in this example, although I can't quite put my finger on why. You didn't ask for help with this code, though, so let me just suggest: I think the supertrait is unnecessary; try making your trait independent and see how it works out. (That is assuming the trait is even necessary in the first place.)

@dingoegret

I always hear disparaging opinions on Inheritance and Java and how it is so inferior. Say what you want, but that model of thinking is natural for people. The data proves it, the history of usage proves it. I even get people telling me that Functional is the way the people naturally think. I’m not a cognitive scientist or anything, but I am pretty sure that people actually tend to think about the world in hierarchies and symmetries of THINGS. The kind of model best fit by OOP and Java style inheritance systems. It is clear where the abstractions begin and end and where the concrete takes over. My biggest pain point with Rust right now is that it is not clear where the lines for abstractions are and how the MANY interconnected rules for traits apply to each other.

I do agree with this, but only up to a point. In general, I think about objects and categories a lot more than about sets and transformations. But I wouldn't go so far as to say that "Java style inheritance" is the way I think. Problems like this one demonstrate situations that are easy for people to describe, but quite difficult to encode into an inheritance hierarchy.

I find, and you may feel differently, that "natural" thinking tends to fall more along the lines of interface-like shared behavior than class-like inheritance. When you need a Chair, you might be fine with a FoldingChair or Recliner, but if I try to give you a HighChair, you would probably prefer to stand. What you really wanted was an impl Seat<You>, and for that purpose, even a Table or a Crate might suffice even though they don't inherit from Chair.

You do get into some level of abstraction hell eventually if you keep trying to make things as general as possible. But that's not a problem unique to Rust; it surfaces in every sufficiently powerful language, Java included (AbstractSingletonProxyFactoryBean nonsense has been sufficiently mocked elsewhere that I don't feel the need to go into more detail). Go is an example of a language that has been intentionally depowered (in some ways) in order to tame this problem from the other side.

[vitalyd]

Even though it seems the problem is resolved, I’d like to see the code and error that @dingoegret hit, preferably in the playground. We can probably explain the interactions/meaning better if there was a concrete piece of code (minimal repro is fine) to talk about.

[dingoegret]

I don't have code saved because this is actually just straight up impossible in Rust without some sort of hackery. Audio transformations using iterators is pretty dull actually, not what I would consider cargo culting. What I would need for this to work is a concept of inheritance based shared properties. Something like a static class variable. Then there is only one place of that meta data and I can transform raw vector data anyway I want within the hierarchy without having to "carry" that meta data with it somehow.

Best possible approach that I think I came up with was to just cave in and work within the confines of Iterator, pass down an Audio data structure with the raw data and audio meta data. It would have the type Item. Interested functions can look inside the data structure for the information they need. Can't use iterators that operate a single items of a container after this though, because now I'm passing along a full or chunked container of data. Or I could just pass a data structures with a single item from that container.. Yeah not happening. Don't know what the overhead of passing down a full fledged structure for what is supposed to be real time data but I'm honestly sapped trying or caring.

I hear "hell" getting attached to every programming thing. Channel hell, callback hell, template hell, monkeypatch hell, unit test hell etc. I'm not fond of arguments that the tool can be abused so the tool is faulty.

Go is an example of a language for many arguments on the internet, just like C++.

AbstractSingletonProxyFactoryBean are very nit picky. Long name, really? What would you name it? Do you understand that Spring is not an application but a framework built primarily for convenience? If you're a strong, powerful programmer, don't need no abstractions then by all means. But if you're a framework trying to cover all cases then "add another layer of abstraction". People who usually make these memes also get sufficiently mocked for being sufficiently misinformed.

[derekdreery]

I often find myself doing Trait::method(val) rather than val.method(), even when it isn't necessary, possibly because I like functional style coding.

For me, the point of iterators is to be able to use combinators. To attach extra info I tend to do .zip(iter::repeat(my_val)) then you have a tuple with the extra info. Not sure if this fits your use-case.

[dingoegret]

That's something I can also do. But I'm just trying to traverse the actual container of data with a common understanding between all the iterators what kind of data that is. Not just merely the container data type. And do it without carrying that extra kind info into each iteration, bouncing around each call of next()

[newpavlov]

First, why do you write AudioIntoIterExt: IntoIterator, instead of just AudioIteratorExt: Iterator? I think this snippet should be quite clear:

trait AudioIteratorExt: Iterator<Item=Sample> { .. }

If you'll need an iterator type, you can just use Self.

Second, I think you chose a slightly wrong approach to your problem. Start with the simple setup with inherent methods:

// in impls you will be able to simply use `I` to reference iterator type
struct AudioIter<I: Iterator<Item=Sample>> {
    iter: I,
    meta_info: MetaInfo,
}

struct AudioChunkIter<'a, I: Iterator<Item=Sample>> {
    iter: &'a mut AudioIter<I>,
   // info to track duration
}

impl AudioIter {
    // note you'll convert raw stream to an iterator right away,
    // so you will not have have to drag `IntoIterator` in your generic code
    fn new(raw: R, meta: MetaInfo) -> Self
       where R: IntoIter<Item=Sample, IntoIter=I>
   { ... }
    fn chunk<'a>(&'a mut self, dur: Duration) -> AudioChunkIter<'a, I> { ... }
}

// generic arguments are skipped here
impl Iterator for AudioIter {
    type item = Sample;
   // ...
}
impl Iterator for AudioChunkIter {
    type item = Sample;
   // ...
}

Now if you'll find that you want duplicate functionality between AudioIter and AudioChunkIter, you can introduce a trait. And you don't have to make it an extension trait.

Yes, writing trait based code after OOP requires switching mindset a bit. But I assure your, it's a matter of experience, after a certain threshold it will just "click" for you. Of course generic trait-based code can be sometimes over-engineered and thus hard to understand, but it's not that different from C++ and Java in which you can create horrible abstract abominations. Ability to design elegant generic APIs without unnecessary complications again comes with the experience.

[dcarosone]

@trentj

When you need a Chair , you might be fine with a FoldingChair or Recliner , but if I try to give you a HighChair , you would probably prefer to stand. What you really wanted was an impl Seat<You> , and for that purpose, even a Table or a Crate might suffice even though they don’t inherit from Chair .

This is one of the better things I've read here lately. 1

[dingoegret]

@newpavlov I don't understand how that would help? Even if AudioIter gets returned by say a function I attach to the Iterator trait, any other Iterator function after it chunk, map, filter, etc would have to return AudioIter or otherwise that meta_info would get lost the very second I call them. map() on Iterator returns type Map, meta_info gone.

[newpavlov]

Ah, so you want to use std combinators and keep meta_info encapsulated in the struct. Unfortunately it is currently impossible. You have the following options:

1. Pass meta_info as a separate argument to methods which need it. The simplest solution, but without encapsulation.
2. Use the following struct and impls:

struct AudioIter<I: Iterator<Item=Sample>> {
    iter: I,
    meta_info: MetaInfo,
}

impl<I> AudioIter<I> {
    fn map<F: ...>(self, f: F) -> AudioIter<Map<I>> {
        let AudioIter { iter, meta_info } = self; 
        AudioIter { iter: iter.map(f), meta_info }
    }
}

[trentj]

@dingoegret

I hear “hell” getting attached to every programming thing. Channel hell, callback hell, template hell, monkeypatch hell, unit test hell etc. I’m not fond of arguments that the tool can be abused so the tool is faulty.

It's a good thing I didn't make that argument. In fact, I'm pretty sure I was making the opposite of that argument: Rust and Java address complexity in different ways, but they both become unwieldy under too much abstraction. I, personally, think Rust's approach is better on the whole, but both ways are valid.

@dingoegret

Go is an example of a language for many arguments on the internet, just like C++.

Now you're just being dismissive. My point was not to bash Go, but to further illustrate the point that there are many ways to deal with too much abstraction. But I'd rather forget about Go altogether than start an argument about it.

@dingoegret

AbstractSingletonProxyFactoryBean are very nit picky. Long name, really? What would you name it?

It's not the name that is bad, it's the fact that such a thing even exists, or needs to. You don't see this kind of thing in Python; surely it's not because Python is a less general or less powerful language. A nice essay on the subject is Execution in the Kingdom of Nouns by Steve Yegge. It's a little bit dated, but I do think it serves to explain one reason why you see stuff like AbstractSingletonProxyFactories in Java but not in other languages.

(And yes, I know that's part of Spring, not Java, and I don't consider myself a Real Programmer who doesn't need no abstractions, but I do consider the proliferation of abstractions to be a problem in its own right. All problems can be solved by adding another layer of abstraction, except the problem of too much abstraction, after all.)

And while I'm linking to blogs, Wizards and Warriors by Eric Lippert is a five-part series that revealed some flaws in the way I used to think about object-oriented design. It may not be relevant to you but honestly, it's pretty short and I would recommend it to most programmers.

[dingoegret]

@newpavlov

Ah, so you want to use std combinators and keep meta_info encapsulated in the struct. Unfortunately it is currently impossible. You have the following options:

Yeah that's what I've come up with as well. Thought I would implement my own map and chunk etc and create wrapper types that hold the meta info and they could return AudioMap, AudioChunk and those could keep instances of real Map and real Chunk and merely delegate to them. Le proxy pattern and create them in AbstractMultitonProxyFactoryIterator : )

But I'm not going to do that because its tedious. Unfortunate there's no hierarchical property values. Raw procedure calls it is

[dingoegret]

When I get home I might but I can't promise I'll read your blogs. It smells like authoritative linking to me and anyone can find a blog for their position. Generally the anti abstraction thing is just posturing that's repeated by hipster game programmers. The sentiment then bubbles down to the rest of the industry and you start hearing people who have no business holding those values parroting it.

No you're not bashing Go. But Go is a go to for "Go doesn't do or does the opposite of feature X because X is bad". Please don't reply that's not what you said, you implied Java's abstraction mechanisms has a bad trait to it and THATS WHY Go runs the other way. I said, you said, no I said etc are very tedious in back and forth posts. Own up to your stated positions, implied or otherwise.

It’s not the name that is bad, it’s the fact that such a thing even exists, or needs to.

And what "thing" is that? Imagine calling a thing out because haha its name is funny. No no, actually not its name, I meant that it's a thing. Like, it exists.

What is the malfunction with AbstractSingletonProxyFactoryBean? That thing existing is very good actually. Maybe you could provide another example you are more familiar with?

Python doesn't have types and the only time I use Python is to prototype, script out a simulation using Simpy or do some tricky computations. I don't use it to build systems. Systems have coupling. Coupling is bad. Abstractions decouple. Certain coupling is so pervasive that their solutions are commonly repeated. We give them names. We say they're called patterns. None of this is bad and is a net good for the industry.

[newpavlov]

@dingoegret

But I’m not going to do that because its tedious.

How many Iterator combinators do you want to use with combinators dependent on meta information? ~5-10 four-line wrapper methods does not seem too bad to me. (probably it's not a bad idea for a crate: iterator wrapper which will encapsulate some information and will handle wrapping combinators) Plus you always can call inner(self) -> I method to get an underlying result iterator and use all remaining combinators on it

[trentj]

I take it that you think the blogs I linked to are hipster game programmer posturing and I have no business holding opinions, so I think I'm going to stop engaging in this thread.

[dingoegret]

@trentj Not what I meant at all. I Apologize it came across like that. Haven't even looked at the blogs actually. The anti Java and anti abstraction train are a meme themselves at this point and I thought it would be clear who we're talking about here. Particularly Mike Acton and Jonathan Blow