I’ve been working on this problem for a few hours every day for about 10 days. I think I’ve exhausted all options and it can’t be solved. I’m hoping someone can put me out of my misery by confirming it can’t be solved or pointing me in the direction of a solution.
The root problem is that the program has to parse billions of records, read from files, and aggregate them based on a composite key, defined as a struct, in a HashMap. And it needs to be fast. Right now it will parse and aggregate 19M records in about 47 seconds. But testing with some dummy values indicates that if I can get rid of most of the .to_string() calls I can get that down to 30 seconds.
To get the best performance I don’t want to convert &str to String unless I absolutely have to. It’s billions of records during each run and .to_string() gets really, really expensive especially considering I only need one copy of the final key for permanent storage in the HashMap.
I thought I could solve the problem by using a generic form of the key that can take &str until it needs to be inserted into the HashMap, then build the owned version of the key with a String. And I’m really close.
You can see my work at https://github.com/ereichert/counter/blob/feature/temp_record_approach/src/record_handling.rs.
But it doesn’t compile because of this line https://github.com/ereichert/counter/blob/feature/temp_record_approach/src/record_handling.rs#L35.
compiling counter v1.0.0 (file:///Users/ereichert/dev/counter) error[E0495]: cannot infer an appropriate lifetime for autoref due to conflicting requirements --> src/record_handling.rs:35:43 | 35 | system_name: self.system_name.as_str(), | ^^^^^^ error: aborting due to previous error error: Could not compile `counter`. To learn more, run the command again with --verbose.
There may be other problems. It seems every time I solve one puzzle another is presented. But that line is the problem I have not been able to solve.
If that can’t be solved is there any way to check
for an existing key using a
as the key?