What you're describing is completely normal and expected. The exact experience you'll get very much depends on your background, but the overall picture is the same with any new language or technology.
I started using Rust before 1.0, with a few years of experience with C and C++ and a little bit of this and that — Lisp, Standard ML, Haskell, Pascal, Java, etc. Rust was much harder to use back then, but I was very intrigued by its design philosophy, and pushed through anyway. Now I don't want to use anything else. Even though It was hard at first to adapt to it's strictness, it taught me invaluable lessons and made me rethink my approach to programming.
Recently, I've been interested in physics simulations, struggling with getting good performance and realising that there is no way I can simulate millions of particles on CPU. That forced me to learn how GPUs work, learn compute and render shaders, which I've been putting off for years. I had to deal with paradigm shift again, struggling to switch from thinking in terms of sequential operation to parallel algorithms. But it wasn't nearly as hard as I had imagined. I just had to be consistent, just like with pre-1.0 Rust.
Something eventually clicks in your mind, and you start to get it. It's only hard in the beginning. Rust still has a relatively steep learning curve, but it pays off big time — especially when you compare it to C++, which gives you a sense of great power, only for you to realise years later that you're been dancing around its seemingly endless quirks and limitations way too much for the benefits you got out of it. Rust is a very strict but very wise teacher.
By the way, even though lifetimes might be hard to grasp at first, you'll still need to understand the concept if you're serious about programming, regardless of the language you use. In practice, you don't need to use references that often anyway after you grasp Rust's ownership model and move semantics, and rarely need to deal with more than one explicit lifetime. References are pervasive in garbage-collected languages because nearly everything lives on the heap, but not nearly to the same degree in Rust.
Keep learning, struggling and making mistakes. You don't get good at programming by only reading literature and thinking. Don't think you're not smart enough because you didn't get a concept immediately, you just need more practice. When you look at great guitar player, you're tempted to think that they're just very talented, but what actually made them great is consistent practice for years.
).