Ah, it's good old Problem Case #3 again. tl;dr: intended to work, not currently supported.
On nightly you can switch to an experimental borrow checker which ought to support it correctly (add -Z polonius to rustc flags).
Here's a previous discussion on the topic. I've mostly given up on understanding the workarounds because I consistently fail at it, but there are workarounds at least in some cases -- maybe one of them will work for you.