Here is a very thorough explanation of how the borrow checker sees it:
Whenever you have any function that links return data with input data using a lifetime, the input reference(s) must be valid as long as the output reference is held.
For example
// this has a lifetime
fn explicit<'a> (borrowed: &'a i32) -> SomeStruct<'a>;
// this has a lifetime too, only it's implicit
fn implicit(borrowed: &i32) -> &i32;
// this would be equivalent and more explicit
// fn implicit<'a> (borrowed: &'a i32) -> &'a i32;
In this case, .into_iter() is essentially a function with this signature
fn into_iter<'a> (self: &'a X) -> XIterator<'a>;
Which means that as long as that the borrow inside that XIterator is held, the reference to that X must remain valid.
After calling .into_iter(), you then call .collect() on the XIterator. In this case, it is essentially a function with the signature:
fn collect<'a, 'b> (self: &'b mut XIterator<'a>) -> Vec<&'a i32>;
As long as any of those &'a i32s are held, the borrow inside XIterator<'a> must be valid, which means the borrow of &'a X must be valid.
Then you call .iter() on your Vec<&'a i32>, which, without going into yet another function signature, ensures that you are still holding that vector (because you need to continue iterating over it) for the entire duration of the loop.
Then you call .set() on the original X, which thanks to it's function signature, requires a mutable reference. So you try to mutably borrow x, but thanks to the aliasing rules, you can't mutably borrow x while it is also borrowed immutably.
It's immutable borrow needs to persist because the lifetime inside the XIterator is still held. The lifetime inside the XIterator is still held because the lifetime inside the references inside the Vec<&i32> is still held. That lifetime is still held because the iterator obtained by calling .iter() is still held. That iterator is still held because the for loop is still in progress.
The borrow checker can reason through all of this without even referencing the code within any of the methods I described. All it needs are the function signatures because they tell you the dependencies between the lifetimes.
Does that clarify the error?