b contains mutable references to instances of X, and pushing those into a and trying to keep them as mutable references would create two separate mutable references to the same value, which is not allowed.
Or, in more empirical code:
b.iter()
Returns &T items, from which one cannot acquire &mut Anything,
b.iter_mut()
Returns &mut T items, from which one can acquire mutable references, however you cannot use b anymore after that (since now the lifetime of the mutable references in a are tied to a mutable borrow of b).
b.into_iter()
Is the same idea as iter_mut().
Note that variance rules would come into consideration should you choose iter_mut() or into_iter().