It's much harder to return a reference I would say. Returning a reference requires the container actually contains the object, this forbids returning a Cow or any kind of guard object. All types in you link to doc falls into the "slice indexing" category, or "contiguous-storage types" if you like this term better. These types are nearly the only ones that is easy to implement Index[Mut] (, beside Index for HashMap-like.
If you search std::ops::Index in the user forum, you can easily see many people having problem with implementing it.
There's even an RFC for improving it.