In the case of AsRef, really the only possible mode of failure is panicking. So to users it means "you may assume this does not panic," and to implementors it means "pretty pretty please do not let this function panic because users will find that very surprising."
Nothing is truly stopping you, however, from writing an implementation that panics.
Another thing that comes to mind is you can't implement AsRef<str> for something like Option<&str>, because you won't always be able to get the str by reference.