Thanks for the answers! I think this solves my problem. Let me summarize the conclusion.
The Panics section documents the function behavior. It doesn't document conditions under which the function should not be called, but would still have a reasonable behavior if it was called. This is opposed to the Safety section which documents when the function should not be called, and would not have a reasonable behavior if it was called. Because of this distinction, the Safety section can be weakened, but the Panics section cannot.
If one wants a safe function with conditions under which it should not be called, it should explicitly mention that it is a user error to call the function under those conditions, but that if by accident it were to happen, the behavior is safe but otherwise unspecified (for example a panic). Only then the behavior can be changed (as long as it remains safe), because it was documented so.
One can see those conditions as safe UBs and they probably deserve a separate section between Panics and Safety (maybe Unspecified, Undefined, Unstable, Invalid, ... I'll need to think about it).