References are not pointers, but temporary locks on data.
You don't have any place where you store clients. In the hash map you only borrow them temporarily, but there's no owned storage for them except a temporary variable inside a function.
Reference can't exit without pointing to an owned value.
Try not using &mut in any data structures, and store owned objects instead. If you need pointers, use Box or Rc pointers.