I get that contains() expects a reference but having the reference symbol (&) in front of a number looked odd. Aren't numbers just copied by value? What does it mean to have a reference to it? Does that mean all numbers we use throughout the program are stored somewhere so that the same number can have the same reference?
It's a reference to a constant number which is stored in your program's binary. If you have a literal like 2 (or true, or "hello"), it gets put in the binary in a section of constants; the reference is a pointer to that, which you can see here.
When you compare two references to the same type, you can think of it as the equivalence operator (PartialEq implementation, for complex types, or compiler intrinsic, for primitives) being able to read the values behind the references and compare them, not the references themselves.
The contains method requires a reference to an integer, and as such, you must pass it a reference. It requires a reference, because the method is generic, and works for both Copy and non-Copy types.