Why do you not just return an Option<&Vec<V>>, or – even better – an Option<&[V]>?
The type &Vec<V> is not an iterator, but it is a collection type that you can iterate over, i.e. that you can convert into an iterator. The relevant trait is IntoIterator. (The same applies to Option.)
Something like -> Option<impl IntoIterator<Item = &V>> probably compiles, or to turn it into an iterator, you could add a call to .map(IntoIter::into_iter).
It is not an iterator because, well, it is not an iterator. It can be turned in to an iterator, but it is not an iterator. You could change your filter_map to return Some(vertexes.iter()) for example.