Closures in API design – theoretical limitations and best practices?

You are holding it totally wrong.

It's not that there's an "undesirable restriction of control flow". The right question to ask is exactly the opposite: why should there be an additional feature to regulate control flow here? Iterators are already lazy, and the fallibility handling is built into the container. Your "ugly" functional example is perfectly expressible as

fn functional_style() -> Option<Vec<u8>> {
        .map(|i| i.checked_mul(10))

And this is the correct way to do it: whoever collects the iterator has the opportunity to simply stop calling next() if that's what s/he pleases, and this is exactly what Option and Result's FromIterator impls do. There's absolutely zero need to run additional circles in order to do exactly the same thing with different syntax.

TL;DR: learn the standard library, don't blame the language.