Copy: the real
Color) has references to heap allocated Vec and buffer types, so it can't be
Copy. I should have added something like that to the minimized example to make it more clear.
I've ended up staying with @mbrubeck's first recommended solution for this, with a few optimizations that bring the "performance cost of safety" down to 7.4% (wall time) in the normal case. One optimization was just to limit the move to a smaller inner struct that is really the desired target for mutations. This is a good direction in any case because it avoids any further possibility of mutations that could invalidate the tree walk. I looked at a bunch other options, including the use of Cell's, but they all appear to have other drawbacks.
The workaround requires a bit of documentation, but I suppose its better than having to explain the potential problems with the unsafe version. See excerpt from the real (as yet unreleased) library below:
/// Perform a depth-first (e.g. children before parent nodes) walk from the
/// specified node ID, applying the provided function.
/// The `Fn` can be a closure or free-function in the form:
/// fn a_filter_fn(pos: NodeRef<'_>, data: &mut NodeData) -> Action;
/// Where `data` provides read-write access to the the `NodeData` of the
/// current node being visited, and `pos` gives a read-only view to the
/// remainder of the `Document`, e.g. parent, children, and siblings of the
/// current node. Note that to avoid aliasing issues, the `NodeData` is
/// actually moved out of the `Document` and replaced with a
/// `NodeData::Hole` value which could be observed via `pos`. The
/// potentially modified `NodeData` is moved back to the `Document` if the
/// function returns `Action::Continue`. The function may also modify the
/// `Document` by returning other [`Action`] values.
pub fn filter_at<F>(&mut self, id: NodeId, mut f: F)
where F: Fn(NodeRef<'_>, &mut NodeData) -> Action;
I learned some things about the borrow checker due to this scenario, but I think we are still friends. Thanks everyone for your suggestions!