There may be ways to get it to work using a reference but it is going to be quite annoying with lifetime handling and 'static bounds.
The simplest solution is just to wrap the Material in an Rc (or Arc if you are doing multi-threading). This provides what you're already trying to do with Box. Unlike Box which is designed for cases where there is only one owner, Rc and Arc are designed for situation like this were you have many references to the same data.