This problem is generally known as finding contours. rten_imageproc::find_contours has an implementation of this based on the algorithm that OpenCV's findContours function uses. It uses a different tensor library (rten-tensor), but there is some code here which shows how to convert an ndarray view into an equivalent NdTensorView. Alternatively you could copy and modify the find_contours implementation.
The rten-imageproc crate also has functions to simplify polygons (that is, find a polygon for the mask with fewer vertices but not as tight a fit to he original mask) and get rotated bounding rectangles containing a polygon.
Edit: One assumption I'm making here is that you want to treat each group of connected pixels in the mask as a separate polygon. If the mask has gaps but you want to get a single polygon as output, you could merge the polygons by taking the convex_hull of all the polygons' points.