For video you'll need to handle non-contiguous planes, so PlanarFrame needs to accept up to 4 planes (YUV+A), at varying resolutions (due to chroma subsampling). And with varying stride/pitch (since sometimes planes are exact width, sometimes even-only width, sometimes rounded to block sizes).
It would be nice if the library was fully aware of BT. 601 and Rec. 709 color spaces, including gamma-accurate conversions between them. It's such a pain to get colorspaces accurate across formats and HD/non-HD videos.