I think you're optimizing for the wrong thing. The ko check is probably one of the cheapest things you do regardless of storage size: hash the board state and look it up in a table, done. It might even be vectorized. Compare that to, say, checking whether a color group is captured, which you will probably do many many times, and could be significantly faster with indexed byte arrays rather than bit-twiddling.
That said, there's another option you might consider: have two
BitVecs, one for black stones and one for white stones. Total size = 736 bits, and I'm guessing it will make certain tasks a bit easier, like finding connected groups. The downside is it becomes possible to represent impossible board states by having a black stone and a white stone in the same space.
(Disclaimer: I've never written a Go bot.)