This mirrors my experience. VP9 is unbearable even with realtime, and VP8 is bearable, but compared to x264 the speed and quality are both much worse. But I guess it's possible that I'm just using it wrong.
Edit: nvm, thanks, @kornel, I was already using deadline=1 but i also had to set cpu_used=6 to have any effect. it's still nowhere near as good as x264 but it's enough for now.