I don't think binary size is usually included when considering performance, though it can of course be a relevant metric to consider in some contexts. The bulk of the difference is due to Rust binaries being statically linked, I believe. This blog post is a few years old, but still relevant I think: Rustlog : Why is a Rust executable large?
If you want to have small binaries you can give UPX(https://upx.github.io/) a chance. But please be careful if execution performance and RAM usage. But it creates small binaries with a cost of higher starting time and its using more RAM.