It's possible to use Rust without Xcode in the same way it's possible to eat a soup with a fork.
In practice, Rust requires developer tools that Apple bundles with Xcode. You can make a DIY developer tools yourself, basically rebuilding everything that Xcode ships, but it's going to cause you a lot of headaches. Cargo dependencies that rely on a C compiler or Apple's system headers will cause problems too.
Homebrew will also complain about lack of Xcode tools. A Mac without Xcode is not suitable for development.
If your macOS is broken and can't have Xcode, reinstall. If you're forbidden from using Xcode by some Enterprise Corporate Policy, complain about it, because it's incredibly counter-productive (or change jobs, if you can).
If all else fails, maybe get a virtual machine and use Linux version of Rust? It will still need a linker, etc. but Linux distros come with