Hi. I have a project idea that I need help validating and fleshing out. I want to know if it's been done, if it's possible, and if it's well scoped for me. The idea I have right now is that I want to make it a lot easier to build and publish (to PyPi) platform-specific Python wheels that include Rust extensions.
Right now it doesn't seem like there is an easy way to publish a Python package with Rust extensions that serve all the major platforms. Is this accurate?
If that's true I'd like to help fix it, but I need help understanding the challenges.
Platform wheels are detailed here, and here and the manylinux wheel is detailed here
The final result in Pypi would look something like this
Right now my plan is to create a cookiecutter project template that would build and publish Python packages after building the necessary Rust binaries in Travis/Appveyor using the template from Trust. I think Trust should handle the complicated bits of the actual platform compilation... All I would need to do is package the Rust binaries into wheels with the Python code. I'll probably rip-off the Rust-in-Python hello world example from here
That project, rust-cpython, looks like it's allowing you to call out to a Python interpreter from within rust. I.e. Rust -> Python -> Rust
What I am trying to facilitate is calling Rust from Python. I.e. Python -> Rust -> Python
Also I am trying to improve the build/package automation. I'm not so concerned with the language bindings themselves, although I'd rather avoid depending on libpython since most users don't have that installed.
Working with the Python community is probably a good idea though.
So, this is a thing that has been done a few times already. For example, I created this project (no longer under active development) which creates a "Rust extension" for Python via CFFI. I then built wheels for it, as you can see on the PyPI page, though not manylinux1 wheels primarily out of laziness. I also know that Armin Ronacher has done this in the past, and I'm sure many others have too.
However, a cookiecutter that makes doing this easier would probably be helpful. To get this to work requires a decent understanding of the Python buildchain as well as a bunch of CI integration to build the various wheels, and centralizing that would likely be really helpful in making this easier, so if that was something you were interested in it sounds like it'd be a really good idea.
Another thing that's worth considering, if you wanted to make this really expert level, would be to either extend or wrap cffi in a way that allows a more graceful integration with Rust. Because Rust lacks C headers, you have to essentially fake them ahead of time and write them all into CFFI, which is a bit annoying. It'd be nicer to be able to auto-generate faked C header files for the purpose of building CFFI, such that you don't need to constantly keep the two interfaces in step where it doesn't matter. That's definitely an expert-level concern though.
However, a cookiecutter that makes doing this easier would probably be helpful. To get this to work requires a decent understanding of the Python buildchain as well as a bunch of CI integration to build the various wheels, and centralizing that would likely be really helpful in making this easier, so if that was something you were interested in it sounds like it'd be a really good idea.
That is basically my goal. I think that people would be a lot more likely to try an extension if they don't have to worry about the various buildchains and CI integrations.
I'm also glad you shared your example. Until now I couldn't find one.
To be clear I'd love to see a tool/guide/cookiecutter that shows me how to writes a rust library and uses ci/cross/Trust to build wheels for as many platforms as possible.
Yes, this was also my first impression, but as said by @Eh2406 it works in both ways.
May be documentation need some improvment.
What more great about this crate, that it allow you to create class in rust code,
and then use it in python, like python class.
Also with one call to py_fn! macro you can get function that possible to call from python.
Not get that part, I build so/dll for linux/windows with cpython to use in my python application,
and this shared library not depend on libpython.
Not get that part, I build so/dll for linux/windows with cpython to use in my python application,
and this shared library not depend on libpython.
My understanding is that it is more difficult when linking libpython. From this post:
What worked well:
Marrying Rust and Python with CFFI. There are some alternatives to this which link against libpython, but it makes for a significantly more complex wheels build.
Hi, author of those last three wheels here. It's kind of a pain, because you can easily set up CI to test on all the platforms your heart desires using trust (cross), but since you need your Linux lib to be manylinux1-compatible, you then have an extra step involving the official manylinux1 Docker image, and running rustup in there (you can't use cross for this step because it's like Docker inception or something – idk if that could be simplified with a custom cross target ).
Currently, the process has three steps for me:
Write Rust lib, push to CI (Travis and Appveyor). If tests pass, push a tag. This publishes zipped binaries to GitHub releases for all the platforms I target. There are lots of subtleties here involving importlibs for Windows, and manylinux1, as I noted.
My Python package CI setup pulls the latest binaries from Github during a preinstall step, integrates them into the wheel-build process, and tests. If tests pass, I push a tag, which causes the wheels to be pushed to Github releases.
I manually upload signed wheels to PyPI using Twine and a simple script, but that's purely down to personal preference; it could easily be automated.
There's also an issue around using MinGW and Python 3.6 wheels for Windows, but even typing those words is causing me to tear up again, so I'll stop.
Anyway, that's definitely not the only way to do it; I couldn't get to grips with cffi when I built all this stuff last year, and it wasn't clear that it was superior to Cython anyway (or ctypes, which is even simpler to set up), and I'll probably revisit it when I have time, but it's robust once you have everything set up – I'm just in the process of migrating all three repos to use trust (from its unnamed predecessor), and the new setup is a lot clearer, and reduces the amount of arcane bash scripts considerably.
I also have an example output project here mckaymatt/trust_pypi_example. The Linux, OSX and Windows releases basically work. I actually kind of screwed up by naming the project trust_pypi_exmple, since in the end I didn't end up using the Trust compiler tools; although I did get a good bit of code from them.
It's pretty easy to get going on locally on OSX and Linux by following the quickstart.
There's still a lot to be done. I would like to have examples of other bindings like PyO3 and rust-cpython
I was messing around with this for a while before I remembered I was in Docker Inception. That's ultimatum why I stopped using Trust. It turned out to just be easier to use Rustup inside the manylinux1 container.
Having the Crate separate from the Python package is probably a better idea in a lot of ways, but I found that PyO3/setuptools-rust made it pretty easy to build/test/bundle them together.
).