Callable from Python inside Rust


#1

I had a look at rust-cpython and PyO3 but did not understand well.

I have the following problem. Let’s say I have a function defined inside my python code:

python.py:

def f(x): return np.mean(x)

And let’s say that I have some Rust function that is supposed to call the function defined inside Python:

main.rs:

pub fn g(f, vector: Vec<f64>) -> f64 {
   f(vector);
}

How can I do that?

Thank you!


#2

Maybe this helps:

https://docs.python.org/3/library/ctypes.html#callback-functions

I tested it successfully but without rust-cython.

My way is creating a “cdylib” Target from Rust

Rust:

#[no_mangle]
pub extern fn noise_texture_ds_middlefunc() -> fn(x1: f32, x2: f32, x3: f32, x4: f32) -> f32 {
	middle_quad
}

#[no_mangle]
pub extern fn noise_texture_ds(width: i32, height: i32, maxred: f32, red: f32, startseed: f32,
							   middlefunc: fn(x1: f32, x2: f32, x3: f32, x4: f32) -> f32)
 -> *mut u8 {
...
}

pythonlib.py:

lib = ctypes.cdll.LoadLibrary("py_render_lib.dll")

lib.noise_texture_ds.args = (c_int, c_int, c_float, c_float, c_float,)
lib.noise_texture_ds.restype = POINTER(c_ubyte)

lib.noise_texture_ds_middlefunc.restype = POINTER(CFUNCTYPE(c_float,c_float,c_float,c_float))

@CFUNCTYPE(c_float,c_float,c_float,c_float,c_float)
def pymiddle(x1, x2, x3, x4):
	return (x1 + x2 + x3 + x4) / 2.5

def run_test():
	print("start noise test")
	w = 513
	#middlefunc = lib.noise_texture_ds_middlefunc()
	middlefunc = pymiddle
	result = lib.noise_texture_ds(c_int(w), c_int(w), c_float(1.0), c_float(0.5), c_float(1.0), middlefunc)
        .....

As you can see that works without any extra crate too.


Current state of GUI development in Rust?
#3

Thanks @bug00r, but I’d not this calling rust from python!
I’m looking for the opposite, the main application is rust, and from it I need to call python function that is existing in some python library.


#4

Oh ok i misunderstand it. Sorry but for this case i have no working example.

Edit: Maybe this could help? http://cffi.readthedocs.io/en/latest/overview.html#embedding


#5

It has been a while since I last used rust_cpython, but here’s some tips.

  • Searching the rust_cpython docs for call I find a couple of results. The only one suitable for calling arbitrary Python objects is this one.
    • This trait is only implemented on PyObject. So your function should take f: &PyObject. (It should probably also take a py: Python<'_>, which represents the Global Interpreter Lock.)
    • The args argument must be a type that implements ToPyObject with an output type of PyTuple. Rust tuples are one example, and if I remember correctly, for an empty argument list there is cpython::NoArgs.
    • (vector,) will probably work
  • Calling f will give you a PyObject, but you want an f64. The README advertises the use of extract to convert Python objects to rust types.
    • output.extract(py) where output is the result of calling f

I’m going to leave it to you to put this all together; but feel free to ask if you need more help.


#6

Thanks @ExpHP

I wrote the Rust code:

myapp/src/main.rs

extern crate cpython;

use cpython::Python;

fn main() {
    let gil = Python::acquire_gil();
    println!("Hello from Rust!");
    let py = gil.python();
    let module = cpython::PyModule::import(py, "fibo").unwrap();

    module.call(py, "fib", (1,), None).unwrap();
}

And saved the Python module as myapp/pyth/fibo.py

filesystem layout

But I get the error:

thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: PyErr { ptype: <class 'ModuleNotFoundError'>, pvalue: Some(ModuleNotFoundError("No module named 'fibo'",)), ptraceback: None }', libcore/result.rs:945:5

The part of the code I’m expecting to know about the directory pyth is: let module = cpython::PyModule::import(py, "fibo").unwrap();


#7

According to some answer on StackOverflow, importing file is a bit more difficult.

Please try:

let gil = Python::acquire_gil();
let py = gil.python();

let locals = PyDict::new(py);
locals.set_item(py, "imp", py.import("imp").unwrap());

let mut path = std::env::current_dir().unwrap();
path.push("pyth");
path.push("fibo.py");

let eval = format!("imp.load_source('{}', r'{}')", "fibo", path.to_str().unwrap());
py.eval(&eval, None, Some(&locals)).unwrap();

let fibo = py.import("fibo").unwrap();
let result: isize = fibo.call(py, "fib", (1,), None).unwrap().extract(py).unwrap();
println!("fibo.fib = {}", result);

Of course, it’d be much better to wrap the code in some fn returning Result, and use the ? operator instead of unwraps.

edit: I don’t know the exact semantics of Python (are simple py files are different from modules?), maybe you’ll need to eval this instead of imp.load_source:

import sys
sys.path.append(r'<insert folder path>')

#8

Here’s how I’d do it.

Option 1: Modify sys.path (for resolving the module path at runtime)

// 'import sys'
let sys = py.import("sys")?;

// 'sys.path = ["./pyth"] + sys.path'
let mut sys_path: Vec<String> = sys.get(py, "path")?.extract(py)?;
sys_path.insert(0, "./pyth".into());
sys.add(py, "path", sys_path);

// // alternatively, all of the above can be written as:
// py.run(r#"
// import sys
// sys.path = ["./pyth"] + sys.path
// "#, None, None)?;

// 'import fibo'; (this should work now)
let m = py.import("fibo")?;

// 'fibo.fib(2)'
let out: i32 = m.call(py, "fib", (2,), None)?.extract(py)?;
println!("successfully found fibo.py at runtime.  Output: {:?}", out);

Because this resolves at runtime, the module will need to be somewhere you can easily find it. The way I wrote it looks in the current directory (as does @withkittens’ answer), meaning your program won’t work if run from somewhere other than the project root.

Option 2: Make your own module (this resolves at compiletime)

(IMPORTANT: This solution is incomplete. See this post!)

// the contents of fibo.py, embedded into the program as string data.
const FIBO_PY: &'static str = include_str!("../pyth/fibo.py");

// make a module with nothing in it
let m = PyModule::new(py, "fibo_2")?;

// run/eval the contents as if they were placed at the top of the module.
// Basically, we use `fibo_2.__dict__` as both our globals and locals, so that
// anything defined by the script appears in there.
let m_locals = m.get(py, "__dict__")?.extract(py)?;
py.run(FIBO_PY, Some(&m_locals), None);

let out: i32 = m.call(py, "fib", (2,), None)?.extract(py)?;
println!("successfully found fibo.py at compiletime.  Output: {:?}", out);

The trick here is that run modifies the locals dict you give it. x.__dict__ is a special object in python used when looking up attributes like x.thing, so adding entries to it makes them appear as members on the object.


#9

Thanks @ExpHP but I got error about every usage of ? and extract(py) in your code :frowning:


#10

I wrote my examples as if they were in a function that returns PyResult<()>. It is the idiomatic way to propagate errors in Rust; you can read about this syntax here: https://doc.rust-lang.org/book/second-edition/ch09-00-error-handling.html

(if you’re not ready to learn that yet, though, for now you can replace ? with .unwrap())

I don’t know what error you’re getting on extract; you’ll have to share it.


#11

Thanks @withkittens
I got an error at: locals.set_item(py, "imp", py.import("imp").unwrap()); telling:

= note: #[warn(unused_must_use)] on by default
= note: this Result may be an Err variant, which should be handled


#12

@ExpHP

I re-wrote the code to be:

 extern crate cpython;

use cpython::{Python, PyResult, PyModule};

fn main() {
    let gil = Python::acquire_gil();
    hello(gil.python()).unwrap();
}

fn hello(py: Python) -> PyResult<()> {
    // the contents of fibo.py, embedded into the program as string data.
    const FIBO_PY: &'static str = include_str!("../pyth/fibo.py");

    // make a module with nothing in it
    let m = PyModule::new(py, "fibo_2")?;

    // run/eval the contents as if they were placed at the top of the module.
    // Basically, we use `fibo_2.__dict__` as both our globals and locals, so that
    // anything defined by the script appears in there.
    let m_locals = m.get(py, "__dict__")?.extract(py)?;
    py.run(FIBO_PY, Some(&m_locals), None);

    let out: i32 = m.call(py, "fib", (2,), None)?.extract(py)?;
    println!("successfully found fibo.py at compiletime.  Output: {:?}", out);
    Ok(())
}

and got this error:

warning: unused std::result::Result which must be used
–> src/main.rs:21:5
|
21 | py.run(FIBO_PY, Some(&m_locals), None);
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
= note: #[warn(unused_must_use)] on by default
= note: this Result may be an Err variant, which should be handled

Finished dev [unoptimized + debuginfo] target(s) in 0.70s
 Running `target/debug/app`

thread ‘main’ panicked at ‘called Result::unwrap() on an Err value: PyErr { ptype: <class ‘NameError’>, pvalue: Some(“name ‘print’ is not defined”), ptraceback: Some(<traceback object at 0x109425d08>) }’, libcore/result.rs:945:5
stack backtrace:


#13

This warning tells you that set_item returns Result – special type that represents a value _or_ an error. Ideally, you’ll want to learn at least the chapter about error handling (as @ExpHP mentioned), or a more basic stuff about Rust itself in earlier chapters (the book is great!)

For now, add .unwrap() in the end of the line:

locals.set_item(py, "imp", py.import("imp").unwrap()).unwrap();

This will return you a value if a Result contains one, or will panic (print an error and exit) if there’s an error.


#14

Oof. I guess my test case was too simple. Apparently my method doesn’t load builtins like print. The best solution should be a mixture of mine and @withkittens’ responses, using include_str! to find the source, and kittens’ method to run it. However I am not at my PC right now to try things out.

(Actually, crud, maybe not. I didn’t realize load_source took a file path. Aaaargh…)

And yes, I forgot to put a ? after the run call


#15

I do not know, I’m new to all this staff, and still getting an rro:

thread ‘main’ panicked at ‘called Result::unwrap() on an Err value: PyErr { ptype: <class ‘NameError’>, pvalue: Some(“name ‘print’ is not defined”), ptraceback: Some(<traceback object at 0x10df60d08>) }’, libcore/result.rs:945:5
stack backtrace:


#16

Right, I know why that’s happening but don’t know how to fix it yet. :stuck_out_tongue: I’ll try to help when I get home.


#17

Okay, the fix turned out to be easier than I thought; I just added a __builtins__ member to the module.

extern crate cpython;

use cpython::{Python, PyResult, PyModule};

const FIBO_PY: &'static str = include_str!("../pyth/fibo.py");

fn main() {
    let gil = Python::acquire_gil();
    let py = gil.python();

    example(py).unwrap();
}

fn example(py: Python<'_>) -> PyResult<()> {
    let m = module_from_str(py, "fibo", FIBO_PY)?;

    let out: i32 = m.call(py, "fib", (2,), None)?.extract(py)?;
    println!("successfully found fibo.py at compiletime.  Output: {:?}", out);

    Ok(())
}

/// Import a module from the given file contents.
///
/// This is a wrapper around `PyModule::new` and `Python::run` which simulates
/// the behavior of the builtin function `exec`. `name` will be used as the
/// module's `__name__`, but is not otherwise important (it does not need
/// to match the file's name).
///
/// Note this compiles and executes the module code each time it is called, as it
/// bypasses the regular import mechanism. No entry is added to the cache in `sys.modules`.
fn module_from_str(py: Python<'_>, name: &str, source: &str) -> PyResult<PyModule> {
    let m = PyModule::new(py, name)?;
    m.add(py, "__builtins__", py.import("builtins")?)?;

    let m_locals = m.get(py, "__dict__")?.extract(py)?;
    py.run(source, Some(&m_locals), None)?;
    Ok(m)
}

#18

Thanks alot, deeply appreciated.