Bad code generation in 1.4?

help
1

I had some FFI code that was working fine with 1.3, but when I upgraded to 1.4 i'm suddenly crashing in the external library. I've reduced it down to be reproducible with he following code:

use std::ptr;
use libc::{c_int, c_uchar, c_ulong, c_void};

extern crate libc;

#[link(name="crypto")]
extern {
    fn BN_set_word(bn : *mut c_void, w : c_ulong) -> c_int;
    
    fn BN_CTX_new() -> *mut c_void;
    fn BN_CTX_free(ctx : *mut c_void);
    
    fn BN_CTX_start(ctx : *mut c_void);
    fn BN_CTX_get(ctx : *mut c_void) -> *mut c_void;
    fn BN_CTX_end(ctx : *mut c_void);    

    fn RSA_new() -> *mut c_void;
    fn RSA_free(rsa : *mut c_void);
    
    fn RSA_generate_key_ex(rsa : *mut c_void, bits : c_int, bn : *mut c_void, bn_gen : *mut c_void) -> c_int;
    fn i2d_RSA_PUBKEY(rsa : *mut c_void, pp : &*mut c_uchar) -> c_int;
}

fn main() {
    unsafe {
        // Create a big number of 3 
        let bn_ctx = BN_CTX_new();
        
        BN_CTX_start(bn_ctx);
        
        let bn = BN_CTX_get(bn_ctx);

        if BN_set_word(bn, 3) != 1
        {
            panic!("Could not set the big number value to 3");
        }            
        
        // Generate the RSA key 
        let ctx = RSA_new();
        
        if RSA_generate_key_ex(ctx, 1024, bn,ptr::null_mut()) != 1 
        {
            panic!("Could not generate RSA token.");    
        }
        
        let bytes = i2d_RSA_PUBKEY(ctx, &ptr::null_mut());
        
        println!("Number of bytes needed for public key: {}", bytes);
        
        RSA_free(ctx);
        BN_CTX_end(bn_ctx);
        BN_CTX_free(bn_ctx);
    }
}

I'm getting the failure when running with Linux Mint 17.1/17.2 x64 (xfce). To compile the above code you also need to install libssl-dev. When compiled with 1.3 it runs fine, when compile with 1.4 it segmentation faults when it's run.

I've traced the problem down to:

let bytes = i2d_RSA_PUBKEY(ctx, &ptr::null_mut());

If I change that code to:

let ptr = ptr::null_mut();
let bytes = i2d_RSA_PUBKEY(ctx, &ptr);

The issue clears up.

2

Since i2d_RSA_PUBKEY writes through the passed reference, it must be declared &mut; otherwise it appears that the first version allocates the null pointer in .rodata. You're leaking memory by the way (according the the manpage, ptr now contains a heap-allocated DER structure).

If you just want the length without actually writing the structure anywhere, declare it pp: *mut *mut c_uchar and call it as i2d_RSA_PUBKEY(ctx, ptr::null_mut()); that is, pass NULL rather than pointer-to-NULL. Since a reference (whether & or &mut) cannot be NULL in Rust, you have to switch to raw pointers to call i2d_RSA_PUBKEY in that way.

1 Like
3

Yep, the const reference in the prototype seems clearly wrong:

int		i2d_RSA_PUBKEY(RSA *a,unsigned char **pp);
4

Okay I'm the bad guy, thanks. I think my problem is that I keep associating & as pointer when I should really be thinking of it as reference.

And I was really happy to leave pointer to pointers behind when I started dealing with JAVA, and I mostly don't have to deal with it in Rust as long as I can stay away these existing libraries.

5

Well, even openssl-sys seems to have gotten this wrong somehow.

6

*const T doesn't have the immutability rule that &T does, so it's not entirely the same.

1 Like