I attempt to generate C header for the Rust code here using the command
cbindgen --config cbindgen.toml --crate tiktoken --lang c --output tiktoken.h
where the cbindgen.toml is just an empty file.
But the command throws several errors, one of them is
ERROR: Cannot use fn tiktoken::byte_pair_encode (Unsupported type: Type::Slice { bracket_token: Bracket, elem: Type::Path { qself: None, path: Path { leading_colon: None, segments: [PathSegment { ident: Ident(u8), arguments: PathArguments::None }] } } }).
It seems to me it's because no Vec type at the C side. So I create a custom Vector struct and modify some corresponded code. Though Rust compilation works, but the error still remains. What might go wrong? And how to fix it? Thanks.
#[repr(C)]
pub struct Vector {
ptr: *mut u32,
len: usize,
capacity: usize,
}
impl Vector {
pub fn to_vec(&self) -> Vec<u32> {
unsafe { Vec::<u32>::from_raw_parts(self.ptr, self.len, self.capacity) }
}
}