Converting Parquet decimal to rust_decimal::Decimal

help
1

Anyone have an example of doing this?

I tried this

fn parquet_decimal_to_rust_decimal(parquet_decimal: &ParquetDecimal) -> Decimal {
    let bytes = parquet_decimal.data(); 
    let scale = parquet_decimal.scale(); 
    let unscaled_int = i128::from_be_bytes(bytes.try_into().unwrap()); 

    Decimal::from_i128_with_scale(unscaled_int, scale as u32)
}

which blows up on the bytes.try_into call.

Looking into it, but, I am thinking this is a basic thing that must be out there somewhere (despite searching not finding it).

2

Assuming ParquetDecimal is actually Decimal from the parquet crate, the .data() returns a slice that can be of any length. Whereas i128::from_be_bytes requires a [u8; 16]. There's no reason to assume the bytes.try_into() call will succeed in most cases. You will need to check and pad or trim the slice to 16 elements before attempting the conversion.

1 Like
3

And so it does:

pub const fn from_le_bytes(bytes: [u8; 16]) -> i128

Seems to me that should be a compile-time error. :slight_smile:

4

It's more annoying than that, even, since you're working with signed numbers.

fn from_be_slice(be: &[u8]) -> Option<i128> {
    // Doesn't attempt conversion of slices larger than an `i128`
    16_usize.checked_sub(be.len()).map(|slop| {
        // Check high bit to see if we're negative or non-negative
        let mut bytes = match be {
            [byte, ..] if *byte >= 0x80 => [0xff; 16],
            _ => [0; 16],
        };
        // overwrite the lower `be.len()` bytes and convert
        bytes[slop..].copy_from_slice(be);
        i128::from_be_bytes(bytes)
    })
}

N.b. I'm not familiar with the crates and the snippet above assumes plain ol' 2's complement integer data.

It's the try_from ... unwrap that panics at runtime, not from_?e_bytes. Slices have dynamic length.

1 Like
5

The fn signature is interesting:

        pub const fn from_be_bytes(bytes: [u8; mem::size_of::<Self>()]) -> Self {

So, one must then know the size of the type for the impl you're using.

Well, I guess this is how systems-level programming works.

6

For anyone interested, this is how I did the padding.

    let bytes: &[u8] = parquet_decimal.data();
    // We need to convert the slice to an array of 16 bytes - it may need to be padded or truncated
    let array: [u8; 16] = bytes
        .iter()
        .cloned()
        .chain(std::iter::repeat(0).take(16))
        .take(16)
        .collect::<Vec<u8>>()
        .try_into()
        .unwrap();