How to do this better being generic?

help
1

Hiho,

for my newest project I am using the num crate in crates.io.

I am in need of several functions that should be generic over Float number types.
For example:

fn tanh_activation_fn<F: Float>(x: F) -> F { x.tanh() }

fn sigmoid_activation_fn<F: Float>(x: F) -> F {
    let half: F = F::from(0.5).unwrap();
    half * (F::one() + (half * x).tanh())
}

This works very well out of the box but has ugliness since I haven't found a way to write the so-called sigmoid function in a sane way. The let statement combined with an ugly never-fail .unwrap() just feels utterly wrong here.

Am I using this crate in a wrong way? Am I doing something wrong in general?

Is there a better way to write this in a sane syntax?

Another thing that bothered me:

fn softplus_activation_fn<F: Float + FloatConst>(x: F) -> F {
    (F::one() + F::E().powf(x)).ln()
}

Here I need to specify F to be generic over Float + FloatConst which isn't very pleasing to me.
I do not want the additional constraint for FloatConst since to my understanding Float should be sufficient but it hasn't got the required E() static method that I need.
Or maybe I am using it incorrectly?

What I really want to write is:

fn softplus_activation_fn<F: Float>(x: F) -> F {
    (1.0 + F::E().powf(x)).ln()
}

Please help! :S

2

I don't know a great solution for arbitrary generic constants. In theory, we don't know the full extent of what types Float may be, so it's possible that some implementation may fail to convert some constants. That's why NumCast::from returns an Option. As ugly as it may be, this should be optimized away.

Perhaps in a future num breaking change, we could add a requirement for Float: From<f32>, so basic constants like this can easily be obtained. It's too bad NumCast::from and From::from collide, but maybe we should break and rename NumCast's method too.

We've tried very hard to keep num stable, since it's fairly popular, even though the API is not ideal. Its various warts are largely why these APIs were booted from std in the first place. There are lots of things that could and probably should make breaking changes in a future num, preferably in one big batch, but I haven't found the time to even start on that.

This too was avoiding a breaking change. FloatConst was just recently added, and I didn't want to break any hypothetical existing Float implementations by adding a new constraint, so for now they're independent. New Float types sound weird, but it could easily happen if someone had a newtype, e.g. struct Velocity(f64); impl Float for Velocity {/*...*/}

In your particular example, F::E().powf(x) is the same as x.exp(). There's also a function for ln(1+n), so this could become x.exp().ln_1p().

1 Like
3

Thank you very much for your detailed answer.

All in all I like the num-crate very much and even though it has some warts as you have described above it is a very nice tool to work generically.
Still a breaking change as external crate isn't soo dramatically in my opinion.

Yes, there may be many dependencies, but these are dependencies to older (current) versions of the library, not dependencies of a future version with breaking changes.
So effectively there will be no breaks, however, it may prevent dependent crates to immediately upgrade to a newer version.

Also thank you very much for your suggestion with the exp and ln_1p methods - very neat!
At least it solved my problem with FloatConts. :smiley:

As a suggestion for some API changes:

  • Maybe rename NumCast::from to NumCast::try_from since it returns an Optional which would be a name with more semantics.
  • The idea with the From<f64> and From<f32> sounds amazing to me - would make much of my code much easier to read. I have to use a lot of F::one() and F::zero() in my code ...
4

It gets trickier when other crates use num in their own public interfaces. For instance, if you put your functions above into a crate, dependent on Float, then anyone using your crate has to agree what version of Float you're talking about.

1 Like
5

Can't you usually just use one() and zero()?

6

This happens when I do something like the following:

fn identity_fn_dx<F: Float>(x: F) -> F { 1.0 }

fn sigmoid_fn<F: Float>(x: F) -> F {
    let half: F = F::from(0.5).unwrap();
    half * (1.0 + (half * x).tanh())
}

When compiling with rustc:

activation_fn.rs:13:46: 13:49 error: mismatched types [E0308]
activation_fn.rs:13 pub fn identity_fn_dx<F: Float>(x: F) -> F { 1.0 }
                                                                             ^~~
activation_fn.rs:13:46: 13:49 help: run `rustc --explain E0308` to see a detailed explanation
activation_fn.rs:13:46: 13:49 note: expected type `F`
activation_fn.rs:13:46: 13:49 note:    found type `_`
activation_fn.rs:39:9: 39:34 error: the trait bound `_: std::ops::Add<F>` is not satisfied [E0277]
activation_fn.rs:39     half * (1.0 + (half * x).tanh())
                                           ^~~~~~~~~~~~~~~~~~~~~~~~~
activation_fn.rs:39:9: 39:34 help: run `rustc --explain E0277` to see a detailed explanation
activation_fn.rs:39:9: 39:34 help: the following implementations were found:
activation_fn.rs:39:9: 39:34 help:   <f64 as std::ops::Add<ndarray::ArrayBase<S, D>>>
activation_fn.rs:39:9: 39:34 help:   <f64 as std::ops::Add<&'a ndarray::ArrayBase<S, D>>>
activation_fn.rs:39:9: 39:34 help:   <f64 as std::ops::Add<&'a num::Complex<f64>>>
activation_fn.rs:39:9: 39:34 help:   <&'a f64 as std::ops::Add<num::Complex<f64>>>