Floating point traits roundtable

Hi!

I'd like to meet and talk about the status of floating point traits in Rust. This is just one topic out of many that are related to numbers, and I hope we can have more of these discussions soon!

This can touch upon all kinds of mathematics, game dev, data sciences and machine learning and other interests, hopefully a lot of you want to participate in some way.

I would like to invite of course @cuviper and @hauleth that have taken up the mantles as num crate maintainers. (Maybe you are more, I don't know!)

I am going to toss in my idea first here. Please tear into it or disregard it. It's fine if we discuss meta-issues before we discuss concrete things. Is it, for example, a fool's errand to try to make different kinds of projects use the same base traits? We do have something to win if we make a better trait, don't we?


Goal

  1. I want to write code once, that works for both f32 and f64 (Float)
  2. I want to write code once, that works for all of f32, f64, Complex<f32>, Complex<f64> (ComplexFloat)
  3. I want to be able to special case for each type when it is needed.
  4. I want to express myself using regular Rust syntax like +, +=, and so on as much as possible.

Design

With a narrow type requirement, we can provide a lot of features.

Examples:

a. Any type that defines addition => we can provide only very rudimentary features
b. Any type that is a primitive floating point type => we can provide a lot of the features they have in common

(1) A float trait should include all the usual floating point operations, and more. I want to be greedy here so that when we have a narrow type requirement (Float), there should be no arbitrary roadblocks. Primitive floating point types implement Display and are 'static, so those should be properties of the trait. Primitive floating point types are Send + Sync, so those should be properties of the trait, so I can send work to a thread without fuss, and so on. ⁵

(2) ComplexFloat should work just like Float: write code once that works for all the four types. It for example allows expressing a complex inner product using conjugation, and for the real numbers, conjugation is simply the identity function.

(3) will be solved by specialization, but it is also solved by the traits by themselves, with special case methods like ComplexFloat::is_complex() -> bool. When you need it, it should be simple to conditionally do an operation.

(4) is addressed by making sure operator traits are included in traits (1) and (2) as far as possible. We can't directly expose the as operator; it is a core Rust feature, but it was not properly available in generic code. Primitive numeric types can be cast with as, so it should be a feature you can use through Float too. I have a draft trait AsPrim for that.

Draft Traits

Prior Art

⁵ We can't even list all the add-on capabilities that f32, f64 have, those that come with each extra crate you add to your project, for example serializable, randomizable, et cetera. Cargo support for opportunistic conditional compilation would do well here, so that a numerics crate can include serialization support only when it's already being used in the project.

5 Likes

It's just the two of us. Maybe you'd like to join? :slight_smile:

So far I've pressed to stay very conservative since num is so widely used. But I think all the OpAssign traits are very compelling to press num forward with breaking changes. Maybe we should start a devel branch to prepare a mega break?

Anyway, I'll try to consider this proposal in depth later tonight...

I've seen also some interest in traits that encompass both floating-point and fixed-point types. I believe these could include most of the functionality of the float traits, except for a few things like infinity.

May I suggest to treat (and advertise) generic numerics as unstable features of libraries for a while?

Seems to me that decisions like "should we include just floats or floats and " are hard, and should not be locked in early, so that we can explore.

As a developer of numerical algorithms in Rust, I want to be locked into bad designs even less than to update my code.

I would like this to start with a solid PrimFloatingPoint trait just for Rust primitive types (akin to PrimInt).

This leaves the door open to a more general FloatingPoint trait in the future that handles user-defined fixed-point types.

I am afraid that if we try to tackle user-defined fixed-point types (or Integer types like BigNums) progress will be really slow, so I'd rather start small with a PrimFloatingPoint/PrimInt RFC and then keep on going from there.

We should keep in mind fixed-point types / bignums when discussing these traits, but being able to abstract over Rust primitive types is IMO already a win.

2 Likes

Are you proposing these things to start life in num? Or go straight to a Rust RFC?

I'm on board with this, and I think treating it similar to PrimInt is a good idea. PrimFloatingPoint is oddly mixed abbreviation though -- either PrimitiveFloatingPoint or I prefer PrimFloat to mimic PrimInt.

It's not great to me that the current Float requires Copy -- I think that belongs in PrimFloat, and leave the possibility that a more abstract Float may not be copyable.

I definitely don't like that your from_real_imag may ignore the imaginary part. If your generic code wants that constructor, it seems not really Float friendly at all -- just use Complex. If you really want this method, I'd make a hard assert rather than ignoring it.
assert!(Self::is_complex() || imag.is_zero())

More generally speaking, I worry that it's unsound to pretend that all of these functions have the same meaning for both Float and Complex types. Most of the complex operations are only returning the principal branch, and I suspect that rigorous use probably needs to consider whether other branches are relevant. But I'm no formal mathematician, so I'd welcome reassurance that this may be ok...

This seems good to me. I only hesitate that the cast_from name doesn't fit in well. T::from_as(x)? T::with_as(x)? Not great either -- I don't know.

In num or another crate for sure, long before RFCs.

Copy is one of the several things that give the impression that the current num::Float is really a trait for just f32, f64 and not for general fractions or generalizations of float. This needs to change. Flexibility is needed for a more general trait and the PrimFloat kind of trait discussed in this topic can provide a lot of features / ergonomics in return for being very narrow.

I wouldn't say that it promises that they have the same meaning.

That's not really the goal, the goal is being able to write code for those four types all at once. I think that will require care for sure, and the trig methods are not expected to be very useful. from_real_imag is useful precisely when you know that you're in a complex-only conditional branch (is_complex method) or you know that the imag ignore is what you want for the real number case.

For example: Cholesky factorization. The operation is on a real or complex-valued matrix, but the diagonal is always real, and you compute square roots of the diagonal. It's a complex-valued problem but you don't need any complex-valued special functions for that, just the conversion support. (Self::from_real(z.real().sqrt()) for example).

Those sound better, improving names sounds good.

This is the user forum, and it's a proposal that we create something better in the library space. I think we need better ergonomics for dealing with f32, f64 specifically. A trait for that can coexist with other kinds of generic tools for numerics.

So it seems that there is at least some consensus that a good first step is to start with a PrimFloat akin to PrimInt.

The only thing I am not sure about is handling Complex<fXX> as PrimFloat. I think there is an important difference between the mathematical concept of a Complex number (and Real, Natural,... numbers), and the "machine" concept of floating point arithmetic.

I can see why it is attractive to abstract over both Complex and Real numbers in e.g. a linear algebra library when implementing matrices. Still I think that PrimInt and PrimFloat job is to abstract over the primitive types. This is already hard enough.

For this reason I would like PrimFloat to start with doing a good job for f32 and f64 first, and once that's done, then consider what would need to be done for Complex to implement PrimFloat. It might well be that it cannot be easily done and that we would need a different Trait for that.

I would be fine with pushing PrimInt and PrimFloat for an RFC as Traits to abstract only about primitive types. These are already useful building blocks that we want anyways, and by making them smaller in scope I think we can make good progress on this front soon.

About where to do this work, the num crate might be a good place for PrimFloat as long as backwards compatibility can be maintained (which can be if PrimFloat is a new trait).

Of course. A trait PrimFloat (a.k.a Float) should be different from a trait like ComplexFloat. The latter caters to four very important types that for example have in common being supported in BLAS.

@bluss I agree that a differnt trait (e.g. ComplexFloat) would be both useful and the best way to proceed.

I decided to throw a 5min strawman so that we have something more concrete to discuss:

This is very rough:

  • all the operators are missing
  • all conversions (e.g. float to int and viceversa) are missing
  • PrimFloat implements all of cmath, c++ math, and c++17 special mathematical functions.
  • all algorithms take &self, but maybe it makes sense to consume the value (take self), whatever is decided should be consistent.
  • some algorithms take a small integer/unsigned integer, should use i32/u32 or isize/usize here? I don't know
  • PrimInt implements all from <cmath>, c++math, and the C++ bitwise manipulation algorithms proposal
  • PrimInt has two associated types although I cannot come up with a usecase where these are required.
  • PrimSignedInt and PrimUnsignedInt are empty (all algorithms I can think of can be lifted to PrimInt, maybe this is not required)
  • there is a PrimNum trait with PrimNumBounds, PrimNumInfo, ... don't know how useful this is and/or whether it should be just PrimInt and PrimFloat, are there usecases for abstracting over primitive ints and floats (e.g. linear algebra) where this would be useful?
  • maybe the special mathematical functions (at the end of PrimFloat) should be factored into their own trait

Something that worries me is that some users will just need minimal abstraction over ints and floats, while others might want e.g. special mathematical functions. The modularization should be done in a way that allows users to pay only for what they need.

EDIT: I don't know if it makes sense to make PrimInt and PrimFloat completely independent or if it makes sense to have them be part of the same trait hierarchy. Some operations like zero and is_signed make sense for both, but since these traits are for abstracting about machine level constructs, it also makes sense to have them be completely split and leave the job of unifying them into a "number" hierarchy to some higher level abstraction.

I lean more towards having the standard library provide low level constructs that are "non-controversial", a "number" hierarchy with/without integer and floating point, complex numbers, fixed-point, ... support is going to be controversial, but two completely independent traits for machine level integers and floats are not.

Did you see that I had these two traits, as seprate entities, in the first post in the thread? A trait for f32, f64 (primitive float) called Float and a trait called ComplexFloat? I'm not saying my strawman is better than yours, just that it sounds like you missed that it was there.

While this is cool, I would like to first focus on not on bringing new numerical features into Rust, but to make a trait that makes it easy to use the features that are already present, of for example primitive floats, in generic code.

A trait for f32, f64 (primitive float) called Float and a trait called ComplexFloat? [...] missed that it was there.

I did missed it at first, but then I realized it but still wanted to know for sure that I understood it correctly.

An open question to me here is: do we consider Complex<f32>, Complex<f64> primitive types or user-defined types?

I think it is very important to make this clear, and also to make clear whether users are allowed to implement Float and ComplexFloat for their own types. I think this should be forbidden, since it doesn't allow us to add some things to the traits in a backwards compatible way, and that's why I would prefer to call them PrimFloat and PrimComplexFloat and consider Complex<f32> and Complex<f64> primitive types.

While this is cool, I would like to first focus on not on bringing new numerical features into Rust, but to make a trait that makes it easy to use the features that are already present, of for example primitive floats, in generic code.

99% of cmath and C++ math are already part of num for floats. The only functionality missing are the 22 special mathematical functions which I would like to see at least as TODOs on the path forward since they are part of C++17 and will be part of the next C standard.

I would also like a clarification between the interaction of PrimFloat and a possible PrimInt proposal. Is there any common functionality that should be factored out? Should they be completely different traits, and a trait for common functionality can be built on top of those later?

I think those two complex types have to be user defined types, not primitive. Num's Complex struct is not unique as complex type definitions in the rust ecosystem either.

I agree here. Those particular traits are simply designed for their specific set of types. I think this makes sense for the primitive float set (f32, f64) and the complex float set (f32, f64, Complex<f32>, Complex<f64>). These are of course not the only numeric traits we need, but these specific ones are important too.


I know that Complex<f32>, Complex<f64> have an ffi problem at this point, but we can solve half of it at least.

  1. Complex is not #[repr(C)], so passing it by pointer to functions that expect a C-complex is not on sound foundation. This practically not wrong at the moment, and can be made theoretically sound by just adding that attribute in the library
  2. Rust does not support the ABI for passing C-complex values by value, so that can not be supported at all, regardless of Complex type definition. It's platform specific, so I believe Complex<f32>, Complex<f64> de facto work on x86-64, but this is rather shaky ground.

Rust may never solve (2) since it requires detailed conformance with a c compiler, in a way that I think llvm does not give for free. Rust could simply choose to only support passing C-complex values by pointer, making (1) sufficient.

So from the discussion until now I think Float/ComplexFloat can move forward independently of PrimInt, and if in the future one might want to provide a trait that abstracts over both Float and PrimInt that can be done independently of that without breaking backwards compatibility.

It would however be wise to try to move a PrimInt RFC forward in parallel with at least Float to at least make sure that methods that perform the same operations have the same names.

I want to create better traits for numerics; we don't necessarily need RFCs for that. In fact it seems best to improve num if we can and continue to try out things in the library space.

Features like additional special functions seem appropriate for an RFC.

Subjective part:

As both a programmer and physicist, I would like to see a (Trait-) model that matches the mathematical properties of numbers (real, complex, …).
Using a programming language where expressions that are a valid mathematical expression, transfered to the programming language, results in an also valid Rust expression, but with an different value/meaning, would be a nightmare.
It is okay if they are not valid, but valid expressions should match their mathematical counterpart.

Having said this, I would like to proceed to propose a model, that uses the mathematical properties of numbers as a reference:

Objective part:
In mathematics there are:

  • natural numbers (approximated by i8 - i64)
  • real numbers (approximated by f32 and f64)
  • complex numbers (approximated by Complex<f32> and Complex<64>)
  • quaternions

… and more. They each have the properties of extending the previous number system. This means once you have defined the rule to transform one value of a number system into the value of the next number system, all operations have the same result.

Let me give an example:

    fn real_to_complex(r: real) -> complex {
       complex { real: r, imag: 0 }
    }

Would be the function to map a real number into a complex number.
Now consider the following (simple) expression:
a + b · c
Assuming that a, b and c are natural numbers, it doesn't matter if we convert them first into real numbers (or complex numbers) and then evaluate the expression, or evaluate the expression first and then convert the result to a real number (complex number).

The reason this will always work is that each number system is a subset of the next higher one.

Proposal:
My suggestion is to use this property of extension to define the Trait model for numbers:

Integer would define:

  • addition
  • subtraction
  • multiplication

Real would add the following:

  • division / reciprocal
  • exponentiation (squaring, cubing, square root, …)

Complex adds:

  • conjugation

Matching the existing system
There are a lot of functions that one would expect from a Real to work, take for example the trigonometric functions. My suggestion would be to define them as generic functions and specialize them for f32 and f64. This way all number systems extending Real would have a correct sin function in place with the possibility replace it with a specialized function if the generic one is too slow.

There are a few other functions left, that do not belong to the number system (they are not inherited be the higher systems):

  • floor / ceil / fract
  • min / max

I do not know an elegant way to generalize them.

One interpretation of this would be that the already present integer division in Rust is a nightmare. For example the fact that 15 / 10 == 1.

I would like to focus on just floating point and related issues in this topic, we don't want to go into the whole numerical traits vs algebra discussion here.

As long as you have to specify that you are using integers for your data (and hopefully know the behaviour of integer division), this fine for me.

If you would add a way to write code that works for integer and float, this kind of division would probably lead to some annoying errors.

For the matter of providing a common Trait for float and complex, there are functions that are not so well defined for complex: floor, ceil, round.

I don't have much valuable technical input - language design is definitely not my forte! I'm currently working on a machine library which would benefit from many of these changes.

I think having access to both Float and Complex float would make my life much easier. For things like eigenvalue decomposition I want my users to be able to put in a real matrix which has complex eigen vectors. This is not really possible right now - from my understanding I would need to always return a complex matrix for the eigen vectors.

I also want my machine learning algorithms to work for both f32 and f64. This is attainable right now with the Float trait in num - but I think some of the proposals here would make things a lot easier.