Need heterogeneous list that's not statically typed

@teohhanhui I am looking at the Typescript interface for the combine function and it does have a type parameter for each source.
(callbag-combine/types.d.ts at master · staltz/callbag-combine · GitHub)

export default function combine<T1>(s1: S<T1>): S<[T1]>;
export default function combine<T1, T2>(s1: S<T1>, s2: S<T2>): S<[T1, T2]>;
...

So it seems that even in the Typescript version, heterogeneous lists are not used. I wonder if you're trying to do something that isn't possible with any statically typed language.

2 Likes

Actually, no, that's because TypeScript didn't have variadic tuple type support until more recently:

I've already experimented before with reactive streams in Rust (using reactive-rs):

Unfortunately, that library is no longer maintained, just like all the other FRP libraries in Rust (except for futures-signals which is lossy, thus does not fit my needs). That's why I've decided to write my own. But since I'm not smart enough to design my own library, callbag seems to be the easiest way forward, especially considering the next version of Cycle.js is built upon callbag (and I really want a Cycle.js equivalent in Rust).

Ok, I see. Well, that is certainly not a feature we have in Rust.

Variadic tuples have definitely been discussed in the past, and even went through RFC, but the RFC was closed for lack of resources and deferred to be revisited at a later time. If we had variadic tuples as per that RFC, it seems like you could have your combine function:

fn combine<(.. T)>(sources: (.. Source<T>)) -> Source<(.. T)> { todo!() }

But the usual way of working around the lack of variadic tuples is to be generic over tuples up to a certain reasonable length, using a macro, like itertools does (albeit it only deals with homogeneous tuples, but it should work similarly):

trait Combine {
    type Output;
    fn combine(self) -> Source<Self::Output>;
}

macro_rules! impl_combine {
    ($first:ident) => {};
    ($first:ident, $($T:ident),+) => (
        impl_combine!($($T),+);
        impl<$($T: Send + Sync + Clone + 'static),+> Combine for ($(Source<$T>),+,) {
            type Output = ($($T),+,);
            fn combine(self) -> Source<Self::Output> {
                Callbag(Box::new(move |_message| {
                    let ($($T),+,) = &self;
                    todo!("combine")
                }))
            }
        }
    )
}
impl_combine!(dummy, T, U, V, W, X, Y, Z);

fn combine<T: Combine>(sources: T) -> Source<<T as Combine>::Output> {
    sources.combine()
}

fn main() {
    let source_a: Source<String> = Callbag(Box::new(|_| todo!()));
    let source_b: Source<i64> = Callbag(Box::new(|_| todo!()));
    let source_c: Source<Vec<u64>> = Callbag(Box::new(|_| todo!()));
    let combined_sources: Source<(String, i64, Vec<u64>)> =
        combine((source_a, source_b, source_c));
}

playground link

The exercise for you is obviously to replace the todo!("combine") which may or may not be possible..

This definition of combine obviously only works with a statically-known tuple input, but then again, that's the only way to have a statically-known output. If you want a heterogeneous tuple as input whose types and arity are only known at runtime, then it seems like any language isn't gonna be able to do better than Vec<Box<dyn Any>>..

4 Likes

Thank you! At a glance that seems like exactly what I need. I only need to support static T output types (at compile time of the program, but not definition time of the combine function).

I've ended up producing this unholy piece of... thing:

which is 9000+ lines of code when expanded:

I hide my face in shame. :see_no_evil:

Will the users of my library hate me for this? (And yeah, I need to fix macro hygiene...)

:woman_shrugging: I've seen far uglier macros. I think the main issue is the amount of duplicated code potentially affecting compile times.. Could you try implementing essentially only the lines which use the macro parameter repetition inside the macro, as individual trait methods, and implement the rest in the combine free function?

Also, instead of a bunch of individual Arc, maybe put all those bits of data in a struct inside one Arc? That would eliminate a lot of relatively expensive atomic operations..

3 Likes

Another thought: you could bring the Rust code more equivalent to the Javascript and possibly move some of the code outside the macro by storing the data in a Arc<[ArcSwapOption<Box<dyn Any + Send + 'static>>]> instead.

3 Likes

The problem is that the Combine trait has to be public, so I'd like to not expose implementation details in the form of trait methods (yeah, I could probably use #[doc(hidden)], but still...). But the main thing is, converting closures to functions is such a pain. Haha...

Thanks for the tip about the struct. I struggled with not being able to get the index necessary for tuple access (cf. RFC: Declarative macro metavariable expressions by markbt · Pull Request #3086 · rust-lang/rfcs · GitHub). Thought of using some map collection but gave up. Somehow struct slipped my mind. :rofl:

I don't understand this phrasing. If the set of types is known at compile time, there should be no need for variadic generics.

Instead of phrasing your question in terms of implementation, could you share the desired usage of the library that demonstrates a scenario in which you need variadic generics?

Is your requirement that the type parameter of the message/event type should itself be an arbitrary generic type, with arbitrarily many type parameters? That is not variadics territory, it's more like higher-kinded types, and in itself, it doesn't seem useful, because messages will need to be instantiated with a concrete type anyway in order to stand a chance of being handled specifically. Instead of trying to make the type parameter itself HKT-generic, you could try making it just an ordinary generic and adding very minimal trait bounds so that it is usable with a broad set of types.

1 Like
let combined: Source<(i32, &str)> = combine(
    from_iter([10, 20, 30]), // Source<i32>
    from_iter(["a", "b", "c"]), // Source<&str>
);

The user can provide any combination of sources with different output types.

I've simplified the implementation, learning from rust/tuple.rs at 84f962a89bac3948ed116f1ad04c2f4793fb69ea · rust-lang/rust · GitHub

The code duplication remains, but at least now I have one ArcSwap for the entire vals tuple, and compile time has improved.

Thanks, that's much clearer. Indeed, in this case, a variadic function could help, but you don't need any of:

  • dynamic typing
  • HKTs
  • heterogeneous lists.

Someone already suggested using tuples instead of proper variadic generics, and I too think that would work. If you don't want to be limited to only a certain number of expressions/types (most of which will be actually unneeded and only elongate compile times), you could try emulating what the compiler does when instantiating generics, and create impls on the fly for only the concrete arities you need.

A procedural macro might come in handy for creating unique names for one-off types, whose sole purpose is to "forward" (by means of an associated type) to an arbitrary-arity tuple and generate the impl (which is why you need the fresh type in the first place).

2 Likes

I'd love to get as much code out of that macro as possible, but I can't yet figure out a way to do so without exposing more public traits in trait bounds for use as type constructors.. I feel you, in terms of #[doc(hidden)] not being a great solution (though as it stands, your Combine trait is in a private module and thus not documented anyway, though I assume based on your previous message that you intend to make that publicly documented).

1 Like

Oh, it's just that the Combine trait needs to be public, because the type is exposed by the combine free function. But yeah, I think it's not documented anyway like you said, which supposedly means it does not constitute part of the public API of the library.

Relevant: https://www.reddit.com/r/rust/comments/ey0ul0/handling_breaking_api_changes/fgesyid?utm_medium=android_app&utm_source=share&context=3

I think one thing that really hinders code de-duplication is the inability to access tuple elements with anything other than a literal...

This looks remarkably like Iterator::zip :laughing:

Indeed, but that's just an example. Haha...

I need to be able to accept Into<Arc<Source<T>>>, but my implementation:

gives errors such as:

error[E0207]: the type parameter `L` is not constrained by the impl trait, self type, or predicates
   --> src/combine.rs:244:17
    |
244 |         (11) -> L
    |                 ^ unconstrained type parameter

I kinda understand why this is happening:

A clause like where SomeType: Index<I> cannot be considered to constrain I , because even if SomeType only implements Index for a single index type, a downstream crate can always add another impl:

struct MyIndex;
impl Index<MyIndex> for that_crate::SomeType {
    ...
}

causing I to become ambiguous.

(from: "the type parameter is not constrained" but it is needed · Issue #56209 · rust-lang/rust · GitHub)

but I've no idea how I can fix this...