Creating a new type from a list of traits

Ah yes, you would indeed need to split Deserialize out from whatever you box - perhaps simply boxing Serialize would be enough, if the other traits are used beforehand.

Indeed. The next problem is that the Transaction storing the payload must be Deserialize, since in another file, Transaction objects are converted to and from JSON strings. That's why I suggest converting the payload to a Vec<u8> when storing it and back into the original type when retrieving it; otherwise, to deserialize a transaction from JSON, you'd need to turn it into a custom opaque type that would somehow represent the original Vec<u8>.

1 Like

You can't make a Box<dyn Serialize>, either, since both traits have generic associated functions.

1 Like

Poop. Well, if you do have a Payload trait, you could add a serialize method that bakes serde_json, but really it seems like just storing the dang bytes is probably better.

For the record: payload: Vec<u8> was just an initial stab at the problem that was really proving to be too difficult to work with. While at the end of the day everything is bytes, the fact is that we KNOW we're working with JSON and we actually want to be able to inspect the payloads as they are saved on disk or in a debugger. So, having an opaque Vec<u8> would require us to also have meta-data to help decode it. Internally, ownership of the data in that payload needs to be entirely the responsibility of whomever put it there.

That pains me to hear

The important requirements are:

  1. That a user of this API can just dump a serialisable thing in as the payload and later, retrieve it just as easily
  2. That when saved to disk (which we do with usage of our add_block_event_listener) the structure is clearly understood JSON - so (and this is asked in another thread) an array of base 10 encoded digits, which are the byte values of something encoded as JSON, is totally contrary to this goal
  3. That internal implementation details (like use of serde) be hidden - Though, this is ideal

If you want to preserve the JSON format, you could store a serde_json::Value as a payload. This would allow objects to be serialized into and deserialized from payloads, and also allow Transactions themselves to be serialized and deserialized.

I am not actively trying to recruit, but it's clear that something that seems simple isn't obviously so. And, I will readily admit to be in over-my-head; my whole team could benefit from an experienced hand.

If anyone is interested, this is a fully endorsed OSF (Open Source Foundation) project, so all are welcome, but there are also paid positions. If you want to get involved in Blockchain, DevSecOps and up your rust foo you'd be a welcome addition.

Hopefully my post doesn't get flagged

Then clearly, you need to make the transaction etc. (whatever you put the "thing" in) generic. I don't think there is any other way of storing a "whatever as long as it's serializable" type and retrieving the value with its exact original type later.

You may also be able to just store the already serialized JSON (Value or string) only, and use a PhantomData<TypeOfThing> to have your container remember the type. Then you can dynamically inspect the structure of your value, and ensure that the value you deserialize in the getter is of the correct type. (Although I'm puzzled as to why you would want to do that – clearly, if you are dealing with arbitrary trees of types/values, then you won't be able to dynamically inspect any useful set of fields/variants, because you simply don't know what they can be.)

Making Transaction generic doesn't really work here. A Transaction contains a payload, a Block contains a Vec<Transaction>, and a Chain contains a Vec<Block>. If a type parameter were used, then every transaction in a chain would have to have the exact same payload type.

I guess this is the main question, regardless of internals, then:

What happens when someone puts in a Foo, and tries to later get a Bar?

If is a compiler error then whatever container type (so far here, Transaction) they are using to get and put must know what the item type is, that is, the container's generic. If you're persisting, though, then that just bumps the question to what happens if you use the wrong item type when defining the container after persisting.

If it's an Err or None (or panic, but please don't) then you can make the get method generic, and use the trait bounds to check the stored form to know that it's the wrong type, whatever that means for you. Maybe just serde_json::from_value().

If they get whatever was stored, then that type has to be able to contain whatever was stored.

1 Like

But those are conflicting requirements! You can't have all of the dynamic chain type checked if its nodes are allowed to be arbitrarily heterogeneous – that's not a value-level linked list anymore, it's now a type-level tuple or cons list or whatever.

If OP wants to write a list where every node can store a different type, and nodes are statically type-checked so that one is guaranteed to get out the same type that they put in, then one has to know the structure of the chain beforehand, i.e. every node has to be typed individually. This means that it's not going to be an arbitrary, run-time-sized collection.

Indeed, static type checking is impossible in this scenario, but I don't think OP was ever asking for static type checking. That's why I've recommended serializing the payloads with either bincode or serde_json, so that the output types at least have to have the same shape as the inputs. For extra safety, one could use monostate to add a ty: MustBe!("Thing") field to every type.

OP's main issue now is that they want the payloads to be observable in their JSON representation. That's why I've recently recommended storing payload as a serde_json::Value. It allows any compatible type to be converted to a payload and back, and it allows the Transactions themselves to be serialized and deserialized. It won't allow full type safety, at least without monostate, but I think it's the best that can be done given the constraints.

2 Likes

A colleague of mine recommended this too; I sort of pushed for something not tied to any given tech.

And, you're right to latch on that it's not possible to guarantee that what went in is compatible with what comes out - this would produce a runtime error and not a compiler error. We'll give this solution a go. Thanks for the input

So, here are my changes:

#[derive(Serialize, Deserialize, Debug, Clone, Hash, PartialEq, Eq)]
pub struct Transaction {
    submitter: Address,
    timestamp: u64,
    payload: Value,
    nonce: u128,
    // Adds a salt to harden
    hash: HashDigest,
    signature: TransactionSignature,
}
#[derive(Debug, Deserialize, Serialize, Hash, PartialEq, Eq)]
struct Thing {
    name: String,
    age: usize
}

pub fn submit_transaction<T, CallBack: 'static + FnOnce(Transaction)>(
    &mut self,
    payload: T,
    on_done: CallBack,
)  -> Transaction where T: Sized + Serialize + Hash + Eq + PartialEq {
    let trans = Transaction::new(
        self.submitter,
        json!(payload),
        &self.keypair,
    );
    self.trans_observers.insert(trans.clone(), Box::new(on_done));
    self.pending_transaction.insert(trans.clone());
    trans.clone()
}

And so, the following examples work

  bc.submit_transaction(Thing{name: String::from("foo"), age: 10}, |t| {
      println!("transaction  accepted {}", t.signature().as_string());
  });

  bc.submit_transaction([1,2,3], |t| {
      println!("transaction  accepted {}", t.signature().as_string());
  });

  let map = BTreeMap::from([
      ("name", String::from("hello")),
      ("age", String::from("10")),
  ]);
  bc.submit_transaction(map, |t| {
      println!("transaction  accepted {}", t.signature().as_string());
  });

But this, does not:

let map  = HashMap::from([
    ("name", String::from("hello")),
    ("age", String::from("10")),
]);
bc.submit_transaction(map, |t| {
    println!("transaction  accepted {}", t.signature().as_string());
});

Which produces an error that Hash is not implemented for HashMap which I find entirely ironic!

error[E0277]: the trait bound `HashMap<&str, std::string::String>: Hash` is not satisfied
   --> src/blockchain/examples/simple_node.rs:57:43
    |
57  |                     bc.submit_transaction(map, |t| {
    |                        ------------------ ^^^ the trait `Hash` is not implemented for `HashMap<&str, std::string::String>`
    |                        |
    |                        required by a bound introduced by this call

and this also doesn't work:

let map  = BTreeMap::from([
    ("name", String::from("hello")),
    ("age", 10),
]);
bc.submit_transaction(map, |t| {
    println!("transaction  accepted {}", t.signature().as_string());
});

Because the tuple types aren't consistent. But, from a JSON perspective, it would be just fine.

When the working samples get saved to disk, it's exactly what I want. The Hash requirement is absolutely a must because the transaction is used as a key in a HashMap and a HashSet.

This is probably good enough, But I would love to be able to accept Maps of mixed types

Why do you need Hash + Eq + PartialEq here? T: Sized + Serialize should be enough. Transaction will be Hash no matter what T is, since it doesn't contain T.

3 Likes

I assume you can't hash Hash* containers because their iteration is not predictable, so it would be inordinately expensive to implement.

Well, hypothetically one could use an order-independent fold, like ^ing the hashes of the elements, rather than hashing the elements themselves.

But that's both collision-prone (the same item there twice undoes the first one) and fits poorly with the API that Rust uses -- the type is passed a &mut impl Hasher, which cannot be cloned so can't get hashes of the individual elements anyway. (Though the hashmap could use its own BuildHasher for that.)

So yeah, overall it's probably better to just not have it to discourage its use. One can use a BTreeMap instead if one needs the whole container to be Hash.

1 Like

You know, I never considered that. I made the changes as you suggest and it works.