Thanks for taking the time to make clarifications! Some of these points are really good, I might rejiggle a few things in the proposal to better match your explanations.
I was thinking of this, but there are a few issues that I saw, the largest one being having to re-twiddle structures during runtime. This might not be a problem for small structs, but for larger, more complex types, the conversion overhead might be quite large. Laying out all structures in an ABI compliant manner before time is results in O(1) transfer times, whereas converting between is O(N) where N is the number of referenced sub-structures.
I was thinking that Vec and Box might need to be special cases, but when you think about it, just about anything can be a special case. An anecdote: A while back, I wrote a NaN-tagging implementation. I won't go into much detail here, but NaN-tags are special pointers to data structures. The problem with NaN-tags is that they aren't explicitly pointers, they're pointers only within certain contexts. When a NaN-tag is made, the data it points to is basically leaked, meaning no pointers to the data exist. To retrieve the data inside a NaN-tag, the original pointer needs to be reconstructed from the bits contained in the NaN-tag.
If you tagged some data in Program A, passed the tag over an FFI boundary to Program B, then tried to reconstitute the data from the tag from program B, chances are something would go wrong. If data was only converted at runtime, the system risks Program B trying to access Non-ABI compliant data. I'm not sure about the exact intricacies of FFIs and ABIs, so correct me if I'm wrong, but Program B might be not be able to even access the data in Program A if they're in different processes.
It seems like due to the unsafe nature of Rust code, guaranteeing that all memory associated with a data structure is transferred over the boundary is hard. This in part can be fixed by using a compile-time ABI, but memory still might be lost if managed in an unsafe manner. An ABI would expose a larger surface area for things to go wrong. This is not to say that it's impossible, it just might take more work. I guess what I'm trying to say is that unless special cases are written, only 'safe' data structures can be transferred across FFI boundaries.
I see. I think the solution would be to develop a generic data-layerouter. Here's what I mean. In Rust, it's possible to define custom layouts for pointers through a very generalized Layout interface. I'm suggesting something similar be done for the ABI.
All a layout really does is indicate which bits belong to what, along with some metadata. Therefore, if a general data structure that can represent these mappings exist, an ABI can be implemented. Take this simple example.
/// A `Layout` is a set of bytes.
/// Each `Layout` contains a number of fields, which may contain a sub-layout.
pub trait Layout {
fn size(&self) -> usize;
fn fields(&self) -> Vec<Field>;
}
/// A `Field` represents a series of bytes in a `Layout` possibly containing a sub-`Layout`.
type Field = (usize, Option<Box<dyn Layout>>);
Then, to define an enum:
pub struct Enum {
size: usize,
discriminant: (usize, Discriminant),
variants: Vec<(usize, Variant)>,
}
pub struct Variant {
size: usize,
contents: Option<Box<dyn Layout>>,
}
pub struct Discriminant {
size: usize,
}
Then we can implement Layout for the Enum:
impl Layout for Enum {
fn size(&self) -> usize { self.size }
fn fields(&self) -> Vec<Fields> {
// ... snip
// use `self.discriminant` and `self.variants` to construct a `Vec<Option<Fields>>`.
}
}
impl Layout for Variant {
// ... snip
// `fields()` would just wrap `self.contents` in a `Vec<...>`.
}
impl Layout for Discriminant {
// ... snip
fn fields(&self) -> Vec<Fields> { vec![] }
}
Then, at compile time, macros would expand enum definitions into Layouts. The compiler would then use the layouts to construct the actual data structures. Note that this is just an example - a real implementation would require a bit more involvement. Needless to say, option types or enums could be used to represent niche optimizations in Enum itself. When the compiler tries to build a layout described by Enum through the Layout trait, the optimizations can be applied, as a Layout simply represents a series of bytes and what they mean. This toy model doesn't show the full range of behaviours that I'm trying to describe, but I hope you see what I'm saying.
We only need to translate structures that go across FFI boundaries, though all Rust structures should be representable. At compile time, ABI compliant data types are translated. Non-ABI compliant datatypes can keep the Rust representation, they just can't be transferred across FFI boundaries. (Of course there's the unsafe caveat I mentioned above.)
B is the way to go, but it shouldn't be done at runtime.
Thanks 