How to merge Slotmaps

help
6

I see, that makes sense.

I assume the keys will be fixed-up once all the data are loaded and merged? in this case, I guess the keys are stored in the nodes in memory only, they don't need to be serialized with the nodes?

due to the nature of slotmap where a key must be allocated/calculated everytime an element is inserted into the container, there's no point to provide a bulk insertion API such as extend() or extend_from_slice() like what Vec does, because slotmaps cannot do similar optimizations for such use cases.

the best you can do is to call insert() in a loop. but this isn't something to avoid in the first place.

in fact, even if you could have merged two slotmaps in bulk, you still needs to fixup the keys afterwards, so you need to iterate through the elements anyway. now you can update the keys the same time as the connections are being inserted into the map, something like this:

impl Project {
    fn merge(&mut self, other: Project) {
        for (_, mut connection) in other.connections {
            self.connections.insert_with_key(move |key| {
                connection.node1.update_connection(key);
                connection.node2.update_connection(key);
                connection
            }); 
        }
    }
}

alternatively, since this is an index to speed up lookups, you can separate it from the main data structure. it doesn't need to be persisted, and doesn't need to be merged during the data loading process. instead, rebuild the index from scratch after the data is loaded.

a pattern I use a lot is to define separate types for in-memory data and serialized data. this also allows me to control the version of the file format. I think it might be adapted and modified for your use case too. here's some contrived snippets how this pattern looks:

/// the in-memory data structure
///
/// - no need to be serializable
/// - can have non-persistent states such as reverse lookup index
/// - no need to use `Option`, default value can be calculated at load time
struct Data {
	foo: bool,
	bar: String,
	baz: Box<dyn Trait>,
}

mod persist {
	/// the on-disk data format, (ab-)use the serde enum tag for versioning
	#[derive(Debug, Serialize, Deserialize)]
	#[serde(tag = "format_version", rename_all = "lowercase")]
	enum Data {
		V1(DataV1),
		V2(DataV2),
		#[serde(other)]
		Unsupported,
	}

	#[derive(Debug, Serialize, Deserialize)]
	struct DataV1 {
		foo: Option<bool>,
	}

	#[derive(Debug, Serialize, Deserialize)]
	struct DataV2 {
		/// for convenience, can use `Deref`
		#[serde(flatten)]
		compat: DataV1,
		bar: Option<String>,
	}
}

/// the load logic, including schema validation, version conversion, index building, etc.
impl TryFrom<persist::Data> for Data {
	type Error = LoadError;

	fn try_from(value: persist::Data) -> Result<Self, LoadError> {
		use persist::Data::*;
		match value {
			V1(v1) => todo!("loaded v1"),
			V2(v2) => todo!("loaded v2"),
			Unsupported => {
				eprintln!("unsupported config file version");
				Err(())
			}
		}
	}
}

/// the save logic, may trim or compact redundent information, etc
impl From<&Data> for persist::DataV2 {
	//...
}

/// save in old format is lossy, not enabled by default
#[cfg(feature = "compat-v1")]
impl From<&Data> for persist::DataV1 {
	// ...
}

/// some convenient wrapper functions
impl Data {
	fn load(path: &Path) -> Result<Self> {
		serde_json::from_slice::<persist::Data>(&fs::read(path)?)?.try_into()
	}

	fn persist(&self, path: &Path) -> io::Result<()> {
		let config = persist::Data::V2(self.into());
		fs::write(path, serde_json::to_string(&config).expect("deserialize"))
	}

	#[cfg(feature = "compat-v1")]
	fn persist_compat(&self, path: &Path) {
		let config = persist::Data::V1(self.into());
		fs::write(path, serde_json::to_string(&config).expect("deserialize"))
	}
}