API Design for Config Override

help
1

I keep encountering this pattern which I have struggled to find a good solution.

  • There is a global config with non-nullable fields
  • Each item can override some per-item config fields

Example: RSS Reader

struct App {
	config: Config,    // global config
	feeds: Vec<Feed>,  // per-item config
}

struct Config {
	max_retry: u8,
	browser_agent: String,
	fetch_interval: std::time::Duration,
}

struct OptionalConfig {
	max_retry: Option<u8>,
	browser_agent: Option<String>,
	fetch_interval: Option<std::time::Duration>,
}

struct Feed {
	source: Source,
	config: OptionalConfig,
}

Here OptionalConfig feels too verbose.

Example: Theme

A nested config is even more complex. All intermediate and leaf attributes should be Option.

pub struct Theme {
    pub title: LabelStyle,
    pub subtitle: LabelStyle,
	...
}

pub struct LabelStyle {
    pub family: FamilyCow,
    pub size: Logical<f32>,
    pub line_height: Option<Logical<f32>>,
    pub color: Color,
    pub weight: u16,
}


struct Grid {
	theme: Theme,
	children: Vec<Box<Grid>>,
}

expands to

pub struct ThemeOverride {
    pub title: Option<LabelStyleOverride>,
    pub subtitle: Option<LabelStyleOverride>,
	...
}

pub struct LabelStyleOverride {
    pub family: Option<FamilyCow>,
    pub size: Option<Logical<f32>>,
    pub line_height: Option<Option<Logical<f32>>>,
    pub color: Option<Color>,
    pub weight: Option<u16>,
}

grid.theme = default_theme;
grid.children[0].theme = default_theme.override(theme_override);

This has many problems.

  • line_height: Option<Option<..>> is difficult to use and serialize. Also ambiguous when exposed through PyO3 or wasm-bindgen.
  • ThemeOverride is large even when there are only few fields to override.
  • Theme already uses more than 30 types, and adding override types for each is too much.
  • The API for grid.children[0].theme is bad; Updating grid.theme doesn't automatically update the child theme. Ideally the child item should only specify the difference, which will be computed by the library

Is there a better approach?

2

w/o knowing the details of your requirements, i'd suggest you make, in the RSS Reader example, the Feed's config field, an impl method instead. something like...

impl Feed {
    fn config(&self) -> Config {...}
    ...
}

the concrete implementation can then inherit the global Config, overrides whatever it needs, cache the result or not, and return the same Config type.

3

there's a crate literally named config, which implements hierarchical and layered configuration resolution.

if your types implement the serde data model, it's very easy to integrate the config crate. you can also just use config directly if you are ok to work with dynamically typed values, i.e. instead of direct field access my_config.some_field, you use a runtime checked call my_config.get::<FieldType>("field_name").

check out the examples for more demos.

4

A crate might well be better than roll-your-own, but here's a couple roll-your-own ideas I had.

You could use shared ownership (Arc<_>, Rc<_>) at select layers. Perhaps with some sort of interning other other bookkeeping to minimize duplicates. It would partially address the following (depending on how granular your Arc placement is), but perhaps not fully.

HashMap<SettingEnum, SettingValue> occured to me but it gives up a lot of type safety.

You can instead use a HashSet without sacrificing much type safety if you have a distinct type that ignores the values for the sake of equality and hashing; that would be amenable to storing deltas.

Example 
pub enum Setting {
    LabelStyleSize(Logical<f32>),
    LabelStyleLineHeight(Option<Logical<f32>>),
    LabelStyleColor(Color),
    LabelStyleWeight(u16),
    // ...
}

pub struct SettingEntry {
    setting: Setting,
}
impl SettingEntry {
    fn id(&self) -> Discriminant<Setting> {
        std::mem::discriminant(&self.setting)
    }
}
impl Eq for SettingEntry {}
impl PartialEq for SettingEntry {
    fn eq(&self, rhs: &Self) -> bool {
        self.id() == rhs.id()
    }
}
impl std::hash::Hash for SettingEntry {
    fn hash<H: std::hash::Hasher>(&self, hasher: &mut H) {
        self.id().hash(hasher)
    }
}

Fetching values without a Setting in hand isn't the greatest, but can be encapsulated somewhat.

Example 
// A little nicer with `hash_brown::{HashSet, Equivalent}`
impl Equivalent<SettingEntry> for Discriminant<Setting> {
    fn equivalent(&self, key: &SettingEntry) -> bool {
        key.id() == *self
    }
}
struct SettingOverride {
    overrides: HashSet<SettingEntry>,
}
impl SettingOverride {
    fn label_style_size(&self) -> Option<&Logical<f32>> {
        const KEY: Discriminant<Setting> = std::mem::discriminant(
            &Setting::LabelStyleSize(Logical(0.0))
        );
        match self.overrides.get(&KEY)? {
            SettingEntry { setting: Setting::LabelStyleSize(sz) } => Some(sz),
            _ => unreachable!(),
        }
    }

    fn override_of(&self, setting: &Setting) -> Option<&Setting> {
        let key = std::mem::discriminant(setting);
        self.overrides.get(&key).map(|se| &se.setting)
    }
}

Using TypeId and Box<dyn Any> instead of Discriminant<_> and an enum is similar in many ways.

6

Hello,

I might consider creating my own enum for the overrides:

enum Revise<T> {
    Preserve,
    Override(T)
}

pub struct ThemeOverride {
    pub title: Revise<LabelStyleOverride>,
    pub subtitle: Revise<LabelStyleOverride>,
	...
}

pub struct LabelStyleOverride {
    pub family: Revise<FamilyCow>,
    pub size: Revise<Logical<f32>>,
    pub line_height: Revise<Option<Logical<f32>>>,
    pub color: Revise<Color>,
    pub weight: Revise<u16>,
}

This might come with its own custom serialization, for example {} for Preserve and { "override": ... } for Override.

7

Thanks for many ideas!

The requirement is serializability and an ergonomic Rust API (type-safety and less boilerplate). For example, if I am a library user I wouldn't want to write

Grid {
    theme: VIRIDIS,
    overrides: ThemeOverride {
        title: TitleOverride {
            font: FontOverride {
                size: 10.0,
                ..Default::default()
            },
            ..Default::default()
        },
        ..Default::default()
    }
    items: vec![
        Grid {
            overrides: ThemeOverride {
                // ..
            },
            Default::default()
        }
    ]
}

Because we know the equivalent is very simple in js/ts:

let plot: Grid = {
    theme: VIRIDIS,
    overrides: {
        "title.font.size": 10.0,
    },
    items: [
        {
            overrides: {
                // ..
            }
        }
    ]
}

I would like to avoid untyped HashMap (string keys). A typed HashMap with Discriminant is interesting but it is hard to maintain if a single enum lists every nested attribute. TypeId can change between builds. A per-type discriminant API is difficult to use:

#[derive(FieldEnum)] // derive per type, beacuse flattened enum is too hard to maintain
struct LabelStyle {
    size: ..,
    color: ..,
)

// generated
enum LabelStyleField {
    Size(..),
    Color(..),
}

type LabelStyleOverride = Vec<LabelStyleField>;

// difficult
Grid {
    theme: VIRIDIS,
    overrides: vec![
        ThemeField::Title(
            vec![TitleField::Font(
                vec![FontField::Size(10.0)]
            )]
        )
    ],
    ...
}

I am currently experimenting with a macro approach to allow dotted-paths.

let plot = obj!(Grid {
    theme: VIRIDIS,
    overrides.title.font.size: 12.0,
    items: vec![
        obj!(Grid {
            overrides.title.outline_size: 2.0,
        }),
    ],
});

For reference, other crates with the similar patterns are

  • GPUI (Style has 38 fields)
  #[derive(Clone, Refineable)]
  struct Style {
      color: Color,
      #[refineable]
      label: LabelStyle,
      line_height: Option<f32>,
      ...
  }

  struct StyleRefinement {
      color: Option<Color>,
      label: LabelStyleRefinement,
      line_height: Option<f32>,
      ...
  }

  style.refine(&child_style);
  • Ratatui
// no override or refinement object. everything is Option<>
let heading_style = Style::new()
    .fg(Color::Black)
    .bg(Color::Green);
  • egui
// no partial type, just style everything every frame
8

IDK, but maybe a simpler approach?

For the "inner working" of the software/crate/app, is it really necessary to know that some config elements originate from a default or common set and some other elements are individual adaptations?

So, for the RSS Reader example, why not just:

struct Config { ... }

struct Feed {
	source: Source,
	config: Config,
}

struct App {
	feeds: Vec<Feed>,
}

No OptionalConfig.

If you would like to provide a user faced language, maybe JSON, to write configs with optional elements, that would be a separate struct:

struct ConfigFileThing {
    field_a: Option<...>,
    field_b: Option<...>,
}

Then you have some mechanism/functions to convert a (hierarchical) config, where all fields are optional, to a concrete list of items with simple Configs.

So the basic idea is: The model, how the user writes a config file, is not identical to the config model used internal. That is, these are two different domains with different needs.

9

understandable. string keys are not type safe, defeating one of the most important aspects of a statically typed language.

a middle ground is usually good enough though, e.g. use string keys under the hood, but use procedurally generated accessor methods to ensure type safety.

the typescript case is interesting. at runtime, the data is just a weakly typed hash map with string keys, but at compile time, there's a type checker which can validate the keys are valid according to the type definition. since it's just a hashmap under the hood, "optional" fields and mandatory fields are only distinguishable at compile time, and the overrides/fallbacks hierachy can be implemented very naturally as js prototype chain.

this flexibility is not built into the rust language, accessing an Option generates different code than accessing a "normal" field. there's no object inheritance or prototype chain, so the hierarchy must be customized and tailored for individual data types.

in addiiton to the maintanence difficulties, another problem with a hash map is the value type must be dynamically typed. the type information for each key is available, but it's very hard to utilize this information to design a safe and ergonomic api around a hash map.

a DSL is an interesting idea, it reminds me of the json!() macro from serde_json.

I can also imagine a different approach: generating "proxy"/"wrapper" types from a "master" type. the master type is just a regular rust type. the wrapper types can provide layered access and fallback logic.

here's a sketch how it might look:

// can be implemented as an attribute or derive macro
#[layered_config]
struct Config {
    max_retry: u8,
    browser_agent: String,
    fetch_interval: Duration,
}

// example generated code

// a trait to combine heterogeneous data sources
trait ConfigDataSource {
    fn max_retry(&self) -> Option<&u8> { None }
    fn browser_agent(&self) -> Option<&String> { None }
    fn fetch_interval(&self) -> Option<&Duration> { None }
}

// a generic wrapper, may be defined in library instead of procedural generated
// note, can use references for layers
struct Layered<Over, Under>(Over, Under);

/// the "bottom" or "base" layer
struct DefaultConfig { ... }
impl Default for DefaultConfig { ... }
/// should all return `Some`
impl ConfigDataSource for DefaultConfig { ... }

/// use the unit type as a placeholder "empty" layer
impl ConfigDataSource for () {}

// can generate other "override" types that can be layered on top of each other
// they don't need to contain all fields of `Config`, nor they need `Option`,
// for example, single field overrides can be defined:
struct ConfigMaxRetry(u8); impl ConfigDataSource for ConfigMaxRetry { ... }
struct ConfigBrowserAgent(String); impl ConfigDataSource for ConfigBroserAgent { ... }
// or, a runtime override, like a "builder"
struct ConfigBuilder {
    max_retry: Option<u8>,
    //...
}
impl ConfigDataSource for ConfigBuilder { ... }

/// the overlay/hierarchy logic using the `Layered` wrapper:
impl<Over: ConfigDataSource, Under: ConfigDataSource> ConfigDataSource for Layered<Over, Under> {
    fn max_retry(&self) -> Option<&u8> {
        self.0.max_retry().or_else(|| self.1.max_retry())
    }
    //...
}

// wrap some type on top of an "empty" layer
impl<O> Layered<O, ()> {
    fn new(layer: O) -> Self { todo!() }
}

// add some type as an overlay on top of the current stack
impl<O, U> Layered<O, U> {
    fn overlay<O2>(self, layer: O2) -> Layered<O2, Self> { todo!() }
}

// the layered wrapper may serve as a builder too
impl<O, U> Layered<O, U> where ... {
    // merge all layers into the final data
    fn merge(self) -> Config {
        let max_retry = self.max_retry().clone().expect("bad base layer");
        //...
        Config {
            max_retry,
            //...
        }
    }
}

I think this design can work, although I'm yet to come up a clear idea about nested data structures.

10

I would probably keep the override type separate from the actual configuration used by the program.

For example:

use std::time::Duration;
#[derive(Clone)]
struct Config {
  max_retry: u8,
  browser_agent: String,
  fetch_interval: Duration,
}

#[derive(Default)]
struct ConfigOverride {
  max_retry: Option<u8>,
  browser_agent: Option<String>,
  fetch_interval: Option<Duration>,
}

impl Config {
fn apply(&self, change: &ConfigOverride) -> Self {
  Self {
    max_retry: change.max_retry.unwrap_or(self.max_retry),
    browser_agent: change
                .browser_agent
                .clone()
                .unwrap_or_else(|| self.browser_agent.clone()),
fetch_interval: change
                .fetch_interval
                .unwrap_or(self.fetch_interval),
        }
    }
}

For example, a Feed only needs to store the values it wants to change:

struct Feed {
  source: Source,
  config: ConfigOverride,
}

let effective_config = app.config.apply(&feed.config);

The nice part is that a change to the global config is automatically visible to
feeds that do not override that particular field. If calculating the effective
config turns out to be expensive, it could be cached later.

For nested values, I don't think it is possible to avoid some kind of
recursive override type while keeping everything statically typed. A separate
Override or Refinement type (perhaps generated by a derive macro) seems

easier to work with than having Option<Option<T>> everywhere. The nested
Option is only needed if “not specified” and “explicitly set to None” have
to mean different things.

So I would use the partial type at the API boundary, and turn it into a
complete Config before passing it to the rest of the code. That keeps the
inheritance behavior in one place.