Sqlx INSERTS and UPDATES with variable fields

I frequently encounter a problem where I need to INSERT or UPDATE table rows that have optional data (not necessary or filled by database itself).
There are a few solutions that I am aware of but none of them feel right.

Here is a postgersql table:

CREATE TABLE ingredient
(
    id          BIGINT GENERATED BY DEFAULT AS IDENTITY (START WITH 1000) PRIMARY KEY,
    version     BIGINT                                                    NOT NULL DEFAULT 1,
    
    -- Timestamps
    creator_id  BIGINT                   NOT NULL,
    created_at  timestamp with time zone NOT NULL DEFAULT now(),
    updater_id  BIGINT,
    updated_at  timestamp with time zone,

    name        varchar(256)             NOT NULL,
    comment     varchar(256), 
    min_amount  FLOAT                    NOT NULL DEFAULT 0,
    max_amount  FLOAT                    NOT NULL DEFAULT 0,

    deleted bool                         NOT NULL DEFAULT false
);
SELECT trigger_updated_at('ingredient');

Its form struct and insert function:

#[derive(Debug, Deserialize)]
pub struct IngredientCreateForm {
	pub name: Option<String>,
	pub min_amount: Option<f64>,
	pub max_amount: Option<f64>,
	pub comment: Option<String>,
}

pub async fn create<'c,E>(
	ctx: &Ctx,
	executor: E,
	entity_c: IngredientCreateForm,
) -> Result<i64>
where
	E: Executor<'c, Database = Postgres>,
{

	let query = r#"
        INSERT INTO ingredient (
        	creator_id,
            name,
            min_amount,
            max_amount,
            comment
        )
        VALUES ($1, $2, $3, $4, $5)
        RETURNING id;"#;
	Ok(sqlx::query_scalar::<_, i64>(query)
		.bind(ctx.user_id())
		.bind(entity_c.name)
		.bind(entity_c.min_amount)
		.bind(entity_c.max_amount)
		.bind(entity_c.comment)
		.fetch_one(executor)
		.await?)
}

The problem in this code is that entity_c.min_amount and entity_c.max_amount could be None. if we try to insert into ingredient without min and max then everything works since it is set by default.

Ideally I want a solution that inserts only values that are Some.

Solutions that I know of:

  1. SeaQuery or something similar (using a separate crate just to build INSERTS is very sad).
  2. Insert whole object always by requesting existing entry from the database in case of UPDATE and manually setting all defaults in case of INSERT (more queries is bad).
  3. require form to include all needed data from the start (it is not possible sometimes).
  4. Manually construct all queries with format (literal torture).
  5. Use code that gets non none fields names and their values. This problem feels like it should have been already solved so writing something your own feels not right... examples and their problems below.

This one needs to be a macros to be complete:

use std::any::Any;

#[derive(Debug)]
struct TestStruct {
    field1: Option<i32>,
    field2: Option<String>,
    field3: Option<f64>,
}

impl TestStruct {
    fn get_non_none_values(&self) -> (Vec<String>, Vec<Box<dyn Any>>) {
        let mut field_names = Vec::new();
        let mut values = Vec::new();

        if let Some(value) = &self.field1 {
            field_names.push("field1".to_string());
            values.push(Box::new(*value) as Box<dyn Any>);
        }
        if let Some(value) = &self.field2 {
            field_names.push("field2".to_string());
            values.push(Box::new(value.clone()) as Box<dyn Any>);
        }
        if let Some(value) = &self.field3 {
            field_names.push("field3".to_string());
            values.push(Box::new(*value) as Box<dyn Any>);
        }

        (field_names, values)
    }
}

Haven't tested this one(Value probably won't work) and it uses additional crates but demonstrates the idea:

use serde::Serialize;
use serde_json::Value;
fn get_non_none_fields<T: Serialize>(s: &T) -> Vec<(String, Value)> {
    let serialized = serde_json::to_value(s).unwrap();
    serialized.as_object().unwrap()
        .iter()
        .filter_map(|(k, v)| {
            if !v.is_null() {
                Some((k.clone(), v.clone()))
            } else {
                None
            }
        })
        .collect()
}

So using a pre-written, 3rd-party crate is not good, but rolling your own isn't good, either? What other possibility is there?

  1. SeaQuery is a big crate and solving such a small task is bad.
  2. I don't like my solution since in my opinion the problem I am facing is already solved and I just can't find an official solution.
    The problem is to find a good solution not just any.

In my opinion, and I'm sorry to say that, the solution would probably be a better design of your data. A better design of your relationships and types would probably solve that issue for you and simplify your queries.

Do you think that is true on the general case, or for this particular example?

Because I've found myself reaching for database defaults quite often to avoid nullable fields

I think that's true here and in the general case.

Well, I don't think wanting automatically populated columns is a design issue. For example, I would absolutely loathe if I had to specify explicit primary keys upon insertion or "last modification" dates upon every write transaction – because I could then specify incorrect values. It is absolutely desirable that you would want this kind of value to be derived by the database.

It's no small task, though. The general phenomenon you are observing is that databases and client languages are hard to interoperate in a type-safe manner. This is not a fully solved problem yet, but ORMs can be a close approximation – the better ones. Using an ORM is probably your best bet if you want type-safe database access.

I don't know how good SeaORM is, though, so I can't recommend it (or against it).

3 Likes

I got the minimalist solution(not perfect but it is fine).
Source: INSERTS and UPDATES with variable fields · launchbadge/sqlx · Discussion #3436 · GitHub
In case of update query:

sqlx::query!(r#"UPDATE ingredient SET
                    creator_id = COALESCE($1, creator_id)
                    WHERE id = $2"#r, Some(2), ing_id).execute(&pool).await.unwrap();
    

In case of an insert query:
use .unwrap_or(custom_default) or .unwrap_or_default() for each value that needs it copying database scheme defaults.

I don't like moving defaults from database to rust, but it is better than adding big dependency where it is not required.

You may want to explore the option of INSERT … RETURNING, which sqlx supports as a query, so that you read back the values actually inserted.

2 Likes

This topic was automatically closed 90 days after the last reply. We invite you to open a new topic if you have further questions or comments.