Sqlx macros with optional newtypes

It seems the very strict compile time type checking of sqlx macros is making it virtually impossible to have a newtype based on a String as an optional TEXT field in Postgres. With the 'assistance' of multiple LLM's I keep running into issues where in the end the advice is to move away from the macro's due to them not being able to figure it out. Are there others here that have the same experience, or maybe even the magic sauce to make it work? I can paste the problematic code upon request, but this is more of a what-is-your-experience-with-this type of question.

Of course I can move away from the macro's but I kinda like the compile time checking bit..

Thanks!

It's kinda hard to follow what you're trying to say, so I would suggest to share your code since that would help other understand the problem.

Sure!

Here is the newtype definition:

/// the RefreshToken newtype definition
#[derive(Debug, Default, Clone, Serialize, Deserialize)]
pub struct RefreshToken(String);

/// Implement the Refresh Token type
impl RefreshToken {
 ...stuff..
}
// Implement From<String> for convenience
impl From<String> for RefreshToken {
    fn from(s: String) -> Self {
        RefreshToken(s)
    }
}

// Encode implementation for sending data to Postgres
impl Encode<'_, Postgres> for RefreshToken {
    fn encode_by_ref(&self, buf: &mut PgArgumentBuffer) -> sqlx::encode::IsNull {
        // Fully qualified call to avoid ambiguity
        <String as Encode<Postgres>>::encode_by_ref(&self.0, buf)
    }

    fn size_hint(&self) -> usize {
        <String as Encode<Postgres>>::size_hint(&self.0)
    }
}

// Decode implementation for reading data from Postgres
impl<'r> Decode<'r, Postgres> for RefreshToken {
    fn decode(value: PgValueRef<'r>) -> Result<Self, BoxDynError> {
        let s = <String as Decode<Postgres>>::decode(value)?;
        Ok(RefreshToken(s))
    }
}

// Type implementation so sqlx knows this maps to TEXT
impl Type<Postgres> for RefreshToken {
    fn type_info() -> PgTypeInfo {
        <String as Type<Postgres>>::type_info()
    }
}

Here is how it is used in a User struct:

#[derive(Debug, Default, Serialize, Deserialize, sqlx::FromRow)]
pub struct User {
    pub name: Name,
    /* lots of other fields */
    pub refresh_token: Option<RefreshToken>,
}

Here is how it is used in a sqlx (postgres) call:

/// Get a user by name
pub async fn get(pool: &PgPool, name: &Name) -> Result<Option<User>> {
    let user = sqlx::query_as!(
        User,
        r#"
        SELECT *
        FROM users
        WHERE name = $1
        "#,
        name
    )
    .fetch_one(pool)
    .await?;

    Ok(Some(user))
}

Here is the error reported on the sqlx query_as!

the trait bound `Option<RefreshToken>: From<Option<String>>` is not satisfied
the following other types implement trait `std::convert::From<T>`:
  `std::option::Option<&T>` implements `std::convert::From<&std::option::Option<T>>`
  `std::option::Option<&mut T>` implements `std::convert::From<&mut std::option::Option<T>>`
  `std::option::Option<&str>` implements `std::convert::From<chumsky::error::SimpleLabel>`
  `std::option::Option<&tracing::Id>` implements `std::convert::From<&EnteredSpan>`
  `std::option::Option<&tracing::Id>` implements `std::convert::From<&Span>`
  `std::option::Option<&tracing::Id>` implements `std::convert::From<&tracing_core::span::Current>`
  `std::option::Option<&tracing::Metadata<'_>>` implements `std::convert::From<&tracing_core::span::Current>`
  `std::option::Option<Mailbox>` implements `std::convert::From<Mailboxes>`
and 22 others
required for `std::option::Option<std::string::String>` to implement `Into<std::option::Option<RefreshToken>>`

I tried to implement a trait for an option like so:


impl From<Option<String>> for Option<RefreshToken> {
    fn from(opt: Option<String>) -> Self {
        opt.map(RefreshToken)
    }
}

But that is not possible:

only traits defined in the current crate can be implemented for types defined outside of the crate
impl doesn't have any local type before any uncovered type parameters
for more information see https://doc.rust-lang.org/reference/items/implementations.html#orphan-rules
define and implement a trait or new type instead

Any advise is greatly appreciated. My current workaround is to make the DB field not optional but an empty string.

Can you try implementing your newtype as this user did? Not sure if there's a difference between the way you implemented things and the way he did, but it seems that sqlx worked better in their case.

It is an interresting approach. I tried it but it does not solve it. That user is not using option DB fields, and that is exactly where the problem is. It seems not using the macro approach would also work. Need to try that even though I do not prefer it.

Another Option (pun intended) is to use a wrapper for the nullable version, as suggested by Jofas here.

1 Like

Yes, I saw that post as well. But I think it is going a bit too far in fighting the type system tbh. I believe the sqlx crate should provide better support for this use-case. I will file an issue and meanwhile use my workaround to avoid having optional DB fields.

I use newtype structs to wrap strings when dealing with sqlx fairly regularly, and have not had the kinds of problems you're encountering. However, I use sqlx's features for this, rather than implementing Encode and Decode myself (where possible). For example, I have:

#[derive(PartialEq, Eq, sqlx::Type)]
#[sqlx(transparent)]
pub struct Secret(String);

pub struct Tokens<'t>(&'t mut SqliteConnection);

impl Tokens<'_> {
    // Issue a new token for an existing login. The issued_at timestamp will
    // be used to control expiry, until the token is actually used.
    pub async fn issue(
        &mut self,
        user: &User,
        issued_at: &DateTime,
    ) -> Result<Secret, sqlx::Error> {
        let id = Id::generate();
        let secret = Uuid::new_v4().to_string();
        let user = user.id();

        let secret = sqlx::query_scalar!(
            r#"
                insert
                into token (id, secret, user, issued_at, last_used_at)
                values ($1, $2, $3, $4, $4)
                returning secret as "secret!: Secret"
            "#,
            id,
            secret,
            user,
            issued_at,
        )
        .fetch_one(&mut *self.0)
        .await?;

        Ok(secret)
    }

    /* ... */
}

Where a Secret is used as a query parameter, sqlx unwraps it to a String for me automatically. The underlying field is a varchar field - analogous in most ways to pg's text type, though I'm using sqlite for this project specifically.

Going the other way, and specifically addressing your nullability point, I have:

#[derive(
    Clone,
    Copy,
    Debug,
    Eq,
    Ord,
    PartialEq,
    PartialOrd,
    serde::Deserialize,
    serde::Serialize,
    sqlx::Type,
)]
#[serde(transparent)]
#[sqlx(transparent)]
pub struct Sequence(i64);

pub struct Messages<'t>(&'t mut SqliteConnection);

impl Messages<'_> {
    // ...
    pub async fn all(&mut self, resume_at: Sequence) -> Result<Vec<History>, sqlx::Error> {
        let messages = sqlx::query!(
            r#"
                select
                    message.channel as "channel: channel::Id",
                    message.sender as "sender: user::Id",
                    message.id as "id: Id",
                    message.body as "body: Body",
                    message.sent_at as "sent_at: DateTime",
                    message.sent_sequence as "sent_sequence: Sequence",
                    deleted.deleted_at as "deleted_at?: DateTime",
                    deleted.deleted_sequence as "deleted_sequence?: Sequence"
                from message
                left join message_deleted as deleted
                    using (id)
                where message.sent_sequence <= $1
                order by message.sent_sequence
            "#,
            resume_at,
        )
        .map(|row| History {
            message: Message {
                sent: Instant::new(row.sent_at, row.sent_sequence),
                channel: row.channel,
                sender: row.sender,
                id: row.id,
                body: row.body.unwrap_or_default(),
                deleted_at: row.deleted_at,
            },
            deleted: Instant::optional(row.deleted_at, row.deleted_sequence),
        })
        .fetch_all(&mut *self.0)
        .await?;

        Ok(messages)
    }
    // ...
}

Note the annotations on deleted.deleted_sequence, in particular. sqlx takes the stored integer, or null, and wraps it in a Some(Sequence(value)) if non-null, or in None if null. (You can ignore the map afterwards; the row structure doesn't perfectly match the struct I'm trying to construct, but that has no bearing on your question, I think.)

this is more of a what-is-your-experience-with-this type of question

It took a bit of doing to pull this understanding out of the sqlx documentation. I would not trust an LLM to provide any of this insight; go to the source, instead.

The relevant bits that I can recall are:

The big surprise for me was that sqlx will not infer the type a field "should have" based on what it gets assigned to, the way Rust itself sometimes will. Your select * will probably have to be select *, refresh_token as "refresh_token: RefreshToken" or similar so that sqlx has the type information it needs to wrap your token.

For completeness' sake, the querying patterns above also work fine with manual implementations of Type and friends. I have

use std::{fmt, string::String as StdString};

#[derive(Clone, Debug, Default, Eq, PartialEq, serde::Serialize, serde::Deserialize)]
#[serde(into = "StdString", from = "StdString")]
#[serde(bound = "N: Normalize")]
pub struct String<N>(StdString, N);


// Type is manually implemented so that we can implement Decode to do
// normalization on read. Implementation is otherwise based on
// `#[derive(sqlx::Type)]` with the `#[sqlx(transparent)]` attribute.
impl<DB, N> Type<DB> for String<N>
where
    DB: Database,
    StdString: Type<DB>,
{
    fn type_info() -> <DB as Database>::TypeInfo {
        <StdString as Type<DB>>::type_info()
    }

    fn compatible(ty: &<DB as Database>::TypeInfo) -> bool {
        <StdString as Type<DB>>::compatible(ty)
    }
}

impl<'r, DB, N> Decode<'r, DB> for String<N>
where
    DB: Database,
    StdString: Decode<'r, DB>,
    N: Normalize,
{
    fn decode(value: <DB as Database>::ValueRef<'r>) -> Result<Self, sqlx::error::BoxDynError> {
        let value = StdString::decode(value)?;
        Ok(Self::from(value))
    }
}

impl<'q, DB, N> Encode<'q, DB> for String<N>
where
    DB: Database,
    StdString: Encode<'q, DB>,
{
    fn encode(
        self,
        buf: &mut <DB as Database>::ArgumentBuffer<'q>,
    ) -> Result<IsNull, sqlx::error::BoxDynError> {
        let Self(value, _) = self;
        value.encode(buf)
    }

    fn encode_by_ref(
        &self,
        buf: &mut <DB as Database>::ArgumentBuffer<'q>,
    ) -> Result<IsNull, sqlx::error::BoxDynError> {
        let Self(value, _) = self;
        value.encode_by_ref(buf)
    }

    fn produces(&self) -> Option<<DB as Database>::TypeInfo> {
        let Self(value, _) = self;
        value.produces()
    }

    fn size_hint(&self) -> usize {
        let Self(value, _) = self;
        value.size_hint()
    }
}

for dealing with strings that must be in a specific normalization form, and I use it in very much the same way as the above #[derive(sqlx::Type)]-annotated structs.

2 Likes

Using refresh_token as "refresh_token: RefreshToken" actually fixes it. And you are right that the documentation actually mentions this here. Basically when using as type checking is disabled. I had to fiddle a bit with the select statement and ended up with specifying all fields specifically instead of *.

Thank you for your help!