Meaning of lifetime errors with generic sqlx function?

Here is a simple function that uses sqlx to update one field in one row of a SQLite database. It works.

struct MyDatabase {
    conn: SqliteConnection,
    rt: tokio::runtime::Runtime
}
fn update_field_plain(db: &mut MyDatabase, id: &str, column_name: &str, arg: i64) {
    let task = async {
        let sql = format!("update cards set {} = $1 where id = $2", column_name);
        let q = sqlx::query(&sql);
        q.bind(arg).bind(id).execute(&mut db.conn).await
    };
    let res = db.rt.block_on(task);
    println!("update_field_plain, res = {res:?}");
}

Now, I think: I'd like to make this function work for any column type, so I make it generic:

fn update_field<T>(db: &mut MyDatabase, id: &str, column_name: &str, arg: T) { ... }

I understand why this is not enough. T has requirements. The compiler complains:

      q.bind(arg).bind(id).execute(&mut db.conn).await
        ---- ^^^ the trait `sqlx::Encode<'_, _>` is not implemented for `T`

And now I get lost. For a simple trait, I know to say where T: Trait or <T: Trait>, but there are multiple traits here, and a lifetime parameter, whose meaning I don't understand. The Rust Book says that lifetime parameters on a struct means that the struct can't outlive the contained references. Eg:

struct Thing<'a> {
    some_ref: &'a str
}

But I don't remember anything about lifetimes on traits, and here, I imagine that an instance of Encode doesn't "contain" a reference to anything. It's just something like i64, String, or &str. I suppose the latter shows that it could be a reference. Still, I'm not sure what the lifetime parameter on Encode means.

I end up with this, but I'm flailing, and it doesn't compile.

fn update_field<'q, T>(db: &mut MyDatabase, id: &str, column_name: &str, arg: T)
    where T: sqlx::Encode<'q, Sqlite> + sqlx::Type<Sqlite>
{
    // The body here is the same as the non-generic fn.
    let task = async {
        let sql = format!("update cards set {} = $1 where id = $2", column_name);
        let q = sqlx::query(&sql);
        q.bind(arg).bind(id).execute(&mut db.conn).await
    };
    let res = db.rt.block_on(task);
    println!("update_field, r = {res:?}");
}

Error message, which I don't understand.

error[E0597]: `sql` does not live long enough
   --> learning/sqlite/src/main.rs:325:29
    |
320 | fn update_field<'q, T>(db: &mut MyDatabase, id: &str, column_name: &str, arg: T)
    |                 -- lifetime `'q` defined here
...
324 |         let sql = format!("update cards set {} = $1 where id = $2", column_name);
    |             --- binding `sql` declared here
325 |         let q = sqlx::query(&sql);
    |                 ------------^^^^-
    |                 |           |
    |                 |           borrowed value does not live long enough
    |                 argument requires that `sql` is borrowed for `'q`
326 |         q.bind(arg).bind(id).execute(&mut db.conn).await
327 |     };
    |     - `sql` dropped here while still borrowed

What is this lifetime that is introduced by the Encode generic bound? And what do I need to do satisfy it and get the function to compile?

~ Rob

It's a parameter to some genetic associated type in the crate's trait system, in this case.

I'd have to get the know the crate better to say much more concrete. They're being generic over borrowing in some sense.

The fix is probably a higher ranked trait bound:

fn update_field<T>(db: &mut MyDatabase, id: &str, column_name: &str, arg: T)
    where for<'q> T: sqlx::Encode<'q, Sqlite> + sqlx::Type<Sqlite>
{

(But I'm afraid I can't confirm that this minute, so correct me if I'm wrong.)

HRTBs let you require bounds involving lifetimes shorter than the function body, like local borrows. Lifetime parameters on functions are chosen by the caller, and always longer than the function body.

1 Like

Wow, that worked. Now I need to read those docs to figure out why. Thanks!

Spoke too soon. It did compile and run when called with an i64, but not with a &str, even though the sqlx crate has an impl for Encode for &str

update_field(db, id, "foo", 33);
update_field(db, id, "bar", "barval");

The error...

error[E0277]: the trait bound `for<'q> &str: sqlx::Encode<'q, Sqlite>` is not satisfied
   --> learning/sqlite/src/main.rs:342:33
    |
342 |     update_field(db, id, "bar", "barval");
    |     ------------                ^^^^^^^^ the trait `for<'q> sqlx::Encode<'q, Sqlite>` is not implemented for `&str`
    |     |
    |     required by a bound introduced by this call
    |

To begin, consider what a T: Encode<'q, DB> bound means. It means that T can be encoded into a database of type DB[1], which is actually a relatively simple meaning, letting types customize their encoding logic for each database.

But what does 'q mean? If you look at where it's used in the Encode trait, you see that it is only used in the form <DB as Database>::ArgumentBuffer<'q>. Database::ArgumentBuffer is (quote)

The concrete type used as a buffer for arguments while encoding.

'q is thus likely the lifetime of that buffer, or the data stored in it. Let's look closer at a concrete example of an ArgumentBuffer<'q> using that lifetime: <Sqlite as Database>::ArgumentBuffer<'q>[2], or Vec<SqliteArgumentValue<'q>>.

SqliteArgumentValue<'q> is an enum, with two variants using that lifetime: Text(Cow<'q, str>), and Blob(Cow<'q, [u8]>). This shows that a SqliteArgumentValue<'q> can contain temporarily borrowed data, which need only exist for 'q. For instance, you could encode a temporary &'q str into that buffer. But thanks to Cow, you could also encode an owned String.

Now we can work our way back up the abstraction: A Database::ArgumentBuffer<'q> means that the data encoded into that buffer must live for at least 'q (or be an owned value).

Finally back at Encode, a T: Encode<'q, DB> means that T can be encoded into a database DB, if the provided argument buffer lives for at least 'q. Essentially, a &'short T can't be encoded into an ArgumentBuffer<'long>, because the 'short borrow of the T will end before the 'long lifetime of the buffer.

Types like i32 work for any lifetime of argument buffer, because they are copied into the buffer and become owned[3], lasting as long as they need to. See impl<'q> Encode<'q, Sqlite> for i32 {}, where i32 can be encoded into any lifetime of ArgumentBuffer.

However, a &'q str can only be encoded into a buffer that lives for 'q[4], shown in the trait impl<'q> Encode<'q, Sqlite> for &'q str {}. Note the difference between this and the impl for i32. The i32 is copied into the buffer, while the borrow of the string is kept in the buffer, so the string data must live for longer than the buffer.

The solution quinedot suggested requires the type T to be encodable into any, arbitrarily short, lifetime of buffer. This rules out any temporary &'a str references being used.

The simple solution is to pass owned Strings into the function.

But. There's more. Your original code that doesn't work is this:

fn update_field<'q, T>(db: &mut MyDatabase, id: &str, column_name: &str, arg: T)
    where T: sqlx::Encode<'q, Sqlite> + sqlx::Type<Sqlite>
{
    let task = async {
        let sql = format!("update cards set {} = $1 where id = $2", column_name);
        let q = sqlx::query(&sql);
        q.bind(arg).bind(id).execute(&mut db.conn).await
    };
    let res = db.rt.block_on(task);
    println!("update_field, r = {res:?}");
}

There are a lot of lifetimes involved in sqlx here, but here's what you need to know.
The signature of the query function is like this: fn query(&'q str) -> Query<'q, DB, DB::Arguments<'q>>. This means that the query string sql must live for the lifetime we will denote as 'q. The bind function is where it finally breaks down.

fn bind(
    self: Query<'q, DB, DB::Arguments<'q>>,
    value: T
) -> Self
where
    T: 'q + Encode<'q, DB> + Type<DB>

Note that the type (and thus lifetime) of the Query does not change. The type T must implement Encode<'q, _>, which as we established before, means it can be written into a buffer of lifetime 'q.

But thanks to the lifetimes on sqlx::query, the local sql variable representing your query must live for whatever 'q is, which is an arbitrary lifetime chosen by the caller of update_field (and is necessarily longer than that function call. The local variable can't live longer than the function).

TLDR: The HRTB for<'q> T: sqlx::Encode<'q, _> is correct. For the function to be correct, T must implement Encode for the same lifetime 'q that the Query<'q, _, _> lives, and thus the same lifetime that the local variable sql in fn update_field lives. Because the 'q in Encode<'q, _> cannot be shortened (something about variance?[5]), T must implement Encode<'q, _> for some arbitrarily short lifetime 'q, so it must be any lifetime 'q, or for<'q> (for all values of lifetime 'q).
Just make your strings owned values (or maybe 'static constant strings will work?).

Anyway, thanks for sending me onto this deep dive into sqlx and how it uses lifetimes.[6]


  1. Database implementors include Postgres, MySql, and Sqlite. They are unit structs that describe which database is used. ↩︎

  2. Sorry about the link not going straight to it, I couldn't find a rustdoc anchor for it ↩︎

  3. in SqliteArgumentValue::Int, for instance ↩︎

  4. exactly equal lifetimes, because 'q in Encode<'q, _> is invariant ↩︎

  5. Edit: Parameters like 'q in traits are always invariant. ↩︎

  6. I really shouldn't have written all this on my phone though ↩︎

2 Likes

Yeah, that implementation doesn't meet the bound, because &'q str implements the trait for one specific lifetime 'q.[1] They probably should have...

impl<'q, 'at_least_q: 'q> Encode<'q, Sqlite> for &'at_least_q str

...assuming that's possible (I don't know the crate well enough to be certain). Then &'static str would meet the bounds.

If you can make your sql a literal instead of a String, you could...

fn update_field<'q, T>(db: &mut MyDatabase, id: &'q str, column_name: &'q str, arg: T)
    where T: 'q + Encode<'q, Sqlite> + sqlx::Type<Sqlite>

as the query function is connecting the lifetime on the sql parameter to the lifetime of the Encode implementation. (A literal "str" is a &'static str that can coerce to a &'q str or any other lifetime.)

N.b. I can't tell you what's canonical/idiomatic with this crate, but the suggestion to use String instead of literals when passing to your function should work if nothing else does.


  1. For other databases, the implementation for &str ignores the lifetime, and presumably just makes it a String. ↩︎

1 Like

Trait parameters are always invariant, regardless of how they're used; trait bounds don't care about variance at all, so T: Encode<'q, T> never means "'q or longer"[1] (though a general enough implementation may make it appear to be the case sometimes).

Variance elsewhere in the crate is probably the determining factor as to whether my suggested more general implementation would fly or not though.

Ouch! :sweat_smile: Kudos.


  1. or shorter ↩︎

1 Like

It should be possible, considering the body of that impl only uses it for getting a &'q str out of it:

impl<'q> Encode<'q, Sqlite> for &'q str {
    fn encode_by_ref(
        &self,
        args: &mut Vec<SqliteArgumentValue<'q>>,
    ) -> Result<IsNull, BoxDynError> {
        args.push(SqliteArgumentValue::Text(Cow::Borrowed(*self)));

        Ok(IsNull::No)
    }
}

[1]

I think my mental model of the code was thinking that the "more general" impl discussed above was implemented, because it logically makes since to exist. (Since &'short str is a subtype of &'long str and all that).

And thank you for correcting me on the variance, I don't really understand how it works that well.

[2]


  1. I should probably make a PR to fix that (even though this question is the most I've ever used sqlx) ↩︎

  2. Sometimes I get hyperfixated on URLO questions when I should really be going to sleep... :sweat_smile: ↩︎

1 Like

Thank you! This seems like a good example for really trying to understand lifetimes. I'm still studying it...

When lifetimes are introduced in the Rust Book, here, their longest example function takes two &str parameters, and while they have the same lifetime parameter, 'a, the actual arguments don't need to have the same lifetime. As explained in that chapter, Rust will use the shorter lifetime for 'a. Is that because there is covariance there, but not here with the 'q?

I'm wondering how anything could ever work then with invariance. Almost every variable has a slightly different lifetime, so if the sqlx API forces both the query and the arg to have the same lifetime, because they both get mapped to 'q ... that seems like it would almost never work. Maybe I should report a github issue for the crate?

Although it's pretty typical, I'm not a fan of the approach the book takes to introducing lifetimes. Rust lifetimes aren't directly tied to lexical scopes for one, but that chapter can't stop talking about scopes.

But just ranting about it isn't going to help you. Let me try to clarify some points instead.

Rust lifetimes -- those '_ things -- are not representative of the liveness scope of a variable. It's an unfortunate overlap in terminology. Rust lifetimes in types are primarily about the duration of borrows. The connection with liveness scopes is that a variable cannot be borrowed when it goes out of scope. There are other uses of places[1] which are incompatible with being borrowed, such as being overwritten or having a &mut _ taken to that place.[2] Going out of scope is just another type of use.

When you take a reference to a variable, the lifetime in the type of the reference doesn't change the liveness scope of the variable. The lifetime is the duration of the borrow. The borrow checker doesn't change the semantics of programs, it just gives them a pass-or-fail verdict.

The way I would explain the longest function is that the referents of both inputs must remain borrowed for as long as the returned value is used. The arguments will have the same lifetime -- the same Rust lifetime, those '_ things -- by the time the function is notionally invoked. The arguments you write at the call site might have longer lifetimes in their types, such as a &'static str. In that case, the values are coerced to a new type at the call site. Shared references implement Copy,[3] so the variable or other expression you wrote at the call site doesn't have to have the exact same lifetime in its type. That's covariance at work. The borrow checker still knows the lifetimes are related, which is how it ensures that the referents remain borrowed so long as the result is in use.

Generic lifetimes parameters of functions are always longer than the function body. It's easy to see why in the case of longest: the inputs have to be usable throughout the function body, and also the return type must be usable after the function returns.

Additionally, one cannot borrow local variables for longer than the function body, because they are moved or go out of scope by the end of the function body. Thus one cannot borrow local variables for the duration of any such lifetime. That's basically what this thread is about -- trying to borrow sql for 'q -- and that's what higher-ranked bounds are for: asserting things about lifetimes shorter than can be named (by asserting things about all lifetimes).


An issue or PR to do the equivalent of

// Implement for all `&str` which can coerce to `&'q str`
impl<'q, 'at_least_q: 'q> Encode<'q, Sqlite> for &'at_least_q str

for every lifetime-bearing implementation that could support it, would AFAICT be an improvement to the crate. A motivating example is this function:

struct MyDatabase {
    conn: SqliteConnection,
    rt: tokio::runtime::Runtime
}

fn update<T>(db: &mut MyDatabase, id: &str, arg: T)
    where T: for<'q> Encode<'q, Sqlite> + Type<Sqlite>,
{
    let sql: String = String::new();
    let query = query(&sql);
    let task = async { query.bind(id).bind(arg).execute(&mut db.conn).await };
    let res = db.rt.block_on(task);
}

Which should start working with T = &'static str, whereas today it cannot.

A ticket against sqlx isn't the ideal place to learn about lifetimes though.


  1. such as variables ↩︎

  2. i.e. creating a new exclusive borrow of that place ↩︎

  3. and &muts have a mechanism called reborrowing ↩︎

3 Likes

I need time to chew on this and try some code before I know what questions to ask, but for starters...

Do you have any reading material you recommend for a good/deep understanding of the Rust type system - in particular, these lifetimes? I've studied many languages over the years. I would read academic papers if some of these concepts originated there.

So for example...

let a: i32 = 123;    
let b: &i32 = &a;
...

I "take a reference" on the second line. Is the "lifetime", aka the "duration of the borrow", the same as the lifetime of the value b in the program - ie, it's path from initialization, to getting dropped, wherever that is?

Unfortunately (or maybe fortunately?) these were designed by Rust team. There are thesis on them… but it's based on Rust.

Rust was started as Technology from the past come to save the future from itself and most if things that Rust used were developed elsewhere… but lifetimes are very much an exception.

1 Like

I don't have any slam dunk resources on the topic, unfortunately. I'll throw out some recommendations though.

Before I do, I will say that understanding bounds, understanding lifetimes in function APIs, and understanding borrows within a function can all somewhat be tackled separately. So the answer also sort of depends on what you want to focus on.


I have this write-up on some basics and common lifetime trip-ups. It's post-book level, like you sorta get lifetimes, but it's still not that advanced. It focuses mainly on APIs I would say, and more practical than theoretical. Some big holes are that I only give a selection of examples about borrows within a function and I don't really cover higher-ranked anything. The meat starts in the "get a feel for" sections.


The best I have to offer on trait solving (higher-ranked or not) is the compiler dev guide. There's probably good material on more of these topics in the guide too; I haven't read all of it. A related topic is coherence/orphan rules/overlap checks, which you can read about here and here.

Here's a more advanced practical article on some current shortcomings of higher-ranked trait bounds and generic associated types.

I do have a short introduction to higher-ranked types over here. All my attempts at a dedicated guide so far have ran into some sort of fractal complexity, like is explored in the article above, and died. Maybe someday I'll accept that it just needs to be incomplete and put something out there.


For borrows within a function, I think it's useful to know the general outline of the borrow checker. Birds eye view:

  • Determine the kind (shared, exclusive) and duration of borrows of places[1]
  • Check every use of every place to see if it conflicts with any active borrows[2][3]

Or in slightly more detail:

  • Look at the uses (liveness) of values whose types contain lifetimes (those '_ things) to determine points where lifetimes must be valid
  • Use that and lifetime constraints[4] to figure out the duration[5] of the lifetimes
  • Associate the borrow of every place with a lifetime and calculate their duration too[6]
  • Check every use of every place to see if it conflicts with any active borrows

A lot of the procedure involves data flow analysis.

I think that high-level understanding can get you a long ways when paired with enough examples. That is to say, I think there's diminishing returns in knowing exactly what the borrow checker does. Actually walking through the algorithms -- e.g. proving why something is accepted -- is a long and tedious process.[7] But I'll provide some resources anyway since you asked.[8]

You can read summaries of the next-gen borrow checker here:[9]

I think these are more approachable than the NLL RFC, even though we don't have Polonius yet. Here's the NLL RFC. It's roughly what we have today. Note however that we didn't actually get location-sensitive lifetimes (put off until Polonius) and so Problem Case #3 is still a thing today.

It's pretty dense and spends a lot of time on specific examples or refinements, so I'll try to highlight some key parts (IMO):


Regarding the book

I just reread that chapter and I would say the main disconnect between the presentation and reality is that it strongly conflates variable liveness and scope with lifetimes. Variable scopes aren't actually assigned Rust lifetimes ('_ things) by the compiler. Their lifetime annotations in the diagram of two variable scopes doesn't actually correspond to any lifetime in the analysis. They refer to the "lifetimes" (liveness scopes) of variables whose types do not have lifetimes, such as Strings, as if they borrow checker is assigning lifetimes based on that scope. But it does not.

Almost all of the borrow check error examples in the chapter revolve around manipulating some inner scope, which corresponds to their assertion that it's mainly about avoiding dangling references. But borrow checking is about more than that, and all their examples can be replaced with ones that don't involve going out of scope. The actual mechanism in the borrow checker that triggers in their examples is: Going out of scope is a use of a variable, and that use conflicts with being borrowed.

You can get rid of the inner scope in all of their examples by replacing going out of scope with

  • moving (if the type is not Copy)
  • overwriting
  • taking a &mut _

and the errors will remain,[11] as those are all uses of a place incompatible with being borrowed. I find "conflicting use" to be a much more general and accurate understanding than "out of scope", especially when reborrowing through references gets involved (which isn't covered in that chapter).

For their longest example, the compiler doesn't "pick the shortest [variable liveness scope]" or "pick the shortest input lifetime" either.[12] Their presentation is sort of backwards on that point. The API says that the referents of the inputs stay borrowed while the result is in use. Back at the call site, the compiler looks at where the result is used to figure out how long the [referents of the] inputs are borrowed. That's the opposite of deciding the lifetime [in the type] of the output based on the inputs!

The Rust lifetime in a reference type and the liveness scope of the referent are different things.

Ultimately the authors are trying to give readers a very general gist about Rust lifetimes and borrowing by concentrating on scopes and dangling pointers. I get that some general gist approach is appropriate here, but I just can't bring myself to accept their take. I've seen too many people confused over the conflation of liveness scopes and Rust lifetimes, and about the relative importance of lexical scopes.

I don't know what the best approach would be for this scenario, but believe it needs to distinguish lifetimes and liveness scope (like the NLL RFC does) and can't be centered solely on going out of scope / dangling references.


  1. variables, fields, dereferences, temporaries... ↩︎

  2. going out of scope is one such use, but not the only such use ↩︎

  3. some borrows can coexist with some uses, like copying something that is shared-borrowed ↩︎

  4. annotations, reborrowing constraints, ... ↩︎

  5. control flow graph region, technically ↩︎

  6. a borrow can be shorter than the associated lifetime when, for example, a reference gets overwritten ↩︎

  7. One can often explain errors with good-enough approximations, so it's not quite as painful. ↩︎

  8. It's not like I can blame anyone for giving it a crack anyway, afterall... ↩︎

  9. listed chronologically, which might not be the best order to read them, not sure ↩︎

  10. the only officialish documentation on reborrowing I'm aware of (!); well, outside the dev guide presumably ↩︎

  11. though the presentation will change ↩︎

  12. there's only one input lifetime... ↩︎

4 Likes