Hey y'all. I threw together this crate that I'm calling lender-loan which is based on a similar mechanism that we have at work for our C code. Basically, it's an Arc that guarantees that the value is dropped on the same thread that it was constructed on by blocking until all other references have dropped.
The version I threw together is for the specific use case of lending a lifetime-erased & reference to other threads. Here's what it looks like:
use lender_loan::{Lender, Loan};
fn use_loan(loan: Loan<Vec<i32>>) {
assert_eq!(*loan, vec![1, 2, 3]);
}
// Create a value that will be lent to other threads.
let mut value = vec![1, 2, 3];
Lender::with(&value, |lender: &Lender<'_, Vec<i32>>| {
for _ in 0..100 {
let loan = lender.lend();
std::thread::spawn(move || use_loan(loan));
}
});
// It is safe to modify the value again; `Lender::with` blocks until all loans are dropped.
value.push(4);
I'm seeking:
Thoughts on the idea or possible use cases I didn't think of.
A quick double-check on my unsafe code. I like to think I know what I'm doing, but ya never know.
Also, has anyone seen this sort of thing before? Is there some other name for this concurrency / smart pointer primitive? I only know it by the name "lender/loan" so I didn't really know what to search for. It feels like a close cousin to yoke, but it's definitely not trying to solve the same problem.
I think that I could extend it to work with other types of "loan principals" (heh). Like Cow<'a, str> instead of &'a T for example. Probably with a very similar trait mechanism to that of yoke.
If you pass ownership of value to Lender::with then return value from that same call I think you can completely eliminate the lifetime. That should eliminate the unsafe code. Though I think you'd end up with two Arcs.
LoanInner<'a, T>: Send should only be satisfied when T: Sync, since it's logically a shared reference to T. This doesn't make a difference to the soundness of the crate, since it only releases Arc<LoanInner<'a, T>>s, and Arc requires both Send and Sync for the value to be transferred between threads.
Lender::with() fails to wait for the loans to be dropped if op() performs an unwinding panic. This can allow the original thread to invalidate the reference while it is still being used:
use lender_loan::Lender;
use std::{
panic::{self, AssertUnwindSafe},
sync::mpsc,
thread,
};
fn main() {
let (sender, receiver) = mpsc::sync_channel(0);
let s = "Hello, world!".to_owned();
panic::catch_unwind(AssertUnwindSafe(|| {
Lender::with(&s[..], |lender| {
let loan = lender.lend();
thread::spawn(move || {
receiver.recv().unwrap();
println!("{loan}"); // use-after-free
});
panic!()
});
}))
.unwrap_err();
drop(s);
sender.send(()).unwrap();
}
So it might be useful to protect the wait() behind a drop guard pattern, either through a third-party crate for that purpose (e.g., scopeguard) or a manually implemented guard type with a Drop impl (explained, e.g., here).
Thanks for taking a look @LegionMammal978 , I had totally forgotten about unwinds.
@H2CO3 this is quite similar to scoped threads, but not the same. For one, it doesn't block waiting for all spawned threads, it only blocks long enough to wait for all Loans to be dropped. Spawned threads can continue running after the Lender has proceeded. Secondly, sharing references between scoped threads requires getting the lifetimes right; this erases the lifetimes to make the code easier to write.
This makes it possible to use with, say, a thread that was spawned earlier in the program that you communicate with via a channel. You can send a Loan through the channel to that longer-lived thread, and the Lender will wait until the thread is done borrowing the value. It basically takes all the hassle out of plumbing thread scopes and lifetimes through your multi-threaded code.
I could be way off, but yoke accomplishes lifetime erasure: crate.
Succinctly, this allows one to “erase” static lifetimes and turn them into dynamic ones, similarly to how dyn allows one to “erase” static types and turn them into dynamic ones.
The motivations are different but may otherwise overlap.
I almost thought they got away without issues after the first potential problem I thought I saw didn’t turn out to be a problem AFAICT… but no, there are (almost) always soundness issues when lifetimes are involved
Edit: Aaaand… here’s a second issue, though as far as I can tell so far, it’s only exploitable on nightly.
Indeed you did. My bad. I only read the first 3/4 of your intro post before reading answers…
Notwithstanding, I look forward to reading how @steffahn found an issue. I was thinking that, just maybe, they found a way to admit safe code, without issue, in the circumstance where they “erasure” the lifetime of data borrowed from a read used to instantiate the app. So, life starts before anything else. I suspect when they need to mutate it they need to make it seem invariant with… or make it seem like the borrow no longer exists when they need to mutate the data?…
Looking at lender-loan now, not a soundness issue, but I’m noticing the Loan<T>: Clone implementation having unnecessarily restrictive bounds (i.e. requiring T: Clone).
<rant>
I am always highly suspicious when a crate is created specifically with the purpose of writing non-idiomatic code that would otherwise not pass the borrow checker, and successful compilation is achieved via unsafe instead. It is my perception that people increasingly assume the borrow checker to be a nuisance rather than the necessary condition of soundness it most frequently is.
We often say that it might be too strict (to remain politically correct, I guess), but my experience shows that even in the corner cases when this is technically true, one would still be better off trying to refactor into a style blessed by borrowck, rather than unsafely working it around, either manually or by means of a crate.
I've just seen way too many soundness holes even in older crates of such origin, and I'm starting to think that their prevalence, along with the sentiment of borrowck being an obstackle rather than a quality standard, hurts the language and its credibility in the long run.
</rant>
As long as such crates are reasonably small, well maintained, and care about fixing all soundness issues being reported, I'd consider the possibility that such crates can be created in Rust a strength not a weakness. In my view, they serve as extensions to the borrow checker, whilst heavily relying on the borrow checker, too, since their APIs are commonly containing closure arguments with HRTB bounds that are necessary for soundness.
The power of such crates is that they help strengthening the argument against individual users "defeating" the borrow checker on a case-by-case and ad-hoc (and usually not well reviewed) basis and instead encapsulate the unsafety necessary for certain constructs, in this case (yet another) form of self-referencing datatype, behind an open source library that can be reviewed and improved, etc...
Of course, you could argue: Just avoid this pattern! I think "if you really need it, please still stick to (probably) sound safe API by using such and such crate" can be a more effective argument to help getting inexperienced Rust users to stop trying to defeat the borrow checker themself (a task that would much more likely be bound to lead to desaster).
I'm much more unhappy with unmaintained instances of such crates though. In particular owning_ref which has an incredible number of known soundness holes, no active maintainers, and is wayyyyy too popular a crate. (AFAIR, a clone of it is even appearing in the source of rustc or at least some tool in the repo; I should probably look back into whether they are still reproducing all those soundness holes there, too, but that's an unrelated discussion).
For example, for the lender-loan library that this thread is about, I appreciate that the API is tiny (yet still useful), and unlike some other libraries discussed above, there are no macros involved in the API either.
I'm happy to see this discussion happening, and I welcome any and all criticism.
I'll note my motivation for this library: I'm tinkering with designing my own programming language aimed at being a "Smaller Rust" as described by withoutboats in this blog post.
One idea I had for how to implement lifetimeless, Rust-like ownership and borrowing was to use this borrowing scheme where threads are blocked until are borrows are finished. So I wrote this library with the intention of trying to use it for the runtime of my language.
I have no particular goals for this in terms of the wider Rust ecosystem or coding patterns. I see @H2CO3 's point about offering beginners ill-advised "outs" from the borrow checker. But I think I agree with @steffahn moreso when they say that the ability to design these constructs is a strength of Rust.
My other primary goal here is to learn. Please continue pointing out other "borrowck-sidestepping" libraries that you know of! TIL about owning_ref.
So, ^ those are the key benefits I see. The key drawback that I see is that it's possible (easy, even) to block your thread indefinitely by extending the lifetime of the loan:
use lazy_static::lazy_static;
use std::sync::Mutex;
type Loan = lender_loan::Loan<String>;
lazy_static! {
static ref LOAN: Mutex<Option<Loan>> = Mutex::new(None);
}
fn main {
let s = String::from("lend me");
lender_loan::Lender::with(&s, |lender| {
let loan = lender.loan();
*LOAN.lock() = Some(loan);
}); // <---- blocks forever
}
Disclaimer: I haven't run this code yet, I'm just assuming it works as I've described
Can we mark such crates on lib.rs and doc.rs? Just like all attempts to prevent UB in C hit the “but it works for me” wall sometimes people would always write unsound code in Rust, but most just honestly look on the crate which solves their problem… and they have no idea that crate is unsound, they are not digging deeply, because this would, kinda defeat the purpose of not writing an ad-hoc implementation.
Although question about soundness and “responsiveness”… what about crates like indoc? It's kinda-sorta popular crate yet soundness issue is ignored (probably because it's not important issue since it can only crash the compiler, probably couldn't do anything more)… it's hard to say what's the proper handling of all that, I guess.
An ICE is not a soundness issue. A soundness issue is when you can cause UB using safe code. An ICE is a compiler bug which should either compiler or generate an error, but it's a problem in the compiler, not in a library's code. The compiler is never supposed to crash on even the wildest and unsafest code.
That's what indoc does there if I'm not mistaken. It produces invalid String (puts invalid UTF-8 inside) and then compiler explodes. Note that it's supposed to produce (according to manual) the exact same string as if you would remove indent — but in that case there are no crash and everything works just fine.
I can not be 100% sure (because I haven't investigated further) but I'm 90% certain it's soundness issue which in this particular case haven't lead to UB, you are correct.
Does it mean that it's Ok for procmacro to create dangling pointers and broken Strings? I was under impression that compiler contract WRT to acceptance of wildest and unsafest code doesn't go quite that far.
That's definitely not OK, but it should not crash the compiler nevertheless. Even if there is a soundness issue in a crate, this should not cause the compiler to crash.
