Explicit lifetime borrows indefinitely

After having reviewed this thread from 2015, I wonder if anyone has ever found a solution to this problem. Have you? Anyone?

The problem I'm running into is that I need to preserve an iterator that points to the last entry in a singly linked list to avoid the self.last() iterator function from having to iterate over the whole list just to find the last entry. Passing the explicit lifetime 'a to the mutable self reference in my setter methods causes the self pointer to never leave scope in those 3 methods thus never allowing the mutable handle to be freed. If I use the self.iter() method to create the iterator from the default lifetime, the compiler assumes its lifetime is too short and cannot create the relatively fixed iterator.

I'm open to ideas.

I'm not sure what your reason to use a singly linked list is, but they're quite inefficient. In addition it's also causing your issues here.

You see it's not really possible to indefinitely keep a borrow to any element in the list in a safe manner, because borrows aren't pointers (even though they compile to pointers). This is because of the lifetimes involved. And even if you could keep a borrow, it would effectively lock it in read-only mode because it would have to be a shared borrow.

So what's the solution?
If you can, switch away from a linked list to a Vec<T>. With that you can keep an index to e.g. the last element, rather than a borrow to it. The advantages there are that it is much more cache efficient than a linked list, and also that the index, being a usize*, is Copy and thus can be shared without any restrictions.
Or you can just not store any the index, since Vecs have a `.last() method IIRC.

*although I encourage you to newtype that usize index, for reasons of type safety.

I'm not totally clear on your issue without seeing your code, but I had no problem implementing a "Last Optimized Iter", providing I could initialize it. The issue that gave me trouble was holding onto the "last" pointer inside the list object, and to solve that I had to use Rc pointers.

Here is the code I quickly threw together

struct List {
    head : Option<Box<ListElement>>,
    //last : & Option<Box<ListElement>> //Can't have self-referential structures without smart-pointers
}
struct ListElement {
    element : u64,
    next : Option<Box<ListElement>>
}

impl List {
    fn new(first_element : u64) -> Self {
    
        let head = ListElement{
            element : first_element,
            next : None
        };
        
        Self {head : Some(Box::new(head))}
    }
}

struct LastOptimizedIter<'a> {
    current : &'a Option<Box<ListElement>>,
    last : &'a Option<Box<ListElement>>,
}

impl <'a>LastOptimizedIter<'a> {
    fn new(list : &'a List) -> Self {
        LastOptimizedIter {
            current : &list.head,
            last : &list.head //NOTE: This should be a last ptr stored with the list
        }
    }
}

impl <'a>Iterator for LastOptimizedIter<'a> {
    type Item=u64;
    
    fn next(&mut self) -> Option<u64> {
        if let Some(current) = self.current {
            let return_val = current.element;
            self.current = &current.next;
            Some(return_val)
        } else {
            None
        }
    }
    
    fn last(self) -> Option<u64> {
        if let Some(last) = self.last {
            Some(last.element)
        } else {
            None
        }
    }
}

fn main() {
    
    let list = List::new(1234);
    
    let the_iter = LastOptimizedIter::new(&list);
    for element in the_iter {
        println!("{}", element);
    }
    
    let the_iter = LastOptimizedIter::new(&list);
    println!("{}", the_iter.last().unwrap())
}

And here is the code with Rc instead of Box, and a List::push() function

use std::rc::Rc;
use core::cell::RefCell;

struct List {
    head : Option<Rc<RefCell<ListElement>>>,
    last : Option<Rc<RefCell<ListElement>>>
}
struct ListElement {
    element : u64,
    next : Option<Rc<RefCell<ListElement>>>
}

impl List {
    fn new(first_element : u64) -> Self {
    
        let head = ListElement{
            element : first_element,
            next : None
        };
        
        let head_ptr = Rc::new(RefCell::new(head));
        Self {
            head : Some(head_ptr.clone()),
            last : Some(head_ptr)
        }
    }
    
    fn push(&mut self, element : u64) {
        let new_element = ListElement{
            element : element,
            next : None
        };
        
        let new_element_ptr = Rc::new(RefCell::new(new_element));
        if let Some(current_last) = &self.last {
            current_last.borrow_mut().next = Some(new_element_ptr.clone());
        }
        self.last = Some(new_element_ptr);
    }
}

struct LastOptimizedIter {
    current : Option<Rc<RefCell<ListElement>>>,
    last : Option<Rc<RefCell<ListElement>>>,
}

impl LastOptimizedIter {
    fn new(list : &List) -> Self {
        LastOptimizedIter {
            current : list.head.clone(),
            last : list.last.clone()
        }
    }
}

impl Iterator for LastOptimizedIter {
    type Item=u64;
    
    fn next(&mut self) -> Option<u64> {
        if let Some(current) = self.current.clone() {
            let return_val = current.borrow().element;
            self.current = current.borrow().next.clone();
            Some(return_val)
        } else {
            None
        }
    }
    
    fn last(self) -> Option<u64> {
        if let Some(last) = self.last {
            Some(last.borrow().element)
        } else {
            None
        }
    }
}

fn main() {
    
    let mut list = List::new(1234);
    list.push(5678);
    
    let the_iter = LastOptimizedIter::new(&list);
    for element in the_iter {
        println!("{}", element);
    }
    
    let the_iter = LastOptimizedIter::new(&list);
    println!("{}", the_iter.last().unwrap())
}

Anyway, as @jjpe said, using the built-in containers is always better if possible.

I also got a lot out of reading Introduction - Learning Rust With Entirely Too Many Linked Lists

1 Like

Thanks! I saw a 17 year old guy was teaching himself programming on GitHub and wanted to give him a tip of how to implement a queue based on his existing singly linked list code but to be efficient, the enqueue and dequeue operations both need to be constant time. He has already written a queue backed by a vector also so I thought I'd send him a pull request with a more efficient linked list and queue in one. I didn't think about the reference counted borrowing so that should be about perfect!

Thanks again!

One aspect of Vec that might not be common knowledge is that removing from its front (or Indeed any other position than the last) is O(n) i.e. with a Vec, dequeueing is going to be linear time, not constant time.

However, if amortized O(1) is good enough then you might want to check out VecDeque and replace the Vec with that.

Also check out the tradeoffs listed in the module documentation.

I'm definitely not going to use a Vec. A singly linked list uses exactly the same amount of memory as a Vec per item plus it doesn't need to reallocate a statically allocated part or risk overflowing by leaving the Vec the same size. I want it totally dynamically allocated and a Singly Linked list is ideal for the job if the last node iterator is cached.

Speaking of iterators, I've finally run into a similar issue to what I ran into last time. I started over from scratch but the lifetime of the iterators are outlasting deleted nodes and the compiler knows it. I'm not using a ref-counted RefCell but just a regular Cell contained in a 3-way enum that lets me special-case the end node (unlike the option type). I may start another thread if needed.

Thanks for trying though.

Vec<u64> uses only 8 bytes per element, while the code above uses 40 bytes per element. (ListElement is 16 bytes wide; RefCell<ListElement> adds an additional 8 bytes, and Rc adds an additional 16 bytes.)

If you could do without the reference-counting types, then you could make a singly-linked list with only one word of overhead per element.

Does the Cell structure take more memory or less memory than a RefCell? The Vec also uses a reference per item plus extra to avoid a reallocation. Does it use the Option type to indicate an empty Vec or is that just a return type? I guess I need to look up the overhead of these internal structures.

Cell has no memory overhead, RefCell has some (1 usize I think).

A Vec does not keep a reference to each item, they are stored inline, packed one after the other. A Vec will sometimes have extra capacity, but probably less than half at any given time. An empty Vec is just a Vec with a length of 0.

Ok. Then the only overhead of a VecDeque are the insert and remove iterators and the expensive grow operation. Thanks. I suppose that due to the underlying structure of a Vec being an array, the cache characteristics are going to be preferable.

I haven't posted my code yet. You're referring to somebody else's example. The overhead in my code is one reference per node using a Cell structure plus metadata.

Oops, thanks for the correction.

1 Like

I just posted my code to GitHub here if anyone wants to look at it. I don't know if I can get it to work at all. The delete function is too complex as it is and there's a bug in the insert method at the end that I just noticed but it looks like it may be a waste of time to finish it. The fact that items may be deleted at random from the middle of a list makes linked lists almost impossible to implement without unsafes. The iterators cannot be guaranteed to have sufficiently long lifetime to actually point to anything in a multithreaded environment.

This topic was automatically closed 90 days after the last reply. We invite you to open a new topic if you have further questions or comments.