Struggle with borrow checker at popping from list implementation

My attempt at linked list implementation in Rust hit at stumbling block of pop method implementation.

I have googled for a solution, but so far found only cheesy versions which avoid the problem by popping an element from head(front) of reversed list.

  • list's element does not implements Clone or Copy
  • order of elements in list is important, new one has to be appended at the end
  • removal of an element is from the end

There is a link to playground, with my sloppy take where it fails at borrow checker at attempt to extract return value from mutable borrow:

Can you provide your List::pop method implementation or give some hint at least how to refactor the code?

Box<&mut T> never makes any sense.

References in Rust aren't general-purpose pointers. They are temporary permissions to view data that is stored somewhere, and the scope of the loan can never be extended. There is no amount of boxing or wrangling with the types that can make a temporary reference any less temporary. Types that contain any reference anywhere become as restricted and short-lived as the reference (always limited to the worst possible case).

Option.take() and std::mem::swap()/std::mem::replace() are useful when working with non-copyable objects.

3 Likes

popping from tail requires looking ahead. but you cannot have two mut references curr and prev as loop carried variables, because prev owns curr.

possible solutions:

CAUTION:

the example code below does NOT handle edge case of popping from a list with a single element. see my follow up answer for more details. the fix is left as an exercise. :wink:

  • use a single loop variable curr, check curr.next.next for the tail node condition.

    // pop an element from list tail
    pub fn pop(&mut self) -> Option<T> {
        let mut curr = self.head.as_mut()?;
        while curr.next.as_deref_mut().and_then(|next| next.next.as_ref()).is_some() {
            curr = curr.next.as_deref_mut().unwrap();
        }
        curr.next.take().map(|node| node.value)
    }
    
  • use recursion

    impl<T> List<T> {
        pub fn pop(&mut self) -> Option<T> {
            self.head.as_mut()?.pop().map(|node| node.value)
        }
    }
    impl<T> Node<T> {
        fn pop(&mut self) -> Option<Box<Self>> {
            let Some(next) = self.next.as_deref_mut() else {
                return None;
            }
            match next.pop() {
                Some(node) => Some(node),
                None => self.next.take(),
            }
        }
    }
    
  • use unsafe raw pointers

    I'm not gonna showing unsafe examples here.

2 Likes

btw, in case you have not read it yet, it is highly recommended to read this book to learn the the borrow checker:

The simpliest solution to this would be having your List type hold a Box to the last node, and each node hold a Box to the previous one. Or, in other words, swap the concept of start and end of the list.

1 Like

Linked lists in Rust are hard. Their difficulty does not reflect well how difficult Rust is in general.

What is your goal with this project? Do you need to do this as an exercise with these exact requirements, do you have a real application you'd need this for, or are you doing it as a personal project with your own requirements?

The real solution will depend on that.

  • If this is an execise, @SkiFire13's solution of reversed list with boxes should be sufficient and simple enough.
  • If you have a real application, it's unlikely you actually want to use a linked list (as outlined in the preface of Learning Rust with Entirely Too Many Linked Lists), and if you really do, you might want to find an application-specific crate instead of rolling your own implementation.
  • If it's a personal project, I recommend against it, because linked lists and trees (and other pointer-based data structures) are particularly thorny in safe Rust. If you want to do it anyway, the Learning Rust with Entirely Too Many Linked Lists book is a good companion.

One eureka moment I had is that a linked list API (with operations that have the same time and space complexity as linked lists) isn’t too hard in Rust, only linked lists themselves are; just use a Vec for the backing storage, and use indexes into the Vec as “pointers”. By no means an original or difficult idea, of course. “Use indices instead of pointers” is a broadly applicable insight.

1 Like

Thanks for your contribution.
In your examples are not handled cases of list with a single element and empty one, but it put me in the right direction.

Use of unsafe rust would be also interesting, maybe make it all that much simpler.

Not sure if understood correctly, but how would look like traversal of the list from first(oldest) to the last(newest)?

You are correct, this is an exercise example. At learning a new language I do a routine of implementation common data structures in each one. In real world code I would probably pick something else like indexed "dynamic array". likely plain Vec.
I just didn't expected it would be so complicated in Rust.

1 Like

Simpler to compile, maybe, but also much easier to be unsound and introduce UB.

If you go that way anyway, look into testing with Miri.

2 Likes

this is indeed the case, but it's an easy fix.

fundamentally, I don't want to change the type definition in the original code, but that data definition is irregular and has some edge cases which lead to unnecessary complication in the algorithm.

typically, singly linked list is most suitable for insertion and deletion at the head, which can be trivially (and elegantly) implemented, when you have some functional progarmming experience:

// this is a "cons" cell, but I call it "node" here so non-lispers know what it is
// struct Cons<T> { car: T, cdr: List<T> }
struct Node<T> {
    value: T,
    next: List<T>,
}
// None is Nil, Some is Cons
// enum List<T> { Nil, Cons(Box<Cons<T>>) }
// need a newtype wrapper (or extension trait) if we want to implement methods for it
// the following example snippets ignore this limitation for brevity
type List<T> = Option<Box<Node<T>>>;

impl<T> List<T> {
    fn nil() -> Self { None }
    fn cons(value: T, next: Self) -> Self {
        Some(Box::new(Node { value, next }))
    }
}

and push() and pop() is trivial:

impl<T> List<T> {
    fn push(&mut self, value: T) {
        let next = self.take();
        *self = Some(Self::cons(value, next))
    }
    fn pop(&mut self) -> Option<T> {
        let node = self.take()?;
        let Node { value, next } = *node;
        *self = next;
        Some(value)
    }
}

with this definitions, even the tail insertion and deletion is very simple. it's still O(n) where n is the length of the list, but we don't need to handle those edge cases like the original code.

here's a recursive version:

impl<T> List<T>
	fn push_tail_node(&mut self, node: Box<Node<T>>) {
		match self.as_deref_mut() {
			Some(curr) => curr.next.push_tail_node(node),
			None => *self = List(Some(node)),
		}
	}
	fn pop_tail_node(&mut self) -> Option<Box<Node<T>>> {
		match self.as_deref_mut()?.next.pop_tail_node() {
			Some(node) => Some(node),
			None => self.take(),
		}
	}

	fn push_tail(&mut self, value: T) {
		let singleton = Box::new(Node {
			value,
			next: Self::nil(),
		});
		self.push_tail_node(singleton);
	}
	fn pop_tail(&mut self) -> Option<T> {
		self.pop_tail_node().map(|node| node.value)
	}
}

an iterative implementation is possible, but unfortunately, the snippet below is rejected by the current borrow checker as is, and polonius is required to compile. on stable, you can use unsafe to work around it, but polonius-the-crab is recommended.

impl<T> List<T> {
    fn push_tail(&mut self, value: T) {
        let singleton = Self::cons(value, Self::nil());
        let mut curr: &mut Self = self;
        while let Some(curr_node) = curr.as_deref() {
            curr = &mut curr_node.next;
        }
        *curr = singleton;
    }
    fn pop_tail(&mut self) -> Option<T> {
        let mut curr: &mut Self = self;
        loop {
            let curr_node = curr.as_deref_mut()?;
            if curr_node.next.is_some() {
                curr = &mut curr_node.next;
            } else {
                return Some(curr.take().unwrap().value);
            }
        }
    }
}

here's a playground link (with unsafe polonius workaround included):

2 Likes

Amazing, thanks for such comprehensive answer. Marking as the solution.

It seems you have to carefully pre-plan design of data structures in accordance how type system of Rust is designed, else it gets too complicated to build upon it. Hope it just requires more experience to avoid such pitfalls, or Rust simply isn't for my mindset :wink:

My mistake was to create a type which would require subsequently two mutable accesses at the same time - struct and its field. By separation in standalone ones this problem can be avoided.

This is an important realization!

On that point, I don't know if you've heard of VecDeque, which allows for O(1) insertion and removal on both ends as well as O(1) random access!

Maybe it's just me, but I only stumbled across it not so long ago and I was kind of mind-blown that this is possible -- I had heard about ring buffers, which is effectively what it is, but I've only seen them implemented with constant sized arrays before, but yeah, it can be dynamic too!

Anyway, tangent over.

This topic was automatically closed 90 days after the last reply. We invite you to open a new topic if you have further questions or comments.