The way you describe works and is fine for the builder pattern. But if you create a builder and then separately (e.g., in if statements) want to change it then things get less appealing.
let t1 = Builder::new().one(1).two(2).build(); // fine
let b = Builder::new();
b = b.one(1);
b = if extra { b.two(2) } else { b };
let t2 = b.build();
Whereas right now using the &mut Self stuff I can do this:
let t1 = Builder::new().one(1).two(2).build(); // fine
let mut b = Builder::new();
b.one(1);
if extra { b.two(2) }
let t2 = b.build();
which I find more appealing.