I've always considered array indexing the biggest Rust footgun. Since 2015, when I started using Rust, I've felt that arr[i] should either return an Option<T> rather than T or require unsafe
Today I learned the hard way that string slicing may be an even bigger footgun, since it operates on byte offsets rather than character positions. I literally panic!ed how many places in my code might be afftected, since I have quite many parsers and other stuff operating on text.
What surprised me even more was how little discussion I could find about it. I found plenty of posts complaining about the inconsistency that s[1..2] is allowed while s[1] is not, but very little discussion about the existence of slicing by byte indexes itself.
I also discovered there's a clippy lint for string slicing, yet it's allowed by default.
Why is this considered an acceptable tradeoff? And why does it seem so under-discussed? Am I in the minority finding it that surprising?
Then it couldn't be a place expression (couldn't be compatible with the Index traits deref based desugaring). To wit, you must practically contain the output type, and str doesn't contain Options.
Or at a higher level, consistency with indexing in most languages.
Too common basically, plus panicking isn't UB.
"Character" (unicode scalar value, char) indexing or grapheme (closer to what a human considers a character) indexing is linear time in the offset value, instead of constant time.
(Grapheme determination also requires a lot of sizable character tables which embedded folks, for example, are opposed to putting in core. The definitions are also more dynamic than those of scalar values.)
The above reasons, AFAIK.
Core Rust aims to serve more than one master, and C-like efficiency/applicability is one of them. If you want proper Unicode handling, use unicode-segmentation or similar crates.
I don't think that slice/string indexing is an undiscussed topic. The Rust Book explains everything about UTF-8 representation in the chapter Storing UTF-8 Encoded Test with Strings, which includes explaining the trade-offs. The previous chapter on slices has following disclaimer.
Note: For the purposes of introducing slices, we are assuming ASCII only in this section; a more thorough discussion of UTF-8 handling is in the “Storing UTF-8 Encoded Text with Strings” section of Chapter 8.
I would say that those basic concepts are explained really well, and quite early on.
I find it quite interesting that someone using Rust since its first public release just discovered one of the most common complains about Rust's strings.
OTOH - considering string is sort of core/basic datatype in many languages - it is completely ommited in the Data Types - The Rust Programming Language
(e.g. a menion why it's not there)
While i as old grumpy programmer with several languages behind my belt understand the chapter ordering etc., i can see why for newcomers easily can easily miss the important difference of Rust's "elementary strings" and working with them (indexing) compared to other languages...
The one and only chapter that has word "string" is the mentioned "Storing UTF-8 Encoded Text with Strings".
This, at least for me imho puts accent on "if you come with UTF-8 string explicitly, look here", wich is quite different to "oh btw, string in Rust is actually..." that the newcomes might be missing.
In terms of safety, making all safe access of element return Option makes sense, since it will force the user to handle out of bounds cases instead of runtime panic. Tho there is different
Rust already has API that returns Option, it is var.get(index). The arr[i] only has 1 branch (checking the len), since if the index exceeds the len, it will call panic quickly from inside, no new branch generated. Where returning Option approach will have 2 branches (checking len, and the Option handling branch in user land code). Both have good parts and bad parts, so it is hard to decide for me :<. So providing both is good too, more flexible
Where the unsafe method is mandatory, because we want to be able to remove any branching check if we are sure the code is safe to make the code faster equal to C/C++ code because they by default do not has those checks. Because the compiler can not remove it if the value is not compile time value
Because the slice is a fundamental feature. Fundamental feature uses lowest level component so that it can be used to create more things. Byte can be seen as a low level data type. So this is Rust providing a view type based on low level bytes. If Rust does not provide this, then you will not be able to create thing that requires slice by byte
C++ also has it, it is also based on bytes -> .substr(start, end)
Golang also has it and byte based too -> var[start : end]
Rust -> &var[start..end]
If Rust only provides slice by char, then said slice can not be used for byte processing. But if Rust provides slice by byte, it can be used to create slice by char the one you want, byte processing, and many other things because it is a low level feature
It is very good feature too to add slice by char to the standard library. Because most high level usage requires by char not byte, like slicing user input, etc
Where returning Option approach will have 2 branches (checking len, and the Option handling branch in user land code)
unless the optimiser inlines the array access and merge the 2 branches, i do know if rust does that optimisation, but it seems reasonable to do (it could be checked checking the resulting assembly (e.g. on godbolt))
It is very good feature too to add slice by char to the standard library. Because most high level usage requires by char not byte, like slicing user input, etc
that depends on the use case, in some languages most/all characters are actually multiple chars that combine to make the actual characters used to write the language, in which case slicing individual chars can result in invalid/unreadable/unexpected characters.
so its possible (and likely) that the high-level usage actually wants graphemes, but it just so happens that graphemes and chars are equivalent in all languages the software was tested with.
(and graphemes are way out of scope for std)
for example slicing user input is not going to go well if its in korean and the start/end of your slice happens to be inside a grapheme.
The indexing-based function is easier to understand, and is thus better if the only thing you can do with Option::None is panic, or if you can prove that indexing will never panic (such as this trivial addition, returning an Option).
That said, the expect(…) variant has a slight advantage - you can grep the linked release mode binary for the string you supplied to expect, and if it appears, you know which expect was not optimized out. Tracking down why your release binary contains code for panicking on an out-of-bounds index is harder.
If rust could have place-references you could have Option<&place T>, that would quite neat, and sounds somewhat similar to the work being investigated in the in-place init effort (I think, not super well read up on that).
I can think of several other cases where this would be useful as well to make non-panicing APIs.
How common is manual string slicing outside of, say, performance-sensitive lexing-like tasks? As opposed to higher-level APIs [1] that don't leak the implementation detail that a &str is a thin wrapper around a &[u8].
In my experience manual slicing of strings is needed only when lexing/parsing and for that the current low level C-array-like API (even converting to &[u8]) works quite well. If one really needs to manage string parts (chars, words) then a higher level library that knows about UNICODE isn't just a good idea but it is absolutely required to avoid an infinite series of common errors.
code unit indexing, which is simple, fast, stable, and works great to long as you get your indices from searching the string or similar. (And if you're not searching the string, where did you get the index from?)
grapheme cluster indexing, which is expensive, harder to store, unstable between unicode versions, and still doesn't necessarily match the "characters" you see in fonts.
So rust uses the former. It really is better for real things, though yes it's harder for silly "reverse a string" tests that nobody ever actually does IRL.
(Reminder that things like Java also use code unit indexing, just on WTF-16 instead of UTF-8, so it's easier to get it wrong and not notice.)
As my standard example of how you don't want to actually index Rust's chars either (though at least they're UTF-32 instead of WTF-16), remember that
(And if you're not searching the string, where did you get the index from?)
I would say it is relatively easy to confuse char index with byte index as they are both usize. E.g. you may have i from some s.chars().enumerate() or similar and then accidentally use this usize you have in s[i..].
Of course at this point we may say that good developers don't do such mistakes but it feels to me that part of the Rust strength is avoiding exactly this kind of errors.
On the other hand, thinking more about this, I guess at this level of the language maturity, there is no realistic way to do anything with that design, even if there was a will (and there is none) to touch it.
While yes you can confuse this -- it's classic Primitive Obsession, like from a purity perspective all the slice methods should take either Fencepost or ElementIndex, not just usize -- it's also that the char index is just fundamentally not that useful in unicode. (I wonder if we should just lint on .chars().enumerate(), actually. We shouldn't make it not work, since it does have a guaranteed behaviour, but a clippy suspicious lint seems plausible.) Getting a char offset into the middle of a emoji typically isn't helpful, for example. Not can you move a char index by one to remove the i from a ffi to get a ff.
And of course str::char_indices exists to give you want you actually want.
I usually do check the assembly, but not using godbolt because it is very hard to use (unresponsive UI) in mobile, I use Rust playground and cargo-asm. I just did logical analysis based the defition ckde that I remembered without search if Rustc enable LLVM's branch merging pass
Someone already did it in the latter comment that Rustc already uses LLVM's branch merging pass, so it is good. Then no reason to keep the arr[i] logically. Not even "it is easier" reason, because arr.get(i) is equally easier, even easier because get is literal english where [..] is symbol
I was talking in literal English language by what I meant char. Like if it is "Hello World", it can split "Hello" regardless of what language, and also automatically compatible with UTF8. I didn't know the technical name for that is graphemes because English is not my main language my vocabulary is limited. What I meant is slicing by char regardless of language is added to STD
Good thing you tried that. So Rustc already uses LLVM's branch merging pass. The result should be the same whether you use unwrap, expect, or manually match because of how the branch merging works. Then arr[i] offers no benefit because if we talk about easier, arr.get(i).unwrap() is equally easier as they are plain English not symbol. More importantly no hidden panic that will getcha surprising if you don't know it's a secret panic yet
I agree with this. We generally prefer iterators over indexes, and indexing can sometimes be a performance gotcha due to the range check, so this syntactic sugar doesn't actually help since it's sugar for an operation that one shouldn't reach for by default. arr.get is sufficient when you do want indexing.
The arr[i] and arr.get[i] are not meant for use inside looping over an element. It is for a single individual access like
fn get_last_item(a: &[i32]) -> i32 {
let i = a.len();
a.get(i).unwrap()
}
You should never use that for looping contigous collection that the value is runtime value, but using iterators or combining multiple iterators. But the 2 methods above serve different purposes, outside usage inside loop, it is for directly get the value of specific index by jumping to there directly without iterating all the values 1 by 1. The arr[i] is redundant to arr.get(i), maybe older Rust just translstes if from previous language because they also have the arr[i] syntax. Then when arr.get() comes, it is already trapped in backward compatibility :<
.substr() is a slice access, to retrieve value between start and end. But .get() is for individual access, so naming it .susbstr() does not make sense
emoji typically isn't helpful, for example. Not can you move a char index by one to remove the i from a ffi to get a ff.