Hi folks. While making some changes to the tokenizer for lisp-rs, I started looking around to see what crates.io had to offer in the way of support for writing tokenizers. Tokenator was kind of in the ballpark, but long story short, I decided to write my own. I think it's pretty much ready to publish as a crate, but I've never done that before and I'm sure I could benefit from having some experienced eyes look it over.
Please take a look at the doc comments in lib.rs to get a feel for what it helps you do:
While random comments in this thread might be helpful, what I'm really hoping for is someone who has published crates to do a bit of mentoring, probably totaling an hour or two in the form of an email thread over the course of a few days. I'm not new to Rust, so I don't think I need much if any help with the language itself, but I have questions about API design, Github actions, documentation (including the Readme), and so on.
Another thing that would really help is a pointer to a relatively simple, straightforward crate (or two) that follows all the best practices for Github actions, documentation, etc.
I should add that I have several other projects in progress, and am hoping to get set up with best practices up front so that I can more easily repeat the process of turning them into published crates.
Worth a try. Perhaps I'm jaded by prior experience with other forums where questions quickly fall from the top of the list into oblivion. It's actually been two days now, but it's helpful to know that you just scan the list twice a week; I just assumed that most people would sign up for a daily digest.
This could be the reason you haven't gotten responses. I read this as, "I don't really want replies here." Asking for an offline mentor is a substantial commitment. An "hour or two" of help doesn't really need to be taken offline, does it?
Can you post them here? I think you'll get good feedback if you ask your questions and have the conversation openly in this forum.
Yes, I can see where someone might read that as not wanting feedback, which isn't the case.
I did ask one specific question; I'll repeat it here:
Could you point me at a relatively simple, straightforward crate (or two) that follows all the best practices for Github actions, documentation, etc. Also module layout, API, and things I don't know to ask about. I think if I just had a few good, simple examples to follow, that would help. It would ESPECIALLY help if there were a document that enumerated such concerns and talked about how they are addressed in specific crates, but I doubt that such a document exists. Love to be proved wrong!
For what it's worth if you want to learn more about good documentation and proper usage of GitHub actions I can really recommend looking into the rustls project, the Rust version of OpenSSL.
The documentation is concise and tells you exactly what you need to know to use the library and I've personally used their approach to GitHub actions to learn about how to use actions for my own projects.
Here's an API question. When I first ran clippy on this crate, it complained about the Lexer having a next() method but not being an Iterator. Apparently clippy thinks that the method name is effectively reserved. Rename it, clippy said, or make Lexer an Iterator.
I thought about other names, but didn't like them. next() is the perfect name for it; it's exactly what you expect the name to be for a method that returns the next character from the line wrapped in an Option. So I implemented Iterator and thought well, maybe it's a good thing, because somebody who still thinks in for-loops could use one on the Lexer.
But in truth, I think anyone trying to use a Lexer as an Iterator is probably doing the wrong thing, and is likely to get confused. You could use one in a for-loop, and you could use Iterator::find() and some other methods, but I can't imagine where that would be more straightforward than using the methods I provided. And using some methods -- the ones that you usually collect() after -- would probably just end in confusion. The idea is not to gather the characters emitted by the Lexer, but rather to use the Lexer to recognize a lexeme. If you need to, you can ask it for the lexeme at the end, but you don't need to collect() the characters to get it, and if you tried, you'd probably get it wrong.
So implementing Iterator feels like it might be a net minus, and I'm facing this choice:
Continue implementing Iterator; most people won't really use it, so it won't hurt, and maybe it will be useful in a way that I haven't recognized.
Remove the Iterator impl and find a way to tell clippy to buzz off.
Remove the Iterator impl and change next() to something I don't like, to make clippy happy.
Or perhaps there is some way to alter the design to make it actually useful (and not confusing) that Lexer implement Iterator. I suspect that that's a unicorn, but I might be wrong.
Another API-ish question. The crate does not currently use the heap, either directly or indirectly, or a filesystem. At least I think it doesn't; there are no Boxes, Vecs, Strings, or the like. Let's assume I'm correct.
I can imagine this crate being useful for CLIs in embedded systems, but I don't know anything about no_std crates. Is whether or not the library uses the heap essentially all you have to worry about? I'm assuming that Iterators, flat_map, tuples, and other stuff that doesn't make use of the heap is all in the core library, right?
Would I be better off just ignoring this for now, releasing it on std and then coming back later to see about making a no_std version, or would I perhaps risk painting myself into a corner in some way if I don't build the no_std-ness in from the start?
Also, is there a good way to check whether a crate makes use of the heap? Perhaps a special allocator you could direct rustc to use which simply stops and prints a message if something tries to allocate. Or maybe just compiling it in the way you would for no_std is the easiest thing?