Wanted: a bit of mentoring for my first crate

Hi folks. While making some changes to the tokenizer for lisp-rs, I started looking around to see what crates.io had to offer in the way of support for writing tokenizers. Tokenator was kind of in the ballpark, but long story short, I decided to write my own. I think it's pretty much ready to publish as a crate, but I've never done that before and I'm sure I could benefit from having some experienced eyes look it over.

Please take a look at the doc comments in lib.rs to get a feel for what it helps you do:

While random comments in this thread might be helpful, what I'm really hoping for is someone who has published crates to do a bit of mentoring, probably totaling an hour or two in the form of an email thread over the course of a few days. I'm not new to Rust, so I don't think I need much if any help with the language itself, but I have questions about API design, Github actions, documentation (including the Readme), and so on.

Another thing that would really help is a pointer to a relatively simple, straightforward crate (or two) that follows all the best practices for Github actions, documentation, etc.

Thanks for any help.

I should add that I have several other projects in progress, and am hoping to get set up with best practices up front so that I can more easily repeat the process of turning them into published crates.

I am in roughly the same boat as you, and if it's not too presumptuous/greedy, I'd appreciate it if you could post a writeup of whatever you learn once you've learned it.

I'm going to list a few resources folks interested in this topic might find useful:

The Rust Book has a chapter on publishing a crate:
https://doc.rust-lang.org/book/ch14-02-publishing-to-crates-io.html

And the Cargo Book has its own chapter on the same topic:
https://doc.rust-lang.org/cargo/reference/publishing.html

And here is an apparently official document on Rust API Guidelines:

There is a tool called cargo-generate that will create a new project for you from a template:

As suggested on this page:

I created my repo with this template:

Among other things, that set up some Github actions for me that, for example, run clippy when I push to the main branch. Here's a blog post by the author on cargo generate and the template:

Bogdan of Let's Get Rusty did a video on publishing crates back in 2021:

Clearly this didn't work. :slight_smile: Is there a better forum for such a request, or is this just not something anyone would want to do?

I suggest waiting more than one day. While I can't speak for others, I visit this forum about twice a week. I suspect I'm not alone.

Worth a try. Perhaps I'm jaded by prior experience with other forums where questions quickly fall from the top of the list into oblivion. :slight_smile: It's actually been two days now, but it's helpful to know that you just scan the list twice a week; I just assumed that most people would sign up for a daily digest.

This could be the reason you haven't gotten responses. I read this as, "I don't really want replies here." Asking for an offline mentor is a substantial commitment. An "hour or two" of help doesn't really need to be taken offline, does it?

Can you post them here? I think you'll get good feedback if you ask your questions and have the conversation openly in this forum.

3 Likes

Yes, I can see where someone might read that as not wanting feedback, which isn't the case.

I did ask one specific question; I'll repeat it here:

Could you point me at a relatively simple, straightforward crate (or two) that follows all the best practices for Github actions, documentation, etc. Also module layout, API, and things I don't know to ask about. :slight_smile: I think if I just had a few good, simple examples to follow, that would help. It would ESPECIALLY help if there were a document that enumerated such concerns and talked about how they are addressed in specific crates, but I doubt that such a document exists. Love to be proved wrong!

I'll post other questions in separate replies.

For what it's worth if you want to learn more about good documentation and proper usage of GitHub actions I can really recommend looking into the rustls project, the Rust version of OpenSSL.
The documentation is concise and tells you exactly what you need to know to use the library and I've personally used their approach to GitHub actions to learn about how to use actions for my own projects.

2 Likes

Here's an API question. When I first ran clippy on this crate, it complained about the Lexer having a next() method but not being an Iterator. Apparently clippy thinks that the method name is effectively reserved. Rename it, clippy said, or make Lexer an Iterator.

I thought about other names, but didn't like them. next() is the perfect name for it; it's exactly what you expect the name to be for a method that returns the next character from the line wrapped in an Option. So I implemented Iterator and thought well, maybe it's a good thing, because somebody who still thinks in for-loops could use one on the Lexer.

But in truth, I think anyone trying to use a Lexer as an Iterator is probably doing the wrong thing, and is likely to get confused. You could use one in a for-loop, and you could use Iterator::find() and some other methods, but I can't imagine where that would be more straightforward than using the methods I provided. And using some methods -- the ones that you usually collect() after -- would probably just end in confusion. The idea is not to gather the characters emitted by the Lexer, but rather to use the Lexer to recognize a lexeme. If you need to, you can ask it for the lexeme at the end, but you don't need to collect() the characters to get it, and if you tried, you'd probably get it wrong.

So implementing Iterator feels like it might be a net minus, and I'm facing this choice:

  1. Continue implementing Iterator; most people won't really use it, so it won't hurt, and maybe it will be useful in a way that I haven't recognized.
  2. Remove the Iterator impl and find a way to tell clippy to buzz off.
  3. Remove the Iterator impl and change next() to something I don't like, to make clippy happy.

Or perhaps there is some way to alter the design to make it actually useful (and not confusing) that Lexer implement Iterator. I suspect that that's a unicorn, but I might be wrong.

Thoughts?

I'll check it out; thanks!

1 Like

That's what I would do. Clippy does have false positives and it's perfectly acceptable to silence them if you don't agree with them (unlike compiler warnings, which I am loathe to ignore).

Another API-ish question. The crate does not currently use the heap, either directly or indirectly, or a filesystem. At least I think it doesn't; there are no Boxes, Vecs, Strings, or the like. Let's assume I'm correct.

I can imagine this crate being useful for CLIs in embedded systems, but I don't know anything about no_std crates. Is whether or not the library uses the heap essentially all you have to worry about? I'm assuming that Iterators, flat_map, tuples, and other stuff that doesn't make use of the heap is all in the core library, right?

Would I be better off just ignoring this for now, releasing it on std and then coming back later to see about making a no_std version, or would I perhaps risk painting myself into a corner in some way if I don't build the no_std-ness in from the start?

Also, is there a good way to check whether a crate makes use of the heap? Perhaps a special allocator you could direct rustc to use which simply stops and prints a message if something tries to allocate. Or maybe just compiling it in the way you would for no_std is the easiest thing?

Some useful info about no_std in the following Reddit thread. Someone asked whether there is such a failing allocator, and it sounds like just trying to build for no_std is the thing to do.

Ah, and I just learned that if you do make a no_std crate, you should tag it as no-std when you publish to crates.io, because crates.io lets you restrict a search to include a tag.

Just discovered "cargo vet"; it looks like something that one might want to use before publishing a crate. Perhaps even a Github action?

https://mozilla.github.io/cargo-vet/

I wondered whether making a crate no_std has some cost to users, and this StackOverflow questions answered the question perfectly:

and gives a pointer to a Github repo that demonstrates how to set up the CI so that it checks whether the crate will actually build without std.