Rust support for Snowball stemmers (was Enabling `clippy::all` for a module)

help
1

Hey folks,

I'm using the Snowball compiler to generate a Rust implementation of our stemmer. This involves copying a couple of modules into my project. Unfortunately Clippy is not happy with one of the modules, and I don't feel confident modifying it. I've corralled the Snowball code into its own module structure and have this snowball.rs file:

mod among;
#[allow(unused)]
#[allow(clippy::all)]
mod snowball_env;
mod pali;

#[must_use]
pub fn pali_stem(input: &str) -> String {
    let mut env = snowball_env::SnowballEnv::create(input);
    pali::stem(&mut env);
    env.get_current().to_string()
}

The #[allow(unused)] suppresses the compiler warnings, but the clippy errors remain.

Your help is again greatly appreciated.

2

The nuclear option is allow(warnings). I don't know exactly what Snowball is doing... obviously the ideal answer would be to actually fix the warnings (possibly by improving the codegen tool)... but if you're not that comfortable with Rust, I suppose that's not an option.

3

clippy::all is, unfortunately, misnamed; it does not refer to every lint that clippy can emit, but every lint that clippy emits by default. If your crate or Cargo.toml lint configuration has enabled non-default lints, you will need to turn those off too, by individual name or by group — for example, #[allow(clippy::pedantic)].

4

I wouldn't say that it's not an option. Indeed it might be a nice thing to do.

The original code for snowball_env and among is found here:

rust_stemmers is used by tantivy and includes the same files:

So a PR to fix the linting issues perhaps?

I've been using pedantic mode, but to get all the linting suppressed required this:

#[allow(unused)]
#[allow(clippy::pedantic)]
#[allow(clippy::style)]
#[allow(clippy::complexity)]
#[allow(clippy::suspicious)]
mod snowball_env;

Thanks to both of you for your help.

5

What is the reason you copy and import the code as a module? Why don't you use the crate as a dependency?

6

A very good question!

First up, which crate? The original code does not appear to be published at crates.io

The rust-stemmers crate might expose the same modules? I'm not sure. The way I ended up here was adapting the tantivy-czech-stemmer approach. I asked the ParadeDB folk about dictionary stemmers and was pointed at this PR of their tantivy fork.

7

It seems like you could include that crate as a path dependency, so that it stays a separate crate for the purpose of clippy lints.

That is, copy the source locally (as you have done) then point your project to use it as a path dependency.

8

Why do you think this is not the same code? It seems to be the same at a glance. It’s true that any GitHub repository could have an arbitrary package name which is unrelated to what’s found on crates.io, but in this case they seem to be the same.

9

As far as I can tell, the code on the snowball project is not available on crates.io and rust_stemmers does not expose the copied code for users of the crate.

10

Oh, actually I've missed something. Tantivy uses the tantivy-stemmers crate. It depends on rust-stemmers:

As its default Stemmer tokenizer depends on a less then alive library rust-stemmers, there are only a very few languages available by default. Nevertheless, Tantivy provides an easy way to build our own custom tokenizers (see the tantivy-tokenizer-api for details).

Which is my main task. I didn't write the snowball Pali stemmer, I'm just experimenting with integration into Tantivy. Now, again from tantivy-stemmers:

All the raw algorithms in this library are written in the Snowball language, then complied into a Rust code using the Snowball compiler - all these generated algorithms are located at src/snowball/algorithms/*. A Snowball environment is then needed to execute the generated algorithm. This environment is comprised of files src/snowball/among.rs and src/snowball/env.rs - both files have been provided (ie. copied) from the official Snowball repository: rust/src/snowball.

11

Well no, it doesn't "depend" on it, it has another copy of the snowball code:

12

So my role in all this is supporting the work of a fellow monk:

I have investigated both a dictionary stemmer, where we read all the terms and their stems from a file and use a HashMap, and an algorithmic stemmer using the algorithm above. Full disclosure: I don't actually know Pali. :blush:

Well, I know a lot of the vocabulary, but nothing much about grammar.

Snowball is very niche, but supported by most of the full-text search engines, including Elastic:

It seems a crate that easily allows new algorithms to be created and tested, along with documentation of the whole process, would be an excellent contribution to the Rust ecosystem.