Implementing trait with static lifetime bound

help
1

G'day folks,

Today I'm working on a dictionary stemmer. The data required for this is several megabytes, so I want to keep as few copies around as possible. There are three traits involved here:

You can see my dummy implementation, cribbed from the existing token filters, here:

The PaliDictStemmer creates the StemmerFilter which in turn creates the StemmerTokenStream. I'd like to pass a reference to the TermStems dictionary, but the Tokenizer trait has the 'static trait bound:

pub trait Tokenizer:
    'static
    + Clone
    + Send
    + Sync {
    type TokenStream<'a>: TokenStream;

    // Required method
    fn token_stream<'a>(&'a mut self, text: &'a str) -> Self::TokenStream<'a>;
}

Does this mean I cannot store a reference to a TermStems instance?

This filter seems to just clone the whole dictionary:

Once again, any help is greatly appreciated!

2

OK, so this does work with a cloned dictionary. I've pushed a new version of the code above:

This is fine for my prototype, but I'd still like to know how I might keep just one copy handy.

3

Use Arc<TermStems> instead? Assuming you don't need &mut TermStems during the streaming.

4

Yeah, I need the map in order to transform the tokens:

impl<T: TokenStream> TokenStream for StemmerTokenStream<T> {
    fn advance(&mut self) -> bool {
        if !self.tail.advance() {
            return false;
        }
        let token = self.tail.token_mut();
        if let Some(entry) = self.term_stems.entries.get(&token.text) && let Some(stem) = entry {
            token.text = stem.clone();
        }
        true
    }
    ...

That's the only place it is used.

5

OK, I'm still not sure if I understand everything you want to accomplish, but here's another stab at it. Can you carry an owned TermStems in StemmerFilter<T> and a borrowed one in the StemmerTokenStream -- removing the clone from token_stream -- or is that still too much duplication for you?

pub struct StemmerTokenStream<'a, T> {
    tail: T,
    buffer: String,
    term_stems: &'a mut TermStems,
}

impl<T: Tokenizer> Tokenizer for StemmerFilter<T> {
    type TokenStream<'a> = StemmerTokenStream<'a, T::TokenStream<'a>>;

    fn token_stream<'a>(&'a mut self, text: &'a str) -> Self::TokenStream<'a> {
        StemmerTokenStream {
            tail: self.inner.token_stream(text),
            buffer: String::new(),
            term_stems: &mut self.term_stems,
        }
    }
}
6

I'll take another look at your second solution when I have time. As a bit of background though:

There's currently no language analyzer for Pali for any full-text search engine that I know of. Currently we use ArangoDB/ArangoSearch, but I wanted to have a go with Rust and Tantivy.

I've already got a tokenizer working, stemming is the next step. Imagine a complete analyzer like this:

let pli_stem = TextAnalyzer::builder(PaliTokenizer::default())
    .filter(LowerCaser)
    .filter(PaliDictStemmer::from(term_stems)
    .build();
  • PaliDictStemmer is the implementer of the TokenFilter trait, with the method transform().
  • In transform() we create a StemmerFilter, which implements Tokenizer.
  • Tokenizer has the method token_stream() where we create a StemmerTokenStream.
  • StemmerTokenStream implements TokenStream, including the advance() method.
  • advance() then finally does something useful. We look up the stem for the token in the term_stems hash map. If it exists, swap the original term for the stem, otherwise leave it be.

Only the StemmerTokenStream needs the ~4MB of data, but we have to get the data from PaliDictStemmer. At this point lifetimes come into play and I got a bit lost.

7

OK, that works!

Thanks again.