Implementing a sink function for Tantivy tokenizer

help
1

Hey folks,

I've just started what will likely be a long journey to provide Pāli language support in Tantivy.

The first stage of the project is tokenization, working towards a dictionary stemmer and algorithmic stemmer testing. I've started looking at the documentation for the tantivy_tokenizer_api crate and instantiated a WhitespaceTokenizer to get started. Calling token_stream() gives me a TokenStream, which in turn has this method:

I'd like to pass in a function or closure that just prints the tokens, but I'm not clear on the signature here: &mut dyn FnMut(&Token). An example function and closure would be great.

Thanks again!

2

This works, of course:

let mut stream = tokenizer.token_stream(segment.as_str());
    while let Some(token) = stream.next() {
        println!("{}", token.text);
    }
3

Does this clear it up?

    // You make a closure, take a `&mut` to it in place,
    // and it coerces to `&mut dyn FnMut(..)`
    mock(that, &mut |token| {
        println!("{token}");
    })
4

Yes indeed!