Tonic server cannot be shutdown via signal

1

I have a Tonic Server which continuously streams hello to its client if connected.

pub mod hello_world {
    tonic::include_proto!("helloworld"); // The string specified here must match the proto package name
}

#[derive(Debug, Default)]
pub struct MyGreeter {}

#[tonic::async_trait]
impl Greeter for MyGreeter {
    type SayHelloStream = ReceiverStream<Result<HelloReply, Status>>;
    async fn say_hello(&self, request: Request<HelloRequest>,) -> Result<Response<Self::SayHelloStream>, Status> {

        let (tx, rx) = mpsc::channel(4);

        let name = request.into_inner().name;
        tokio::spawn(async move {
            loop {
                tx.send(Ok(HelloReply { message: format!("Hello {}", name.clone()) })).await.unwrap();
                tokio::time::sleep(Duration::from_secs(1)).await;
            }
        });

        Ok(Response::new(ReceiverStream::new(rx)))
    }
}

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let addr = "[::1]:50051".parse()?;
    let greeter = MyGreeter::default();

    let (tx, mut rx) = mpsc::channel::<()>(1);

    let handler = tokio::spawn(async move {
        Server::builder()
            .add_service(GreeterServer::new(greeter))
            .serve_with_shutdown(addr, rx.recv().map(|_|()))
            .await
    });

    tokio::time::sleep(Duration::from_secs(5)).await;
    println!("Shutting down server");

    tx.send(()).await.unwrap();

    let _ = handler.await;

    println!("Finished");

    Ok(())
}

The data is generated by a fixed interval in a busy loop that runs in a separated thread.

loop {
    tx.send(Ok(HelloReply { message: format!("Hello {}", name.clone()) })).await.unwrap();
    tokio::time::sleep(Duration::from_secs(1)).await;
}

The code works smoothly if no client connects.

However, when the server is serving its client, the shutdown signal fails its duty and the server gets stuck in a hanging state.

I wonder if there is an event that can be tracked when the server is running into shutdown. Upon that, the busy loop can capture that and quit immediately.

2

How does it look like exactly when you are thinking of gets stuck in a hanging state?
Does it continue to serve new requests or more exactly new connections (Keep-Alive) and only the shutdown gets stuck? Or doesn't it continue to work and shutdown also doesn't proceed?

The point is, servers are doing often graceful shutdowns, so they are waiting for connections being closed, before they continue to exit.
There are some config options for timeouts on connections and keep-alive.

Never have used tonic, so don't know how it is in this case, and if there is a special signal to shutdown not graceful (next to kill -9).

Well you can run it in a debugger and set a break point in the shutdown handler, to see what is going on. If there is a problem you should create an issue for it.

3

I tested this again, and find that the server doesn't get stuck exactly. New client connection will be refused, and the client which has connected before are still be served. The server seems to wait for all connected clients to quit.

I think I have some misunderstanding about the graceful.

4

I think the "graceful" bit means it will try to shut down the server without killing any existing connections.

If you want the server to stop immediately, wrap the future returned by serve_with_shutdown() in an AbortHandle.

1 Like
5

Yup. This means that the full fix is slightly involved.

Your continuous serving functions need to listen for and receive the shutdown signal, then tell the clients to go away and reconnect. Then, kill the connection after the clients get a chance to comply. Once that is all complete, the graceful shutdown will finish and the server will exit.

6

This topic was automatically closed 90 days after the last reply. We invite you to open a new topic if you have further questions or comments.