Hi, I was extracting a service I haven't worked for quite some time and then noticed that upon shutdown that it throws an error saying it cannot drop a runtime.
Cannot drop a runtime in a context where blocking is not allowed.
This happens when a runtime is dropped from within an asynchronous context
I loosely remember having fixed something similar ages ago and I believe it was related to mixing sync and async, but I cannot remember anything anymore.
In a nutshell, the service builds a gRPC service, attaches a gRPC health handler, builds a http service to expose metrics via autometrics and then stats them jointly with a signal handler to listen to shutdown signals. Fairly basic. The main method looks like this:
dbg_print("Setup autoconfiguration");
let ctx_manager = async { CtxManager::new() }.await;
let dns_manager = async { DnsManager::new(&ctx_manager) }.await;
let cfg_manager = async {
CfgManager::new(
SVC_ID,
smdb_specs::smdb_service_config(),
&ctx_manager,
&dns_manager,
)
}
.await;
dbg_print(&format!("Detected context: {}", ctx_manager.env_type()));
dbg_print("Auto-configure service ip and port for context");
let service_addr = cfg_manager
.get_svc_socket_addr()
.expect("DBGW: Failed to get host and port");
dbg_print("Set up socket address for gRPC and HTTP");
let grpc_addr = service_addr
.parse()
.expect("[SMDB]: Failed to parse address");
dbg_print(&format!("Socket address: {}", service_addr));
dbg_print("Construct gRPC health_service");
let (mut health_reporter, health_service) = tonic_health::server::health_reporter();
health_reporter
.set_serving::<SmdbServiceServer<SMDBServer>>()
.await;
dbg_print("Construct gRPC server");
let grpc_svc = SmdbServiceServer::new(SMDBServer::new());
let signal = shutdown_utils::signal_handler("gRPC server");
let grpc_server = Server::builder()
.add_service(grpc_svc)
.add_service(health_service)
.serve_with_shutdown(grpc_addr, signal);
dbg_print("Configure http endpoint");
let (metrics_addr, metrics_uri) = cfg_manager
.get_metrics_socket_addr_uri()
.expect("[SMDB]: Failed to get metric host, uri, and port");
dbg_print(&format!(" metrics_addr: {}", &metrics_addr));
dbg_print(&format!(" metrics_uri: {}", &metrics_uri));
dbg_print("Build http metrics endpoint");
let routes = warp::get()
.and(warp::path(metrics_uri.clone()))
.and(warp::path::end())
.and_then(metrics_handler);
dbg_print("Build http server");
let web_addr: SocketAddr = metrics_addr.parse().expect("Failed to parse web address");
let signal = shutdown_utils::signal_handler("metric server");
let (_, web_server) = warp::serve(routes).bind_with_graceful_shutdown(web_addr, signal);
// Create a handler for each server https://github.com/hyperium/tonic/discussions/740
dbg_print("Create an async handler for gRPC server");
let grpc_handle = tokio::spawn(grpc_server);
dbg_print("Create an async handler for http server");
let http_handle = tokio::spawn(web_server);
// Start all servers jointly
dbg_print("Start all servers jointly");
print_utils::print_start_header(&SVC_ID, &service_addr, &metrics_addr, &metrics_uri);
match tokio::try_join!(grpc_handle, http_handle) {
Ok(_) => {}
Err(e) => {
println!("[SMDB]: Failed to start gRPC and HTTP server: {:?}", e);
}
}
Full code in this repo.
The service starts happily, does it thing, but when I hit ctrl-c, it throws the error that it cannot drop a runtime. From memory, the sync stuff at the very top is all wrapped in an async context to prevent issues with Tokio, but I am not so sure anymore if that is the right way so maybe that is the culprit?
If I were to re-write those mangers all as async, would that solve the issue for good?