Tokio executors in custom thread pool

So I'm having a own library that I use to spawn threads and have a thread pool, mainly because I need strictly specific actions on delegating tasks and when the program is closing. Recently I implemented into most of my methods Tokio crate so I got to a point at which I had to implement async/await into the closures that were being sent for execution over to the thread pool. Unfortunately I can't quite figure out how to properly use both and I even started to think that I should go with one of the two (since the default runtime on Tokio is spawning tasks on different threads if I remember correct).

What I tried:

  • I tried executor::block_on() and executor::LocalPool::new().run_until() on the received closure in the pool, but that's panicking the spawned thread with "cannot execute LocalPool executor from within another executor: EnterError" (are spawned threads copying the fn main() of the main thread and that's why they already have an executor?)
  • I tried changing the thread pool to accept Box<Future<...> + Send> and also return a Box<Future<...> + Send> that would be executed on thread spawn so we can spin the thread. The problem here is with the closure, it doesn't implement Unpin so it can't be moved. I've not been able to figure out a different approach how to accept DST (dyn Future<...>) and return such while covering it behind a moveable allocated memory.

Am I thinking all of this the wrong way? Any guides/documentation would be appreciated.

It's not really clear what you are trying to do. If you are using Tokio, you should not be using the executor from the futures crate, but the one from Tokio.

You can configure the Tokio runtime with tokio::runtime::Builder, including having it not spawn any threads.

@alice I apologize for not being clear enough. Appreciation once again for the time you are taking to answer.

I'm using the default tokio runtime for the main function via #[tokio::main] however I have a separate library that is thread pool which uses std::thread::Builder to spawn threads in a pool which is waiting for jobs via mpsc. I'm trying to implement in the function that passes the jobs to the thread pool to support futures execution in the closure as well.

So assuming that we use the thread pool from the book here: 20.02 Turning Our Single-Threaded Server into a Multithreaded Server
That's the closest example I can provide: ThreadPool and tokio example

Why are you trying to run futures on a custom thread pool when Tokio already provides one? Your Thread::new doesn't have to be async, because there is no .await inside it. Additionally you need to wrap boxed futures in Pin with Box::pin instead of Box::new.

Finally I don't see how you are executing the future. Your spawned thread just immediately returns a boxed future.

1 Like

@alice Ah.. I've tried the exact same approach the other day and seems like a couple foolish mistakes have been keeping me away from getting it to work. I've redone it again now and its working, probably one of the key things was not using futures executor on top of tokio executor.

As I mentioned above I need to control certain things upon Drop and when the ThreadPool can't accept any more jobs at the moment. Also I was seeing this as the most flexible environment having all three ways of spawning a task in a separate thread or in a future and also having futures in the spawned threads. (therefore having ability to spawn whatever I want wherever I want)
Isn't that better than having only futures available(even though they are in separate threads, which can't be controlled in the way I need), do I have to choose between the two and if yes - why?

To be honest I was really doubting about this situation if I have handled it properly and you just raised my doubts again.

This topic was automatically closed 90 days after the last reply. We invite you to open a new topic if you have further questions or comments.