Real time Server with mio


#1

Hello,

I am trying to write a simple server for a real time multiplayer and with the UdpSocket from mio.
So from all examples that I have found for mio version 0.6 the receiving part is really straight forward. But I can’t figure out a smart way to send messages. (The send function should be called from the main thread without blocking it).

In my current implementation the Poll is in a separated thread and for sending messages I use a Mutex (to_send). The send function looks like this:

pub fn send(&mut self, addr: SocketAddr, buffer: Vec<u8>) {
    let data_send = DataSend::new(Some(addr), buffer, false);

    self.to_send.lock().unwrap().push(data_send);
    self.poll.reregister(&*self.sender, SENDER, Ready::writable(), PollOpt::edge()).unwrap();
}

The thread for receiving (and sending) looks like this:

while running {
    poll.poll(&mut events, Some(Duration::from_secs(1))).unwrap();
    for event in events.iter() {
        match event.token() {
            SENDER => {
                // ---------- Multiple locks here
                if to_send.lock().unwrap().len() == 0 {
                    continue;
                }

                let data_send = to_send.lock().unwrap().remove(0);

                let _bytes_sent = match sender.send_to(data_send.data(), &data_send.to().unwrap()) {
                    Ok(bytes_sent) => bytes_sent,
                    Err(ref e) if e.kind() == ErrorKind::WouldBlock => {
                        poll.reregister(&*sender, SENDER, Ready::writable(), PollOpt::edge()).unwrap();
                        to_send.lock().unwrap().insert(0, data_send);
                        continue;
                    }
                    Err(ref e) => panic!("Error {:?}", e)
                };

                if to_send.lock().unwrap().len() != 0 {
                    poll.reregister(&*sender, SENDER, Ready::writable(), PollOpt::edge()).unwrap();
                }
                // ---------- Multiple locks here
            }
            RECEIVER => {
                // Receiver Code
            }
            _ => unreachable!()
        }
    }
}

(Most of the unwraps will be removed with real error handling of course)

So now I am stuck with an additional thread and a Mutex for sending data which is slow for sending messages(?).
Is there a more intuitive way?

Thanks


#2

You can use an mpsc::channel to send the Vec<u8> to the mio thread, rather than using a mutex-protected queue.


#3

Yes, i had the same idea but I am not sure if the channel will always be faster than the poll.reregister() call. If the channel has a delay, the message would not processed until the next message triggers the reregister again. Or can I somehow guarantee that this won’t happen?


#4

That’s a good question. My expectation would be that when Sender::send() returns, the message (buffer) is fully visible to any other thread that would attempt to poll/recv on the queue. To that end, waking up/notifying the mio event loop ought to work. The mio thread, upon wakeup, would keep draining the channel until there’re no more messages. If reregister is sticky in that if a reregister that happens while a mio loop isn’t waiting but when it goes to poll is immediately notified, then I think it should work. But I’d need to go read the mio docs closely to see if I can confirm that.


#5

Ok, thank you. Maybe I have time tomorrow to write some tests if using a channel is reliable.

Regarding the general implementation: Is there a better/smarter design or is this ok?
I am new to mio and non-blocking APIs so I am not really sure. It seems a little bit clunky to me.


#6

Have you looked at using tokio instead of mio? It uses mio internally, but provides a higher-level API over it.


#7

Yes, I have. And I have also read that it is easier to use, but I liked the simple event loop design of mio.
I think it was easier to implement than tokio with its futures, because I don’t want futures in the public API of my networking crate.


#8

Ok, I wrote some simple tests and it seems to works out, at least on Linux and Windows.


#9

Ok, thanks for following up. I am slightly nervous about this assumption (cause I don’t know mio thoroughly enough), so cc @carllerche - hopefully he’ll confirm/deny that this is reasonable (and/or offer up a better suggestion). You can also try pinging the mio github and see if other, more intimately familiar with mio, people can help out.


#10

Just discovered https://crates.io/crates/mio-extras.
Now I am using the channel from this crate to send a message to the mio thread. In the mio thread the message triggers an event and from there I register the socket-write interest. This seems safe to me. As a negative side effect there is one more iteration of the poll, until a message is actually sent.


#11

Interesting - I had not seen that crate before (but as mentioned, I’ve not touched mio directly much either :slight_smile:). It certainly has the right author(s) on it.

Can you attempt a write and only register (and defer the write) if it returns WouldBlock?


#12

Depending on how many messages come in at one time, this saves a lot additional reregisters.
For example with a delay of 5 milliseconds, the send (almost) never returns Wouldblock.
With no delay just throwing in ~250 messages, the send (almost) always returns Wouldblock.
(Tested on Windows)


#13

What’s the 5ms delay? A delay you’re artificially adding?

I would send all the messages I have until WouldBlock. There’s no inherent issue with exhausting the kernel’s send buf and reregistering/deferring the remaining messages until it’s writable again. This keeps latency down and utilizes the stack better.


#14

The 5ms are to simulate a normal game, so there are outside the networking crate. Inside the networking crate I have no delay and just sending until i get WouldBlock.
Messages that are sent are dropped immediately after this, all others go into a queue.


#15

Ah ok, yup - makes sense to me. So overall, you shouldn’t need extra reregister and poll calls.