Dealing with tons of sockets, how do you do testing?

help
#1

Hey folks, I am building a SIP stack and in case you don't know it, SIP has tons of transports. At the moment I am dealing with TCP, TLS and UDP but there will be more like WS(S), SCTP and potentially in the future QUIC..

My question is what people do in order to run (integration?) tests: do you create sockets? This is not very efficient for various reasons:

  • allocating an actual socket/port for just running a test feels like a bit too much
  • you are restricted to localhost, unless you touch the /etc/hosts but then that's way out of scope of the Rust testing system
  • for running tests in parallel, you need to make sure you don't allocate the same ports for tests, or just run the tests serially

I would like to avoid that. I want to create a fake socket: something that doesn't even bind to an actual system socket/port and something that I can fully control and inspect during testing time, so that I can either inject stuff in there to assert certain things on my library, or assert certain things on the faked socket itself that I expect to happen due to side effects of actions I do on my library. The idea is that there will be a specific entry point where you can inject that "fake" socket along with some other information (socket local/remote addresses, and socket type) and then the same code will be used as the regulars sockets.

Potentially this "entry point" might be used by library users as well (for instance, instantiating a TCP socket but from a different library, like socket2 that provides tons of options), so it's good to have.

For TCP and TLS and I guess WS(S) (but not for UDP), I can somehow abstract the socket using tokio::io::AsyncRead + tokio::io::AsyncWrite. Having that, then I split it (as I do with regular sockets), and then the regular code path is followed. But then my problem becomes, how can I create a "test" type that adheres to those traits (plus Send + Unpin I think), but it is also a fake one and something that I can fully control. I don't mind if this type is a bit bloated (has mutexes etc internally), as it is used for testing, performance is not the key here. Anyway, long story short: I have abstracted TCP/TLS streams like that, but as I said, I can't create the testing type to set up the testing scenario when testing. I tried but I have to dive deep into the futures, and I couldn't figure this out. So for me that was a deadend. My first question: has anyone built a testing implementation that I could potentially reuse or take a peek at it ?

Then I was thinking abstracting the sockets as a a unified futures::Stream + futures::Sink. Again it won't be used only for testing, but also to allow library users to create the socket ends (stream/sink) from other libraries like socket2 for whatever reason they want. As long as they provide me a Stream + Sink and the type of the socket (like, TCP) we are good. I haven't tried this yet, but sounds like something that could be done.

Note that, even when you develop SIP applications, you want to run some kind of "integration" tests: tests that test a specific flow (INVITE, REGISTER, etc) so even if you are a user of that SIP server library I am building, you are going to need to deal with sockets: again, ideally, you don't want to bind real sockets. Instead you want to bind "fake" sockets that you can control and assert things on them, or use them to manipulate states on your SIP element.

So I would say, my second and main question is: how other people deal with these kind of socket problems ? What do you suggest of me doing here ?

I should also mention that I am on Tokio ecosystem.

#2

A file is a fake socket for what it's worth - it implements all the interfaces, behaves exactly the same as a socket (I mean it can't randomly fail, but you can introduce latencies and random fails by wrapping around a tokio::fs::File if you please).

#3

Ok, not perfect but definitely helpful. But what about Udp sockets, can I fake those with files? Because they don't seem to follow the AsyncRead + AsyncWrite traits.

#4

For an integration test, not at all. The entire point of integration testing is to test as much of the real operation of the system as is feasible. If you implement a fake that's not actually a socket, you're not testing any of the unique behaviors of sockets (e.g. how they behave when set nonblocking, which can't be done with files). And it is very unlikely that creating a socket will be slow enough that it affects your overall test suite speed.

In my opinion, the levels of testing you should be doing are

  • real sockets, and
  • sans-IO unit tests — tests of a pure algorithm, calling functions and seeing what they return, that don't involve having it make any syscalls. If it's hard to write these tests, refactor until it's easier — there are many more benefits to separating your IO from your core algorithms, such as having an easier time recovering from IO errors.
  • you are restricted to localhost, unless you touch the /etc/hosts but then that's way out of scope of the Rust testing system

True, and this means it's difficult to exercise hostname-dependent logic (like TLS/SSL certificates). But you can do that in a separate test system which runs more test setups than the default — something like cargo test --features use_test_servers which explicitly contacts a subdomain of your project web site, or something like that. (By the way — DNS addresses can point to 127.0.0.1!) And most of your test cases shouldn't care about the hostname.

  • for running tests in parallel, you need to make sure you don't allocate the same ports for tests, or just run the tests serially

Here is how to handle ports for testing:

  1. Create test server, specifying port number 0 which means the OS should choose a port.
  2. Ask it what port it got assigned.
  3. Have the client under test connect to that port.
1 Like
#5

Hi,

First of all thanks for your replies. I would say the main drive for faking the sockets was
a) parallelizing the tests: that can be solved by the 0 port that you mention
b) to test logic on SIP messages that is based on DNS addresses (either in the request line, or in Contact/Via headers. SIP has a lot of logic around that, especially when building a SIP element like a proxy. So when you get a SIP message that you are suppose to route to example.com, I want to handle/own that part, without mocking the DNS resolver (the SIP server I am building allows you to inject your own SIP DNS resolver). I agree that "integration" tests need to be as close to reality as possible, but in practice you will need to "fake" something, and between DNS or sockets, I was hoping part of the sockets. Because DNS can point to an IP address, but we have a single IP address and we can't map DNS to different ports, right ?