Is a Rest API with REST hooks available for Rust?


#1

I am writing a REST API, at some point it needs to wait around for the server to finish something, then get a response alerting it that the server running the REST API is ready for the next step. I’m assuming that we’ll need to just poll the REST API, but wondering if there is a better method that could be alerted upon this operation completing.

I’m looking at Nickel, which looks great and traditionally I know that REST needs polling but is it possible to send a request to a Nickel REST API (or a Nickel alternative) that just waits for a response?


#2

Are you talking about a caller of your REST API waiting for the server to finish something?

Traditionally a REST API would go out of its way to not do anything outside the realms of stateless HTTP, so you could pass a callback URL to your long-running task and have your API call that when it’s done. But that means your client needs to be a server too.

Another option could be to ditch REST for this long-running task and use websockets instead. That’s probably not supported in Nickel though (I’ve only just started looking into it myself).

So in that case you could have a separate websocket service, so a client calls your REST API to kick off some long-running task, which immediately returns a correlation id, your client can then choose to send that correlation id over a websocket to a ‘Watch API’ that keeps alive until results are available.

That way you have a bit of graceful enhancement, where long-running tasks can be tracked in realtime, or left to complete in their own time.

Sorry to be so fuzzy in my response, I’m interested in approaches to this scenario in Rust too :smile:


#3

When I have long-running tasks in a ReST-API, I usually do it as follows:

--> POST /jobs/              (doesn't block)
<-- 201 Created
    Location: /jobs/<some ID>/

...                          (client can do something different in between)

--> GET /jobs/<some ID>/     (blocks until the task is finished)
<-- 102 Progress: X%         (optionally, if the client supports 1XX messages)
<-- 102 Progress: X%         (client can close connection and try later)
...
<-- 200 OK
--> DELETE /jobs/<some ID>/  (if the client doesn't delete the job resource,
<-- 204 No content            the server can clean it up after some time)

That way you can maintain the stateless nature of ReST.


#4

Great idea, unfortunately in my case I don’t really have a progression as I’m just waiting for user input during this period.

Thinking I’ll support both polling and callback URLS and let developers decide which they want to implement with callback being all around preferred, except for the ease of use by the programmers.

Would love to hear more thoughts though.


#5

I often deal with a small variation on @troplin’s suggestion, more in line with KodrAus’s description, that is used to request large/expensive-to-query reports via REST APIs. Instead of a blocking “-> GET /jobs//”, your API returns a status (busy, completed, error), an expiration date for the requested resource, and in the case of “completed”, it also returns a url for the client to use. The url will sometimes point to a cdn/csv or back to their API with a generic /jobs//download/, that serves as the “next step” for the client.

The client is expected to poll the status of the job via that url, at their discretion/convenience, until the job completes. You can cache the job status to make this polling fast/cheap and then update that cache when the job status changes. This approach also allows the client to request multiple jobs and save the job id/details locally. Your api can enforce how many jobs it will queue simultaneously for any given client (and total for all clients). You can also implement an internal job queue for processing the requests. The client won’t care about the internals.

I’m new to rust, however, so I do not know what is available or possible, via existing crates, but I’ve seen a project providing amqp bindings if you explore that route.


#6

As I understand it, @KodrAus suggested that the server should establish a reverse connection to the client, like active mode in FTP. I would not recommend going that route, it introduces many new problems.

I think those are the crucial points that are common to your suggestion an mine:

  1. Job creation and status notification have to be separated.
  2. Remain stateless by using URLs for jobs and polling/notification
  3. Plain HTTP/ReST is sufficient, additional protocols like WebSockets are not necessary.

#7

Interesting thoughts, my suggestion was based on a ‘no polling’ approach for long-ish lived tasks. It was actually the same kind of idea, except that instead of polling for a 200 you ask for a socket to that task, so updates are pushed down as they occur.

In this case where you’re waiting on user input (nobody can say for certain when a user will provide that needed input) I would definitely prefer a callback. It’d be a waste to sit and poll for progress continuously for maybe a couple of days while waiting. That’s a reverse connection, but the client is in control of where and how that connection happens.


#8

I don’t know how much of this is a hack: the server uses chunked transfer encoding, and sends a new chunk each time it wants to report progress to the user, and between two chunks just keeps the connection open and idle. It might need proper timeout configuration, but technically this should be feasible.


#9

My suggestion doesn’t use polling at all. It’s just one single GET request, but the server delays the final 200 reply until the job is finished.

The intermediate 102 progress messages are optional and belong to the same single GET request. 1XX answers in HTTP are just this, informational status messages. They are not “real” replies but there has to a final reply with a status >= 200.
Unfortunately, I think many HTTP clients don’t support 1XX messages even if they are part of the HTTP protocol.

I think using a reverse connection is very problematic:

  • Every client has to be a server
  • NAT/Firewall issues

The connection should always come from the client. And since we already have a connection, why create a new one? It’s just wasting resources.


How to alert running REST API when a separate function is called?
#10

Ah I see, I haven’t encountered 1XX messages so I thought each one was in response to a request instead of a 200. That’s neat.

Yeh, another assumption I’d made was that the API was internal so @mczarnek is also the client. Now that I actually think about it there’s no reason for me to think that :smile: I’ll have to spend some more time looking into your design.


#11

Using 1xx is a great idea.

Checking the IANA registry, it appears that 102 is registered as Processing. It used to be part of WebDAV, see RFC 2518, section 10.1, for exactly this purpose, but was actually removed in a newer version of the spec RFC 4918, section 21.4. 102 is still indicated by IANA as “Processing” regardless.

Apparently an HTTP-compliant client implementation “MUST be able to parse one or more 1xx responses received prior to a final response, even if the client does not expect one.” RFC 7231, section 6.2, but you’re right this should be tested.


#12

I agree using 102 progress messages is a great idea. Thanks Troplin!

Different clients using different libraries may see timeouts differently, I’m guessing a 102 should reset timeouts though. Then I can just send one every few seconds, until I finally have the actual alert for the client waiting for this and it can then be sent as fast as possible. Additionally, no need for others to set up a server to support the callback URL. Love it!

Someone mentioned some clients might not support 102? Do I need to worry about this?