Announcing the `http` crate

[alexcrichton]

Good afternoon Rustlers! Today we’d like to announce a crate that a number of us including @carllerche, @seanmonstar, @withoutboats, and I have been working on for some time now: the http crate! This crate is intended to be sort of the “standard library for HTTP types” in the sense that it provides common types shared amongst all HTTP implementations: requests, responses, headers, etc. It does not cover transport, but is rather intended for use in libraries like Hyper, and to support an ecosystem of crates that can share HTTP-related types.

This crate is not currently published on crates.io and we’d like to get your feedback before doing so! The current plan is to publish an 0.1 release before RustConf on August 19.

We hope that the http crate can become a stable foundation for the emerging HTTP ecosystem in Rust, usable for both synchronous and asynchronous servers, clients, and middleware. This crate has been designed from the start with long-term API stability in mind, leaning heavily on principles learned throughout the ongoing Libz Blitz. We’ve also focused on making it as flexible as possible for various use cases, striving for ergonomics at compile time and speed at run time.

You can get started today with the http crate first by reading the documentation and then by using a git dependency:

[dependencies]
http = { git = 'https://github.com/carllerche/http' }

We don’t expect much API-level breakage before 0.1 is released, but if you uncover any issues please feel free to report them! We want to be sure to give everyone a chance to provide feedback before the publication to crates.io. After the 0.1 publication we plan to start integrating the http crate throughout the ecosystem where possible, followed by a 1.0 release after everything looks like it’s going smoothly! (and possibly with updates from what we’ve learned along the way)

Please feel free to leave any feedback here or on GitHub!

[josh]

Awesome! Having common types across the various HTTP client and server crates will help hugely with portability and helper-crate development.

[theduke]

After scanning over the API:
It looks a lot like what hyper has currently but streamlined and some of the awkwardness removed, so .

It’s a great step for the ecosystem.

[KodrAus]

Well done! I’m really keen to start yanking out some bespoke http request and response types

[bfrog]

I think this is a great idea, and a much sought after feature. My only thought would be why not put it under the hyper organizational account rather than a personal account? Personal accounts tend to have issues at some point when someone vanishes for any number of possible reasons.

[floatingatoll]

To clarify: Are you asking why this pre-release code isn’t being released formally through the org account, or are you asking what the future plans are for the formal release of this code once the community discussion period concludes?

[newpavlov]

It’s an important crate, but I think it’s not the best idea to publish it as http. I agree with the opinion voiced on reddit that something like http-types would fit better. For example if we imagine new Rust users, I think most of them will expect something like reqwest under http crate and not collection of types and traits not directly usable for them. So in the proposed form http crate can be a source of confusion and frustration for them.

[seanmonstar]

I believe we will move it to github.com/hyperium/http when 0.1 is published.

[radix]

It’s an important crate, but I think it’s not the best idea to publish it as http.

I think I agree, since it’s clear that people here and elsewhere are already getting confused about what this crate is, based on the name.

[mmrath]

Good step, but http names doesn’t seem good.

[seanmonstar]

I don’t believe there will be that much confusion. There won’t be a Client, or a Server, or similar. And the documentation can clarify that this is a ‘link to’ library, not an implementation.

[lltapp]

A very good step.

I saw a lot of lib or trait powered by the team, That’s good, But who will implement it for the users to use?
We can’t always wait for someone.

I suggest the Team consider that. it’s not a Ask, Just suggestion.

[josh]

Quoting a response I made to one such comment:

I think http actually makes sense here, for two reasons.

First, this crate provides a common layer for the HTTP protocol; if you’re interacting with HTTP, you’d use this crate and its types, often directly. Note, in particular, that this crate provides builders to construct HTTP requests and responses, which is often a large part of client and server libraries, respectively.

And second, I don’t think any specific HTTP client or server software should claim the crate name http. A common layer like this, built around the protocol itself, seems to have a much more reasonable claim to it.

[jsanders]

Neat! Does this, or will this, include any types for HTTP 2?

[seanmonstar]

The goal is to be agnostic of the underlying version from the user’s point of view, but we’ve been testing this crate in an HTTP2 implementation, and making sure it’s optimized well for it.

Not really sure if we’d add some standardized types for referring to H2 things to this crate. It might make sense as we get closer to a fully featured H2 implementation.

[jsanders]

Ok thanks. Is the implementation you’ve been testing with publicly
available someplace?

[winsut]

The crate seems quite good for the most frequent use cases, but it seems to require several extensions to be truly suitable for all use cases.

1. Use case: avoid copies for data that comes from sources other than bytes::Bytes objects for requests/responses that are passed to APIs that don’t need them to live on the heap (e.g. synchronous I/O APIs)

Requirements: support using borrowed strings instead of heap-allocated strings

Suggested design: add a lifetime 'a parameter to Bytes in the bytes crate, ByteStr, and all structures containing ByteStr. When the lifetime is 'static, the structures behaves as the current ones. Otherwise, they can include borrowed pointers with lifetime 'a instead of lifetime 'static

2. Use case: using strings already on the heap in structures other than bytes::Bytes that are passed to APIs that don’t need sharing or multithreading support

Requirements: support using non-Arc heap-allocated data, like Rc-based data, boxed data, normal vecs, String, etc.

Solution: replace ByteStr with a T parameter that can be borrowed to an &'a [u8], where 'a is the lifetime introduced in (1); introduce a ByteStr and bytes::Bytes variant that uses Rc instead of Arc, and allow directly using Vec or Cow too.

3. Use case: having Decode+Encode always produce the starting byte sequence, implementing transparent proxies that modify the request as little as possible, having an encoder that can produce all the byte sequences that decode to a logical request/response

Requirements: whitespace, case for case-insensitive strings, leading zeros and redundant plus signs for numbers that can have them, exact ordering for lists and other non-semantic characteristics need to be recorded

Solution: add FormattedRequest, FormattedResponse structs that include such data and are convertible or include Request/Response

4. Use case: interoperability of body handling between libraries (including support for chunked encoding and in general streaming)

Requirements: provide guidance on what the T for body should be set to, possibly including some structures to use there OR replace T with something more structured

5. Use case: support using websockets

Requirements: websocket-related structures are provided, ws and wss protocols in Uri, etc.

[netroby]

Hope support both TLS, HTTP2.0

All in one package.
No more need Native-TLS.

Cross compile Openssl sucks.

[cundd]

Seeing the headline I thought: Cool a standard HTTPClient - so I agree that the name could lead a bit to confusion.

Another question: Why is the URIs query not some kind of map? Wouldn’t it be easier to work with the URI, if the query was already parsed?

[periklis]

Great work, thanks for making http a first-class good citizen in the rust ecosystem.