HTTP request works with CLI curl & wget, but not with Rust's tinyget or native-tls

help
1

I try to make a HTTP request from within a Rust application, but get other results as in the command line.

This works fine on the command line, giving a HTTP2 response:

$ curl -v --tls-max 1.2 'https://www.example.com/settings'
[...]
> GET /settings HTTP/2
> Host: www.example.com
> user-agent: curl/7.88.1
> accept: */*
> 
< HTTP/2 200 
[...]

This as well, also CLI, this time a HTTP1 response:

$ wget -S 'https://www.example.com/settings'
[...]
  HTTP/1.1 200 OK
[...]

Mimicking the same with crate tinyget fails:

use tinyget;

let request = tinyget::Request::new("https://www.example.com/settings")
              .with_header("user-agent", "curl/7.88.1")
              .with_header("accept", "*/*");
let response = request.send().unwrap();
println!("Response code: {}", response.status_code);
// Prints: Response code: 403 [...]

Next try using quite low level crate native-tls:

use std::net::TcpStream;
use native_tls::TlsConnector;
use native_tls::TlsStream;

let tcp_stream = TcpStream::connect_timeout(&address, Duration::from_secs(15)).unwrap();
let connector = TlsConnector::new().unwrap();
let mut stream = connector.connect("www.example.com", tcp_stream).unwrap();
let get_message = format!("\
  GET /settings HTTP/2\r\n\
  Host: www.example.com\r\n\
  User-Agent: curl/7.88.1\r\n\
  Accept: */*\r\n\
  \r\n\
");
stream.write_all(get_message.as_bytes()).unwrap();
let mut response = Vec::new();
stream.read_to_end(&mut response).unwrap();
let response = String::from_utf8_lossy(&response).to_string();
println!("response {}", response);
// Prints: response HTTP/1.1 505 HTTP Version Not Supported [...]
// (changing "HTTP/2" to "HTTP/1.0" leads to the 403 response like with `tinyget`)

(example.com is a replacement, can't reveal the actual URL)

Well, all happens on the same PC (Debian), all TCP stuff works fine, all TLS stuff works fine, yet I'm out of ideas on what might cause that server to behave so differently across these four attempts. Especially with the first and last version, it should see the exactly same request.

Any ideas on how to get CLI behavior inside Rust?

2

Did you enable the https feature for tinyget?

3

Yes, HTTPS works.

4

You probably need to configure ALPN in the TLS backend to negotiate h2.

1 Like
5

Thanks for the answer, @sfackler. So I enabled this feature and changed this line:

let connector = TlsConnector::new().unwrap();

to this:

let connector = TlsConnector::builder()
                .request_alpns(&["h2", "http/1.1"])
                .build().unwrap();

Now I no longer get text as response, but 57 binary bytes (even if that was some kind of encryption, not enough to hold these ~830 characters of expected response):

&response = [
  0, 0, 18, 4, 0, 0, 0, 0, 0, 0, 3, 0, 0, 0, 100, 0,
  4, 0, 1, 0, 0, 0, 5, 0, 255, 255, 255, 0, 0, 4, 8, 0,
  0, 0, 0, 0, 127, 255, 0, 0, 0, 0, 8, 7, 0, 0, 0, 0,
  0, 0, 0, 0, 0, 0, 0, 0, 1,
]

That said, curl in the command line works without ALPN as well, same result as in the opening post:

$ curl -v --tls-max 1.2 --no-alpn 'https://www.example.com/settings'

Never had such odd behavior before. This is so weird I start to consider using std::process::Command to grab that info I need from the actual command line.

6

Oh I missed this the first time. HTTP 2 is a binary protocol, not a text protocol. If you want to hand-write a request like that you are there you're going to need to use HTTP/1.1.

1 Like
7

If you are receiving a 403 response, then your HTTP code is working. The problem is that the HTTP server you are communicating with is telling you that the request is not authorized, but from an HTTP-implementing point of view, this is a request that is successfully sent and processed.

I would wager that the request you are sending differs from the requests curl or wget are sending in some way, as the server is responding to those requests differently. The best ways I know of to identify those differences are with tools like wireshark, or by sending the "same requests" to a local web server to look for those differences by hand.

You might also want to look at the body of that 403 response, to see if the server tells you why it rejected your request, or talk to the operators of the server you're sending requests to, to ask how to send a properly-authorized request.

1 Like
8

I get a 404 (not 403) when running that code, if that helps at all.

9

You would; OP probably isn't actually trying to connect to IANA's example.com service.

1 Like
10

Thanks for the help everybody. As no magic insight happened and performance isn't critical, I ended up using the command line curl. Catching stderr rather than stdout, because I need the headers:

// Run the `curl` command, catch stderr(!).
let curl = match Command::new("curl").arg("-v")
                                     .arg("https://www.example.com/setting")
                                     .output() {
  Ok(output) => {
    let stderr = String::from_utf8_lossy(&output.stderr).to_string();
    if ! output.status.success() {
      return Err(format!(
        "`curl` returned {}, '{}'",
        output.status, stderr.lines().next().unwrap_or("(nothing)")
      ));
    }

    stderr
  },
  Err(error) => return Err(format!(
    "'curl' failed on the command line: {}", error
  )),
};

Not really a solution for the particular problem here, yet it works!