How can i read line by line from tokio::net::TcpStream?

[ous51m1]

i am trying to do an http request, and read the response line by line, this is my piece of code

use std::time::Instant;
use std::error::Error;

use url::Url;
use futures::join;
use console::style;
use tokio::net::TcpStream;
use tokio::io::AsyncWriteExt;
use tokio::io::AsyncBufReadExt;

async fn len(url: &str) {
    let url = Url::parse(url).unwrap();
    let host = url.host_str().unwrap();
    let path = url.path();

    let mut stream = TcpStream::connect(format!("{}:80", host).as_str()).await.unwrap();
    let req = vec![
        format!("GET {} HTTP/1.1", path).as_str(),
        format!("Host: {}", host).as_str(),
        "\r\n",
    ].join("\r\n");

    stream.write_all(req.as_bytes()).await.unwrap();

    let line = String::new();
    stream.read_line(&mut line).await.unwrap();

    println!("{}", line);
}

and this is tokio in Cargo.toml

tokio = { version = "0.2.11", features = ["time", "macros", "tcp", "rt-threaded", "dns", "io-util"] }

when i run, i get this error

error[E0599]: no method named `read_line` found for type `tokio::net::tcp::stream::TcpStream` in the current scope
  --> src/main.rs:26:12
   |
26 |     stream.read_line(&mut line).await.unwrap();
   |            ^^^^^^^^^ method not found in `tokio::net::tcp::stream::TcpStream`
   |
   = note: the method `read_line` exists but the following trait bounds were not satisfied:
           `tokio::net::tcp::stream::TcpStream : tokio::io::util::async_buf_read_ext::AsyncBufReadExt`

error: aborting due to previous error

For more information about this error, try `rustc --explain E0599`.
error: could not compile `gutenberg`.

To learn more, run the command again with --verbose.

[alice]

The AsyncBufReadExt::read_line method that you are trying to use is only available for types that implement the AsyncBufRead trait, however the TcpStream type does not implement this trait, which you can see by inspecting the Trait implementations section of the documentation on the TcpStream type.

Note that it is true that TcpStream implements the AsyncRead trait, but this is in fact not the same trait as AsyncBufRead. You can wrap your stream in a BufReader to enhance the stream with buffering, which is required to use the read_line function.

use tokio::io::BufReader;
use tokio::io::AsyncWriteExt;
use tokio::io::AsyncBufReadExt;

let stream = TcpStream::connect(...);
let mut stream = BufReader::new(stream);

// We can still write!
stream.write_all(req.as_bytes()).await.unwrap();

// And we can now use read_line
let mut line = String::new();
stream.read_line(&mut line).await.unwrap();

playground

Note that you can still write to the wrapped stream, as BufReader will implement AsyncWrite when the inner stream implements AsyncWrite. However this will not result in any buffering for writes: that would require a BufWriter.

If you run into this issue again in the future, I would like to propose some tips for reading the error message that the compiler has given you:

note: the method `read_line` exists but the following trait bounds were not satisfied:
      `tokio::net::tcp::stream::TcpStream : tokio::io::util::async_buf_read_ext::AsyncBufReadExt`

So the reason it fails is as it said: TcpStream does not implement AsyncBufReadExt, however this way of saying it sounds a bit odd, considering that you do not normally implement the Ext traits directly. To come to the bottom of this kind of error, you can inspect the implementors section in the documentation of this trait to see what types do in fact implement AsyncBufReadExt. You will see a single line:

impl<R: AsyncBufRead + ?Sized> AsyncBufReadExt for R

What this means is that Tokio provides an implementation of the AsyncBufReadExt trait for any type that implements the AsyncBufRead trait. Since there is only one option, it is reasonable to inspect what implements the AsyncBufRead trait. Again, this type provides an implementors section, in which you can see a few options: Empty, Take<R>, BufReader<R>, BufStream<R> and BufWriter<W>.

You can now proceed to look closer at these five options, which would hopefully lead you to the BufReader type.

[alice]

Just a comment regarding the Tokio features: I generally recommend using "full" for applications, and not using specific features unless you are writing a library.

[ous51m1]

thank you @alice for the detailed explanation, that really helped me out

as for the full feature, when i compile my app, is it going to include all the features even if i am not using them in my application, or is it going to include only the necessary bits that the application require to run?

[alice]

The compiled crate will likely be slightly larger if you compile it with "full". I don't know how good the optimizer is at this.

[TomP]

Even so, it's much simpler to get the code working with "full" and then see what features can be dropped without the compile erroring out, than it is to randomly try subsets of features to see which might be unneeded. This is likely another case of Knuth's "Premature optimization is the root of all evil."