A year in Rust. And I still can't read the docs!

This morning I found the forum had noticed that it was the first anniversary of my signing up here and I saw slices of cake everywhere. Which was nice as it's also my birthday.

As it happened I was struggling to make the most simple, first experiment, creating a web socket client with tokio-tungsten. All I wanted to do was see how I would connect to my ws server, send an auth message and then print the endless stream of data I expect back. How simple could it be?

This took me about 4 hours from first light this morning! Grrrr.... my birthday is not starting well...

Why did I have the difficulty? I'm not sure.

It all starts with the tokio_tungstenite docs : https://docs.rs/tokio-tungstenite/0.10.1/tokio_tungstenite/

There I find we have a "connect_async" function https://docs.rs/tokio-tungstenite/0.10.1/tokio_tungstenite/fn.connect_async.html great, just what I need. But there is no clue as to how to use it or what to do with the connection one gets.

Getting desperate I look in the source and find they have considerately included a client example: https://docs.rs/crate/tokio-tungstenite/0.10.1/source/examples/client.rs

That is nice but it is full of stuff I don't need, at least not for now, and uses mysterious things like 'futures' and 'pin' and futures::channel::mpsc:. I don't need any channels for this, and why does it not use tokio channels anyway?

The example builds and runs but I delete all the stuff I don't need and try to make what I want.

Oops, nothing builds. No error messages make any sense. Nothing I find in any docs after an hour or so helps. The rust analyzer in VS Code has nothing useful to say.

After some hours and much frustration I'm seemingly down to one problem. I can't send anything. Not even a simple string. Grrr.

write.send(msg_string).await.unwrap();

Does not work. Errors tell me I need to send a 'tokio_tungstenite::tungstenite::Message'. But I can't make one of those because it is a private enum or whatever.

Almost time to give up.

Then I remembered something alice said somewhere in response to some unrelated question "Use .into()'"

write.send(msg.into().await.unwrap();

BINGO! It works! It writes, it authenticates, it reads. I have no idea why!

So simple. But how on Earth would anyone ever be expected to find that out from the docs. I'm sure it's in there somewhere. Everything is. But how does anyone connect the dots in a case like this? This is the most recent of many such tribulations I have had with the docs. Which are usually resolved only by asking here.

Anyway, it works, it's a happy birthday for me after all, I'm going to celebrate with some London Pride :slight_smile:

Thanks alice and everyone here over the last year. It's been a blast.

Oh, and here is my code in case any other lost soul has the same problems:

#![feature(duration_constants)]
#![feature(or_patterns)]
#![warn(rust_2018_idioms)]
#[macro_use]
extern crate log;
use clap::{App, Arg, SubCommand};
use futures::{SinkExt, StreamExt};
use tokio::io::AsyncWriteExt;
use tokio_tungstenite::connect_async;
use url::Url;
use hmac::{Hmac, NewMac};
use jwt::SignWithKey;
use sha2::Sha256;
use std::collections::BTreeMap;
use serde::{Deserialize, Serialize};
use serde_json::Result;
...
// Some message definitions
....
#[tokio::main]
async fn main() {
    ...
    // Some command line arg processing with clap to get url
    ...
    // Connect to web socket server at given URL.
    let url = matches.value_of("url").unwrap();
    let url = Url::parse(url).unwrap();
    let (ws_stream, _) = connect_async(url).await.expect("Failed to connect");
    info!("WebSocket handshake has been successfully completed");
    let (mut write, read) = ws_stream.split();

    // Create a JWT.
    let key: Hmac<Sha256> = Hmac::new_varkey(b"some-secret").unwrap();
    let mut claims = BTreeMap::new();
    claims.insert("some claim", "claim something");
    let token_str = claims.sign_with_key(&key).unwrap();

    // Build an authentication message in JSON from the JWT
    let msg = AuthenticateMsg::make("zicog".to_string(), token_str.clone());
    let msg = serde_json::to_string(&msg).unwrap();
    
    // Send authentication message to server
    write.send(msg.into().await.unwrap();

    // Endlessly read what we get back from the server.
    let ws_to_stdout = {
        read.for_each( |message| async {
            let data = message.unwrap().into_data();
            tokio::io::stdout().write_all(&data).await.unwrap();
        })
    };

    ws_to_stdout.await;
}

Cheers!

10 Likes

Happy birthday :birthday:

2 Likes

Oh, except congratulations: I feel your pain. As a total beginner at Rust I hope, it will improve over time but I too find crate documentation hard. Over at R language, the repository (CRAN) has a concept of a vignette. Next to the (R)oxygen documentation of your functions you can upload pdf or html documents as an introduction to your package. Some use it intensively, some not at all. I always check out packages with vignettes first as I assume, their authors will generally be more inclined to produce good documentation.
I wish there was a way that CRAN and crates.io could prefer well documented packaged over less will documented, jsut to slightly nudge authors towards longer Readmes or vignettes not only in the sense of "documentation" but "explanation".

2 Likes

Congratulations for your anniversary!

Docs are just. Hard. Like old manchego cheese on baby mouse tooth :mouse2:.

I find your learning curve pretty impressive. Things take time to be understood and assimilated. I think you did a good job!

6 Likes

Definitely this, but lots of programmers in general (i.e. it's broader than just Rust) also just really don't like writing documentation, or at least a lot less than they like programming.

@zicog I try to look at docs.rs as more or less an encyclopedia about the public API of a crate, which manages my expectations a bit in terms of what I can and cannot get out of the docs of any given crate.

3 Likes

I found the tungstenite message enum:
https://docs.rs/tungstenite/0.10.0/tungstenite/enum.Message.html

Hiding away in the tungstenite crate (re-exported by tokio-tungstenite), it is public so you could have made it yourself it turns out...

I struggle with this as well, particularly in the async world, the number and complexity of types seem to explode!

I tried recently to use tungstenite (or anything) to make a websocket connection from a wasm project running in the browser. I failed! :frowning: Probably should have asked for help here.

I feel your pain, and congratulations on getting it working the same day :slight_smile:

Another tip beyond "try .into()" is "check the docs of crates under re-exports".

I also quite like the big "See all $crate's items" button for an overview. I wonder if this could also show re-exported items? Probably collapsed by crate by default :slight_smile:

Happy Birthday :tada:

1 Like

But I that is a tungstenite message not a tokio_tunstenite message. Compiler complains:

error[E0308]: mismatched types
   --> src/bin/conq_ws_client_cli.rs:157:16
    |
157 |     write.send(msg).await.unwrap();
    |                ^^^ expected enum `tokio_tungstenite::tungstenite::Message`, found enum `tungstenite::Message`
    |
    = note: perhaps two different versions of crate `tungstenite` are being used?

So I try using "tokio_tungstenite::tungstenite::Message":

// Build an authentication message in JSON from the JWT
    let msg = AuthenticateMsg::make("zicog".to_string(), token_str.clone());
    let msg = serde_json::to_string(&msg).unwrap();
    let msg=  tokio_tungstenite::tungstenite::Message::Text(msg);
    // Send authentication message to server
    write.send(msg).await.unwrap();

BAM, it works!

Now I don;t recall exactly what I did before or why it was complaining about a 'private type'

That's nice, I would rather have an explicit message type being obviously created in my code that some inscrutable '.into' for any future reader to puzzle over.

Thanks.

1 Like

I suspect that this compiler note is the source of the problem; is tokio using a different version of tungstenite than you use directly?

2 Likes

That is my approach.

I'm sure everything one needs to know is in there somewhere. But so far I fail to see how anyone could figure out how to use this kind of thing from reading those docs.

There are thousands of parts and adapters to fit different parts together and I'm far away from seeing how one to connects them together.

By way of analogy: I have a square peg, which I understand just fine. I have a round hole which is similarly clear. What I want to do is connect round peg to square hole. I cannot. There is a square to round adapter somewhere but I have no idea of existence, what it might be called or where to look for it.

A similar case in the code I posted above is the JWT stuff.

There I create 'claims' as a BTreeMap. I have no idea why it needs to be a BTreeMap but I can go with it.

Then I magically create a JWT out of that with a '.sign_with_key(...) method.

But wait a minute. Where did that come from? I'm sure a BTreeMap has no idea about JWTs. There is no mention of JWT in my code there that actually creates one.

How did that happen?

Some how by adding "use jwt::SignWithKey;" at the top of my code a magic 'round peg to square peg' adapter appeared and made it possible.

Inscrutable.

1 Like

I don't have a regular tunstenite in my build.

I did put one in at some point this morning, for some reason I forget now. It confused things no end. Well, me anyway. So I removed it.

The message I quoted above was me trying to put back in again. It's gone again now.

I want to go async here...

I think this is an optimistic but flawed perspective. I think rustdoc is fantastic, but it is not magical and cannot produce high-quality documentation out of thin air. There's room for improving auto-generated documenation, especially regarding (in my opinion) re-exported types and trait methods. I suspect solving these sorts of things generically is pretty challenging.

The inherently strict nature of Rust often results in passable "reference" or "encyclopedia" level documentation. This is great and really elevates the minimum documentation quality, as compared with other languages. Unfortunately, writing quality guide-level documentation is:

  1. difficult
  2. (often) thankless
  3. less enjoyable than hacking on code

Without human-crafted guide-level documentation for a crate, it often feels as though I'm paging through a glossary of technical terms. I'm not sure what the solution is for this. Contribute documentation pull requests? :joy:

8 Likes

I was originally going to defend the quality of the docs, but I then realized you were talking mostly about 3rd-party crates, and I haven’t ventured outside std very much. As I’ve never looked at jwt at all, this seems like reasonable practice. It’s not exactly representative, because you’ve already given the solution, and I just have to figure out how it works.

According to the root crate docs,

Claims can be any serde::Serialize type, usually derived with serde_derive .

The code snippet there uses sign_with_key, which is a member of the SignWithKey trait. I’m unsure what the type parameter is for, but there are two blanket implementations listed for anything that implements the ToBase64 trait, which in turn has a blanket implementation:

impl<T: serde::Serialize> ToBase64 for T

So, then it’s off to the serde docs to find out what types implement Serialize, and BTreeMap is on the list.


The information is all there but, as you noted, not particularly easy to follow. I probably had to review a dozen pages to work this out, and I already had a good idea of the ultimate answer when I started.

2 Likes

Hmmm... my perspective is coming from languages like C.

There life is simple. We have some simple data types, integer/float, signed/unsigned, of various sizes. We have arrays, structs as strings.

With that one can read almost any terse technical description of the library functions and other API's and see what you can do with it and how to do it. See the 'man' pages for example. This is the function name, these are it's parameters, this is what it returns. Go at it.

My feeling is that in the Rust world we now have a billion types, provided by std libs and many others. We have billions of adapters (traits/macros) to enable composing all this together. The result being I don't find it possible to glean any useful information on how to use a thing from it's terse technical description.

Or is that just me?

2 Likes

FWIW, i think the latest version of the compiler is printing out a better error message here that actually gives a good suggestion.

error[E0308]: mismatched types
  --> src\main.rs:11:16
   |
11 |     write.send(msg).await;
   |                ^^^
   |                |
   |                expected enum `tungstenite::protocol::message::Message`, found struct `std::string::String`
   |                help: try using a variant of the expected enum: `tungstenite::protocol::message::Message::Text(msg)`

In terms of debugging the issue, this one is a bit tough. A relevant piece of information from the docs is the following definition

impl<T> Sink<Message> for WebSocketStream<T> where
    T: AsyncRead + AsyncWrite + Unpin, 

Which basically says that when writing data to WebSocketStream, Message is the expected data type. If you click on Message in the docs it will actually take you to the tungstenite crate correctly. Its hard to know though that that is the place to look since the .send() function is part of the SinkExt trait and not Sink. Additionally you have the .split() in there which further obfuscates the type.

Happy Birthday :crab: though. I find that every day I use Rust it gets a bit easier.

2 Likes

I'm using nightly here. How latest can I get?

The error message, above, suggested using ' enum: `tungstenite::protocol::message::Message::Text(msg)' but doing so fails as shown above.

Oh not sure why nightly would be giving seemingly worse error messages than latest stable. But following the suggestion like so makes the compiler happy

write.send(tungstenite::message::Message::Text(msg)).await
1 Like

Nevermind just noticed the protocol in there. I must have instinctively removed it when i typed it in or something. I got nothing. This is a tough one and the types do seem to obfuscate the issue quite a bit here.

1 Like

Thanks yes. That is pretty much what I have now:

    let msg=  tokio_tungstenite::tungstenite::Message::Text(msg);
    write.send(msg).await.unwrap();

I like to do things one step at a time, to keep lines short, make it clear, and save my sanity.

My experience with C documentation is that it usually also includes a list of preconditions that the programmer is expected to ensure before calling the function. Rust moves most of these to the type system, which is where the “if it compiles, it works” effect comes from— If a Rust function takes a parameter of type FooBar, I have high confidence that most any FooBar I can come up with will work.

The flip side of this is that the API to create a FooBar needs to force you to satisfy the preconditions that were in the C documentation, which can make it a convoluted process.

1 Like

For what it's worth, you can find into() via searching by type signatures: https://docs.rs/tungstenite/0.10.0/tungstenite/enum.Message.html?search=String%20->%20Message. Rustdoc could probably do a better job documenting that feature, I don't think I've seen it written up anywhere.

10 Likes