Parsing incoming text streams

I'm using ssh2 to connect to an embedded system, issue commands to it and extract data from it's responses.

So of course I should ideally parse the incoming text stream looking for the shell prompt and then reading lines of the output of the commands I issue.

Before I start hacking a solution to this I thought there is likely a crate or two that already does such things.

Any suggestions?

ssh2 has bindings to libssh2 which if that works would almost certainly save you a bunch of trouble

I'm using ssh2. It works very well.

I connect to my target, I do a session handshake, I authenticate with user name and password, I make a session channel and start a shell on it. All using those nice ssh2 methods.

So far so good.

Currently after that I just write the command I want, on the assumption that it did actually send me a shell prompt.

Then loop forever reading the response. These commands don't stop outputting unless you send an escape.

But that is a bit flaky, I'd like to parse what I get back and check for the shell prompt properly.

What I have now only works because each read I do happens to be one line of the output, but there is no guarantee that is always the case, so I should at least be looking for '\r' i there to make sure I'm reading lines correctly.

I haven’t used any of them myself, but the nom, combine, lalrpop, and peg crates all look like reasonable parser generators. They might be overkill for your application, though.

If all you want to do is reliably split the output into lines, you should be able to wrap the ssh2::Channel object in a std::io::BufReader, which provides a lines() iterator.

To add to @2e71828, you can create a channel per command to run i believe using the same ssh session that can each exec a single command if you want to run multiple commands. Probably easier than dropping to a shell. If you need the shell, you could also look at rexpect which I've never used though I've used pexpect which its based on to some success in the past.

Yeah, I’d strongly recommend avoiding the shell if at all possible. You should be able to execute the desired command directly. (I haven’t done this with rust, but have with the ruby net/ssh gem. It’s definitely supported by the ssh protocol.)

Splitting on lines does not work for detecting the command prompt, which has no line ending. Perhaps I can use std::io::BufReader when I have the command started and it is outputting lines contiually.

I can indeed create two channels for the same ssh session.

I'm pretty sure I need to start a shell though. I'm talking to an embedded device running Linux, busybox and Dropbear for ssh. When one logs into this device manually with ssh one does not end up in Linux command line shell. Rather one is presented with a very minimal shell with only a handful of commands. I believe it is the device's application TUI, the same one that it used when it was running bare metal without Linux many years ago. I have to issue my commands to that TUI.

So far I have not managed to get any output from the thing using .exec() instead of starting a shell. Not even a simple .exec("ls").

rexpect is just the sort of 'expect' like thing I was looking for.

Thanks all.

It sounds like they've just overridden the default shell in the SSH server's config file to use a custom program instead of /bin/bash (or whatever). It sounds like Channel::exec() should execute a command directly instead of going through the shell. From there you could keep reading until EOF to get the output.

Do you get any output on stderr? It may be that you need to execute /bin/ls instead of just ls.

I was experimenting with this. I can't get Channel::exec() to return any response no matter what command I give it, a linux command or an application TUI command.

Never seen any output unless I start a shell.

Where would I look for an output on stderr?

Why is that?

Another thing to try is something like .exec(“/bin/sh -c \”command to run\””)

Hmmm...

Thing is the "command to run” we want is not any kind of Linux program you can run from a regular Linux shell. It's a command served up by the devices application when you connect to it. It has it's own home made command line style TUI.

We did promise the owners of this device, our clients, that we would not hack into it and would do our best to keep it secure from the outside world. We are only supposed to be making use of a couple of commands from their application's TUI and securely delivering the output to the 'cloud'.

They will be wanting to audit our code.

But hey, you got me curious. For sure there is a proper shell in there somewhere!

We cannot do that in the delivered software though.

Maybe just use the shell then but only ever run one command on it. And use a second channel with a second shell to run another command. That at least eliminates the need to do parsing to check for the shell prompt after one command is done

That will be my next experiment. Currently I make two totally independent connections to the device.

But for sure if I create two sessions from the one connection, and then run a shell on each I will have a command prompt to check on both of them.

So I was tinkering around with running two commands on two shells on one SSH connection and I saw something I have never seen Rust do before, segfault with no explanation. This in Debian in WSL in Windows:

$ ./target/release/experiment
SSH dual session experimet.
Connected.
Authenticated.
Chan 1: [76, 105, 110, 117, 120, 32, 83, 109, 97, 114, 116, 84, 114, 97, 102, 102, 105, 99, 32, 52, 46, 57, 46, 48, 45, 56, 45, 97, 109, 100, 54, 52, 32, 35, 49, 32, 83, 77, 80, 32, 68, 101, 98, 105, 97, 110, 32, 52, 46, 57]
Segmentation fault (core dumped)
$

Here is the code:

use std::io::prelude::*;
use std::net::{TcpStream};
use ssh2::Session;
use std::thread;

fn main () {
    println!("SSH dual session experiment.");

    let tcp = TcpStream::connect(address).unwrap();
    println!("Connected.");

    let mut sess = Session::new().unwrap();
    sess.set_tcp_stream(tcp);
    sess.handshake().unwrap();
    sess.userauth_password(user, password).unwrap();
    println!("Authenticated.");

    let commands = vec!["./gstat", "./dlstat"]; 
    let mut threads = Vec::<std::thread::JoinHandle<_>>::new();
    for command in commands {
        let mut channel = sess.channel_session().unwrap();
        channel.shell();

        threads.push(thread::spawn(move || {
            channel.write_fmt(format_args!("{}\n", command)).unwrap();

            loop {
                let mut buf = vec![0u8; 50];
                channel.read(&mut buf).unwrap();
                println!("Chan {} : {:?}", command, buf);
                println!("{}", channel.exit_status().unwrap());
            }
        }));
    }
}

Of course what is missing there is the joining the threads before exiting.

    for thread in threads {
        thread.join();
    }

You might want to edit out ips, usernames and passwords

Oops...thanks.

Luckily only throw away test servers. Changed now anyway.

The ssh2 library wraps libssh2, which is written in C. It's a really high quality set of bindings, but no project is perfect. I'd suggest creating an issue against the ssh2 repo because there may be a bug in their unsafe code (or libssh2 itself).

OK. Issue raised here: https://github.com/alexcrichton/ssh2-rs/issues/193

1 Like

This topic was automatically closed 90 days after the last reply. We invite you to open a new topic if you have further questions or comments.