Reading a Protocol Buffer file -- What header should I skip?

Hi
I have a file which is a dnstap file captured using the fstrm-capture log file Github .

I am able to read the file but when I try to process it I get an error

The proto definition is present here .proto definition GitHub .

I have the following code that tries to read the file and print the ProtoBuf encoded messages. When I execute I get the error

Error: DecodeError { description: "invalid tag value: 0", stack: [] }

I am guessing I need to skip some bytes before I get to the actual ProtoBuf encoded messages. I looked at the fstrm_dump and figured that the actual bytes are 22+6 . Since I guess Rust is doing zero indexing

Let me know where I am going wrong .

//main.rs use std::error::Error; use bytes::{Bytes};

// Include the dnstap module, which is generated from dnstap.proto.
pub mod dnstap {
include!(concat!(env!("OUT_DIR"), "/dnstap.rs"));
}

fn main() -> Result<(), Box> {

println!("Hello, world!");
let buffer = std::fs::read("dnstap.log")?;
let  buf1 = Bytes::from(buffer);
let mut buf2=&buf1[28 .. ];
println!(" Length of buf1 is {:?} , buf2  is {:?} ", buf1.len(),buf2.len());
let message:dnstap::Dnstap = prost::Message::decode(&mut buf2)?;
println!("Finished Reading 1! {:?}",message);
Ok(())

}

My build.rs file looks like this //build.rs extern crate prost_build;

fn main() {
prost_build::compile_protos(&["proto/dnstap.proto"],
&["proto/"]).unwrap();
}

and the Cargo.toml

//Cargo.toml

[dependencies]
bytes = "1.0.1"
prost = "0.7.0"
prost-types = "0.7.0"

[build-dependencies]
prost-build = "0.7.0"

The od -cx dump of the file ( starting bytes looks like this ). The actual file is present here log file Github .

0000000 \0 \0 \0 \0 \0 \0 \0 " \0 \0 \0 002 \0 \0 \0 001
0000 0000 0000 2200 0000 0200 0000 0100
0000020 \0 \0 \0 026 p r o t o b u f : d n s
0000 1600 7270 746f 626f 6675 643a 736e
0000040 t a p . D n s t a p \0 \0 \0 w \n \b
6174 2e70 6e44 7473 7061 0000 7700 080a

If I run a fstrm_dump on the file this is what I get
FSTRM_CONTROL_START.
FSTRM_CONTROL_FIELD_CONTENT_TYPE (22 bytes).
 "protobuf:dnstap.Dnstap"
Data frame (119) bytes.
 "\x0a\x08harihara\x12\x0cBIND 9.16.13r[\x08\x05\x10\x01\x18\x01\"\x04\x7f\x00\x
00\x01*\x04\x7f\x00\x00\x010\xc4\x8d\x038\x00@\xa7\xa8\xc6\x83\x06Mq~w\x03R6\xc7
\x9a\x01 \x00\x01\x00\x00\x00\x00\x00\x01\x03www\x05yahoo\x03com\x00\x00\x01\x00
\x01\x00\x00)\x10\x00\x00\x00\x00\x00\x00\x0c\x00\x0a\x00\x08~'a4*\xed\xea\x0ex\
x01"
Data frame (122) bytes.
 "\x0a\x08harihara\x12\x0cBIND 9.16.13r^\x08\x03\x10\x01\x18\x01\"\x04\xc0\xa8\x
01\x06*\x04\x08\x08\x04\x040\xc9\xe1\x0285@\xa7\xa8\xc6\x83\x06Mq~w\x03R6\xa6S\x
01\x00\x00\x01\x00\x00\x00\x00\x00\x01\x03www\x05yahoo\x03com\x00\x00\x01\x00\x0
1\x00\x00)\x02\x00\x00\x00\x80\x00\x00\x0c\x00\x0a\x00\x08|\x98\x92\xbd\xaau\x1e
\xecZ\x01\x00x\x01"
Data frame (108) bytes.
 "\x0a\x08harihara\x12\x0cBIND 9.16.13rP\x08\x03\x10\x01\x18\x01\"\x04\xc0\xa8\x
01\x06*\x04\xc7\x09\x0e\xc90\x87\x95\x0285@\xa7\xa8\xc6\x83\x06Mq~w\x03R(\x82\xb
5\x00\x00\x00\x01\x00\x00\x00\x00\x00\x01\x00\x00\x02\x00\x01\x00\x00)\x02\x00\x
00\x00\x80\x00\x00\x0c\x00\x0a\x00\x08\xa2\xc9\xf6\xf9VI=\xf3Z\x01\x00x\x01"

From here FSTRM Python Receiver

# Frame Streams Control Frame Format - Data frame length equals 00 00 00 00

# |------------------------------------|----------------------|
# | Data frame length                  | 4 bytes              |  
# |------------------------------------|----------------------|
# | Control frame length               | 4 bytes              |
# |------------------------------------|----------------------|
# | Control frame type                 | 4 bytes              |
# |------------------------------------|----------------------|
# | Control frame content type         | 4 bytes (optional)   |
# |------------------------------------|----------------------|
# | Control frame content type length  | 4 bytes (optional)   |
# |------------------------------------|----------------------|
# | Content type payload               | xx bytes             |     
# |------------------------------------|----------------------|

# Frame Streams Data Frame Format

# |------------------------------------|----------------------|
# | Data frame length                  | 4 bytes              |
# |------------------------------------|----------------------|
# | Payload - Protobuf                 | xx bytes             |
# |------------------------------------|----------------------|

So my code becomes

fn main() -> Result<(), Box> {

println!("Hello, world!");
let buffer = std::fs::read("dnstap.log")?;
let  buf1 = Bytes::from(buffer);
data_frame_length = u32::from_be_bytes(buf1[42 .. 46].try_into().unwrap());

println!("Next Data frame length = {}",data_frame_length);
let mut data_frame = &buf1[46 .. (46 + data_frame_length as usize)];

let message:dnstap::Dnstap = prost::Message::decode(&mut data_frame)?;


println!("Finished Reading 1! {:?}",message);
Ok(())

}

This topic was automatically closed 90 days after the last reply. We invite you to open a new topic if you have further questions or comments.