I have a web server written on top of Axum. I'm using tracing for logging, and recently made a layered approach with tracing_subscriber to split off access_log type events to their own file. I was able to pin down values for every field in the standard "Combined" log entry with the exception of the size. It seems Apache will log a transfer byte count for each request.
My routing has a few layers, including an http compression layer. I've been poking around in my outermost layer at the Response object to see if I could get it to tell me how big it was. There's a size_hint() function with lower and upper bounds, but these never seem to have legit data on them.
Is there a not-terrible way to interrogate for how much data we're going to be sending out?
No, it wouldn't. I know that because I have worked with it. It couldn't. It writes size when it knows it, and just writes - when that's impossible.
Suppose I'm into retro-computing and am writing chat for Netscape 3.0 (pre XmlHttpRequest version). That's not as hard as it sounds: just never close the connection and send one space every second if no one writes anything in your chat or send javascript command that appends text to window with chat. Been there, done that.
Now, the amount of infomation that one, single request would send depends on my chat activity and time I'm spendong in that chat: it could be as little as 86K in 24 hours or maybe megabytes in a few minutes… depends entirely on how active my chatroom is.
Apache wouldn't, of course, write anything about size of my requests for such module and the same would be true for axum.
Except if you want to pass all that data (which may need hours or even days to arrive!) through the log handler.
Have fun organizing everything in a fashion that wouldn't lose these empty spaces sent every second or Netscape 3.0 would detect a timeout and would close the connection.
I'm no expert in anything Apache or any web technology, really. I base my knowledge of the log format on their documentation:
2326 (%b)
The last part indicates the size of the object returned to the client, not including the response headers. If no content was returned to the client, this value will be "-". To log "0" for no content, use %B instead.
I'm not doing web socket type communication. The system is pretty standard "get a request, serve a response" structure. A simplified version of my outermost middleware is mainly for logging and looks like this:
async fn mw_response_log(
ConnectInfo(addr): ConnectInfo<SocketAddr>,
request: axum::extract::Request,
next: Next,
) -> Response {
let start_time = std::time::Instant::now();
let path = match request.uri().path_and_query() {
Some(p_and_q) => { p_and_q.as_str().to_owned() },
None => { request.uri().path().to_string() }
};
let method = request.method().to_owned();
let version = request.version();
let req_headers = request.headers();
let user_agent = req_headers.get("user-agent").cloned();
let referer = req_headers.get("referer").cloned();
let forward_addr = req_headers.get("X-Forwarded-For").cloned();
let addr = forward_addr.map_or(addr.ip().to_string(), |addr| {
String::from_utf8_lossy(addr.as_bytes()).to_string()
});
let response = next.run(request).await;
let status = response.status();
log_access(
status.as_u16(),
method.as_str(),
version,
path.as_str(),
addr.as_str(),
user_agent.map(|v|v.to_str().expect("user agent extract").to_string()),
referer.map(|v|v.to_str().expect("referer extract").to_string()),
);
response
}
That response: Response<Body> has a size_hint() function, but the returned object says lower = 0, upper = None. Is that my fault? Did I need to do something different in my inner handlers to provide it with that data?
What is “a socket type communication” here. Netscape 3.0 is year 1996 product[1], it had no support for sockets!
The exact same code was used for normal static web pages and chats with it.
Yes, but what kind of a response? In axum, the same as in Apache Response can be a Stream of byte chunks and said stream could be generated asynchronyously.
No.
That just means that said response is not ready yet. And it could be generated, in an asynchronous fashion, later… maybe even over few hours (like in old chats).
You need to decide how the whole thing should work first.
Do you want to disable asynchronious generation of responses? If so then you can convert that response into sequence of bytes and then measure its size.
Do you want support it and then log data after it was served to the client. Need a pass-through with counter then.
Or you may just write - in the log like Apache did. Because doing anything else is hard.
What you shouldn't expect is some API that magically would conjure something from the thin air that doesn't even exist yet, at this point in the execution of a program.
We had to support it in XXI century because someone invested too much into machines that couldn't run Windows 95, but that's different story. ↩︎
I was sitting here looking at the HEAD responses, which includes accurate "content-length" values, and thinking "some part of this system knows the answer". I did some experiments. I added trace logs for the size_hint(), checking the "content-length" header, and adding my own size information through the response.extensions() mapping (on the paths that I control). It was interesting because the results were effectively random. Asynchronous, I suppose, as you say. Like when I go to sample it, some other parts of the system are trying to figure it out. I don't think logging it is more important than speed, so I wouldn't want to hold things up waiting for the answer.
What I'm doing for now is querying all 3 of those sources and looking for any of them to not give me None, with 0 as a fallback.