Update. This still isn't resolved but I'm making progress at a glacial pace.
The bug only presents itself in the release build, and the release build is optimized (and there was much rejoicing 
) which means most of the variables I want to peek at are <optimized out>, which further means that gdb's print var won't work most of the time.
Instead, I step through the assembly, correlate the assembly with the Rust code (which is not-trivially difficult since the release build reordered the machine code), and identify a register expression (e.g. $rbp-0x148) that allows me to see the value otherwise stored in my <optimized out> variable.
Thanks goes out to breeden4 and kmc on #rust irc who introduced me to test::blackbox(&x). I use it to access a variable of interest in my code, recompile, rr record, and track down the asm for the call. It's fairly trivial to break on the test::black_box() line and extract the register expression from the disassembly.
Maybe I can feed it a Rust block expression, and force the optimizer to ignore the code under scrutiny. I might try that later.
One other detour I got snagged on is setting a conditional breakpoint in GDB for my register expression. break <file>:<line> if *($rbp-0x148) > 1000 produced a "generic pointer dereference" error. So I'm thinking C casting is in order (gdb is written in C), but break <file>:<line> if *((unsigned long*)($rbp-0x148)) > 1000 produced a "syntax error".
The correct way to cast in GDB is to use the syntax of the language your debugging. GDB provides a show language command to let you know what language it thinks it's working with. The correct break expression is break <file>:<line> if *(($rbp-0x148) as &usize) > 1000.
When the SSCCE runs correctly, the MySQL packet size never exceeds 1000. The "Packet out of order" error occurs when the program glitches and finds a packet size upwards of 32kB in the MySQL packet header, and goes off to the races. The conditional breakpoint above will stop at this exact moment in the program.
From here, I hope to extract the MySQL packets from a Wireshark capture so I know what is supposed to end up in my program's buffer, and identify whether my program is looking at garbage in the buffer or looking at the wrong bytes in the buffer.
The mysql_async library my program uses has a multiple buffer approach to processing the data from TcpStream. First it reads 4kB chunks (or less) into a local variable let buf = [0u8: 4096];, after each read it copies this local buffer into a std::collections::vec_deque::VecDeque<u8> using its .extend method, then repeats until zero bytes are read from the stream. Then it's onward to parsing the MySQL packet.
I'm trying to step through the source code for mysql_async and vec_deque, but the latter compiled with a source path of /checkout/src/libcollections/vec_deque/veq_deque.rs, which causes problems with the rustup's rust-src component, which doesn't have /checkout in the path. Now I'm off to sort that matter out.
Cheers