Feel free to share a playground with the working implementation and the test-case, together with the definition (or a simplified definition) of the BTree type, as that simplifies experimentation. (For example, I can’t write macros, even less so bug-free, without being able to ask the compiler “does this work”.)
it's not possible for the syntax you are using, and this is the exact reason, the tt meta variable is a sledgehammer, it will capture anything, so you cannot use the optional modifier on it (e.g. $($x:tt)? will never work unless at the tail position)
that said, if you can accept a different syntax, the macro can be simplified a little bit. what I'd suggest is to ditch the comma separator and use a s-expression like syntax instead, which is easier to parse.
example:
let nil = tree!();
let example = tree!(1 (2 () ()) (3 (4 () ()) ()));
// possible shorter convenient syntax
let tree1 = tree!(1); // short for `tree!(1 () ())`
let tree2 = tree!(1 2 3); // short for `tree!(1 (2) (3))`, which is short for `tree!(1 (2 () ()) (3 () ()))`
let tree3 = tree!(1 () 3); // you get the idea
let tree4 = tree!(1 2 ()); // or maybe just `tree!(1 2)`
btw, round parenthesizes, square brackets, and curly braces are all tt group tokens, you can mix and match them, the s-expr doesn't have to be use single type of brackets.