I think you still need to explicitly disable Unicode support, (?-u), to match a byte that would be invalid UTF-8. From the docs:
Hexadecimal notation can be used to specify arbitrary bytes instead of Unicode codepoints. For example, in ASCII compatible mode, \xFF matches the literal byte \xFF , while in Unicode mode, \xFF is a Unicode codepoint that matches its UTF-8 encoding of \xC3\xBF . Similarly for octal notation when enabled.