The question is not whether it could, but whether it should be allowed. IMO, it should not. Raw pointers were never designed to be "ergonomic"; they were designed to be unambiguous and explicit so as to avoid memory safety problems.
(Anyway, I deeply despise conflating ergonomics with arbitrarily cooked up syntactic sugar. That's what most often people actually mean, but there is so much more to ergonomics than that. Having your raw pointers be offset explicily is probably more ergonomic in the bigger picture of things, because it will pre-empt many kinds of hard-to-debug abuse.)
Back to the topic: if you are trying to describe arbitrary graphs, use indices/node IDs/adjacency matrices/etc., basically any other representation instead of raw pointers. Or just use an existing graph library that abstracts away the boilerplate for you.