Where should I store the data in my parser?

[robin]

I am re-writting a small C compiler (chibicc) from C to Rust. I have some trouble converting the C code into idiomatic Rust.

typedef enum {
  TK_PUNCT, // Punctuators
  TK_NUM,   // Numeric literals
  TK_EOF,   // End-of-file markers
} TokenKind;

// Token type
typedef struct Token Token;
struct Token {
  TokenKind kind; // Token kind
  int val;        // If kind is TK_NUM, its value
  /* ... other fields */
};

A simple convertion could be:

enum TokenKind {
  Punctuator,
  Numeric,
  Eof, /// End-of-file markers
};
struct Token {
  kind: TokenKind,
  val: i64, // If kind is Numeric, its value
  /* ... other fields */
}

However,

val: i64 // If kind is Numeric, its value

This comment feels very wrong for Rust, but I fail to re-write it in a more idiomatic way.

Should I had the data into TokenKind?

enum TokenKind {
  Punctuator,
  Numeric(i64),
  Eof,
};
struct Token {
  kind: TokenKind,
  /* ... other fields */
}

This make it harder to compare the type of the token: if tk == TokenKind::NUMERIC doesn't work anymore.

[asymmetrikon]

You can compare it like:

if let tk.kind = TokenKind::Numeric(_)

Do any other fields care about what TokenKind is?

[idkravitz]

This is a perfect example what one does in C with a lack of sum types. TokenKind is actually an enum tag or discriminant, which should be thrown away in rust implementation, since you have a nice sum types in Rust, make it simply a Token enum.

[pandaman64]

You can also write if matches!(tk.kind, TokenKind::Numeric(_)).