Wrong EOS Masking ?

[sarvam31]

I noticed that eos_token_id is unmasked only when no other token_id is unmasked (referring to get_next_token_acceptance_for_single_stack in IncrementalTokenRecognizer)

But suppose we have a grammar to generate 1 or 2 digit number expressed as
root ::= [0-9] ( [0-9] | )

Now top of stack allows generating 0-9 which is expected but after 1 time step when a token for single digit is sampled by llm stack updates to point to next 0-9 only and no stack entry accounts for epsilon. So, we cannot generate 1 digit number even if grammar allows it.

I thought it should be handled in a way where another stack entry is present, corresponding to epsilon, which points to next possible byte that can be generated if epsilon is chosen for current rule and in case epsilon is one of the last alternates in grammar then unmask eos token too.

Please let me know if I understood it right or you need further information.