Question:
So I am making a programming language (RedStone) in Python. The syntax is similar to Python’s but a bit different. At the start of every line there should be a tilde (~).
The input ~ print("Hello world") fails with an illegal character error.
Looking at Lexer.make_tokens, it will always return an error for input with a tilde.
A solution would be to check for the tilde (and any whitespace following it) and then remove it or ignore it, in the Lexer.text string.
In Lexer.make_tokens, have a bool variable start_tilde. Reset it every newline. After or before the newline elif case, put a new case for the tilde character that raises an error if one was already encountered this line, else set start_tilde to True. After both of those cases, put a check for start_tilde, raising an error if it is False (missing).
Your current method doesnt work because keywords can only be lexed after the lexer encounters a letter. The best way to fix this would be to remove the tilde from every token and just consume a tilde before parsing a statement.