Token

A string with meta-information, that is produced by the lexer.

When parsing text, the resulting chunks of the input that haven't been discarded,
will end up in the tree as Token instances. The Token class inherits from Python's ``str``,
so normal string comparisons and operations will work as expected.

Attributes:
    type: Name of the token (as specified in grammar)
    value: Value of the token (redundant, as ``token.value == token`` will always be true)
    start_pos: The index of the token in the text
    line: The line of the token in the text (starting with 1)
    column: The column of the token in the text (starting with 1)
    end_line: The line where the token ends
    end_column: The next column after the end of the token. For example,
        if the token is a single character with a column value of 4,
        end_column will be 5.
    end_pos: the index where the token ends (basically ``start_pos + len(token)``)

A string with meta-information, that is produced by the lexer.

When parsing text, the resulting chunks of the input that haven't been discarded,
will end up in the tree as Token instances. The Token class inherits from Python's ``str``,
so normal string comparisons and operations will work as expected.

Attributes:
    type: Name of the token (as specified in grammar)
    value: Value of the token (redundant, as ``token.value == token`` will always be true)
    start_pos: The index of the token in the text
    line: The line of the token in the text (starting with 1)
    column: The column of the token in the text (starting with 1)
    end_line: The line where the token ends
    end_column: The next column after the end of the token. For example,
        if the token is a single character with a column value of 4,
        end_column will be 5.
    end_pos: the index where the token ends (basically ``start_pos + len(token)``)