Lexer Token化SKIP和NEWLINE值，如何隐藏？

Question

我下面的词法分析器代码会生成“SKIP”和“NEWLINE”这两个值，而我其实是想把它们隐藏起来。有人能告诉我该怎么做才能隐藏它们吗？

        tok_regex = '|'.join('(?P<%s>%s)' % pair for pair in token_specification)
        get_token = re.compile(tok_regex, re.DOTALL).match
        mo = get_token(self.input, self.p)

        self.p = mo.end()

        if self.p >= len(self.input):
            self.c = "EOF"
        else:
            if mo is not None:
                self.c = mo.lastgroup
                self.val = mo.group(self.c)
                if self.c == 'NEWLINE':
                    self.line += 1

    def next_token(self):

        keywords = {'digraph', 'subgraph', 'node', 'edge', 'strict', 'graph'}
        self.tokenize()
        if self.c != 'SKIP' and self.c != "NEWLINE":

            if self.c == 'ID' and self.val in keywords:
                self.c = self.val
        return TokenTypes.Token(self.c, self.val)
        # if self.p != len(self.input):
        #    raise RuntimeError('Unexpected character %r on line %d' %(self.input[self.p], self.line))

def main():
    statements = '''
        digraph G {
            main [shape=box]; 
        }
    '''
    lexer = Lexer(statements)
    t = lexer.next_token()
    i = 0
    while (t.typ != "EOF"):
        i += 1
        print(t)
        t = lexer.next_token()

if __name__ == "__main__":
    sys.exit(main())

我正在尝试通过导入上面词法分析器生成的标记来为点语法生成一个抽象语法树。但是那些不需要的SKIP和NEWLINE标记让我遇到了麻烦。

这里有一个输入示例：

digraph G { main [shape=box]; }

这个输入会产生以下输出：

NEWLINE digraph SKIP ID SKIP LBRACE NEWLINE ID SKIP LBRACKET ID EQUAL ID RBRACKET END SKIP NEWLINE SKIP RBRACE

但它应该输出：

digraph ID LBRACE ID LBRACKET ID EQUAL ID RBRACKET END RBRACE

Answer 1

我觉得你想要修改一下 next_token() 这个方法，让它在遇到 NEWLINE 或 SKIP 的时候去解析另一个标记。比如说：

def next_token(self):
    keywords = {'digraph', 'subgraph', 'node', 'edge', 'strict', 'graph'}

    # Parse next token.
    self.tokenize()
    while self.c == "SKIP" or self.c == "NEWLINE":
        # Discard this token and parse another token.
        self.tokenize()

    if self.c == 'ID' and self.val in keywords:
        self.c = self.val

    return TokenTypes.Token(self.c, self.val)
    # if self.p != len(self.input):
    #    raise RuntimeError('Unexpected character %r on line %d' %(self.input[self.p], self.line))