使用生成器解析fasta文件（Python）

Question

我正在尝试解析一个很大的fasta文件，但遇到了内存不足的错误。如果能给一些改善数据处理的建议，我会很感激。目前程序可以正确打印出名称，但在文件处理到一半时，我就会出现MemoryError。

这是生成器的代码

def readFastaEntry( fp ):
    name = ""
    seq = ""
    for line in fp:
        if line.startswith( ">" ):
            tmp = []
            tmp.append( name )
            tmp.append( seq )
            name = line
            seq = ""
            yield tmp
        else:
            seq = seq.join( line )

这是调用的代码示例，等这个部分正常工作后会添加更多内容

fp = open( sys.argv[1], 'r' )

for seq in readFastaEntry( fp ) :
    print seq[0]

对于不熟悉fasta格式的人，这里有一个例子

>1 (PB2)
AATATATTCAATATGGAGAGAATAAAAGAACTAAGAGATCTAATGTCACAGTCTCGCACTCGCGAGATAC
TCACCAAAACCACTGTGGACCACATGGCCATAATCAAAAAGTACACATCAGGAAGGCAAGAGAAGAACCC
TGCACTCAGGATGAAGTGGATGATG
>2 (PB1)
AACCATTTGAATGGATGTCAATCCGACTTTACTTTTCTTGAAAGTTCCAGCGCAAAATGCCATAAGCACC
ACATTTCCCTATACTGGAGACCCTCC

每个条目以一个">"开头，后面跟着名称等信息，然后接下来的N行是数据。数据没有明确的结束标志，只有下一行以">"开头时才算结束。

Answer 1

如果你对自己在做什么没有很深的理解，我可能会这样写代码：

def readFastaEntry( fp ):
    name = ""
    while True:
        line = name or f.readline()
        if not line:
            break
        seq = []
        while True:
            name = f.readline()
            if not name or name.startswith(">"):
                break
            else:
                seq.append(name)
        yield (line, "".join(seq))

这段代码的意思是从一个起始行开始，收集数据直到下一个起始行。把seq设置成一个数组，这样可以尽量减少字符串拼接的次数，直到最后一刻。返回一个元组比返回一个列表更合理。

Answer 2

这个格式的pyparsing解析器只需要几行代码就能完成。请看下面代码中的注释：

data = """>1 (PB2) 
AATATATTCAATATGGAGAGAATAAAAGAACTAAGAGATCTAATGTCACAGTCTCGCACTCGCGAGATAC 
TCACCAAAACCACTGTGGACCACATGGCCATAATCAAAAAGTACACATCAGGAAGGCAAGAGAAGAACCC 
TGCACTCAGGATGAAGTGGATGATG 
>2 (PB1) 
AACCATTTGAATGGATGTCAATCCGACTTTACTTTTCTTGAAAGTTCCAGCGCAAAATGCCATAAGCACC 
ACATTTCCCTATACTGGAGACCCTCC"""

from pyparsing import Word, nums, QuotedString, Combine, OneOrMore

# define some basic forms
integer = Word(nums)
key = QuotedString("(", endQuoteChar=")")

# sequences are "words" made up of the characters A, G, C, and T
# we want to match one or more of them, and have the parser combine
# them into a single string (Combine by default requires all of its
# elements to be adjacent within the input string, but we want to allow
# for the intervening end of lines, so we add adjacent=False)
sequence = Combine(OneOrMore(Word("AGCT")), adjacent=False)

# define the overall pattern to scan for - attach results names
# to each matched element
seqEntry = ">" + integer("index") + key("key") + sequence("sequence")

for seq,s,e in seqEntry.scanString(data):
    # just dump out the matched data
    print seq.dump()
    # could also access fields as seq.index, seq.key and seq.sequence

输出结果是：

['>', '1', 'PB2', 'AATATATTCAATATGGAGAGAATAAAAGAACTAAGAGATCTAATGTCACAGTCTCGCACTCGCGAGATACTCACCAAAACCACTGTGGACCACATGGCCATAATCAAAAAGTACACATCAGGAAGGCAAGAGAAGAACCCTGCACTCAGGATGAAGTGGATGATG']
- index: 1
- key: PB2
- sequence: AATATATTCAATATGGAGAGAATAAAAGAACTAAGAGATCTAATGTCACAGTCTCGCACTCGCGAGATACTCACCAAAACCACTGTGGACCACATGGCCATAATCAAAAAGTACACATCAGGAAGGCAAGAGAAGAACCCTGCACTCAGGATGAAGTGGATGATG
['>', '2', 'PB1', 'AACCATTTGAATGGATGTCAATCCGACTTTACTTTTCTTGAAAGTTCCAGCGCAAAATGCCATAAGCACCACATTTCCCTATACTGGAGACCCTCC']
- index: 2
- key: PB1
- sequence: AACCATTTGAATGGATGTCAATCCGACTTTACTTTTCTTGAAAGTTCCAGCGCAAAATGCCATAAGCACCACATTTCCCTATACTGGAGACCCTCC

Answer 3

你有没有考虑过使用BioPython呢？他们有一个序列读取器，可以用来读取fasta文件。如果你有兴趣自己写一个，也可以看看BioPython的代码。

编辑：代码已添加

def read_fasta(fp):
    name, seq = None, []
    for line in fp:
        line = line.rstrip()
        if line.startswith(">"):
            if name: yield (name, ''.join(seq))
            name, seq = line, []
        else:
            seq.append(line)
    if name: yield (name, ''.join(seq))

with open('f.fasta') as fp:
    for name, seq in read_fasta(fp):
        print(name, seq)