提高Python代码速度

Question

有人知道如何提高这部分python代码的速度吗？ 它是为处理小文件而设计的（只有几行，而且速度非常快），但我想用大文件（大约50Gb，数百万行）来运行它。在

此代码的主要目标是从文件（.txt）中获取刺并在输入文件中搜索这些刺，并打印出这些刺在输出文件中出现的次数。在

下面是代码：infile、seqList和{}是由optparse在代码开头确定的选项（未显示）

def novo (infile, seqList, out) :
    uDic = dict()
    rDic = dict()
    nmDic = dict()

    with open(infile, 'r') as infile, open(seqList, 'r') as RADlist :
        samples = [line.strip() for line in RADlist]
        lines = [line.strip() for line in infile]

    #Create dictionaires with all the samples
        for i in samples:
            uDic[i.replace(" ","")] = 0
            rDic[i.replace(" ","")] = 0
            nmDic[i.replace(" ","")] = 0

        for k in lines:
            l1 = k.split("\t")
            l2 = l1[0].split(";")
            l3 = l2[0].replace(">","")
            if len(l1)<2:
                continue
            if l1[4] == "U":
                for k in uDic.keys():
                    if k == l3:
                        uDic[k] += 1

            if l1[4] == "R":
                for j in rDic.keys():
                    if j == l3:
                        rDic[j] += 1

            if l1[4] == "NM":
                for h in nmDic.keys():
                    if h == l3:
                        nmDic[h] += 1

    f = open(out, "w")
    f.write("Sample"+"\t"+"R"+"\t"+"U"+"\t"+"NM"+"\t"+"TOTAL"+"\t"+"%R"+"\t"+"%U"+"\t"+"%NM"+"\n")
    for i in samples:
        U = int()
        R = int()
        NM = int ()
        for k, j in uDic.items():
            if k == i:
                U = j
        for o, p in rDic.items():
            if o == i:
                R = p
        for y,u in nmDic.items():
            if y == i:
                NM = u
        TOTAL = int(U + R + NM)
        try:
         f.write(i+"\t"+str(R)+"\t"+str(U)+"\t"+str(NM)+"\t"+str(TOTAL)+"\t"+str(float(R) / TOTAL)+"\t"+str(float(U) / TOTAL)+"\t"+str(float(NM) / TOTAL$
        except:
         continue

    f.close()

Answer 1

在处理50gb文件时，问题不是如何使其更快，而是如何使其可运行 完全。在

主要的问题是，内存不足，需要修改代码来处理文件 没有在内存中保存所有文件，而是在内存中只有一行，这是必需的。在

您的问题中的以下代码正在读取两个文件中的所有行：

with open(infile, 'r') as infile, open(seqList, 'r') as RADlist :
    samples = [line.strip() for line in RADlist]
    lines = [line.strip() for line in infile]
# at this moment you are likely to run out of memory already

#Create dictionaires with all the samples
for i in samples:
    uDic[i.replace(" ","")] = 0
    rDic[i.replace(" ","")] = 0
    nmDic[i.replace(" ","")] = 0

#similar loop over `lines` comes later on

你应该推迟阅读这几行，直到最晚可能的时候，像这样：

^{pr2}$

注意：您想使用line.strip()还是{}？在

这样，您就不必将所有内容保存在内存中。在

有更多的优化选项，但这一个将让你起飞和运行。在

Answer 2

如果您将脚本转换为函数（这使分析更容易），然后查看在编写评测代码时它做了什么：我建议使用runsnake:runsnakerun

Answer 3

如果您提供一些示例输入，它会变得更容易。因为你没有，我还没有测试过这个，但是这个想法很简单——每个文件只迭代一次，使用迭代器，而不是把整个文件读入内存。使用高效的collections.Counter对象来处理计数并最小化内部循环：

def novo (infile, seqList, out):
    from collections import Counter
    import csv


    # Count
    counts = Counter()
    with open(infile, 'r') as infile:
        for line in infile:
            l1 = line.strip().split("\t")
            l2 = l1[0].split(";")
            l3 = l2[0].replace(">","")
            if len(l1)<2:
                continue
            counts[(l1[4], l3)] += 1


    # Produce output
    types = ['R', 'U', 'NM']
    with open(seqList, 'r') as RADlist, open(out, 'w') as outfile:
        f = csv.writer(outfile, delimiter='\t')
        f.writerow(types + ['TOTAL'] + ['%' + t for t in types])
        for sample in RADlist:
            sample = sample.strip()
            countrow = [counts((t, sample)) for t in types]
            total = sum(countrow)
            f.writerow([sample] + countrow + [total] + [c/total for c in countrow])

        samples = [line.strip() for line in RADlist]
        lines = [line.strip() for line in infile]