优化CSV处理脚本 - Python、Perl和Java

Question

我正在尝试写一个简单的脚本，快速处理一个嘈杂的CSV文件。
我只想从一个很大的CSV文件（经过gzip压缩）中提取几个列，然后写入一个新的CSV文件，里面是经过筛选的数据。还添加了一个简单的过滤方法，检查第一列的长度是否等于15。

我比较了perl、java和python的脚本，发现Java的速度比其他语言快很多。我在想，是否还有其他方法可以优化每种语言的这个简单过程呢？

对于一个800MB的gzip文件，各种语言的基准测试时间如下：
1. Java: 74秒
2. Python: 197秒
3. Perl: 7分钟

Python代码：

import gzip
import csv
import time

def getArray(row): 
    columns = [0,4,5,26,33,34,35,36,39,41,42,47,54,65,66,72,73,91]
    row_filt = []
    for i in columns:
        row_filt.append(row[i])
    return row_filt

filename = 'Very_large_csv.gz' 
outfile = filename + '.csv'
csv.register_dialect('wifi', delimiter='|', quoting=csv.QUOTE_NONE, quotechar = '')
start_time = time.time()

try:
    f = gzip.open(filename, 'rb')
    f2 = open(outfile, 'wb')
    reader = csv.reader(f, dialect = 'wifi')
    writer = csv.writer(f2, dialect = 'wifi')
    header = reader.next()
    writer.writerow(getArray(header))
    for row in reader:
        if (len(row[0]) != 15):
            continue
        writer.writerow(getArray(row))
    print(time.time() - start_time)

finally:
    f.close()

Perl代码：

use strict;
use warnings;
use Cwd;
use IO::Uncompress::Gunzip qw($GunzipError);
use Text::CSV_XS;
use Time::Piece;
use Time::Seconds;

my @COLUMNS = (0,4,5,26,33,34,35,36,39,41,42,47,54,65,66,72,73,91);

my $csv = Text::CSV_XS->new ({  binary => 1,
                                sep_char => '|',
                                escape_char => undef,
                                eol => "\n",
                                quote_char => undef
                                });

my $infile='Very_large_csv.gz';

my $fh = IO::Uncompress::Gunzip->new($infile) or die "IO::Uncompress::Gunzip failed: $GunzipError\n";

my $outfile = $infile . ".csv";
open my $out, ">", $outfile or die "$outfile: $!\n";

my @header_row = split(/\|/,<$fh>);
my @header = ();
foreach my $column (@COLUMNS)
{
    push @header, $header_row[$column];
}
my $header_filter = \@header;   
$csv->print ($out, $header_filter);

print "Start.\n";
while (my $row = $csv->getline($fh))
{
    length($row->[0]) == 15 or next;
    my @data = ();
    foreach my $column (@COLUMNS)
    {
        push @data, $row->[$column];
    }
    my $row_filter = \@data;

    $csv->print($out, $row_filter); 
}
$csv->eof or $csv->error_diag ();

close $fh;
close $out or die "$outfile: $!";

Java代码：

public class NoiseFilter {
    static final int[] columns = {0,4,5,26,33,34,35,36,39,41,42,47,54,65,66,72,73,91};

    public static void main(String[] args) throws IOException {
        fname='Very_large_csv.gz';
        GZIPInputStream gzip = new GZIPInputStream(new FileInputStream(fname));
        BufferedReader reader = new BufferedReader(new InputStreamReader(gzip));

        String line = reader.readLine(); // Header
        String[] header = line.split("\\|");
        PrintWriter ww = new PrintWriter(fname + ".csv");
        printRow(header, ww);

        while ((line = reader.readLine()) != null) {
            String[] data = line.split("\\|",-1);
            if (data[0].length() != 15 ) { continue; }
            printRow(data, ww);
        }

        ww.close();
        reader.close();
    }

    private static void printRow(String[] row, PrintWriter writer) {
        for (int i = 0; i < columns.length; i++) {
            if (i == 0) {
                writer.print(row[columns[i]]);
            } else {
                writer.print("|" + row[columns[i]]);
            }
        }

        writer.print("\n");
    }
}

我对Python代码进行了修改，运行时间变成了95秒，和Java差不多。

def getArray(line): 
    string=''
    row=line.split(',')
    for i in columns:   
        string+=(row[i]+',')
    return string+'\n'

try:
    f = gzip.open(filename, 'rb')
    f2 = open(outfile, 'wb')

    header = f.readline() 
    f2.write(getArray(header))

    for line in f:
        f2.write(getArray(line))
finally:
    f.close()

性能优化脚本优化编程语言比较数据过滤基准测试数据筛选 CSV处理 gzip压缩

优化CSV处理脚本 - Python、Perl和Java

2 个回答

解释：

撰写回答