基于列表在Python Numpy中快速排列数据的方法

Question

我在使用numpy整理数据时遇到了问题。

比如我有一组数据范围：

numpy.array([1,3,5,4,6])

还有一些数据：

numpy.array([1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19])

我需要把这些数据整理成：

numpy.array([

[1,9999,9999,9999,9999,9999,9999]

[2,3,4,9999,9999,9999]

[5,6,7,8,9,9999]

[10,11,12,13,9999,9999]

[14,15,16,17,18,19]

])

我觉得这跟diag（对角线）或trace（迹）功能有点像。

我通常是通过基本的循环来完成这个工作……那么numpy有没有类似的功能，可以让这个过程更快呢？

Answer 1

又一次，Sven把我们都甩在了后面 :) 我来试试我的简单方法，

from numpy import arange,array,split
from numpy import concatenate as cat
from numpy import repeat as rep

a = arange(1,20)

i = array([1,3,5,4,6])
j = max(i) - i

s = split(a,i.cumsum())
z = array([cat((t,rep(9999,k))) for t,k in zip(s[:-1],j)])

print z

可以实现，

[[   1 9999 9999 9999 9999 9999]
 [   2    3    4 9999 9999 9999]
 [   5    6    7    8    9 9999]
 [  10   11   12   13 9999 9999]
 [  14   15   16   17   18   19]]

Answer 2

这里有一种方法，可以在不使用任何Python循环的情况下，通过高级索引来实现：

r = numpy.array([1,3,5,4,6])
data = numpy.array([1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19])

result = numpy.empty((len(r), r.max()), data.dtype)
result.fill(9999)
cum_r = numpy.r_[0, r.cumsum()]
i = numpy.arange(len(r)).repeat(r)
j = numpy.arange(cum_r[-1]) - cum_r[:-1].repeat(r)
result[i, j] = data
print result

输出结果是

[[   1 9999 9999 9999 9999 9999]
 [   2    3    4 9999 9999 9999]
 [   5    6    7    8    9 9999]
 [  10   11   12   13 9999 9999]
 [  14   15   16   17   18   19]]

Answer 3

这里有一些整理数据的方法：

from numpy import arange, array, ones, r_, zeros
from numpy.random import randint

def gen_tst(m, n):
    a= randint(1, n, m)
    b, c= arange(a.sum()), ones((m, n), dtype= int)* 999
    return a, b, c

def basic_1(a, b, c):
    # some assumed basic iteration based
    n= 0
    for k in xrange(len(a)):
        m= a[k]
        c[k, :m], n= b[n: n+ m], n+ m

def advanced_1(a, b, c):
    # based on Svens answer
    cum_a= r_[0, a.cumsum()]
    i= arange(len(a)).repeat(a)
    j= arange(cum_a[-1])- cum_a[:-1].repeat(a)
    c[i, j]= b

def advanced_2(a, b, c):
    # other loopless version
    c[arange(c.shape[1])+ zeros((len(a), 1), dtype= int)< a[:, None]]= b

还有一些时间记录：

In []: m, n= 10, 100
In []: a, b, c= gen_tst(m, n)
In []: 1.* a.sum()/ (m* n)
Out[]: 0.531
In []: %timeit advanced_1(a, b, c)
10000 loops, best of 3: 99.2 us per loop
In []: %timeit advanced_2(a, b, c)
10000 loops, best of 3: 68 us per loop
In []: %timeit basic_1(a, b, c)
10000 loops, best of 3: 47.1 us per loop

In []: m, n= 50, 500
In []: a, b, c= gen_tst(m, n)
In []: 1.* a.sum()/ (m* n)
Out[]: 0.455
In []: %timeit advanced_1(a, b, c)
1000 loops, best of 3: 1.03 ms per loop
In []: %timeit advanced_2(a, b, c)
1000 loops, best of 3: 1.06 ms per loop
In []: %timeit basic_1(a, b, c)
1000 loops, best of 3: 227 us per loop

In []: m, n= 250, 2500
In []: a, b, c= gen_tst(m, n)
In []: 1.* a.sum()/ (m* n)
Out[]: 0.486
In []: %timeit advanced_1(a, b, c)
10 loops, best of 3: 30.4 ms per loop
In []: %timeit advanced_2(a, b, c)
10 loops, best of 3: 32.4 ms per loop
In []: %timeit basic_1(a, b, c)
1000 loops, best of 3: 2 ms per loop

所以基本的循环看起来效率还不错。

更新:
当然，基于基本循环的实现性能还可以进一步提升。作为一个起点建议，可以考虑这个（基于减少加法的基本循环）：

def basic_2(a, b, c):
    n= 0
    for k, m in enumerate(a):
        nm= n+ m
        c[k, :m], n= b[n: nm], nm