如何在剩下10%数据的情况下生成索引集

2024-03-28 22:02:02 发布

您现在位置:Python中文网/ 问答频道 /正文

我试图实现jackknife方法来计算大量数据(几百万个数据点)的均值及其相应的方差。因为我有大量的数据,如果每次只漏掉一个元素,那就没什么帮助了。我遗漏了单个元素的代码:

def jackknife(x, func):
    """Jackknife estimate of the estimator func"""
    x = np.asarray(x)
    n = len(x)
    idx = np.arange(n)
    return np.sum(func(x[idx!=i]) for i in range(n))/float(n)

def jackknife_var(x, func):
    """Jackknife estiamte of the variance of the estimator func."""
    x = np.asarray(x)
    n = len(x)
    idx = np.arange(n)
    j_est = jackknife(x, func)
    return j_est, (n-1)/(n + 0.0) * np.sum((func(x[idx!=i]) - j_est)**2.0 for i in range(n))

考虑到庞大的数据集,这是相当缓慢的。有人知道如何有效地实现10%数据遗漏的jackknife方法吗?你知道吗


Tags: ofthe数据方法元素defnpfunc
1条回答
网友
1楼 · 发布于 2024-03-28 22:02:02

或许可以尝试使用^{}?你知道吗

import numpy as np
from sklearn.cross_validation import KFold
import time

def jackknife(x, func):
    """Jackknife estimate of the estimator func"""
    x = np.asarray(x)
    n = len(x)
    idx = np.arange(n)
    return np.sum(func(x[idx!=i]) for i in range(n))/float(n)


def jackknife_v2(x, func):
    """Jackknife estimate of the estimator func"""
    x = np.asarray(x)
    n = len(x)

    kf = KFold(n, n_folds=10)
    return np.mean([np.sum(func(x[idx])) for idx, _ in kf])

x = np.random.normal(12,3, 100000)

start = time.time()
jack1 = jackknife(x, np.var)
end = time.time()
print('jackknife time elapsed:    {:>10f}'.format(end-start))

start = time.time()
jack2 = jackknife_v2(x, np.var)
end = time.time()
print('jackknife_v2 time elapsed: {:>10f}'.format(end-start))

print(jack1, jack2)

## jackknife time elapsed:     59.567203
## jackknife_v2 time elapsed:   0.005295
## 8.98020789924 8.98019104673

相关问题 更多 >