我有下面的数据框,我正在对每个ID在一个月内的所有工作日和周末的所有日子之间执行t检验
> +-----+------------+-----------+---------+-----------+ | id | usage_day | dow | tow | daily_avg |
> +-----+------------+-----------+---------+-----------+ | c96 | 01/09/2020 | Tuesday | week | 393.07 |
> +-----+------------+-----------+---------+-----------+ | c96 | 02/09/2020 | Wednesday | week | 10.38 |
> +-----+------------+-----------+---------+-----------+ | c96 | 03/09/2020 | Thursday | week | 429.35 |
> +-----+------------+-----------+---------+-----------+ | c96 | 04/09/2020 | Friday | week | 156.20 |
> +-----+------------+-----------+---------+-----------+ | c96 | 05/09/2020 | Saturday | weekend | 346.22 |
> +-----+------------+-----------+---------+-----------+ | c96 | 06/09/2020 | Sunday | weekend | 106.53 |
> +-----+------------+-----------+---------+-----------+ | c96 | 08/09/2020 | Tuesday | week | 194.74 |
> +-----+------------+-----------+---------+-----------+ | c96 | 10/09/2020 | Thursday | week | 66.30 |
> +-----+------------+-----------+---------+-----------+ | c96 | 17/09/2020 | Thursday | week | 163.84 |
> +-----+------------+-----------+---------+-----------+ | c96 | 18/09/2020 | Friday | week | 261.81 |
> +-----+------------+-----------+---------+-----------+ | c96 | 19/09/2020 | Saturday | weekend | 410.30 |
> +-----+------------+-----------+---------+-----------+ | c96 | 20/09/2020 | Sunday | weekend | 266.28 |
> +-----+------------+-----------+---------+-----------+ | c96 | 23/09/2020 | Wednesday | week | 346.18 |
> +-----+------------+-----------+---------+-----------+ | c96 | 24/09/2020 | Thursday | week | 20.67 |
> +-----+------------+-----------+---------+-----------+ | c96 | 25/09/2020 | Friday | week | 222.23 |
> +-----+------------+-----------+---------+-----------+ | c96 | 26/09/2020 | Saturday | weekend | 449.84 |
> +-----+------------+-----------+---------+-----------+ | c96 | 27/09/2020 | Sunday | weekend | 438.47 |
> +-----+------------+-----------+---------+-----------+ | c96 | 28/09/2020 | Monday | week | 10.44 |
> +-----+------------+-----------+---------+-----------+ | c96 | 29/09/2020 | Tuesday | week | 293.59 |
> +-----+------------+-----------+---------+-----------+ | c96 | 30/09/2020 | Wednesday | week | 194.49 |
> +-----+------------+-----------+---------+-----------+
我的脚本如下,但不幸的是,它太慢了,不是熊猫做事的方式。 我怎样才能做得更有效
from scipy.stats import ttest_ind, ttest_ind_from_stats
p_val = []
stat_flag = []
all_ids = df.id.unique()
alpha = 0.05
print(len(all_ids))
for id in all_ids:
t = df[df.id == id]
group1 = t[t.tow == 'week']
group2 = t[t.tow == 'weekend']
t, p_value_ttest = ttest_ind(group1.daily_avg, group2.daily_avg, equal_var=False)
if p_value_ttest < alpha:
p_val.append(p_value_ttest)
stat_flag.append(1)
else:
p_val.append(p_value_ttest)
stat_flag.append(0)
p-val给出每个id的p值
数据集
根据您提供的数据集:
为了
groupby
清晰起见,我添加了一个具有类似数据的新id
:MCVE
您不必求助于任何显式循环,而是利用^{} 方法,该方法对帧进行操作,也可与^{} 一起使用
为此,我们定义了一个函数,该函数在数据帧上执行所需的测试(
groupby
将为对应于分组键组合的每个子数据帧调用此方法):然后,在
groupby
调用之后链接apply
就足够了:结果是:
重构
一旦您了解了这种方法的威力,就可以将上述代码重构为可重用的函数,例如:
它允许您根据需要调整统计测试和数据帧列
我无法在没有样本数据的情况下进行基准测试,但也许您可以尝试使用groupby而不是for循环:
相关问题 更多 >
编程相关推荐