Scikit-Learn中对Polars DataFrame进行置换和更新

Question

我正在尝试重写scikit-learn的置换重要性的源代码，目标是：

1. 与Polars兼容
2. 支持特征的聚类

import polars as pl
import polars.selectors as cs
import numpy as np

from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split

X, y = make_classification(
    n_samples=1000,
    n_features=10,
    n_informative=3,
    n_redundant=0,
    n_repeated=0,
    n_classes=2,
    random_state=42,
    shuffle=False,
)
X_train, X_test, y_train, y_test = train_test_split(X, y, stratify=y, random_state=42)
feature_names = [f"feature_{i}" for i in range(X.shape[1])]

X_train_polars = pl.DataFrame(X_train, schema=feature_names)
X_test_polars = pl.DataFrame(X_test, schema=feature_names)
y_train_polars = pl.Series(y_train, schema=["target"])
y_test_polars = pl.Series(y_test, schema=["target"])

为了获得一组特征的未来重要性，我们需要同时打乱这组特征，然后将其传入评分器，以便与基准分数进行比较。

不过，我在替换多个Polars数据框的列时遇到了困难，因为我需要检查特征的聚类：

from sklearn.utils import check_random_state
random_state = check_random_state(42)
random_seed = random_state.randint(np.iinfo(np.int32).max + 1)

X_train_permuted = X_train_polars.clone()
shuffle_arr = np.array(X_train_permuted[:, ["feature_0", "feature_1"]])

random_state.shuffle(shuffle_arr)
X_train_permuted.replace_column( # This operation is in place
                0, 
                pl.Series(name="feature_0", values=shuffle_arr))

通常情况下，shuffle_arr的形状是(n_samples,)，这样可以很容易地使用polars.DataFrame.replace_column()替换Polars数据框中相关的列。但在这个例子中，shuffle_arr的形状是多维的(n_samples, n_features in a cluster)。那么，有什么有效的方法可以替换相关的列呢？

Answer 1

简而言之

X_train_permuted = (
    X_train_permuted.with_columns(
        pl.DataFrame(shuffle_arr, schema=features)
    )
)

---

我们来做一个简单的例子。

X_train_permuted

import polars as pl
import numpy as np

np.random.seed(0)

data = {f'feature_{i}': np.random.rand(4) for i in range(0,3)}

X_train_permuted = pl.DataFrame(data)

X_train_permuted

shape: (4, 3)
┌───────────┬───────────┬───────────┐
│ feature_0 ┆ feature_1 ┆ feature_2 │
│ ---       ┆ ---       ┆ ---       │
│ f64       ┆ f64       ┆ f64       │
╞═══════════╪═══════════╪═══════════╡
│ 0.548814  ┆ 0.423655  ┆ 0.963663  │
│ 0.715189  ┆ 0.645894  ┆ 0.383442  │
│ 0.602763  ┆ 0.437587  ┆ 0.791725  │
│ 0.544883  ┆ 0.891773  ┆ 0.528895  │
└───────────┴───────────┴───────────┘

打乱 feature_0 和 feature_1

用一个列表来记录你要打乱的特征：features = ["feature_0", "feature_1"]。

features = ["feature_0", "feature_1"]

shuffle_arr = np.array(X_train_permuted[:, features])

from sklearn.utils import check_random_state

random_state = check_random_state(42)
random_seed = random_state.randint(np.iinfo(np.int32).max + 1)

random_state.shuffle(shuffle_arr)

shuffle_arr

array([[0.71518937, 0.64589411],
       [0.60276338, 0.43758721],
       [0.5488135 , 0.4236548 ],
       [0.54488318, 0.891773  ]])

用 shuffle_arr 的值替换 X_train_permuted 中相关的列

• 使用 pl.DataFrame.with_columns，并传入一个带有 schema=features 的 pl.DataFrame。

X_train_permuted = (
    X_train_permuted.with_columns(
        pl.DataFrame(shuffle_arr, schema=features)
    )
)

X_train_permuted

shape: (4, 3)
┌───────────┬───────────┬───────────┐
│ feature_0 ┆ feature_1 ┆ feature_2 │
│ ---       ┆ ---       ┆ ---       │
│ f64       ┆ f64       ┆ f64       │
╞═══════════╪═══════════╪═══════════╡
│ 0.715189  ┆ 0.645894  ┆ 0.963663  │
│ 0.602763  ┆ 0.437587  ┆ 0.383442  │
│ 0.548814  ┆ 0.423655  ┆ 0.791725  │
│ 0.544883  ┆ 0.891773  ┆ 0.528895  │
└───────────┴───────────┴───────────┘