Simon's questions

我有一个包含多个 ID 和对应状态的数据框。我想分析状态随时间的变化情况，并有效地呈现这些信息。

以下是一个例子：

import polars as pl

df = pl.DataFrame({
    "ID": [1, 2, 3],
    "T0": ["A", "B", "C"],
    "T1": ["B", "B", "A"],  
})

一种方法是“连接”列，然后执行value_counts()更改列

df = df.with_columns(
    (pl.col("T0") + " -> " + pl.col("T1")).alias("Change")
)

但是，可能有更好的方法，甚至有一个内置函数可以更有效地实现我的需要。

电流输出：

shape: (3, 4)
┌─────┬─────┬─────┬────────┐
│ ID  ┆ T0  ┆ T1  ┆ Change │
│ --- ┆ --- ┆ --- ┆ ---    │
│ i64 ┆ str ┆ str ┆ str    │
╞═════╪═════╪═════╪════════╡
│ 1   ┆ A   ┆ B   ┆ A -> B │
│ 2   ┆ B   ┆ B   ┆ B -> B │
│ 3   ┆ C   ┆ A   ┆ C -> A │
└─────┴─────┴─────┴────────┘

shape: (3, 2)
┌────────┬───────┐
│ Change ┆ count │
│ ---    ┆ ---   │
│ str    ┆ u32   │
╞════════╪═══════╡
│ C -> A ┆ 1     │
│ B -> B ┆ 1     │
│ A -> B ┆ 1     │
└────────┴───────┘

我有一个包含 ID、DATE 和 OS 的 polars 数据框。对于每一天，我想计算出到那一天为止有多少个唯一 ID。

import polars as pl
df = (
    pl.DataFrame(
        {
            "DAY": [1,1,1,2,2,2,3,3,3],
            "OS" : ["A","B","A","B","A","B","A","B","A"],
            "ID": ["X","Y","Z","W","X","J","K","L","X"]
        }
    )
)

期望输出：

shape: (3, 3)
┌─────┬─────┬─────┐
│ DAY ┆ A   ┆ B   │
│ --- ┆ --- ┆ --- │
│ i64 ┆ i64 ┆ i64 │
╞═════╪═════╪═════╡
│ 1   ┆ 2   ┆ 1   │
│ 2   ┆ 2   ┆ 3   │
│ 3   ┆ 3   ┆ 4   │
└─────┴─────┴─────┘

它看起来应该是这样的，因为在第 1 天，有 3 个值和 3 个 ID。在第 2 天，ID“X”使用相同的 OS 重复，因此，列 A 保持不变，而其他 2 个不同，因此将 2 添加到 B。在第 3 天，ID X 与 A 重复，而其他 2 个不同，因此它再次对每列求和。

我认为可以采用以下方法解决：

(
    df
    .pivot(
        index="DAY",
        on="OS",
        aggregate_function=(pl.col("ID").cum_sum().unique())
    )
)

我正在尝试使用 scikit-learnLabelEncoder和 Polars DataFrame 来编码分类列。我正在使用以下代码。

import polars as pl

from sklearn.preprocessing import LabelEncoder

df = pl.DataFrame({
    "Color" : ["red","white","blue"]
})

enc = LabelEncoder()

但是，会出现错误。

ValueError: y should be a 1d array, got an array of shape () instead.

接下来，我尝试将该列转换为 NumPy。

df.with_columns(
    enc.fit_transform(pl.col("Color").to_numpy()) 
)

现在，出现了一个不同的错误。

AttributeError: 'Expr' object has no attribute 'to_numpy'

注意。我发现.cast(pl.Categorical).to_physical()可以使用来获得所需的结果。不过，我更喜欢transform()在我的测试数据集上使用类似的东西。

df.with_columns(
    pl.col("Color").cast(pl.Categorical).to_physical().alias("Color_encoded")
)

我正在研究文本分类任务，并决定为此使用 PyTorch 模型。该过程主要涉及以下步骤：

1. 加载并处理文本。
2. 使用 TF-IDF 矢量化器。
3. 建立神经网络并保存 TF-IDF 矢量化器和模型以预测新数据。

但是，我每天都需要对新的评论进行分类，并纠正任何错误的分类。

目前，我的方法是将具有正确分类的新评论添加到数据集并重新训练整个模型。这个过程很耗时，而且新评论可能会在验证过程中丢失。我想用新分类的文本创建一个新的数据集，并继续对这些新数据进行训练（新评论是手动分类的，因此每个标签都是正确的）。

使用 GPT 和一些在线代码，我编写了所需的流程，但是，我不确定它是否按预期工作，或者我犯了一些不应该发生的愚蠢错误。

因此主要问题是：

1. 我如何检查解决该问题的建议方法是否如我预期的那样有效？
2. 当矢量化器面临新的标记时，我该怎么做呢？我只能做一个吗.fit_transform()？否则我会丢失原始的矢量化器吗？

以下是完整的训练过程：

import torch
from torch import nn
from torch.utils.data import Dataset, DataLoader, random_split
from sklearn.preprocessing import LabelEncoder
import polars as pl
from sklearn.model_selection import train_test_split
from sklearn.feature_extraction.text import TfidfVectorizer
import joblib

set1 = (
    pl
    .read_csv(
        "set1.txt",
        separator=";",
        has_header=False,
        new_columns=["text","label"]
    )
)

# since the dateset its unbalanced, im going to force to have more balance

fear_df = set1.filter(pl.col("label") == "fear")
joy_df = set1.filter(pl.col("label") == "joy").sample(n=2500)
sadness_df = set1.filter(pl.col("label") == "sadness").sample(n=2500)
anger_df = set1.filter(pl.col("label") == "anger")

train_df = pl.concat([fear_df,joy_df,sadness_df,anger_df])

"""
The text its already clean, so im going to change the labels to numeric
and then split it on train, test ,val
"""

label_mapping = {
    "anger": 0,
    "fear": 1,
    "joy": 2,
    "sadness": 3
}

train_mapped = (
    train_df
    .with_columns(
        pl.col("label").replace_strict(label_mapping, default="other").cast(pl.Int16)
    )
   
)

train_set, pre_Test = train_test_split(train_mapped,
                                    test_size=0.4,
                                    random_state=42,
                                    stratify=train_mapped["label"])

test_set, val_set = train_test_split(pre_Test,
                                    test_size=0.5,
                                    random_state=42,
                                    stratify=pre_Test["label"]) 

# Vectorize text data using TF-IDF
vectorizer = TfidfVectorizer(max_features=30000, ngram_range=(1, 2))

X_train_tfidf = vectorizer.fit_transform(train_set['text']).toarray()
X_val_tfidf = vectorizer.transform(val_set['text']).toarray()
X_test_tfidf = vectorizer.transform(test_set['text']).toarray()

y_train = train_set['label']
y_val = val_set['label']
y_test = test_set['label']

class TextDataset(Dataset):
    def __init__(self, texts, labels):
        self.texts = texts
        self.labels = labels
    
    def __len__(self):
        return len(self.texts)
    
    def __getitem__(self, idx):
        text = self.texts[idx]
        label = self.labels[idx]
        return text, label
    
train_dataset = TextDataset(X_train_tfidf, y_train)
val_dataset = TextDataset(X_val_tfidf, y_val)
test_dataset = TextDataset(X_test_tfidf, y_test)

batch_size = 32
train_loader = DataLoader(train_dataset, batch_size=batch_size, shuffle=True)
val_loader = DataLoader(val_dataset, batch_size=batch_size)
test_loader = DataLoader(test_dataset, batch_size=batch_size)

class TextClassificationModel(nn.Module):
    def __init__(self, input_dim, num_classes):
        super(TextClassificationModel, self).__init__()
        self.fc1 = nn.Linear(input_dim, 64)
        self.dropout1 = nn.Dropout(0.5)
        self.fc2 = nn.Linear(64, 32)
        self.dropout2 = nn.Dropout(0.5)
        self.fc3 = nn.Linear(32, num_classes)

    def forward(self, x):
        x = torch.relu(self.fc1(x))
        x = self.dropout1(x)
        x = torch.relu(self.fc2(x))
        x = self.dropout2(x)
        x = torch.softmax(self.fc3(x), dim=1)
        return x
    
input_dim = X_train_tfidf.shape[1]
model = TextClassificationModel(input_dim, 4)

# Define loss and optimizer
criterion = nn.CrossEntropyLoss()
optimizer = torch.optim.Adamax(model.parameters())

# Training loop
num_epochs = 17
best_val_acc = 0.0
best_model_path = "modelbest.pth"

for epoch in range(num_epochs):
    model.train()
    for texts, labels in train_loader:
        texts, labels = texts.float(), labels.long()
        outputs = model(texts)
        loss = criterion(outputs, labels)
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

    # Validation
    model.eval()
    correct, total = 0, 0
    with torch.no_grad():
        for texts, labels in val_loader:
            texts, labels = texts.float(), labels.long()
            outputs = model(texts)
            _, predicted = torch.max(outputs.data, 1)
            total += labels.size(0)
            correct += (predicted == labels).sum().item()
    val_acc = correct / total
    if val_acc > best_val_acc:
        best_val_acc = val_acc
        torch.save(model.state_dict(), best_model_path)

    print(f'Epoch [{epoch+1}/{num_epochs}], Loss: {loss.item():.4f}, Val Acc: {val_acc:.4f}')

# Load the best model
model.load_state_dict(torch.load(best_model_path))

# Load the best model
model.load_state_dict(torch.load(best_model_path))

# Test the model
model.eval()
correct, total = 0, 0
with torch.no_grad():
    for texts, labels in test_loader:
        texts, labels = texts.float(), labels.long()
        outputs = model(texts)
        _, predicted = torch.max(outputs.data, 1)
        total += labels.size(0)
        correct += (predicted == labels).sum().item()
test_acc = correct / total
print(f'Test Acc: {test_acc:.3f}')


# Save the TF-IDF vectorizer
vectorizer_path = "tfidf_vectorizer.pkl"
joblib.dump(vectorizer, vectorizer_path)

# Save the PyTorch model
model_path = "text_classification_model.pth"
torch.save(model.state_dict(), model_path)

建议代码：

import torch
import joblib
import polars as pl
from sklearn.model_selection import train_test_split
from torch import nn
from torch.utils.data import Dataset, DataLoader

# Load the saved TF-IDF vectorizer
vectorizer_path = "tfidf_vectorizer.pkl"
vectorizer = joblib.load(vectorizer_path)

input_dim = len(vectorizer.get_feature_names_out())

class TextClassificationModel(nn.Module):
    def __init__(self, input_dim, num_classes):
        super(TextClassificationModel, self).__init__()
        self.fc1 = nn.Linear(input_dim, 64)
        self.dropout1 = nn.Dropout(0.5)
        self.fc2 = nn.Linear(64, 32)
        self.dropout2 = nn.Dropout(0.5)
        self.fc3 = nn.Linear(32, num_classes)

    def forward(self, x):
        x = torch.relu(self.fc1(x))
        x = self.dropout1(x)
        x = torch.relu(self.fc2(x))
        x = self.dropout2(x)
        x = torch.softmax(self.fc3(x), dim=1)
        return x
    
# Load the saved PyTorch model
model_path = "text_classification_model.pth"
model = TextClassificationModel(input_dim, 4)
model.load_state_dict(torch.load(model_path))

# Map labels to numeric values
label_mapping = {"anger": 0, "fear": 1, "joy": 2, "sadness": 3}
sentiments = ["fear","joy","sadness","anger"]

new_data = (
    pl
    .read_csv(
        "set2.txt",
        separator=";",
        has_header=False,
        new_columns=["text","label"]
    )
    .filter(pl.col("label").is_in(sentiments))
    .with_columns(
        pl.col("label").replace_strict(label_mapping, default="other").cast(pl.Int16)
    )
    
)
# Vectorize the new text data using the loaded TF-IDF vectorizer
X_new = vectorizer.transform(new_data['text']).toarray()
y_new = new_data['label']

class TextDataset(Dataset):
    def __init__(self, texts, labels):
        self.texts = texts
        self.labels = labels
    
    def __len__(self):
        return len(self.texts)
    
    def __getitem__(self, idx):
        text = self.texts[idx]
        label = self.labels[idx]
        return text, label

batch_size = 10
   
# Create DataLoader for the new training data
new_train_dataset = TextDataset(X_new, y_new)
new_train_loader = DataLoader(new_train_dataset, batch_size=batch_size, shuffle=True)

# Define loss and optimizer
criterion = nn.CrossEntropyLoss()
optimizer = torch.optim.Adamax(model.parameters())

num_epochs = 5
new_best_model_path = "modelbest.pth"
for epoch in range(num_epochs):
    model.train()
    for texts, labels in new_train_loader:
        texts, labels = texts.float(), labels.long()
        outputs = model(texts)
        loss = criterion(outputs, labels)
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()
        torch.save(model.state_dict(), new_best_model_path)
        
print(f'Epoch [{epoch+1}/{num_epochs}], Loss: {loss.item():.4f}')

# Save the PyTorch model
new_best_model_path = "new_moedl.pth"
torch.save(model.state_dict(), new_best_model_path)

数据集可以在这里找到

我正在尝试将 DataFrame 中的浮点数截断为所需的小数位数。我发现这可以在这里使用 Pandas 和 NumPy 来完成，但我还发现使用 也可以polars.Config.set_float_precision。

以下是我当前的方法，但我想我可能会采取额外的步骤。

import polars as pl

data = {
    "name": ["Alice", "Bob", "Charlie"],
    "grade": [90.23456, 80.98765, 85.12345],
}

df = pl.DataFrame(data)
df.with_columns(
# Convert to string
    pl.col("grade").map_elements(lambda x: f"{x:.5f}", return_dtype=pl.String).alias("formatted_grade")
).with_columns(
# Slice to get my desired decimals
    pl.col("formatted_grade").str.slice(0, length = 4)
).with_columns(
# Convert it again to Float
    pl.col("formatted_grade").cast(pl.Float64)
)

我有一段Python脚本，它从一个JSON文件中加载搜索词，并处理Pandas DataFrame以添加新列，指示文本数据中是否存在某些词。然而，我想修改脚本，使用Polars而不是Pandas，并可能去除对JSON的依赖。这是我原始的代码： ```python import pandas as pd import json class SearchTermLoader: def __init__(self, json_file): self.json_file = json_file def load_terms(self): with open(self.json_file, 'r') as f: data = json.load(f) terms = {} for phase_name, phase_data in data.items(): terms[phase_name] = ( phase_data.get('words', []), phase_data.get('exact_phrases', []) ) return terms class DataFrameProcessor: def __init__(self, df: pd.DataFrame, col_name: str) -> None: self.df = df self.col_name = col_name def add_contains_columns(self, search_terms): columns_to_add = ["type1", "type2"] for column in columns_to_add: self.df[column] = self.df[self.col_name].apply( lambda text: any( term in text for term in search_terms.get(column, ([], []))[0] + search_terms.get(column, ([], []))[1] ) ) return self.df # 示例用法 data = {'text_column': ['The apple is red', 'I like bananas', 'Cherries are tasty']} df = pd.DataFrame(data) term_loader = SearchTermLoader('word_list.json') search_terms = term_loader.load_terms() processor = DataFrameProcessor(df, 'text_column') new_df = processor.add_contains_columns(search_terms) new_df ``` 这是JSON文件的示例： ```json { "type1": { "words": ["apple", "tasty"], "exact_phrases": ["soccer ball"] }, "type2": { "words": ["banana"], "exact_phrases": ["red apple"] } } ``` 我知道我可以使用`.str.contains()`函数，但我想用它来匹配特定的词和确切的短语。你能提供一些如何开始的指导吗？