主页 / coding / 问题 / 79022157
Accepted
elokema
Asked: 2024-09-25 17:40:01 +0800 CST

Pyspark：如何将过去几个月的所有数据获取到当前月份？

  • 772

在 pyspark 中，我有这张表，我想获取到当前月份的所有过去月份 =‘九月’

一月 二月 行进 四月 可能 六月 七月 八月 九月 十月 十一月 十二月
10 20 三十 40 50 60 70 80 100 110 120 150

然后，我最后会得到这个表格：

一月 二月 行进 四月 可能 六月 七月 八月 九月
10 20 三十 40 50 60 70 80 100

这是我的代码：

months = ["January", "February", "March", "April", "May", "June", "July", "August", "September", "October", "November", "December"]

for month in months:
   if month < (F.month(F.current_date())):
      df = df.filter(F.col(month))
```
python

2 个回答

  1. Best Answer

    定义当前月份

    import datetime
index_of_current_month = datetime.datetime.now().month

    选择当前月份之前的月份（包括当前月份）

    df_selected = df.select(*months[:index_of_current_month])
    • 1

  2. 您可以尝试使用此代码片段

    # initialising spark 
spark = SparkSession.builder.appName("FilterPastMonths").getOrCreate()

months = ["January", "February", "March", "April", "May", "June", "July", "August", "September", "October", "November", "December"]

# Create a dict to map month-names to their corresponding month nos
month_mapping = {month: index + 1 for index, month in enumerate(months)}

current_month_row = spark.sql("SELECT month(current_date()) as current_month").collect()
current_month = current_month_row[0]["current_month"]

# filter the months to only include months up to the current month (e.g., September)
filtered_months = [
    month for month, month_num in month_mapping.items() if month_num <= current_month
]

# This should be your actual df 
data = [(10, 20, 30, 40, 50, 60, 70, 80, 100, 110, 120, 150)]
columns = months

df = spark.createDataFrame(data, columns)

# Select only the columns for the filtered months
df_filtered = df.select(filtered_months)

# Show the result
df_filtered.show()

    结果：测试的 Python 版本：3.8

    
+-------+--------+-----+-----+---+----+----+------+---------+                   
|January|February|March|April|May|June|July|August|September|
+-------+--------+-----+-----+---+----+----+------+---------+
|     10|      20|   30|   40| 50|  60|  70|    80|      100|
+-------+--------+-----+-----+---+----+----+------+---------+
    • 1

相关问题