主页 / coding / 问题 / 76995118
Accepted
user3472
Asked: 2023-08-29 02:07:00 +0800 CST

pyspark 将可变长度数组类型的列拆分为两个较小的数组

  • 772

如果我有一个可变长度的数组类型的列，例如：

  [ [1,2,3,4,6] ] 
  [ [0,4,5,4,6,8,9,1] ]  
  [ [1,2,3,4,6,2,4,5,6] ]  
  ...

如何拆分它，使第一个索引与其余索引分开，例如：

  [ [1] ], [ [2,3,4,6] ] 
  [ [0] ], [ [4,5,4,6,8,9,1] ]  
  [ [1] ], [ [2,3,4,6,2,4,5,6] ]

在纯Python中我可能会这样做：

  new_list = list[0]
  second_list = list[1:]
pyspark

1 个回答

  1. Best Answer

    在 PySpark 中，您可以使用 expr() 来实现此转换

    代码：-

    from pyspark.sql import SparkSession
from pyspark.sql.functions import col, expr
spark = SparkSession.builder.appName("ArraySplitting").getOrCreate()
data = [
    [[1, 2, 3, 4, 6]],
    [[0, 4, 5, 4, 6, 8, 9, 1]],
    [[1, 2, 3, 4, 6, 2, 4, 5, 6]]
]
columns = ["ArrayColumn"]
df = spark.createDataFrame(data, columns)
df_split = df.withColumn("FirstIndex", expr("ArrayColumn[0]"))
df_split = df_split.withColumn("RestArray", expr("slice(ArrayColumn, 2, size(ArrayColumn))"))
df_split.show()

    输出：-

    +--------------------+----------+--------------------+
|         ArrayColumn|FirstIndex|           RestArray|
+--------------------+----------+--------------------+
|     [1, 2, 3, 4, 6]|         1|        [2, 3, 4, 6]|
|[0, 4, 5, 4, 6, 8...|         0|[4, 5, 4, 6, 8, 9...|
|[1, 2, 3, 4, 6, 2...|         1|[2, 3, 4, 6, 2, 4...|
+--------------------+----------+--------------------+
    • 0

相关问题