主页 / coding / 问题 / 76977555
Accepted
Surender Raja
Asked: 2023-08-25 21:12:46 +0800 CST

为什么当我们进行转换时 Spark sql 会跳过毫秒

  • 772
 I dont know why this spark sql skips the last 3 000s after that decimal point

    df = spark.sql("""
      select cast('1900-01-01 00:00:00.000' as timestamp) as a  from dbo.products
      """)

  df.show(1,False)

上面的查询生成 1900-01-01 00:00:00 它以某种方式错过了小数点后的最后 3 个零

预计是

 1900-01-01 00:00:00.000

有人可以帮忙吗

pyspark

1 个回答

  1. Best Answer

    看来当毫秒是.000Spark 时默认会跳过该部分：

    >>> spark.sql("SELECT CAST('1900-01-01 00:00:00.000' AS TIMESTAMP)").show(truncate=False)
+------------------------------------------+
|CAST(1900-01-01 00:00:00.000 AS TIMESTAMP)|
+------------------------------------------+
|1900-01-01 00:00:00                       |
+------------------------------------------+

    例如，如果将毫秒设置为.999，则会显示：

    >>> spark.sql("SELECT CAST('1900-01-01 00:00:00.999' AS TIMESTAMP)").show(truncate=False)
+------------------------------------------+
|CAST(1900-01-01 00:00:00.999 AS TIMESTAMP)|
+------------------------------------------+
|1900-01-01 00:00:00.999                   |
+------------------------------------------+

    如果您确实需要显示该部分，请使用date_format：

    >>> df = spark.sql("SELECT CAST('1900-01-01 00:00:00.000' AS TIMESTAMP) datecol")
>>> df.select(F.date_format("datecol", "yyyy-MM-dd HH:mm:ss.SSS").alias("datecol")).show(truncate=False)
+-----------------------+
|datecol                |
+-----------------------+
|1900-01-01 00:00:00.000|
+-----------------------+
    • 1

相关问题