主页 / coding / 问题 / 77937329
Accepted
oppressionslayer
Asked: 2024-02-05 03:38:00 +0800 CST

Pandas/Numpy 中 bitwise_xor 累加的 for 循环的主要加速问题

  • 772

好的，我正在使用如下所示的 for 循环，使用异或累加将此数据转换为下面的数据。对于我有（830401）行的条目，这非常非常慢。有没有什么方法可以加速 pandas 中的这种累积或使用 numpy 然后将其返回 numpy 数组本身



In [122]: acctable[0:20]
Out[122]: 
    what  dx1  dx2  dx3  dx4  dx5  dx6  dx7  dx8  dx9
0      4    2   10    8    0    5    7    1   13   11
1      4    0    0    0    0    0    0    0    0    0
2      6    0    0    0    0    0    0    0    0    0
3     14    0    0    0    0    0    0    0    0    0
4     12    0    0    0    0    0    0    0    8    0
5      4    0    0    0    0    0    0    0    0    0
6      1    0    0    0    0    0    0    0    0    0

...      ...  ...  ...  ...  ...  ...  ...  ...  ...  ...
830477    15    0    0    0    0    0    0    0    0    0
830478     3    0    0    0    0    0    0    0    0    0
830479    11    0    0    0    0    0    0    0    0    0
830480     9    0    0    0    0    0    0    0    0    0
830481    11    0    0    0    0    0    0    0    0    0

[830482 rows x 10 columns]

这是我尝试过的，它实际上可能需要一整分钟，而且我有更大的数据集可以使用，所以任何快捷方式或最佳方法都会很有帮助：

# Update: Instead of all 800k of 'what', i put the first 5 numbers in rstr so you can see how i'm xor accumulating. You should be able to copy/paste the first 6 elements of the data from with pd.read_clipboard() and assign to acctable. 

In [121]: rstr
Out[121]: array([ 4,  4, 12, 14,  6,  4], dtype=int8)
  
dt = np.int8
rstr = np.array(acctable.loc[:5, ('what')], dtype=dt)
for x in range(4): # # Prime Sequencing Functions
   wuttr = np.bitwise_xor.accumulate(np.r_[[rstr[-(x+1)]], acctable.loc[x, 'what':]], dtype=dt)
   acctable.loc[x+1, "what":] = wuttr[:end]

后：


In [122]: acctable[0:20]
Out[122]: 
    what  dx1  dx2  dx3  dx4  dx5  dx6  dx7  dx8  dx9
0      4    2   10    8    0    5    7    1   13   11
1      4    0    2    8    0    0    5    2    3   14
2      6    2    2    0    8    8    8   13   15   12
3     14    8   10    8    8    0    8    0   13    2
4     12    2   10    0    8    0    0    8    8    5
5      4    8   10    0    0    8    8    8    0    8
6      1    5   13    7    7    7   15    7   15   15
...      ...  ...  ...  ...  ...  ...  ...  ...  ...  ...
830477    15   15    7    0    0    5    9   14   10    3
830478     3   12    3    4    4    4    1    8    6   12
830479    11    8    4    7    3    7    3    2   10   12
830480     9    2   10   14    9   10   13   14   12    6
830481    11    2    0   10    4   13    7   10    4    8

[830482 rows x 10 columns]

这是一个简单的累加，但您需要前一行才能继续累加，而我能做的唯一方法是使用 for 循环。另外，“rstr”变量实际上是“什么”列。

谢谢！

我从人工智能收到了这个结果，但它只适用于第一行：

what_arr = acctable['what'].to_numpy().reshape(-1)  # Reshape to ensure 1D array

# Modified XOR accumulation:
all_what_arr = np.concatenate([[what_arr[0]], what_arr[1:]])
cumulative_xor = np.bitwise_xor.accumulate(all_what_arr)
shifted_xor = cumulative_xor[1:].reshape(-1, 1)
acctable.iloc[1:, 1:] = shifted_xor ^ acctable.iloc[1:, 1:]


In [171]: acctable
Out[171]: 
        what  dx1  dx2  dx3  dx4  dx5  dx6  dx7  dx8  dx9
0          4    2   10    8    0    5    7    1   13   11
1          6    0    2    8    0    0    5    2    3   14
2         14    4    6    6   12   14   12   11   11   10
3         12    2   10    0   10   10    8    8   15    8
4          4   12   14    4    4   14    4   12    4   11

以下是 timeit 值，您可以看到 Andrej 的修改和 njit 的使用是加速的一个重要因素！

In [262]:  import timeit
     ...: 
     ...:  setup = """
     ...:  import numpy as np
     ...:  import pandas as pd
     ...:  from numba import njit
     ...:  
     ...:  
     ...:  def do_work_no_njit(df):
     ...:      dt = np.int8
     ...:      end = -1
     ...:      rstr = np.array(df.loc[:, 0], dtype=dt)
     ...:      for x in range(len(df)):
     ...:          wuttr = np.bitwise_xor.accumulate(np.r_[[rstr[-(x+1)]], df.loc[x, 0:]], dtype=dt)
     ...:          df.loc[x+1, 0:] = wuttr[:end]
     ...:  
     ...:  @njit
     ...:  def do_work(vals):
     ...:      for row in range(vals.shape[0] - 1):
     ...:          for i in range(vals.shape[1] - 1):
     ...:              vals[row + 1, i + 1] = vals[row, i] ^ vals[row + 1, i]
     ...:  
     ...:  # Replace with your DataFrame creation code
     ...:  df = pd.DataFrame(np.random.randint(0, 15, size=(1000000, 10)), dtype=np.int8) # Example DataFrame, dtype=np.int8) # Example DataFrame
     ...:  """
     ...: 
     ...:  stmt = """
     ...:  do_work(df.values)
     ...:  """
     ...: 
     ...:  stmtnonjit = """
     ...:  do_work_no_njit(df.copy())
     ...:  """
     ...: 
     ...:  number = 1  # Adjust the number of repetitions as needed
     ...: 
     ...:  time = timeit.timeit(stmtnonjit, setup, number=number)
     ...:  print(f"Average time per execution no njit: {time / number:.4f} seconds")
     ...: 
     ...:  time = timeit.timeit(stmt, setup, number=number)
     ...:  print(f"Average time per execution with njit and optimized code by Andrej: {time / number:.4f} seconds")
     ...: 
Average time per execution no njit: 73.3801 seconds
Average time per execution with njit and optimized code by Andrej: 0.0442 seconds
python

1 个回答

  1. Best Answer

    您可以尝试使用来加快计算速度：

    from numba import njit


@njit
def do_work(vals):
    for row in range(vals.shape[0] - 1):
        for i in range(vals.shape[1] - 1):
            vals[row + 1, i + 1] = vals[row, i] ^ vals[row + 1, i]


do_work(df.values)
print(df)

    印刷：

       what  dx1  dx2  dx3  dx4  dx5  dx6  dx7  dx8  dx9
0     4    2   10    8    0    5    7    1   13   11
1     4    0    2    8    0    0    5    2    3   14
2     6    2    2    0    8    8    8   13   15   12
3    14    8   10    8    8    0    8    0   13    2
4    12    2   10    0    8    0    0    8    8    5

    最初的df：

       what  dx1  dx2  dx3  dx4  dx5  dx6  dx7  dx8  dx9
0     4    2   10    8    0    5    7    1   13   11
1     4    0    0    0    0    0    0    0    0    0
2     6    0    0    0    0    0    0    0    0    0
3    14    0    0    0    0    0    0    0    0    0
4    12    0    0    0    0    0    0    0    0    0
    • 1

相关问题