主页 / server / 问题

问题[io](server)

Martin Hope
eheu
Asked: 2022-02-03 01:41:32 +0800 CST
  • 0

我正在管理一个 Debian Linux x86_64 服务器，它有 125GB RAM、10GB 交换分区和 60 的交换值。

free -gw输出打印：

              total        used        free      shared     buffers       cache   available
Mem:            125          20           1          25           0         103          78
Swap:             9           1           7

每天晚上，都会运行一个 cron 作业，该作业对磁盘 R/W 非常密集，并且在执行期间将使用的内存增加 2GB，然后返回到 ~20GB。

在同一时间范围内，某些长期运行的容器化 Python 进程（尤其是 gunicorn）被分页。

随着时间的推移，这些进程将逐渐填满交换空间——几周后，交换空间将达到 99%。

swapoff我可以通过重新启动进程或使用and禁用和重新启用来清除交换swapon。但是 Python 进程的交换使用将再次缓慢上升。

我想部分解释可能是，由于每晚的 cron 作业，内核将文件系统缓存的优先级高于 Python 进程。但我也猜想这些 Python 进程的内存处理中的一些软件错误可能是罪魁祸首。

一位朋友建议我购买更多 RAM，以便为文件系统缓存提供更多空间。这对我来说似乎太过分了。我想更准确地诊断交换的原因并找到一些解决问题的软件。

所以现在我将问题传递给 Serverfault - 你如何看待这种现象，我应该从这里去哪里？

linux io memory python swap
Martin Hope
Marcell Bernát
Asked: 2021-05-12 03:15:04 +0800 CST
  • 2

我在 Windows Server 2016 主机（Dell PowerEdge R530，Intel Xeon E5-2620 @ 2.1GHz，96GB RAM）上有一个 Ubuntu 18.04 虚拟机，在 VirtualBox 6.1.10 中运行。主机系统通过 PERC H730 Mini 控制器在 RAID 1 中有 2x2TB 硬盘驱动器。VM 运行一个 Elasticsearch 节点和一些其他用于处理传入数据的软件。它有两个连接到它的虚拟磁盘（155GB、1.5TB），但它只有第二个正在运行。几个月前它都很好，但最近它开始看似随机地抛出 I/O 错误。

这是昨天的一些 dmesg 输出：

[Mon May 10 20:37:04 2021] ata4.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[Mon May 10 20:37:04 2021] ata4.00: irq_stat 0x40000000
[Mon May 10 20:37:04 2021] ata4.00: failed command: WRITE DMA EXT
[Mon May 10 20:37:04 2021] ata4.00: cmd 35/00:40:00:09:41/00:05:16:00:00/e0 tag 21 dma 688128 out
                                   res 41/10:40:00:09:41/00:05:16:00:00/e0 Emask 0x81 (invalid argument)
[Mon May 10 20:37:04 2021] ata4.00: status: { DRDY ERR }
[Mon May 10 20:37:04 2021] ata4.00: error: { IDNF }
[Mon May 10 20:37:04 2021] ata4.00: configured for UDMA/133
[Mon May 10 20:37:04 2021] ata4: EH complete
[Mon May 10 20:37:04 2021] ata4.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[Mon May 10 20:37:04 2021] ata4.00: irq_stat 0x40000000
[Mon May 10 20:37:04 2021] ata4.00: failed command: WRITE DMA EXT
[Mon May 10 20:37:04 2021] ata4.00: cmd 35/00:c0:40:0e:41/00:02:16:00:00/e0 tag 15 dma 360448 out
                                   res 41/10:c0:40:0e:41/00:02:16:00:00/e0 Emask 0x81 (invalid argument)
[Mon May 10 20:37:04 2021] ata4.00: status: { DRDY ERR }
[Mon May 10 20:37:04 2021] ata4.00: error: { IDNF }
[Mon May 10 20:37:04 2021] ata4.00: configured for UDMA/133
[Mon May 10 20:37:04 2021] ata4: EH complete
[Mon May 10 20:37:04 2021] ata4.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[Mon May 10 20:37:04 2021] ata4.00: irq_stat 0x40000000
[Mon May 10 20:37:04 2021] ata4.00: failed command: WRITE DMA EXT
[Mon May 10 20:37:04 2021] ata4.00: cmd 35/00:40:00:11:41/00:05:16:00:00/e0 tag 31 dma 688128 out
                                   res 41/10:40:00:11:41/00:05:16:00:00/e0 Emask 0x81 (invalid argument)
[Mon May 10 20:37:04 2021] ata4.00: status: { DRDY ERR }
[Mon May 10 20:37:04 2021] ata4.00: error: { IDNF }
[Mon May 10 20:37:04 2021] ata4.00: configured for UDMA/133
[Mon May 10 20:37:04 2021] ata4: EH complete
[Mon May 10 20:37:04 2021] ata4.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[Mon May 10 20:37:04 2021] ata4.00: irq_stat 0x40000000
[Mon May 10 20:37:04 2021] ata4.00: failed command: WRITE DMA EXT
[Mon May 10 20:37:04 2021] ata4.00: cmd 35/00:40:00:09:41/00:05:16:00:00/e0 tag 7 dma 688128 out
                                   res 41/10:40:00:09:41/00:05:16:00:00/e0 Emask 0x81 (invalid argument)
[Mon May 10 20:37:04 2021] ata4.00: status: { DRDY ERR }
[Mon May 10 20:37:04 2021] ata4.00: error: { IDNF }
[Mon May 10 20:37:04 2021] ata4.00: configured for UDMA/133
[Mon May 10 20:37:04 2021] ata4: EH complete
[Mon May 10 20:37:04 2021] ata4.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[Mon May 10 20:37:04 2021] ata4.00: irq_stat 0x40000000
[Mon May 10 20:37:04 2021] ata4.00: failed command: WRITE DMA EXT
[Mon May 10 20:37:04 2021] ata4.00: cmd 35/00:c0:40:0e:41/00:02:16:00:00/e0 tag 15 dma 360448 out
                                   res 41/10:c0:40:0e:41/00:02:16:00:00/e0 Emask 0x81 (invalid argument)
[Mon May 10 20:37:04 2021] ata4.00: status: { DRDY ERR }
[Mon May 10 20:37:04 2021] ata4.00: error: { IDNF }
[Mon May 10 20:37:04 2021] ata4.00: configured for UDMA/133
[Mon May 10 20:37:04 2021] ata4: EH complete
[Mon May 10 20:37:04 2021] ata4.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[Mon May 10 20:37:04 2021] ata4.00: irq_stat 0x40000000
[Mon May 10 20:37:04 2021] ata4.00: failed command: WRITE DMA EXT
[Mon May 10 20:37:04 2021] ata4.00: cmd 35/00:40:00:11:41/00:05:16:00:00/e0 tag 23 dma 688128 out
                                   res 41/10:40:00:11:41/00:05:16:00:00/e0 Emask 0x81 (invalid argument)
[Mon May 10 20:37:04 2021] ata4.00: status: { DRDY ERR }
[Mon May 10 20:37:04 2021] ata4.00: error: { IDNF }
[Mon May 10 20:37:04 2021] ata4.00: configured for UDMA/133
[Mon May 10 20:37:04 2021] ata4: EH complete
[Mon May 10 20:37:04 2021] ata4.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[Mon May 10 20:37:04 2021] ata4.00: irq_stat 0x40000000
[Mon May 10 20:37:04 2021] ata4.00: failed command: WRITE DMA EXT
[Mon May 10 20:37:04 2021] ata4.00: cmd 35/00:40:00:09:41/00:05:16:00:00/e0 tag 0 dma 688128 out
                                   res 41/10:40:00:09:41/00:05:16:00:00/e0 Emask 0x81 (invalid argument)
[Mon May 10 20:37:04 2021] ata4.00: status: { DRDY ERR }
[Mon May 10 20:37:04 2021] ata4.00: error: { IDNF }
[Mon May 10 20:37:04 2021] ata4.00: configured for UDMA/133
[Mon May 10 20:37:04 2021] ata4: EH complete
[Mon May 10 20:37:04 2021] ata4.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[Mon May 10 20:37:04 2021] ata4.00: irq_stat 0x40000000
[Mon May 10 20:37:04 2021] ata4.00: failed command: WRITE DMA EXT
[Mon May 10 20:37:04 2021] ata4.00: cmd 35/00:c0:40:0e:41/00:02:16:00:00/e0 tag 8 dma 360448 out
                                   res 41/10:c0:40:0e:41/00:02:16:00:00/e0 Emask 0x81 (invalid argument)
[Mon May 10 20:37:04 2021] ata4.00: status: { DRDY ERR }
[Mon May 10 20:37:04 2021] ata4.00: error: { IDNF }
[Mon May 10 20:37:04 2021] ata4.00: configured for UDMA/133
[Mon May 10 20:37:04 2021] ata4: EH complete
[Mon May 10 20:37:04 2021] ata4.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[Mon May 10 20:37:04 2021] ata4.00: irq_stat 0x40000000
[Mon May 10 20:37:04 2021] ata4.00: failed command: WRITE DMA EXT
[Mon May 10 20:37:04 2021] ata4.00: cmd 35/00:40:00:11:41/00:05:16:00:00/e0 tag 16 dma 688128 out
                                   res 41/10:40:00:11:41/00:05:16:00:00/e0 Emask 0x81 (invalid argument)
[Mon May 10 20:37:04 2021] ata4.00: status: { DRDY ERR }
[Mon May 10 20:37:04 2021] ata4.00: error: { IDNF }
[Mon May 10 20:37:04 2021] ata4.00: configured for UDMA/133
[Mon May 10 20:37:04 2021] ata4: EH complete
[Mon May 10 20:37:04 2021] ata4.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[Mon May 10 20:37:04 2021] ata4.00: irq_stat 0x40000000
[Mon May 10 20:37:04 2021] ata4.00: failed command: WRITE DMA EXT
[Mon May 10 20:37:04 2021] ata4.00: cmd 35/00:40:00:09:41/00:05:16:00:00/e0 tag 31 dma 688128 out
                                   res 41/10:40:00:09:41/00:05:16:00:00/e0 Emask 0x81 (invalid argument)
[Mon May 10 20:37:04 2021] ata4.00: status: { DRDY ERR }
[Mon May 10 20:37:04 2021] ata4.00: error: { IDNF }
[Mon May 10 20:37:04 2021] ata4.00: configured for UDMA/133
[Mon May 10 20:37:04 2021] ata4: EH complete
[Mon May 10 20:37:04 2021] ata4.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[Mon May 10 20:37:04 2021] ata4.00: irq_stat 0x40000000
[Mon May 10 20:37:04 2021] ata4.00: failed command: WRITE DMA EXT
[Mon May 10 20:37:04 2021] ata4.00: cmd 35/00:c0:40:0e:41/00:02:16:00:00/e0 tag 7 dma 360448 out
                                   res 41/10:c0:40:0e:41/00:02:16:00:00/e0 Emask 0x81 (invalid argument)
[Mon May 10 20:37:04 2021] ata4.00: status: { DRDY ERR }
[Mon May 10 20:37:04 2021] ata4.00: error: { IDNF }
[Mon May 10 20:37:04 2021] ata4.00: configured for UDMA/133
[Mon May 10 20:37:04 2021] ata4: EH complete
[Mon May 10 20:37:04 2021] ata4.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[Mon May 10 20:37:04 2021] ata4.00: irq_stat 0x40000000
[Mon May 10 20:37:04 2021] ata4.00: failed command: WRITE DMA EXT
[Mon May 10 20:37:04 2021] ata4.00: cmd 35/00:40:00:11:41/00:05:16:00:00/e0 tag 17 dma 688128 out
                                   res 41/10:40:00:11:41/00:05:16:00:00/e0 Emask 0x81 (invalid argument)
[Mon May 10 20:37:04 2021] ata4.00: status: { DRDY ERR }
[Mon May 10 20:37:04 2021] ata4.00: error: { IDNF }
[Mon May 10 20:37:04 2021] ata4.00: configured for UDMA/133
[Mon May 10 20:37:04 2021] ata4: EH complete
[Mon May 10 20:37:04 2021] ata4.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[Mon May 10 20:37:04 2021] ata4.00: irq_stat 0x40000000
[Mon May 10 20:37:04 2021] ata4.00: failed command: WRITE DMA EXT
[Mon May 10 20:37:04 2021] ata4.00: cmd 35/00:40:00:09:41/00:05:16:00:00/e0 tag 25 dma 688128 out
                                   res 41/10:40:00:09:41/00:05:16:00:00/e0 Emask 0x81 (invalid argument)
[Mon May 10 20:37:04 2021] ata4.00: status: { DRDY ERR }
[Mon May 10 20:37:04 2021] ata4.00: error: { IDNF }
[Mon May 10 20:37:04 2021] ata4.00: configured for UDMA/133
[Mon May 10 20:37:04 2021] ata4: EH complete
[Mon May 10 20:37:04 2021] ata4.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[Mon May 10 20:37:04 2021] ata4.00: irq_stat 0x40000000
[Mon May 10 20:37:04 2021] ata4.00: failed command: WRITE DMA EXT
[Mon May 10 20:37:04 2021] ata4.00: cmd 35/00:c0:40:0e:41/00:02:16:00:00/e0 tag 2 dma 360448 out
                                   res 41/10:c0:40:0e:41/00:02:16:00:00/e0 Emask 0x81 (invalid argument)
[Mon May 10 20:37:04 2021] ata4.00: status: { DRDY ERR }
[Mon May 10 20:37:04 2021] ata4.00: error: { IDNF }
[Mon May 10 20:37:04 2021] ata4.00: configured for UDMA/133
[Mon May 10 20:37:04 2021] ata4: EH complete
[Mon May 10 20:37:04 2021] ata4.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[Mon May 10 20:37:04 2021] ata4.00: irq_stat 0x40000000
[Mon May 10 20:37:04 2021] ata4.00: failed command: WRITE DMA EXT
[Mon May 10 20:37:04 2021] ata4.00: cmd 35/00:40:00:11:41/00:05:16:00:00/e0 tag 10 dma 688128 out
                                   res 41/10:40:00:11:41/00:05:16:00:00/e0 Emask 0x81 (invalid argument)
[Mon May 10 20:37:04 2021] ata4.00: status: { DRDY ERR }
[Mon May 10 20:37:04 2021] ata4.00: error: { IDNF }
[Mon May 10 20:37:04 2021] ata4.00: configured for UDMA/133
[Mon May 10 20:37:04 2021] ata4: EH complete
[Mon May 10 20:37:04 2021] ata4.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[Mon May 10 20:37:04 2021] ata4.00: irq_stat 0x40000000
[Mon May 10 20:37:04 2021] ata4.00: failed command: WRITE DMA EXT
[Mon May 10 20:37:04 2021] ata4.00: cmd 35/00:40:00:09:41/00:05:16:00:00/e0 tag 18 dma 688128 out
                                   res 41/10:40:00:09:41/00:05:16:00:00/e0 Emask 0x81 (invalid argument)
[Mon May 10 20:37:04 2021] ata4.00: status: { DRDY ERR }
[Mon May 10 20:37:04 2021] ata4.00: error: { IDNF }
[Mon May 10 20:37:04 2021] ata4.00: configured for UDMA/133
[Mon May 10 20:37:04 2021] sd 3:0:0:0: [sdb] tag#18 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[Mon May 10 20:37:04 2021] sd 3:0:0:0: [sdb] tag#18 Sense Key : Illegal Request [current]
[Mon May 10 20:37:04 2021] sd 3:0:0:0: [sdb] tag#18 Add. Sense: Logical block address out of range
[Mon May 10 20:37:04 2021] sd 3:0:0:0: [sdb] tag#18 CDB: Write(10) 2a 00 16 41 09 00 00 05 40 00
[Mon May 10 20:37:04 2021] blk_update_request: I/O error, dev sdb, sector 373360896 op 0x1:(WRITE) flags 0x4000 phys_seg 168 prio class 0
[Mon May 10 20:37:04 2021] ata4: EH complete
[Mon May 10 20:37:04 2021] ata4.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[Mon May 10 20:37:04 2021] ata4.00: irq_stat 0x40000000
[Mon May 10 20:37:04 2021] ata4.00: failed command: WRITE DMA EXT
[Mon May 10 20:37:04 2021] ata4.00: cmd 35/00:c0:40:0e:41/00:02:16:00:00/e0 tag 1 dma 360448 out
                                   res 41/10:c0:40:0e:41/00:02:16:00:00/e0 Emask 0x81 (invalid argument)
[Mon May 10 20:37:04 2021] ata4.00: status: { DRDY ERR }
[Mon May 10 20:37:04 2021] ata4.00: error: { IDNF }
[Mon May 10 20:37:04 2021] ata4.00: configured for UDMA/133
[Mon May 10 20:37:04 2021] sd 3:0:0:0: [sdb] tag#1 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[Mon May 10 20:37:04 2021] sd 3:0:0:0: [sdb] tag#1 Sense Key : Illegal Request [current]
[Mon May 10 20:37:04 2021] sd 3:0:0:0: [sdb] tag#1 Add. Sense: Logical block address out of range
[Mon May 10 20:37:04 2021] sd 3:0:0:0: [sdb] tag#1 CDB: Write(10) 2a 00 16 41 0e 40 00 02 c0 00
[Mon May 10 20:37:04 2021] blk_update_request: I/O error, dev sdb, sector 373362240 op 0x1:(WRITE) flags 0x0 phys_seg 88 prio class 0
[Mon May 10 20:37:04 2021] EXT4-fs warning (device sdb1): ext4_end_bio:315: I/O error 10 writing to inode 10095089 (offset 8388608 size 1871872 starting block 46670368)
[Mon May 10 20:37:04 2021] Buffer I/O error on device sdb1, logical block 46669856
[Mon May 10 20:37:04 2021] Buffer I/O error on device sdb1, logical block 46669857
[Mon May 10 20:37:04 2021] Buffer I/O error on device sdb1, logical block 46669858
[Mon May 10 20:37:04 2021] Buffer I/O error on device sdb1, logical block 46669859
[Mon May 10 20:37:04 2021] Buffer I/O error on device sdb1, logical block 46669860
[Mon May 10 20:37:04 2021] Buffer I/O error on device sdb1, logical block 46669861
[Mon May 10 20:37:04 2021] Buffer I/O error on device sdb1, logical block 46669862
[Mon May 10 20:37:04 2021] Buffer I/O error on device sdb1, logical block 46669863
[Mon May 10 20:37:04 2021] Buffer I/O error on device sdb1, logical block 46669864
[Mon May 10 20:37:04 2021] Buffer I/O error on device sdb1, logical block 46669865
[Mon May 10 20:37:04 2021] ata4: EH complete
[Mon May 10 20:37:04 2021] ata4.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[Mon May 10 20:37:04 2021] ata4.00: irq_stat 0x40000000
[Mon May 10 20:37:04 2021] ata4.00: failed command: WRITE DMA EXT
[Mon May 10 20:37:04 2021] ata4.00: cmd 35/00:40:00:11:41/00:05:16:00:00/e0 tag 7 dma 688128 out
                                   res 41/10:40:00:11:41/00:05:16:00:00/e0 Emask 0x81 (invalid argument)
[Mon May 10 20:37:04 2021] ata4.00: status: { DRDY ERR }
[Mon May 10 20:37:04 2021] ata4.00: error: { IDNF }
[Mon May 10 20:37:04 2021] ata4.00: configured for UDMA/133
[Mon May 10 20:37:04 2021] sd 3:0:0:0: [sdb] tag#7 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[Mon May 10 20:37:04 2021] sd 3:0:0:0: [sdb] tag#7 Sense Key : Illegal Request [current]
[Mon May 10 20:37:04 2021] sd 3:0:0:0: [sdb] tag#7 Add. Sense: Logical block address out of range
[Mon May 10 20:37:04 2021] sd 3:0:0:0: [sdb] tag#7 CDB: Write(10) 2a 00 16 41 11 00 00 05 40 00
[Mon May 10 20:37:04 2021] blk_update_request: I/O error, dev sdb, sector 373362944 op 0x1:(WRITE) flags 0x4000 phys_seg 168 prio class 0
[Mon May 10 20:37:04 2021] ata4: EH complete
[Mon May 10 20:37:04 2021] ata4.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[Mon May 10 20:37:04 2021] ata4.00: irq_stat 0x40000000
[Mon May 10 20:37:04 2021] ata4.00: failed command: WRITE DMA EXT
[Mon May 10 20:37:04 2021] ata4.00: cmd 35/00:08:40:16:41/00:01:16:00:00/e0 tag 13 dma 135168 out
                                   res 41/10:08:40:16:41/00:01:16:00:00/e0 Emask 0x81 (invalid argument)
[Mon May 10 20:37:04 2021] ata4.00: status: { DRDY ERR }
[Mon May 10 20:37:04 2021] ata4.00: error: { IDNF }
[Mon May 10 20:37:04 2021] ata4.00: configured for UDMA/133
[Mon May 10 20:37:04 2021] ata4: EH complete
[Mon May 10 20:37:05 2021] ata4.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[Mon May 10 20:37:05 2021] ata4.00: irq_stat 0x40000000
[Mon May 10 20:37:05 2021] ata4.00: failed command: WRITE DMA EXT
[Mon May 10 20:37:05 2021] ata4.00: cmd 35/00:08:40:16:41/00:01:16:00:00/e0 tag 30 dma 135168 out
                                   res 41/10:08:40:16:41/00:01:16:00:00/e0 Emask 0x81 (invalid argument)
[Mon May 10 20:37:05 2021] ata4.00: status: { DRDY ERR }
[Mon May 10 20:37:05 2021] ata4.00: error: { IDNF }
[Mon May 10 20:37:05 2021] ata4.00: configured for UDMA/133
[Mon May 10 20:37:05 2021] ata4: EH complete
[Mon May 10 20:37:05 2021] ata4.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[Mon May 10 20:37:05 2021] ata4.00: irq_stat 0x40000000
[Mon May 10 20:37:05 2021] ata4.00: failed command: WRITE DMA EXT
[Mon May 10 20:37:05 2021] ata4.00: cmd 35/00:08:40:16:41/00:01:16:00:00/e0 tag 5 dma 135168 out
                                   res 41/10:08:40:16:41/00:01:16:00:00/e0 Emask 0x81 (invalid argument)
[Mon May 10 20:37:05 2021] ata4.00: status: { DRDY ERR }
[Mon May 10 20:37:05 2021] ata4.00: error: { IDNF }
[Mon May 10 20:37:05 2021] ata4.00: configured for UDMA/133
[Mon May 10 20:37:05 2021] ata4: EH complete
[Mon May 10 20:37:05 2021] ata4.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[Mon May 10 20:37:05 2021] ata4.00: irq_stat 0x40000000
[Mon May 10 20:37:05 2021] ata4.00: failed command: WRITE DMA EXT
[Mon May 10 20:37:05 2021] ata4.00: cmd 35/00:08:40:16:41/00:01:16:00:00/e0 tag 28 dma 135168 out
                                   res 41/10:08:40:16:41/00:01:16:00:00/e0 Emask 0x81 (invalid argument)
[Mon May 10 20:37:05 2021] ata4.00: status: { DRDY ERR }
[Mon May 10 20:37:05 2021] ata4.00: error: { IDNF }
[Mon May 10 20:37:05 2021] ata4.00: configured for UDMA/133
[Mon May 10 20:37:05 2021] ata4: EH complete
[Mon May 10 20:37:05 2021] ata4.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[Mon May 10 20:37:05 2021] ata4.00: irq_stat 0x40000000
[Mon May 10 20:37:05 2021] ata4.00: failed command: WRITE DMA EXT
[Mon May 10 20:37:05 2021] ata4.00: cmd 35/00:08:40:16:41/00:01:16:00:00/e0 tag 19 dma 135168 out
                                   res 41/10:08:40:16:41/00:01:16:00:00/e0 Emask 0x81 (invalid argument)
[Mon May 10 20:37:05 2021] ata4.00: status: { DRDY ERR }
[Mon May 10 20:37:05 2021] ata4.00: error: { IDNF }
[Mon May 10 20:37:05 2021] ata4.00: configured for UDMA/133
[Mon May 10 20:37:05 2021] ata4: EH complete
[Mon May 10 20:37:05 2021] ata4.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[Mon May 10 20:37:05 2021] ata4.00: irq_stat 0x40000000
[Mon May 10 20:37:05 2021] ata4.00: failed command: WRITE DMA EXT
[Mon May 10 20:37:05 2021] ata4.00: cmd 35/00:08:40:16:41/00:01:16:00:00/e0 tag 11 dma 135168 out
                                   res 41/10:08:40:16:41/00:01:16:00:00/e0 Emask 0x81 (invalid argument)
[Mon May 10 20:37:05 2021] ata4.00: status: { DRDY ERR }
[Mon May 10 20:37:05 2021] ata4.00: error: { IDNF }
[Mon May 10 20:37:05 2021] ata4.00: configured for UDMA/133
[Mon May 10 20:37:05 2021] sd 3:0:0:0: [sdb] tag#11 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[Mon May 10 20:37:05 2021] sd 3:0:0:0: [sdb] tag#11 Sense Key : Illegal Request [current]
[Mon May 10 20:37:05 2021] sd 3:0:0:0: [sdb] tag#11 Add. Sense: Logical block address out of range
[Mon May 10 20:37:05 2021] sd 3:0:0:0: [sdb] tag#11 CDB: Write(10) 2a 00 16 41 16 40 00 01 08 00
[Mon May 10 20:37:05 2021] blk_update_request: I/O error, dev sdb, sector 373364288 op 0x1:(WRITE) flags 0x0 phys_seg 33 prio class 0
[Mon May 10 20:37:05 2021] EXT4-fs warning (device sdb1): ext4_end_bio:315: I/O error 10 writing to inode 10095089 (offset 8388608 size 1871872 starting block 46670569)
[Mon May 10 20:37:05 2021] ata4: EH complete
[Mon May 10 20:37:05 2021] JBD2: Detected IO errors while flushing file data on sdb1-8

HD Sentinel 报告说两个磁盘都很好，我在 SMART 数据中看不到任何问题。通过 iDRAC 进行的诊断没有发现任何东西，RAID 控制器及其电池都很好。

谷歌提出的所有问题都指向物理硬盘故障或 Hyper-V 问题，但这是一个虚拟磁盘，Hyper-V 已禁用。也许它与VirtualBox的设置有关？它在相同的设置下运行了几个月。毕竟，也许它是控制器？

一些关于如何继续追查这个问题的根源的指示将不胜感激。

io virtualbox raid windows-server-2016
Martin Hope
Maxim
Asked: 2021-05-09 14:37:50 +0800 CST
  • -1

有什么方法可以跟踪特定进程并获取磁盘 IO 的统计信息，例如队列深度、读/写线程总数、读/写百分比等？

主要目标是使用所有这些信息通过 fio 工具模拟 IO 活动。
或者可能有任何其他方式（工具）来估计哪些硬件可以更好地满足特定负载？

当然测试是最好的选择，但对我来说并不完全可用，我无法购买所有可能的硬件。
因此，我必须与购买前已经做出一些假设的产品进行比较。

performance hard-drive linux io fio
Martin Hope
func0der
Asked: 2021-01-25 06:16:41 +0800 CST
  • 0

我在 Debian 10（无 SELinux）上使用 Dovecot 2.3.4.1 和 EncFS（encfs 版本 1.9.5）加密邮箱

基本设置是：

EncFS 安装有

encfs --public /var/encrypted-mail /var/decrypted-mail -o nonempty --verbose

邮箱设置：

/var/decrypted-mail/domain.com/[email protected]/[cur,new...]

drwx------ 8 mail mail 4096 Dec 29 2019 /var/decrypted-mail（这些权限是递归的）

所有 dovecot 进程都以mail用户身份运行：

mail 5194 0.0 0.0 6308 4312 ? S 14:52 0:00 dovecot/imap

尽管一切似乎都配置正确，但我无法创建新文件夹或重命名现有文件夹。

我在以下内容中收到此错误/var/log/dovecot.log：

Jan 24 14:59:04 imap([email protected])<6010><Us21sxWS5OZaC6Ex>: Error: Couldn't create mailbox list lock /var/decrypted-mail/domain.com/[email protected]/mailboxes.lock: file_create_locked(/var/decrypted-mail/domain.com/[email protected]/mailboxes.lock) failed: link(/var/decrypted-mail/domain.com/[email protected]/mailboxes.lockba53af3795974a32, /var/decrypted-mail/domain.com/[email protected]/mailboxes.lock) failed: Operation not permitted

该错误消息的源代码位于此处：https ://github.com/dovecot/core/blob/4d09884528e592a0fa5d38bf479b3ddab1a4a0ef/src/lib/file-create-locked.c#L109-L123

如果我没看错，它就不能使用“常规锁定”机制并尝试将临时文件符号链接到mailboxes.lock. 这表明它可以创建临时文件mailboxes.lockba53af3795974a32并且无法对其进行符号链接。

当我在 shell 中运行这个命令时： sudo -u mail bash -c 'touch mailboxes.lockba53af3795974a32 && ln -s mailboxes.lockba53af3795974a32 mailboxes.lock'

它工作正常，我没有错误。

鸽舍的行为怎么可能与普通贝壳如此不同？

我唯一的解释是时间问题（之前访问过临时文件，但是为什么这没有出现在 bash 复制中？）。

我也尝试过chmod -R g+rwx /var/decrypted-mail，以确保所有这些 gid 都不是问题，但这只是寻找修复的绝望尝试。

io hardlink dovecot
Martin Hope
ogbofjnr
Asked: 2020-08-28 15:09:46 +0800 CST
  • 0

在磁盘基准测试工具 fio 中有 brandswitch 的统计信息

Run status group 0 (all jobs):
   READ: bw=20.9MiB/s (21.9MB/s), 10.4MiB/s-10.8MiB/s (10.9MB/s-11.3MB/s), io=64.0MiB (67.1MB), run=2973-3069msec

根据文档：

Values outside of brackets are power-of-2 format and those within are the equivalent value in a power-of-10 format.

据我所知 1 MB/s = 8MiB/s。但这里 20.9MiB/s 符合 21.9MB/s。

hard-drive io filesystems benchmark
Martin Hope
code_worker
Asked: 2020-06-21 02:21:36 +0800 CST
  • 4

我的服务器包含 128 个监视器，这导致提交了许多随机写入 IO。如果我使用 SSD，许多写入很快就会损坏。

我读了一些关于 F2FS 的文章，知道它在闪存盘和随机/小写上效果更好。

我的服务器中有很多随机写入 IO。它总是会降低我的服务器的性能。

如果我在硬盘上安装 F2FS？这比 ext4 好用吗？

====== 更新 ======

RAID0 只是为了性能基准，实际上我会使用 RAID5 或 RAID6。

在我搜索了很多之后，有两种方法可能有效：

  1. 缓存 IO
  2. 从随机 IO 到顺序 IO 的队列 IO <-- F2FS支持
linux io filesystems
Martin Hope
PraMiD
Asked: 2020-06-15 07:49:04 +0800 CST
  • 1

我目前正在将大型目录树从外部 HDD 复制到服务器的内部存储。复制操作的性能基本相当不错，即通过USB2连接外置硬盘时为35MB/s。

但是，我只获得了大约 30 秒的性能，然后复制过程没有得到任何 CPU 时间。这意味着，rsync（以及 cp）不会复制任何数据，不会更新其标准输出，或对 SIG-TERM 做出反应。大约 1 分钟后，它再次被调度并以 35MB/s 的速度再次复制 30 秒，然后再没有任何东西，依此类推。

有谁知道这里会发生什么或给我命名任何工具来调查正在发生的事情？我已经将 rsync 进程的 nice 值增加到 19，但这并没有改变任何东西。在rsync的空闲时间内，整个系统处于空闲状态（没有其他IO，没有其他占用CPU的进程。基本上整个系统都是空闲的）。在活动期间，rsync 进程具有三个线程，它们在“运行”、“不可中断睡眠”和“可中断睡眠”之间切换。在非活动期间，所有线程要么处于“不可中断睡眠”状态，要么处于“可中断睡眠”状态。正因为如此，我认为 rsync 必须等待一些持久的 IO 操作。但是我怎么能确定什么操作需要这么多时间呢？

该系统在 5.4.34-1-pve linux 内核上运行，内部存储是 LVM 上的 LUKS 加密 ZFS 文件系统。LVM 位于 mdadm RAID 5 之上。我用来启动 rsync 的命令是：rsync -ah --no-compress --progess

感谢您的任何建议！

io rsync copy scheduler
Martin Hope
Daniel Bandeira
Asked: 2020-04-25 05:54:26 +0800 CST
  • 0

通过 ctrl+alt+del 进行错误重置后，我的系统出现了一些问题。从那时起，报告了与 ACPI 和 ata6 相关的错误。为了解决这个问题（以一种野蛮的方式：我是 n00b），我开始测试目录 /sys/class/ata_port/ata6/ 中的竞争。我的意思是测试是寻找有用的信息或其他东西这可以激励我学习或在论坛中提出更清晰的问题。不管怎样，除了这个问题，还有一件奇怪的事情发生了：

当我将cat或文件与文件 /sys/class/ata_port/ata6/autosuspend_delay_ms 一起使用时，我得到"I/O erro"。但是...使用vi我读取了"1"文件的内容。

主要问题：为什么以及如何发生这种情况？（这真的把我吓坏了！vi更像是“用户空间”，而cat和文件更像是“内核空间”）

附带问题：如果有人知道如何解决这个系统问题，我会很高兴！现在，系统仅使用启动参数“acpi_enforce_responses=lax”启动（祝你好运！有时甚至这还不足以启动并且监视器失去通信（NULL 输出））”

细节：

系统： Linux debian 4.9.0-11-amd64 #1 SMP Debian 4.9.189-3+deb9u2 (2019-11-11) x86_64 GNU/Linux

错误系统日志：acpi PNP0A03:00：_OSC 失败（AE_NOT_FOUND）；禁用 ASPM

输出：dmesg|grep '失败\|错误'

[0.234571] acpi PNP0A03:00：_OSC 失败（AE_NOT_FOUND）；禁用 ASPM

[2.239147]ata6.00：识别失败（I/O错误，err_mask=0x100）

[7.971272]ata6.00：识别失败（I/O错误，err_mask=0x100）

[13.603109] ata6.00：识别失败（I/O 错误，err_mask=0x100）

[20.228845] EXT4-fs (sda1): 重新安装。选项：errors=remount-ro

[23.170590] r8169 0000:05:00.0：固件：加载 rtl_nic/rtl8168e-1.fw (-2) 失败

[23.174073] r8169 0000:05:00.0：rtl_nic/rtl8168e-1.fw 的直接固件加载失败，错误 -2

输出：dmesg|grep -i 'ata'

[0.000000] BIOS-e820: [mem 0x00000000cffb0000-0x00000000cffbdfff] ACPI 数据

[0.000000] NODE_DATA(0) 分配 [mem 0x22fff9000-0x22fffdfff]

[0.000000]内存：8089844K/8387896K可用（6288K内核代码，1162K rwdata，2880K rodata，1424K init，656K bss，298052K保留，0K cma-reserved）

[0.235157] pci 0000:00:11.0: 将 SATA 设置为 AHCI 模式

[1.692680]写保护内核只读数据：12288k

[1.744345] 已加载 libata 3.00 版。

[1.746053]scsi主机0：pata_atiixp

[1.747319]scsi host1:pata_atiixp

[1.747387] ata1: PATA max UDMA/100 cmd 0x1f0 ctl 0x3f6 bmdma 0xff00 irq 14

[1.747388] ata2: PATA max UDMA/100 cmd 0x170 ctl 0x376 bmdma 0xff08 irq 15

[1.748026] ahci 0000:00:11.0: AHCI 0001.0200 32 个插槽 4 个端口 6 Gbps 0xf impl SATA 模式

[1.749170] ata3: SATA max UDMA/133 abar m1024@0xfe7ffc00 端口 0xfe7ffd00 irq 25

[1.749172] ata4: SATA max UDMA/133 abar m1024@0xfe7ffc00 端口 0xfe7ffd80 irq 25

[1.749173] ata5: SATA max UDMA/133 abar m1024@0xfe7ffc00 端口 0xfe7ffe00 irq 25

[1.749175] ata6: SATA max UDMA/133 abar m1024@0xfe7ffc00 端口 0xfe7ffe80 irq 25

[1.928821] ata1.01：ATA-9：WDC WDS240G2G0A-00JH30，UF450000，最大 UDMA/133

[1.928823] ata1.01：468877312 个扇区，多 1：LBA48 NCQ（深度 0/32）

[1.929427]ata1.01：由于 40 线电缆，仅限于 UDMA/33

[1.952293]ata1.01：为 UDMA/33 配置

[1.952499] scsi 0:0:1:0：直接访问 ATA WDC WDS240G2G0A-0000 PQ：0 ANSI：5

[2.076991] ata5：SATA 链接断开（SStatus 0 SControl 300）

[2.077014] ata4：SATA 链接断开（SStatus 0 SControl 300）

[2.237428] ata6：SATA 链接速度为 1.5 Gbps（SStatus 113 SControl 300）

[2.237447] ata3：SATA 链接速度为 3.0 Gbps（SStatus 123 SControl 300）

[2.238304] ata3.00：ATA-7：ST3160815AS，3.CHH，最大 UDMA/100

[2.238306] ata3.00：312581808 个扇区，多 16：LBA48 NCQ（深度 31/32）

[2.239147]ata6.00：识别失败（I/O错误，err_mask=0x100）

[7.493471]ata3.00：为 UDMA/100 配置

[7.493630] scsi 2:0:0:0：直接访问 ATA ST3160815AS H PQ：0 ANSI：5

[7.969453] ata6：SATA 链接速度为 1.5 Gbps（SStatus 113 SControl 300）

[7.971272]ata6.00：识别失败（I/O错误，err_mask=0x100）

[7.971275]ata6：将 SATA 链接速度限制为 1.5 Gbps

[13.601480] ata6：SATA 链接速度为 1.5 Gbps（SStatus 113 SControl 310）

[13.603109] ata6.00：识别失败（I/O 错误，err_mask=0x100）

[19.233505] ata6：SATA 链接速度为 1.5 Gbps（SStatus 113 SControl 310）

[19.884310] EXT4-fs (sda1)：以有序数据模式挂载的文件系统。选项：（空）

[20.449088]acpi_cpufreq：覆盖 BIOS 提供的 _PSD 数据

debian io dmesg firmware bug
Martin Hope
Paul Smith
Asked: 2020-03-04 08:55:42 +0800 CST
  • 1

我有一个带有 6 个 SSD 的服务器和一个支持 RAID 10 和 RAID 50 的 RAID 控制器卡，计划将其用作我们的构建服务器。它将引入 NPM 包和许多小代码文件，编译和上传人工制品。

我们目前有一台服务器正在执行此操作，它正在针对 IO 瓶颈运行（它当前使用的是 RAID1 配置中的非 SSD 驱动器）。

RAID 50 和 RAID 10 中哪种 RAID 配置最适合性能？

从用例来看，IO 将主要是小文件的写入（随机写入）。磁盘空间和正常运行时间不是主要问题，因为我们有故障转移并且重建服务器很简单。所以我不关心在拆除阵列之前有多少驱动器故障等，唯一的考虑是性能。

RAID0 已被官僚机构排除在外。

我想真正的问题是奇偶校验的计算是否比总是写入同一个镜像磁盘需要更长的时间？

performance io raid raid10 write
Martin Hope
Kusal Arthanayake
Asked: 2020-02-12 09:17:27 +0800 CST
  • 0

在我的 Ubuntu 18.04 服务器上，我每小时都会在完全相同的时间挂断 IO。iotop 表明 jbd2 在这个时候飙升到 30%。挂断只持续了几秒钟。当 mysql_slow_query 日志在这个时候也被填满时，我开始注意到这一点。我没有运行 cron 作业。

Linode服务器中的Ubuntu 18.04 mysql 5.7 apache 2.4 php 7.2

iotop

iotop 2

是否有 Linux 工具可以检测导致此问题的程序？

performance mysql io ubuntu-18.04