主页 / server / 问题

问题[hangs](server)

Martin Hope
brardong
Asked: 2020-09-26 01:14:35 +0800 CST
  • 1

我的服务器在 RAID1 中有两个 HDD，在 RAID0 中有两个 SSD 和一个 NVME。但有时硬盘会随机挂起几分钟。上次 HDD 20 分钟没有响应。我不知道什么可能是这个问题的原因。我检查了节点导出器的数据，没有什么奇怪的。这是晚上，所以服务器很平静。CPU 使用率、内存使用率或 IOPS 没有峰值。

发生此问题时，HDD 完全被阻止，但我不确定 SSD 和 NVME 肯定不受影响。智能数据看起来不错。我不确定这是硬件问题还是软件问题。

任何想法在哪里寻找更多信息？

谢谢你。

Debian Strech 4.19.0-0.bpo.1-amd64 来自日志的相关信息：

INFO: task md0_raid1:248 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
jbd2/dm-17-8    D    0  4937      2 0x80000000
Call Trace:
 ? __schedule+0x3f5/0x880
 schedule+0x32/0x80
 rwsem_down_read_failed+0x12e/0x190
 ? call_rwsem_down_read_failed+0x14/0x30
 call_rwsem_down_read_failed+0x14/0x30
 down_read+0x1c/0x30
 dm_thin_find_block+0x2e/0x70 [dm_thin_pool]
 thin_map+0x168/0x270 [dm_thin_pool]
 __map_bio+0x42/0x170 [dm_mod]
 __split_and_process_non_flush+0x12c/0x220 [dm_mod]
 ? __process_bio+0x170/0x170 [dm_mod]
 __split_and_process_bio+0xb2/0x1a0 [dm_mod]
 __dm_make_request.isra.31+0x3f/0xa0 [dm_mod]
 generic_make_request+0x1e7/0x410
 ? submit_bio+0x6c/0x140
 submit_bio+0x6c/0x140
 ? guard_bio_eod+0x36/0x100
 submit_bh_wbc+0x163/0x190
 ? jbd2_journal_begin_ordered_truncate+0xa0/0xa0 [jbd2]
 jbd2_journal_commit_transaction+0x5ec/0x18a0 [jbd2]
 ? __switch_to_asm+0x40/0x70
 ? __switch_to_asm+0x34/0x70
 ? __switch_to_asm+0x40/0x70
 ? kjournald2+0xc1/0x260 [jbd2]
 kjournald2+0xc1/0x260 [jbd2]
 ? remove_wait_queue+0x60/0x60
 kthread+0xf8/0x130
 ? commit_timeout+0x10/0x10 [jbd2]
 ? kthread_create_worker_on_cpu+0x70/0x70
 ret_from_fork+0x35/0x40
INFO: task rs:main Q:Reg:5075 blocked for more than 120 seconds.
      Not tainted 4.19.0-0.bpo.1-amd64 #1 Debian 4.19.12-1~bpo9+1
 ? alloc_set_pte+0x3f8/0x5b0
 __do_page_fault+0x255/0x4f0
RAX: 00007f6d6e592638 RBX: 00007fff92429f80 RCX: 00007f6d6e4e57c0

智能数据

=== START OF INFORMATION SECTION ===
Model Family:     Western Digital Re
Device Model:     WDC WD2004FBYZ-01YCBB1
[...]

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x00) Offline data collection activity
                    was never started.
                    Auto Offline Data Collection: Disabled.
Self-test execution status:      (   0) The previous self-test routine completed
                    without error or no self-test has ever 
                    been run.
Total time to complete Offline 
data collection:        (    0) seconds.
Offline data collection
capabilities:            (0x7b) SMART execute Offline immediate.
                    Auto Offline data collection on/off support.
                    Suspend Offline collection upon new
                    command.
                    Offline surface scan supported.
                    Self-test supported.
                    Conveyance Self-test supported.
                    Selective Self-test supported.
SMART capabilities:            (0x0003) Saves SMART data before entering
                    power-saving mode.
                    Supports SMART auto save timer.
Error logging capability:        (0x01) Error logging supported.
                    General Purpose Logging supported.
Short self-test routine 
recommended polling time:    (   2) minutes.
Extended self-test routine
recommended polling time:    ( 216) minutes.
Conveyance self-test routine
recommended polling time:    (   5) minutes.
SCT capabilities:          (0x003d) SCT Status supported.
                    SCT Error Recovery Control supported.
                    SCT Feature Control supported.
                    SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x002f   200   200   051    Pre-fail  Always       -       0
  3 Spin_Up_Time            0x0027   182   182   021    Pre-fail  Always       -       3866
  4 Start_Stop_Count        0x0032   100   100   000    Old_age   Always       -       62
  5 Reallocated_Sector_Ct   0x0033   200   200   140    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x002e   100   200   000    Old_age   Always       -       0
  9 Power_On_Hours          0x0032   058   058   000    Old_age   Always       -       31119
 10 Spin_Retry_Count        0x0032   100   100   000    Old_age   Always       -       0
 11 Calibration_Retry_Count 0x0032   100   100   000    Old_age   Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       62
 16 Unknown_Attribute       0x0022   007   193   000    Old_age   Always       -       390630975468
183 Runtime_Bad_Block       0x0032   100   100   000    Old_age   Always       -       0
192 Power-Off_Retract_Count 0x0032   200   200   000    Old_age   Always       -       42
193 Load_Cycle_Count        0x0032   200   200   000    Old_age   Always       -       86
194 Temperature_Celsius     0x0022   119   104   000    Old_age   Always       -       28
196 Reallocated_Event_Count 0x0032   200   200   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0032   200   200   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0030   100   253   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x0032   200   200   000    Old_age   Always       -       0
200 Multi_Zone_Error_Rate   0x0008   100   253   000    Old_age   Offline      -       0

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Extended offline    Completed without error       00%     29834         -

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

=== START OF INFORMATION SECTION ===
Model Family:     Western Digital Re
Device Model:     WDC WD2004FBYZ-01YCBB1
[...]

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x00) Offline data collection activity
                    was never started.
                    Auto Offline Data Collection: Disabled.
Self-test execution status:      (   0) The previous self-test routine completed
                    without error or no self-test has ever 
                    been run.
Total time to complete Offline 
data collection:        (    0) seconds.
Offline data collection
capabilities:            (0x7b) SMART execute Offline immediate.
                    Auto Offline data collection on/off support.
                    Suspend Offline collection upon new
                    command.
                    Offline surface scan supported.
                    Self-test supported.
                    Conveyance Self-test supported.
                    Selective Self-test supported.
SMART capabilities:            (0x0003) Saves SMART data before entering
                    power-saving mode.
                    Supports SMART auto save timer.
Error logging capability:        (0x01) Error logging supported.
                    General Purpose Logging supported.
Short self-test routine 
recommended polling time:    (   2) minutes.
Extended self-test routine
recommended polling time:    ( 216) minutes.
Conveyance self-test routine
recommended polling time:    (   5) minutes.
SCT capabilities:          (0x003d) SCT Status supported.
                    SCT Error Recovery Control supported.
                    SCT Feature Control supported.
                    SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x002f   200   200   051    Pre-fail  Always       -       0
  3 Spin_Up_Time            0x0027   180   180   021    Pre-fail  Always       -       3991
  4 Start_Stop_Count        0x0032   100   100   000    Old_age   Always       -       62
  5 Reallocated_Sector_Ct   0x0033   200   200   140    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x002e   200   200   000    Old_age   Always       -       0
  9 Power_On_Hours          0x0032   058   058   000    Old_age   Always       -       31118
 10 Spin_Retry_Count        0x0032   100   100   000    Old_age   Always       -       0
 11 Calibration_Retry_Count 0x0032   100   100   000    Old_age   Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       62
 16 Unknown_Attribute       0x0022   006   194   000    Old_age   Always       -       370886226476
183 Runtime_Bad_Block       0x0032   100   100   000    Old_age   Always       -       0
192 Power-Off_Retract_Count 0x0032   200   200   000    Old_age   Always       -       42
193 Load_Cycle_Count        0x0032   200   200   000    Old_age   Always       -       81
194 Temperature_Celsius     0x0022   116   106   000    Old_age   Always       -       31
196 Reallocated_Event_Count 0x0032   200   200   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0032   200   200   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0030   100   253   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x0032   200   200   000    Old_age   Always       -       0
200 Multi_Zone_Error_Rate   0x0008   100   253   000    Old_age   Offline      -       0

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Extended offline    Completed without error       00%     29833         -

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.



=== START OF INFORMATION SECTION ===
Device Model:     INTEL SSDSC2KG480G8
[...]

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x02) Offline data collection activity
                    was completed without error.
                    Auto Offline Data Collection: Disabled.
Self-test execution status:      (   0) The previous self-test routine completed
                    without error or no self-test has ever 
                    been run.
Total time to complete Offline 
data collection:        (   72) seconds.
Offline data collection
capabilities:            (0x79) SMART execute Offline immediate.
                    No Auto Offline data collection support.
                    Suspend Offline collection upon new
                    command.
                    Offline surface scan supported.
                    Self-test supported.
                    Conveyance Self-test supported.
                    Selective Self-test supported.
SMART capabilities:            (0x0003) Saves SMART data before entering
                    power-saving mode.
                    Supports SMART auto save timer.
Error logging capability:        (0x01) Error logging supported.
                    General Purpose Logging supported.
Short self-test routine 
recommended polling time:    (   1) minutes.
Extended self-test routine
recommended polling time:    (   2) minutes.
Conveyance self-test routine
recommended polling time:    (   2) minutes.
SCT capabilities:          (0x003d) SCT Status supported.
                    SCT Error Recovery Control supported.
                    SCT Feature Control supported.
                    SCT Data Table supported.

SMART Attributes Data Structure revision number: 1
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  5 Reallocated_Sector_Ct   0x0032   100   100   000    Old_age   Always       -       0
  9 Power_On_Hours          0x0032   100   100   000    Old_age   Always       -       9754
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       4
170 Unknown_Attribute       0x0033   100   100   010    Pre-fail  Always       -       0
171 Unknown_Attribute       0x0032   100   100   000    Old_age   Always       -       0
172 Unknown_Attribute       0x0032   100   100   000    Old_age   Always       -       0
174 Unknown_Attribute       0x0032   100   100   000    Old_age   Always       -       4
175 Program_Fail_Count_Chip 0x0033   100   100   010    Pre-fail  Always       -       21474773638
183 Runtime_Bad_Block       0x0032   100   100   000    Old_age   Always       -       0
184 End-to-End_Error        0x0033   100   100   090    Pre-fail  Always       -       0
187 Reported_Uncorrect      0x0032   100   100   000    Old_age   Always       -       0
190 Airflow_Temperature_Cel 0x0022   070   066   000    Old_age   Always       -       30 (Min/Max 17/36)
192 Power-Off_Retract_Count 0x0032   100   100   000    Old_age   Always       -       4
194 Temperature_Celsius     0x0022   100   100   000    Old_age   Always       -       30
197 Current_Pending_Sector  0x0012   100   100   000    Old_age   Always       -       0
199 UDMA_CRC_Error_Count    0x003e   100   100   000    Old_age   Always       -       0
225 Unknown_SSD_Attribute   0x0032   100   100   000    Old_age   Always       -       25630
226 Unknown_SSD_Attribute   0x0032   100   100   000    Old_age   Always       -       61
227 Unknown_SSD_Attribute   0x0032   100   100   000    Old_age   Always       -       69
228 Power-off_Retract_Count 0x0032   100   100   000    Old_age   Always       -       585288
232 Available_Reservd_Space 0x0033   100   100   010    Pre-fail  Always       -       0
233 Media_Wearout_Indicator 0x0032   100   100   000    Old_age   Always       -       0
234 Unknown_Attribute       0x0032   100   100   000    Old_age   Always       -       0
235 Unknown_Attribute       0x0033   100   100   010    Pre-fail  Always       -       21474773638
241 Total_LBAs_Written      0x0032   100   100   000    Old_age   Always       -       25630
242 Total_LBAs_Read         0x0032   100   100   000    Old_age   Always       -       58487
243 Unknown_Attribute       0x0032   100   100   000    Old_age   Always       -       132255

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Extended offline    Completed without error       00%      9516         -

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.



=== START OF INFORMATION SECTION ===
Device Model:     INTEL SSDSC2KG480G8
[...]

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x02) Offline data collection activity
                    was completed without error.
                    Auto Offline Data Collection: Disabled.
Self-test execution status:      (   0) The previous self-test routine completed
                    without error or no self-test has ever 
                    been run.
Total time to complete Offline 
data collection:        (   72) seconds.
Offline data collection
capabilities:            (0x79) SMART execute Offline immediate.
                    No Auto Offline data collection support.
                    Suspend Offline collection upon new
                    command.
                    Offline surface scan supported.
                    Self-test supported.
                    Conveyance Self-test supported.
                    Selective Self-test supported.
SMART capabilities:            (0x0003) Saves SMART data before entering
                    power-saving mode.
                    Supports SMART auto save timer.
Error logging capability:        (0x01) Error logging supported.
                    General Purpose Logging supported.
Short self-test routine 
recommended polling time:    (   1) minutes.
Extended self-test routine
recommended polling time:    (   2) minutes.
Conveyance self-test routine
recommended polling time:    (   2) minutes.
SCT capabilities:          (0x003d) SCT Status supported.
                    SCT Error Recovery Control supported.
                    SCT Feature Control supported.
                    SCT Data Table supported.

SMART Attributes Data Structure revision number: 1
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  5 Reallocated_Sector_Ct   0x0032   099   099   000    Old_age   Always       -       1
  9 Power_On_Hours          0x0032   100   100   000    Old_age   Always       -       9754
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       9
170 Unknown_Attribute       0x0033   099   099   010    Pre-fail  Always       -       0
171 Unknown_Attribute       0x0032   100   100   000    Old_age   Always       -       0
172 Unknown_Attribute       0x0032   100   100   000    Old_age   Always       -       0
174 Unknown_Attribute       0x0032   100   100   000    Old_age   Always       -       9
175 Program_Fail_Count_Chip 0x0033   100   100   010    Pre-fail  Always       -       42949609951
183 Runtime_Bad_Block       0x0032   100   100   000    Old_age   Always       -       0
184 End-to-End_Error        0x0033   100   100   090    Pre-fail  Always       -       0
187 Reported_Uncorrect      0x0032   100   100   000    Old_age   Always       -       0
190 Airflow_Temperature_Cel 0x0022   069   065   000    Old_age   Always       -       31 (Min/Max 17/36)
192 Power-Off_Retract_Count 0x0032   100   100   000    Old_age   Always       -       9
194 Temperature_Celsius     0x0022   100   100   000    Old_age   Always       -       31
197 Current_Pending_Sector  0x0012   100   100   000    Old_age   Always       -       0
199 UDMA_CRC_Error_Count    0x003e   100   100   000    Old_age   Always       -       0
225 Unknown_SSD_Attribute   0x0032   100   100   000    Old_age   Always       -       25308
226 Unknown_SSD_Attribute   0x0032   100   100   000    Old_age   Always       -       61
227 Unknown_SSD_Attribute   0x0032   100   100   000    Old_age   Always       -       69
228 Power-off_Retract_Count 0x0032   100   100   000    Old_age   Always       -       585269
232 Available_Reservd_Space 0x0033   099   099   010    Pre-fail  Always       -       0
233 Media_Wearout_Indicator 0x0032   100   100   000    Old_age   Always       -       0
234 Unknown_Attribute       0x0032   100   100   000    Old_age   Always       -       0
235 Unknown_Attribute       0x0033   100   100   010    Pre-fail  Always       -       42949609951
241 Total_LBAs_Written      0x0032   100   100   000    Old_age   Always       -       25308
242 Total_LBAs_Read         0x0032   100   100   000    Old_age   Always       -       58473
243 Unknown_Attribute       0x0032   100   100   000    Old_age   Always       -       133623

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Extended offline    Completed without error       00%      9516         -

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.
hard-drive linux hangs
Martin Hope
SledgehammerPL
Asked: 2020-02-11 01:41:14 +0800 CST
  • 0

远程 debian buster 服务器，所以不能去控制台。

昨天还正常，今天就没有反应了。我可以远程重启它，所以重启后我可以重新连接，但我只能运行像 top 这样的短命令并且我失去了连接。最重要的是，我发现 100% 是 clamd，但是当我尝试这样做时，/etc/init.d/clamd-daemon sto我在输入最后一个字母之前失去了连接。所以我下次复制它，但它没有帮助。我想知道我应该尝试获取哪些日志（通过 scp）以获得更好的诊断....

hangs fail2ban debian-buster
Martin Hope
Alex Flo
Asked: 2012-06-27 08:09:13 +0800 CST
  • 1

我有 2 个 Debian Linux 6.0.4 服务器有一个奇怪的行为：5-7-10 天后它们挂起。我的意思是服务器需要重新启动，在此之前 ping 不会回答。

几个月来我一直在努力解决这个问题，这里有一些想法/我尝试过但无法解决问题。

  • 我更改了服务器上的 RAM。作为两台不同的服务器，我怀疑它可能与硬件有关，因为第三台相同的服务器不会有这个问题。
  • 我记录了服务器负载，当它崩溃时负载很好（非常低）
  • 我在服务器日志中找不到任何内容，在服务器冻结之前日志都很好。
  • 不幸的是，我无权访问控制台。

虽然我有多年的管理经验，但我从未遇到过这样的问题，现在我不知道还能去哪里调查。

如果您知道我可以尝试什么来解决问题，请与我分享:-)

debian hangs
Martin Hope
Asked: 2010-01-20 06:23:37 +0800 CST
  • 0

我在一台服务器上使用 apache2+fastcgi+suexec+php5 托管多个（~30）个不同的站点。站点具有不同的脚本负载和不同的执行时间（其中一些处理请求 5-7 秒，一些 <1sek）。

有时，当单个站点接收到非常高的负载时（该站点的所有 php 实例都已创建和使用）-整个 apache 服务器挂起。Apache (worker mpm) 创建新进程达到上限。看起来它开始对每个站点的所有新请求进行排队，不仅是具有高负载并快速达到进程限制的站点......重新启动 apache 解决了问题......

配置： FastCgiConfig -singleThreshold 1 -multiThreshold 10 -listen-queue-depth 30 -maxProcesses 80 -maxClassProcesses 12 -idle-timeout 30 -pass-header HTTP_AUTHORIZATION -pass-header If-Modified-Since -pass-header If-None-Match

（早期有默认 -listen-queue-depth = 100，但它没有改变任何东西......）

有什么建议么？

另一个问题 - 这个监听队列是如何实现的？它是整个 apache 的一个队列，还是每个定义的 php 应用程序（suexec 站点）的唯一队列？

我想实现这样的目标：当一个站点接收到高负载并且它的队列已满时 - 服务器反弹下一个请求，但仅针对这个站点..其他站点应该正常工作......

apache-2.2 hangs fastcgi php5
Martin Hope
FerranB
Asked: 2009-07-09 13:36:50 +0800 CST
  • 1

我有一个工作站突然挂起在共享资源上工作的应用程序。但不适用于其他 TCP 连接。

例如，我打开 Windows 资源管理器并浏览 Z:\documents。我还打开了一个终端服务器连接。第一个挂起，第二个完美运行。

这仅发生在该工作站上，而不会发生在公司的任何其他工作站上。我放弃了防病毒软件，因为我已禁用它。

有什么建议吗？

UPDATE1： @Jay，它在浏览后或在使用其他使用共享文件夹的软件时挂起。

UPDATE2： @Russ，没有防火墙比 windows 防火墙更多，它与其他工作站完全相同。

file-sharing windows-xp hangs