AskOverflow.Dev

AskOverflow.Dev Logo AskOverflow.Dev Logo

AskOverflow.Dev Navigation

  • 主页
  • 系统&网络
  • Ubuntu
  • Unix
  • DBA
  • Computer
  • Coding
  • LangChain

Mobile menu

Close
  • 主页
  • 系统&网络
    • 最新
    • 热门
    • 标签
  • Ubuntu
    • 最新
    • 热门
    • 标签
  • Unix
    • 最新
    • 标签
  • DBA
    • 最新
    • 标签
  • Computer
    • 最新
    • 标签
  • Coding
    • 最新
    • 标签
主页 / server / 问题 / 1058056
Accepted
yankee
yankee
Asked: 2021-03-24 13:49:03 +0800 CST2021-03-24 13:49:03 +0800 CST 2021-03-24 13:49:03 +0800 CST

目录被 > 500 万个文件淹没后,目录列表挂起

  • 772

配置错误的 logrotate 在我的服务器上的一个目录中产生了很多文件。ls | wc -l显示 5,387,172 个文件,据此du -sh总计约 8 GB。dmesg显示了许多这样的错误:

[16718682.749947] EXT4-fs warning (device dm-5): ext4_dx_add_entry:2209: Directory (ino: 4194315) index full, reach max htree level :2
[16718682.750028] EXT4-fs warning (device dm-5): ext4_dx_add_entry:2213: Large directory feature is not enabled on this filesystem

配置现在已修复,我删除了所有不应该存在的文件。清理后,我得到了这个:

# ls -l roundcube/logs/
total 20
-rw-r--r-- 1 www-data www-data   0 Mär 21 06:25 errors.log
-rw-r--r-- 1 www-data www-data 315 Mär 21 06:25 errors.log.1
-rw-r--r-- 1 www-data www-data   0 Jan  7 06:36 errors.log.10.gz
-rw-r--r-- 1 www-data www-data   0 Dez 18 06:26 errors.log.11.gz
-rw-r--r-- 1 www-data www-data   0 Dez 10 06:25 errors.log.12.gz
-rw-r--r-- 1 www-data www-data 321 Mär 14 06:25 errors.log.2.gz
-rw-r--r-- 1 www-data www-data 272 Mär  7 06:25 errors.log.3.gz
-rw-r--r-- 1 www-data www-data 354 Feb 28 06:25 errors.log.4.gz
-rw-r--r-- 1 www-data www-data  20 Feb 18 16:07 errors.log.5.gz
-rw-r--r-- 1 www-data www-data   0 Feb 18 18:14 errors.log.6.gz
-rw-r--r-- 1 www-data www-data   0 Feb  5 13:36 errors.log.7.gz
-rw-r--r-- 1 www-data www-data   0 Jan 29 09:17 errors.log.8.gz
-rw-r--r-- 1 www-data www-data   0 Jan 14 06:49 errors.log.9.gz
# du -sh roundcube/logs/
632M    roundcube/logs/

然而,ls通话仍然需要很长时间(我没有看时钟,但我认为它超过了 10 分钟)。也可能 > 60 分钟。而且很奇怪,du仍然报告 632MB。

我启动了一个救援系统并运行 fsck:

root@rescue /dev/mapper # fsck.ext4 vg0-mail 
e2fsck 1.44.5 (15-Dec-2018)
vg0-mail: clean, 300078/13107200 files, 13579986/52428800 blocks
root@rescue /dev/mapper # fsck.ext4 -f vg0-mail 
e2fsck 1.44.5 (15-Dec-2018)
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
vg0-mail: 300078/13107200 files (0.5% non-contiguous), 13579986/52428800 blocks

所以看起来一切都很好。我重新启动回到正常系统,但ls仍然很慢。

只是为了确保没有进程干扰,我重新启动到救援系统,在那里安装卷并尝试列出目录内容。它也挂了。

strace ls显示很多行。最后几行是:

fstat(3, {st_mode=S_IFREG|0444, st_size=0, ...}) = 0
read(3, "nodev\tsysfs\nnodev\ttmpfs\nnodev\tbd"..., 1024) = 309
read(3, "", 1024)                       = 0
close(3)                                = 0
access("/etc/selinux/config", F_OK)     = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/usr/lib/locale/locale-archive", O_RDONLY|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=5547600, ...}) = 0
mmap(NULL, 5547600, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f0bd79c6000
close(3)                                = 0
ioctl(1, TCGETS, {B38400 opost isig icanon echo ...}) = 0
ioctl(1, TIOCGWINSZ, {ws_row=66, ws_col=271, ws_xpixel=0, ws_ypixel=0}) = 0
stat("roundcube/logs/", {st_mode=S_IFDIR|0755, st_size=662331392, ...}) = 0
openat(AT_FDCWD, "roundcube/logs/", O_RDONLY|O_NONBLOCK|O_CLOEXEC|O_DIRECTORY) = 3
fstat(3, {st_mode=S_IFDIR|0755, st_size=662331392, ...}) = 0
getdents64(3

所以它似乎挂了getdents64?我有点困惑,它以看似不完整的线条结尾。

我读到这可能是由硬件缺陷引起的,所以我尝试smartctl -t long在两个磁盘上运行分析(raid 1)。结果如下:

/dev/sda

smartctl 6.6 2017-11-05 r4594 [x86_64-linux-4.19.0-14-amd64] (local build)
Copyright (C) 2002-17, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     Seagate Constellation ES.3
Device Model:     ST2000NM0033-9ZM175
Serial Number:    Z1X0QWWG
LU WWN Device Id: 5 000c50 064a202c3
Firmware Version: SN07
User Capacity:    2.000.398.934.016 bytes [2,00 TB]
Sector Size:      512 bytes logical/physical
Rotation Rate:    7200 rpm
Form Factor:      3.5 inches
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ACS-2 (minor revision not indicated)
SATA Version is:  SATA 3.0, 6.0 Gb/s (current: 3.0 Gb/s)
Local Time is:    Tue Mar 23 21:24:30 2021 CET
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x82) Offline data collection activity
                    was completed without error.
                    Auto Offline Data Collection: Enabled.
Self-test execution status:      (   0) The previous self-test routine completed
                    without error or no self-test has ever 
                    been run.
Total time to complete Offline 
data collection:        (  592) seconds.
Offline data collection
capabilities:            (0x7b) SMART execute Offline immediate.
                    Auto Offline data collection on/off support.
                    Suspend Offline collection upon new
                    command.
                    Offline surface scan supported.
                    Self-test supported.
                    Conveyance Self-test supported.
                    Selective Self-test supported.
SMART capabilities:            (0x0003) Saves SMART data before entering
                    power-saving mode.
                    Supports SMART auto save timer.
Error logging capability:        (0x01) Error logging supported.
                    General Purpose Logging supported.
Short self-test routine 
recommended polling time:    (   1) minutes.
Extended self-test routine
recommended polling time:    ( 245) minutes.
Conveyance self-test routine
recommended polling time:    (   2) minutes.
SCT capabilities:          (0x50bd) SCT Status supported.
                    SCT Error Recovery Control supported.
                    SCT Feature Control supported.
                    SCT Data Table supported.

SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x000f   070   063   044    Pre-fail  Always       -       11974882
  3 Spin_Up_Time            0x0003   096   096   000    Pre-fail  Always       -       0
  4 Start_Stop_Count        0x0032   100   100   020    Old_age   Always       -       25
  5 Reallocated_Sector_Ct   0x0033   100   100   010    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x000f   088   060   030    Pre-fail  Always       -       706711710
  9 Power_On_Hours          0x0032   033   033   000    Old_age   Always       -       59193
 10 Spin_Retry_Count        0x0013   100   100   097    Pre-fail  Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   020    Old_age   Always       -       24
184 End-to-End_Error        0x0032   100   100   099    Old_age   Always       -       0
187 Reported_Uncorrect      0x0032   100   100   000    Old_age   Always       -       0
188 Command_Timeout         0x0032   100   100   000    Old_age   Always       -       0
189 High_Fly_Writes         0x003a   059   059   000    Old_age   Always       -       41
190 Airflow_Temperature_Cel 0x0022   060   050   045    Old_age   Always       -       40 (Min/Max 36/44)
191 G-Sense_Error_Rate      0x0032   100   100   000    Old_age   Always       -       0
192 Power-Off_Retract_Count 0x0032   100   100   000    Old_age   Always       -       18
193 Load_Cycle_Count        0x0032   099   099   000    Old_age   Always       -       2459
194 Temperature_Celsius     0x0022   040   050   000    Old_age   Always       -       40 (0 21 0 0 0)
195 Hardware_ECC_Recovered  0x001a   042   015   000    Old_age   Always       -       11974882
197 Current_Pending_Sector  0x0012   100   100   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0010   100   100   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x003e   200   200   000    Old_age   Always       -       0

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Extended offline    Completed without error       00%     59192         -
# 2  Short offline       Interrupted (host reset)      10%     59185         -
# 3  Extended offline    Completed without error       00%     53208         -
# 4  Extended offline    Completed without error       00%     53191         -
# 5  Extended offline    Completed without error       00%     53178         -
# 6  Short offline       Completed without error       00%     53170         -
# 7  Extended offline    Completed without error       00%     53056         -
# 8  Extended offline    Completed without error       00%     53040         -
# 9  Extended offline    Completed without error       00%     53025         -
#10  Short offline       Completed without error       00%     53017         -
#11  Extended offline    Completed without error       00%     51660         -
#12  Extended offline    Completed without error       00%     51643         -
#13  Extended offline    Completed without error       00%      2411         -
#14  Extended offline    Completed without error       00%      2383         -

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.




/dev/sdb

smartctl 6.6 2017-11-05 r4594 [x86_64-linux-4.19.0-14-amd64] (local build)
Copyright (C) 2002-17, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     Hitachi/HGST Ultrastar 7K4000
Device Model:     HGST HUS724020ALA640
Serial Number:    PN2138P2GNK30J
LU WWN Device Id: 5 000cca 24bc9579a
Firmware Version: MF6OAA70
User Capacity:    2.000.398.934.016 bytes [2,00 TB]
Sector Size:      512 bytes logical/physical
Rotation Rate:    7200 rpm
Form Factor:      3.5 inches
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ATA8-ACS T13/1699-D revision 4
SATA Version is:  SATA 3.0, 6.0 Gb/s (current: 3.0 Gb/s)
Local Time is:    Tue Mar 23 22:38:27 2021 CET
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x84) Offline data collection activity
                    was suspended by an interrupting command from host.
                    Auto Offline Data Collection: Enabled.
Self-test execution status:      (   0) The previous self-test routine completed
                    without error or no self-test has ever 
                    been run.
Total time to complete Offline 
data collection:        (   24) seconds.
Offline data collection
capabilities:            (0x5b) SMART execute Offline immediate.
                    Auto Offline data collection on/off support.
                    Suspend Offline collection upon new
                    command.
                    Offline surface scan supported.
                    Self-test supported.
                    No Conveyance Self-test supported.
                    Selective Self-test supported.
SMART capabilities:            (0x0003) Saves SMART data before entering
                    power-saving mode.
                    Supports SMART auto save timer.
Error logging capability:        (0x01) Error logging supported.
                    General Purpose Logging supported.
Short self-test routine 
recommended polling time:    (   1) minutes.
Extended self-test routine
recommended polling time:    ( 314) minutes.
SCT capabilities:          (0x003d) SCT Status supported.
                    SCT Error Recovery Control supported.
                    SCT Feature Control supported.
                    SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x000b   100   100   016    Pre-fail  Always       -       0
  2 Throughput_Performance  0x0005   136   136   054    Pre-fail  Offline      -       80
  3 Spin_Up_Time            0x0007   163   163   024    Pre-fail  Always       -       367 (Average 396)
  4 Start_Stop_Count        0x0012   100   100   000    Old_age   Always       -       28
  5 Reallocated_Sector_Ct   0x0033   100   100   005    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x000b   100   100   067    Pre-fail  Always       -       0
  8 Seek_Time_Performance   0x0005   145   145   020    Pre-fail  Offline      -       24
  9 Power_On_Hours          0x0012   095   095   000    Old_age   Always       -       37639
 10 Spin_Retry_Count        0x0013   100   100   060    Pre-fail  Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       27
192 Power-Off_Retract_Count 0x0032   099   099   000    Old_age   Always       -       1311
193 Load_Cycle_Count        0x0012   099   099   000    Old_age   Always       -       1311
194 Temperature_Celsius     0x0002   150   150   000    Old_age   Always       -       40 (Min/Max 22/60)
196 Reallocated_Event_Count 0x0032   100   100   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0022   100   100   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0008   100   100   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x000a   200   200   000    Old_age   Always       -       0

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Extended offline    Completed without error       00%     37638         -
# 2  Short offline       Interrupted (host reset)      90%     37630         -
# 3  Extended offline    Completed without error       00%     31654         -
# 4  Extended offline    Completed without error       00%     31637         -
# 5  Extended offline    Completed without error       00%     30951         -
# 6  Extended offline    Completed without error       00%     30932         -
# 7  Extended offline    Completed without error       00%     30927         -
# 8  Short offline       Completed without error       00%     30919         -
# 9  Extended offline    Completed without error       00%     30882         -
#10  Extended offline    Completed without error       00%     30866         -
#11  Extended offline    Completed without error       00%     19849         -
#12  Extended offline    Completed without error       00%     19832         -

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

我大部分都不懂,但对我来说看起来并不“坏”?!

最终我刚刚执行mv logs logs-old; mv logs-old/* logs; rm -r logs-old了这似乎确实“解决了”我现在的问题。但我想知道这是否真的是这样,或者是否可能还有更多。

这里可能发生了什么导致这种目录列表极慢的行为?现在问题可以解决了吗?我应该找到一些损坏吗?

hard-drive ext4
  • 1 1 个回答
  • 129 Views

1 个回答

  • Voted
  1. Best Answer
    shodanshok
    2021-06-18T12:28:46+08:002021-06-18T12:28:46+08:00

    这是一个已知问题:当有这么多文件时,目录条目(dentry)变得非常大,删除文件不会缩小它。

    更简单的解决方案是删除并重新创建目录本身(即:在您的情况下,为rm -rf roundcube/logs/ ; mkdir roundcube/logs/)

    请参阅此处了解更多信息

    • 2

相关问题

  • 总大小(磁盘)与总大小(媒体)

  • Linux:“发现重复的 PV XXXXYYYYY:用户 /dev/sdb1 而不是 /dev/sda1”

  • Windows C:驱动器大小

  • 了解磁盘队列长度

  • md5sum 重复为同一台机器上的同一文件提供不同的校验和

Sidebar

Stats

  • 问题 205573
  • 回答 270741
  • 最佳答案 135370
  • 用户 68524
  • 热门
  • 回答
  • Marko Smith

    新安装后 postgres 的默认超级用户用户名/密码是什么?

    • 5 个回答
  • Marko Smith

    SFTP 使用什么端口?

    • 6 个回答
  • Marko Smith

    命令行列出 Windows Active Directory 组中的用户?

    • 9 个回答
  • Marko Smith

    什么是 Pem 文件,它与其他 OpenSSL 生成的密钥文件格式有何不同?

    • 3 个回答
  • Marko Smith

    如何确定bash变量是否为空?

    • 15 个回答
  • Martin Hope
    Tom Feiner 如何按大小对 du -h 输出进行排序 2009-02-26 05:42:42 +0800 CST
  • Martin Hope
    Noah Goodrich 什么是 Pem 文件,它与其他 OpenSSL 生成的密钥文件格式有何不同? 2009-05-19 18:24:42 +0800 CST
  • Martin Hope
    Brent 如何确定bash变量是否为空? 2009-05-13 09:54:48 +0800 CST
  • Martin Hope
    cletus 您如何找到在 Windows 中打开文件的进程? 2009-05-01 16:47:16 +0800 CST

热门标签

linux nginx windows networking ubuntu domain-name-system amazon-web-services active-directory apache-2.4 ssh

Explore

  • 主页
  • 问题
    • 最新
    • 热门
  • 标签
  • 帮助

Footer

AskOverflow.Dev

关于我们

  • 关于我们
  • 联系我们

Legal Stuff

  • Privacy Policy

Language

  • Pt
  • Server
  • Unix

© 2023 AskOverflow.DEV All Rights Reserve