主页 / server / 问题

问题[server-crashes](server)

Martin Hope
Zak
Asked: 2021-12-11 10:58:38 +0800 CST
  • 0

我花了很长时间才弄清楚这一点。Apache，每隔几天就会挂起大约 20 分钟左右，然后“恢复活力”。这发生在中午，它发生在半夜。该服务器是一个强大的 Web 服务器，具有 4 个 CPU、8GB RAM 和另一个 12GB 交换空间。我在错误日志中看不到任何突出的内容。有人可以在这些日志中看到任何指示问题的内容吗？因为我看到的一切看起来都是问题或症状的结果！

系统日志

Dec 10 05:25:02 admin kernel: [397885.197196] [UFW BLOCK] IN=ens32 OUT= MAC=00:50:56:08:0d:03:00:50:56:08:09:19:08:00 SRC=10.2.6.60 DST=10.2.6.80 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=23622 DF PROTO=TCP SPT=59284 DPT=4949 WINDOW=29200 RES=0x00 SYN URGP=0
Dec 10 05:25:03 admin kernel: [397886.194931] [UFW BLOCK] IN=ens32 OUT= MAC=00:50:56:08:0d:03:00:50:56:08:09:19:08:00 SRC=10.2.6.60 DST=10.2.6.80 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=23623 DF PROTO=TCP SPT=59284 DPT=4949 WINDOW=29200 RES=0x00 SYN URGP=0
Dec 10 05:25:05 admin kernel: [397888.198941] [UFW BLOCK] IN=ens32 OUT= MAC=00:50:56:08:0d:03:00:50:56:08:09:19:08:00 SRC=10.2.6.60 DST=10.2.6.80 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=23624 DF PROTO=TCP SPT=59284 DPT=4949 WINDOW=29200 RES=0x00 SYN URGP=0
Dec 10 05:25:08 admin kernel: [397891.641472] [UFW BLOCK] IN=ens32 OUT= MAC=00:50:56:08:0d:03:00:a0:c9:27:01:01:08:00 SRC=20.115.4.12 DST=10.2.6.80 LEN=40 TOS=0x00 PREC=0x00 TTL=115 ID=967 DF PROTO=TCP SPT=52268 DPT=80 WINDOW=2045 RES=0x00 ACK FIN URGP=0
Dec 10 05:25:08 admin kernel: [397891.974137] [UFW BLOCK] IN=ens32 OUT= MAC=00:50:56:08:0d:03:00:a0:c9:27:01:01:08:00 SRC=18.206.39.189 DST=10.2.6.80 LEN=40 TOS=0x00 PREC=0x00 TTL=116 ID=28012 DF PROTO=TCP SPT=50916 DPT=443 WINDOW=0 RES=0x00 ACK RST URGP=0
Dec 10 05:25:08 admin kernel: [397891.974230] [UFW BLOCK] IN=ens32 OUT= MAC=00:50:56:08:0d:03:00:a0:c9:27:01:01:08:00 SRC=18.206.39.189 DST=10.2.6.80 LEN=40 TOS=0x00 PREC=0x00 TTL=116 ID=28013 DF PROTO=TCP SPT=50866 DPT=80 WINDOW=1021 RES=0x00 ACK FIN URGP=0
Dec 10 05:25:09 admin kernel: [397892.210857] [UFW BLOCK] IN=ens32 OUT= MAC=00:50:56:08:0d:03:00:50:56:08:09:19:08:00 SRC=10.2.6.60 DST=10.2.6.80 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=23625 DF PROTO=TCP SPT=59284 DPT=4949 WINDOW=29200 RES=0x00 SYN URGP=0
Dec 10 05:25:09 admin CRON[8325]: (root) CMD (sh /etc/apache2/websitesCron/apacheTest.sh)
Dec 10 05:25:09 admin CRON[8326]: (root) CMD (if [ -x /etc/munin/plugins/apt_all ]; then /etc/munin/plugins/apt_all update 7200 12 >/dev/null; elif [ -x /etc/munin/plugins/apt ]; then /etc/munin/plugins/apt update 7200 12 >/dev/null; fi)
Dec 10 05:25:10 admin sm-mta[1501]: rejecting connections on daemon MTA-v4: load average: 388
Dec 10 05:25:10 admin sm-mta[1501]: rejecting connections on daemon MSP-v4: load average: 388
Dec 10 05:25:17 admin kernel: [397900.226738] [UFW BLOCK] IN=ens32 OUT= MAC=00:50:56:08:0d:03:00:50:56:08:09:19:08:00 SRC=10.2.6.60 DST=10.2.6.80 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=23626 DF PROTO=TCP SPT=59284 DPT=4949 WINDOW=29200 RES=0x00 SYN URGP=0
Dec 10 05:25:25 admin sm-mta[1501]: rejecting connections on daemon MTA-v4: load average: 401
Dec 10 05:25:25 admin sm-mta[1501]: rejecting connections on daemon MSP-v4: load average: 401
Dec 10 05:25:27 admin kernel: [397910.285120] [UFW BLOCK] IN=ens32 OUT= MAC=00:50:56:08:0d:03:00:a0:c9:27:01:01:08:00 SRC=20.115.4.12 DST=10.2.6.80 LEN=40 TOS=0x00 PREC=0x00 TTL=115 ID=985 DF PROTO=TCP SPT=52268 DPT=80 WINDOW=0 RES=0x00 ACK RST URGP=0
Dec 10 05:25:33 admin kernel: [397916.242478] [UFW BLOCK] IN=ens32 OUT= MAC=00:50:56:08:0d:03:00:50:56:08:09:19:08:00 SRC=10.2.6.60 DST=10.2.6.80 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=23627 DF PROTO=TCP SPT=59284 DPT=4949 WINDOW=29200 RES=0x00 SYN URGP=0
Dec 10 05:25:34 admin sendmail[8390]: 1BABPXbK008390: from=root, size=425, class=0, nrcpts=1, msgid=<202112101125.1BABPXbK008390@admin.yourwebpro.com>, bodytype=8BITMIME, relay=root@localhost
Dec 10 05:25:34 admin sendmail[8390]: 1BABPXbK008390: to=root, delay=00:00:01, mailer=relay, pri=30425, stat=queued
Dec 10 05:25:40 admin sm-mta[1501]: rejecting connections on daemon MTA-v4: load average: 413
Dec 10 05:25:40 admin sm-mta[1501]: rejecting connections on daemon MSP-v4: load average: 413
Dec 10 05:25:55 admin sm-mta[1501]: rejecting connections on daemon MTA-v4: load average: 419
Dec 10 05:25:55 admin sm-mta[1501]: rejecting connections on daemon MSP-v4: load average: 419
Dec 10 05:26:04 admin CRON[8459]: (root) CMD (cp /var/spool/cron/crontabs/root /var/www/crontab/root && chmod 777 /var/www/crontab/root && chown zak:zak /var/www/crontab/root)
Dec 10 05:26:05 admin kernel: [397948.305893] [UFW BLOCK] IN=ens32 OUT= MAC=00:50:56:08:0d:03:00:50:56:08:09:19:08:00 SRC=10.2.6.60 DST=10.2.6.80 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=23628 DF PROTO=TCP SPT=59284 DPT=4949 WINDOW=29200 RES=0x00 SYN URGP=0
Dec 10 05:26:10 admin sm-mta[1501]: rejecting connections on daemon MTA-v4: load average: 423
Dec 10 05:26:10 admin sm-mta[1501]: rejecting connections on daemon MSP-v4: load average: 423
Dec 10 05:26:25 admin sm-mta[1501]: rejecting connections on daemon MTA-v4: load average: 428
Dec 10 05:26:25 admin sm-mta[1501]: rejecting connections on daemon MSP-v4: load average: 428
Dec 10 05:26:26 admin kernel: [397969.985342] [UFW BLOCK] IN=ens32 OUT= MAC=00:50:56:08:0d:03:00:a0:c9:27:01:01:08:00 SRC=20.115.4.12 DST=10.2.6.80 LEN=40 TOS=0x00 PREC=0x00 TTL=115 ID=990 DF PROTO=TCP SPT=62843 DPT=80 WINDOW=2045 RES=0x00 ACK FIN URGP=0
Dec 10 05:26:29 admin kernel: [397972.391234] [UFW BLOCK] IN=ens32 OUT= MAC=00:50:56:08:0d:03:00:a0:c9:27:01:01:08:00 SRC=20.115.4.12 DST=10.2.6.80 LEN=40 TOS=0x00 PREC=0x00 TTL=115 ID=995 DF PROTO=TCP SPT=62843 DPT=80 WINDOW=2045 RES=0x00 ACK FIN URGP=0
Dec 10 05:26:34 admin kernel: [397977.203821] [UFW BLOCK] IN=ens32 OUT= MAC=00:50:56:08:0d:03:00:a0:c9:27:01:01:08:00 SRC=20.115.4.12 DST=10.2.6.80 LEN=40 TOS=0x00 PREC=0x00 TTL=115 ID=999 DF PROTO=TCP SPT=62843 DPT=80 WINDOW=2045 RES=0x00 ACK FIN URGP=0
Dec 10 05:26:40 admin sm-mta[1501]: rejecting connections on daemon MTA-v4: load average: 436
Dec 10 05:26:40 admin kernel: [397983.346629] apache2 invoked oom-killer: gfp_mask=0x26000c0, order=2, oom_score_adj=0
Dec 10 05:26:40 admin kernel: [397983.346635] apache2 cpuset=/ mems_allowed=0
Dec 10 05:26:40 admin kernel: [397983.346644] CPU: 0 PID: 8270 Comm: apache2 Not tainted 4.4.0-119-generic #143-Ubuntu
Dec 10 05:26:40 admin kernel: [397983.346646] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 12/12/2018
Dec 10 05:26:40 admin kernel: [397983.346649]  0000000000000286 553e9d17649eeffd ffff8800138a7b10 ffffffff81400443
Dec 10 05:26:40 admin kernel: [397983.346652]  ffff8800138a7cc8 ffff8800b8e5aa00 ffff8800138a7b80 ffffffff8121086e
Dec 10 05:26:40 admin kernel: [397983.346655]  ffff88023fc1ad70 ffff88023fc1ad60 ffffea0006a80b00 0000000100000001
Dec 10 05:26:40 admin kernel: [397983.346657] Call Trace:
Dec 10 05:26:40 admin kernel: [397983.346670]  [<ffffffff81400443>] dump_stack+0x63/0x90
Dec 10 05:26:40 admin kernel: [397983.346677]  [<ffffffff8121086e>] dump_header+0x5a/0x1c5
Dec 10 05:26:40 admin kernel: [397983.346682]  [<ffffffff81196f32>] oom_kill_process+0x202/0x3c0
Dec 10 05:26:40 admin kernel: [397983.346685]  [<ffffffff81197359>] out_of_memory+0x219/0x460
Dec 10 05:26:40 admin kernel: [397983.346689]  [<ffffffff8119d3a5>] __alloc_pages_slowpath.constprop.88+0x965/0xb00
Dec 10 05:26:40 admin kernel: [397983.346692]  [<ffffffff8119d7c8>] __alloc_pages_nodemask+0x288/0x2a0
Dec 10 05:26:40 admin kernel: [397983.346694]  [<ffffffff8119d87b>] alloc_kmem_pages_node+0x4b/0xc0
Dec 10 05:26:40 admin kernel: [397983.346699]  [<ffffffff8108077e>] copy_process+0x1be/0x1bb0
Dec 10 05:26:40 admin kernel: [397983.346703]  [<ffffffff811a44f7>] ? lru_cache_add_active_or_unevictable+0x27/0xa0
Dec 10 05:26:40 admin kernel: [397983.346707]  [<ffffffff811c6178>] ? handle_mm_fault+0xcc8/0x1820
Dec 10 05:26:40 admin kernel: [397983.346709]  [<ffffffff81082300>] _do_fork+0x80/0x360
Dec 10 05:26:40 admin kernel: [397983.346712]  [<ffffffff81082689>] SyS_clone+0x19/0x20
Dec 10 05:26:40 admin kernel: [397983.346717]  [<ffffffff8184f708>] entry_SYSCALL_64_fastpath+0x1c/0xbb
Dec 10 05:26:40 admin kernel: [397983.346718] Mem-Info:
Dec 10 05:26:40 admin kernel: [397983.346723] active_anon:1463690 inactive_anon:290454 isolated_anon:132
Dec 10 05:26:40 admin kernel: [397983.346723]  active_file:17208 inactive_file:13369 isolated_file:0
Dec 10 05:26:40 admin kernel: [397983.346723]  unevictable:913 dirty:0 writeback:987 unstable:0
Dec 10 05:26:40 admin kernel: [397983.346723]  slab_reclaimable:19387 slab_unreclaimable:41949
Dec 10 05:26:40 admin kernel: [397983.346723]  mapped:25866 shmem:18736 pagetables:141112 bounce:0
Dec 10 05:26:40 admin kernel: [397983.346723]  free:26269 free_pcp:0 free_cma:0
Dec 10 05:26:40 admin kernel: [397983.346728] Node 0 DMA free:15864kB min:132kB low:164kB high:196kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15904kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:8kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
Dec 10 05:26:40 admin kernel: [397983.346734] lowmem_reserve[]: 0 2937 7928 7928 7928
Dec 10 05:26:40 admin kernel: [397983.346737] Node 0 DMA32 free:44712kB min:24988kB low:31232kB high:37480kB active_anon:2037256kB inactive_anon:515680kB active_file:24624kB inactive_file:19936kB unevictable:1228kB isolated(anon):348kB isolated(file):0kB present:3129216kB managed:3048436kB mlocked:1228kB dirty:0kB writeback:2620kB mapped:38304kB shmem:28716kB slab_reclaimable:25488kB slab_unreclaimable:70356kB kernel_stack:15136kB pagetables:261600kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Dec 10 05:26:40 admin kernel: [397983.346743] lowmem_reserve[]: 0 0 4990 4990 4990
Dec 10 05:26:40 admin kernel: [397983.346745] Node 0 Normal free:44500kB min:42460kB low:53072kB high:63688kB active_anon:3817504kB inactive_anon:646136kB active_file:44208kB inactive_file:33540kB unevictable:2424kB isolated(anon):180kB isolated(file):0kB present:5242880kB managed:5110492kB mlocked:2424kB dirty:0kB writeback:1328kB mapped:65160kB shmem:46228kB slab_reclaimable:52060kB slab_unreclaimable:97432kB kernel_stack:13728kB pagetables:302848kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Dec 10 05:26:40 admin kernel: [397983.346750] lowmem_reserve[]: 0 0 0 0 0
Dec 10 05:26:40 admin kernel: [397983.346752] Node 0 DMA: 0*4kB 1*8kB (U) 1*16kB (U) 1*32kB (U) 1*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15864kB
Dec 10 05:26:40 admin kernel: [397983.346762] Node 0 DMA32: 6731*4kB (UM) 2257*8kB (UM) 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 44980kB
Dec 10 05:26:40 admin kernel: [397983.346768] Node 0 Normal: 10918*4kB (UMEH) 49*8kB (UMH) 0*16kB 1*32kB (H) 1*64kB (H) 3*128kB (H) 0*256kB 1*512kB (H) 0*1024kB 0*2048kB 0*4096kB = 45056kB
Dec 10 05:26:40 admin kernel: [397983.346777] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Dec 10 05:26:40 admin kernel: [397983.346779] 68538 total pagecache pages
Dec 10 05:26:40 admin kernel: [397983.346781] 18641 pages in swap cache
Dec 10 05:26:40 admin kernel: [397983.346783] Swap cache stats: add 383468753, delete 383450112, find 229519152/378197444
Dec 10 05:26:40 admin kernel: [397983.346784] Free swap  = 1252280kB
Dec 10 05:26:42 admin kernel: [397983.346785] Total swap = 11717628kB
Dec 10 05:26:42 admin kernel: [397983.346787] 2097022 pages RAM
Dec 10 05:26:42 admin kernel: [397983.346788] 0 pages HighMem/MovableOnly
Dec 10 05:26:42 admin kernel: [397983.346789] 53314 pages reserved
Dec 10 05:26:42 admin kernel: [397983.346790] 0 pages cma reserved
Dec 10 05:26:42 admin kernel: [397983.346791] 0 pages hwpoisoned
Dec 10 05:26:42 admin kernel: [397983.346793] [ pid ]   uid  tgid total_vm      rss nr_ptes nr_pmds swapents oom_score_adj name
Dec 10 05:26:42 admin kernel: [397983.346800] [  409]     0   409    10976      822      24       3      602             0 systemd-journal
Dec 10 05:26:42 admin kernel: [397983.346803] [  451]     0   451    23693       83      17       3       50             0 lvmetad
Dec 10 05:26:42 admin kernel: [397983.346805] [  470]     0   470    11415      242      22       3      485         -1000 systemd-udevd
Dec 10 05:26:42 admin kernel: [397983.346808] [  784]     0   784    47527      409      52       3      223             0 vmtoolsd
Dec 10 05:26:42 admin kernel: [397983.346811] [  838]     0   838     5884        0      16       3       51             0 rpc.idmapd
Dec 10 05:26:42 admin kernel: [397983.346813] [  842]   100   842    25081      132      20       3       52             0 systemd-timesyn
Dec 10 05:26:42 admin kernel: [397983.346816] [  934]     0   934    11906      140      27       3      109             0 rpcbind
Dec 10 05:26:42 admin kernel: [397983.346818] [  948]     0   948   151028      166      28       4      213             0 lxcfs
Dec 10 05:26:42 admin kernel: [397983.346821] [  953]     0   953     7252      456      19       3       48             0 cron
Dec 10 05:26:42 admin kernel: [397983.346823] [  959]     0   959     1099      335       8       3       40             0 acpid
Dec 10 05:26:42 admin kernel: [397983.346826] [  961]     0   961     7165      239      19       3       57             0 systemd-logind
Dec 10 05:26:42 admin kernel: [397983.346828] [  967]   108   967    10725      566      25       3       76          -900 dbus-daemon
Dec 10 05:26:42 admin kernel: [397983.346831] [ 1047]     0  1047    21359      342      32       3      347             0 VGAuthService
Dec 10 05:26:42 admin kernel: [397983.346833] [ 1049]     0  1049    68974      456      37       3      224             0 accounts-daemon
Dec 10 05:26:42 admin kernel: [397983.346835] [ 1051]     0  1051     6511      391      18       3       47             0 atd
Dec 10 05:26:42 admin kernel: [397983.346838] [ 1056]   104  1056    64099      457      29       3      367             0 rsyslogd
Dec 10 05:26:42 admin kernel: [397983.346841] [ 1148]     0  1148     3343       51      11       3       23             0 mdadm
Dec 10 05:26:42 admin kernel: [397983.346843] [ 1179]     0  1179     6011      251      16       3       89             0 vsftpd
Dec 10 05:26:42 admin kernel: [397983.346845] [ 1189]     0  1189    69278      506      40       4      122             0 polkitd
Dec 10 05:26:42 admin kernel: [397983.346848] [ 1194]     0  1194     1305      358       9       3       61             0 iscsid
Dec 10 05:26:42 admin kernel: [397983.346850] [ 1195]     0  1195     1430      879      10       3        0           -17 iscsid
Dec 10 05:26:42 admin kernel: [397983.346853] [ 1207]     0  1207     9494        0      22       3      190             0 rpc.mountd
Dec 10 05:26:42 admin kernel: [397983.346856] [ 1221]     0  1221    16378      339      36       3      196         -1000 sshd
Dec 10 05:26:42 admin kernel: [397983.346859] [ 1296]     0  1296    16458      453      37       4      128             0 login
Dec 10 05:26:42 admin kernel: [397983.346861] [ 1326]     0  1326     4905      284      14       3       39             0 irqbalance
Dec 10 05:26:42 admin kernel: [397983.346864] [ 1420]     0  1420    13514      229      30       3     2438             0 munin-node
Dec 10 05:26:42 admin kernel: [397983.346866] [ 1501]     0  1501    26199      440      51       4      470             0 sendmail-mta
Dec 10 05:26:42 admin kernel: [397983.346869] [ 1681]     0  1681   109282      257     171       3     1936             0 php-fpm7.0
Dec 10 05:26:42 admin kernel: [397983.346871] [ 1687]    33  1687   109282      184     152       3     1961             0 php-fpm7.0
Dec 10 05:26:42 admin kernel: [397983.346873] [ 1688]    33  1688   109282      184     152       3     1961             0 php-fpm7.0
Dec 10 05:26:42 admin kernel: [397983.346876] [16727]  1000 16727    11330      422      25       3      217             0 systemd
Dec 10 05:26:42 admin kernel: [397983.346878] [16731]  1000 16731    52186        0      37       3      501             0 (sd-pam)
Dec 10 05:26:42 admin kernel: [397983.346881] [16734]  1000 16734     5900      357      18       4      725             0 bash
Dec 10 05:26:42 admin kernel: [397983.346884] [11281]   119 11281    69349      594      88       3     1257             0 freshclam
Dec 10 05:26:42 admin kernel: [397983.346886] [32663]     0 32663    23229      459      48       3      258             0 sshd
Dec 10 05:26:42 admin kernel: [397983.346889] [32740]  1000 32740    23668      373      48       3      743             0 sshd
Dec 10 05:26:42 admin kernel: [397983.346892] [22943]     0 22943    23229      330      51       3      272             0 sshd
Dec 10 05:26:42 admin kernel: [397983.346894] [23020]  1000 23020    23229      332      49       3      247             0 sshd
Dec 10 05:26:42 admin kernel: [397983.346897] [23023]  1000 23023     3220      393      12       3       51             0 sftp-server
Dec 10 05:26:42 admin kernel: [397983.346899] [24027]  1000 24027     3220      381      12       3       67             0 sftp-server
Dec 10 05:26:42 admin kernel: [397983.346902] [27215]  1000 27215     3220      378      12       3       95             0 sftp-server
Dec 10 05:26:42 admin kernel: [397983.346904] [13006]     0 13006   168834     2419     280       3    29858             0 apache2
Dec 10 05:26:42 admin kernel: [397983.346907] [15772]    33 15772   385874    34986     583       4    67131             0 apache2
Dec 10 05:26:42 admin kernel: [397983.346909] [15773]    33 15773   381008    40856     579       4    53799             0 apache2
Dec 10 05:26:42 admin kernel: [397983.346912] [15776]    33 15776   297940    35288     395       4    53305             0 apache2
Dec 10 05:26:42 admin kernel: [397983.346915] [15777]    33 15777   303389    37051     415       4    49856             0 apache2
Dec 10 05:26:42 admin kernel: [397983.346919] [15779]    33 15779   303184    39806     403       4    46606             0 apache2
Dec 10 05:26:42 admin kernel: [397983.346922] [15782]    33 15782   304265    46518     404       4    43859             0 apache2
Dec 10 05:26:42 admin kernel: [397983.346925] [15786]    33 15786   371258    41084     557       4    49806             0 apache2
Dec 10 05:26:42 admin kernel: [397983.346929] [15787]    33 15787   302813    34962     403       4    53776             0 apache2
Dec 10 05:26:42 admin kernel: [397983.346931] [15789]    33 15789   380748    35019     567       4    52932             0 apache2
Dec 10 05:26:42 admin kernel: [397983.346934] [15799]    33 15799   384260    39566     561       4    55415             0 apache2
Dec 10 05:26:42 admin kernel: [397983.346936] [15800]    33 15800   374314    29895     544       4    52106             0 apache2
Dec 10 05:26:42 admin kernel: [397983.346939] [15801]    33 15801   380377    34701     565       4    53467             0 apache2
Dec 10 05:26:42 admin kernel: [397983.346941] [15802]    33 15802   306345    35364     424       4    57003             0 apache2
........
Dec 10 05:33:29 admin sm-mta[1501]: rejecting connections on daemon MTA-v4: load average: 570
Dec 10 05:33:29 admin sm-mta[1501]: rejecting connections on daemon MSP-v4: load average: 570
Dec 10 05:33:29 admin sm-mta[1501]: rejecting connections on daemon MTA-v4: load average: 556
Dec 10 05:33:29 admin sm-mta[1501]: rejecting connections on daemon MSP-v4: load average: 556
Dec 10 05:33:29 admin sm-mta[1501]: rejecting connections on daemon MTA-v4: load average: 539
Dec 10 05:33:29 admin sm-mta[1501]: rejecting connections on daemon MSP-v4: load average: 539
Dec 10 05:33:29 admin sm-mta[1501]: rejecting connections on daemon MTA-v4: load average: 525
Dec 10 05:33:29 admin sm-mta[1501]: rejecting connections on daemon MSP-v4: load average: 525
......
Dec 10 06:27:15 admin kernel: [401618.309140] Out of memory: Kill process 15801 (apache2) score 14 or sacrifice child
Dec 10 06:27:15 admin kernel: [401618.309960] Killed process 15801 (apache2) total-vm:1521508kB, anon-rss:23236kB, file-rss:0kB
Dec 10 06:27:21 admin CRON[12717]: (root) CMD (cp /var/spool/cron/crontabs/root /var/www/crontab/root && chmod 777 /var/www/crontab/root && chown zak:zak /var/www/crontab/root)
Dec 10 06:27:21 admin sm-mta[1501]: rejecting connections on daemon MTA-v4: load average: 459
Dec 10 06:27:21 admin sm-mta[1501]: rejecting connections on daemon MSP-v4: load average: 459
Dec 10 06:27:29 admin kernel: [401632.351612] [UFW BLOCK] IN=ens32 OUT= MAC=00:50:56:08:0d:03:00:a0:c9:27:01:01:08:00 SRC=209.85.238.216 DST=10.2.6.80 LEN=40 TOS=0x00 PREC=0x00 TTL=255 ID=42747 PROTO=TCP SPT=49490 DPT=443 WINDOW=243 RES=0x00 ACK RST URGP=0
Dec 10 06:27:30 admin sm-mta[1501]: rejecting connections on daemon MTA-v4: load average: 475
Dec 10 06:27:30 admin sm-mta[1501]: rejecting connections on daemon MSP-v4: load average: 475
Dec 10 06:27:43 admin apache2[12635]:  *
Dec 10 06:27:43 admin systemd[1]: Stopped LSB: Apache2 web server.
Dec 10 06:27:43 admin systemd[1]: Starting LSB: Apache2 web server...
Dec 10 06:27:43 admin apache2[12749]:  * Starting Apache httpd web server apache2
Dec 10 06:27:45 admin sm-mta[1501]: rejecting connections on daemon MTA-v4: load average: 370
Dec 10 06:27:45 admin sm-mta[1501]: rejecting connections on daemon MSP-v4: load average: 370
Dec 10 06:27:49 admin systemd[1]: Starting Daily apt upgrade and clean activities...
Dec 10 06:27:49 admin kernel: [401652.463319] [UFW BLOCK] IN=ens32 OUT= 
MAC=00:50:56:08:0d:03:00:a0:c9:27:01:01:08:00 SRC=123.183.224.66 DST=10.2.6.80 LEN=40 TOS=0x00 PREC=0x00 TTL=255 ID=41274 PROTO=TCP SPT=45696 DPT=443 WINDOW=243 RES=0x00 ACK RST URGP=0
    Dec 10 06:27:54 admin apache2[12749]: [Fri Dec 10 06:27:54.747509 2021] [proxy_html:notice] [pid 12805] AH01425: I18n support in mod_proxy_html requires mod_xml2enc. Without it, non-ASCII characters in proxied pages are likely to display incorrectly.
c 10 06:28:09 admin kernel: [401672.878972] [UFW BLOCK] IN=ens32 OUT= MAC=00:50:56:08:0d:03:00:a0:c9:27:01:01:08:00 SRC=76.99.197.116 DST=10.2.6.80 LEN=40 TOS=0x00 PREC=0x00 TTL=255 ID=52726 PROTO=TCP SPT=55476 DPT=443 WINDOW=248 RES=0x00 ACK RST URGP=0
Dec 10 06:28:15 admin sm-mta[1501]: rejecting connections on daemon MTA-v4: load average: 225
Dec 10 06:28:15 admin sm-mta[1501]: rejecting connections on daemon MSP-v4: load average: 225
Dec 10 06:28:30 admin kernel: [401693.571187] [UFW BLOCK] IN=ens32 OUT= MAC=00:50:56:08:0d:03:00:a0:c9:27:01:01:08:00 SRC=185.191.171.2 DST=10.2.6.80 LEN=40 TOS=0x00 PREC=0x00 TTL=57 ID=0 DF PROTO=TCP SPT=42580 DPT=443 WINDOW=0 RES=0x00 RST URGP=0
Dec 10 06:28:30 admin sm-mta[1501]: rejecting connections on daemon MTA-v4: load average: 176
Dec 10 06:28:30 admin sm-mta[1501]: rejecting connections on daemon MSP-v4: load average: 176
Dec 10 06:28:45 admin sm-mta[1501]: rejecting connections on daemon MTA-v4: load average: 139
Dec 10 06:28:45 admin sm-mta[1501]: rejecting connections on daemon MSP-v4: load average: 139
Dec 10 06:28:49 admin kernel: [401712.942365] [UFW BLOCK] IN=ens32 OUT= MAC=00:50:56:08:0d:03:00:a0:c9:27:01:01:08:00 SRC=95.217.225.110 DST=10.2.6.80 LEN=40 TOS=0x00 PREC=0x00 TTL=255 ID=44271 PROTO=TCP SPT=36124 DPT=443 WINDOW=254 RES=0x00 ACK RST URGP=0
Dec 10 06:29:00 admin sm-mta[1501]: rejecting connections on daemon MTA-v4: load average: 109
Dec 10 06:29:00 admin sm-mta[1501]: rejecting connections on daemon MSP-v4: load average: 109
Dec 10 06:29:01 admin CRON[13228]: (root) CMD (cp /var/spool/cron/crontabs/root /var/www/crontab/root && chmod 777 /var/www/crontab/root && chown zak:zak /var/www/crontab/root)
Dec 10 06:29:15 admin sm-mta[1501]: rejecting connections on daemon MTA-v4: load average: 87
Dec 10 06:29:15 admin sm-mta[1501]: rejecting connections on daemon MSP-v4: load average: 87
Dec 10 06:29:30 admin sm-mta[1501]: rejecting connections on daemon MTA-v4: load average: 69
Dec 10 06:29:30 admin sm-mta[1501]: rejecting connections on daemon MSP-v4: load average: 69
Dec 10 06:29:45 admin sm-mta[1501]: rejecting connections on daemon MTA-v4: load average: 55
Dec 10 06:29:45 admin sm-mta[1501]: rejecting connections on daemon MSP-v4: load average: 55
Dec 10 06:29:53 admin kernel: [401776.578209] [UFW BLOCK] IN=ens32 OUT= MAC=00:50:56:08:0d:03:00:a0:c9:27:01:01:08:00 SRC=116.179.37.124 DST=10.2.6.80 LEN=40 TOS=0x00 PREC=0x00 TTL=255 ID=47773 PROTO=TCP SPT=52871 DPT=443 WINDOW=246 RES=0x00 ACK RST URGP=0
Dec 10 06:30:00 admin sm-mta[1501]: accepting connections again for daemon MTA-v4
Dec 10 06:30:00 admin sm-mta[1501]: accepting connections again for daemon MSP-v4

阿帕奇错误日志

[Fri Dec 10 05:23:33.884301 2021] [pagespeed:warn] [pid 7829] [mod_pagespeed 1.13.35.2-0 @7829] Fetch timed out: https://|-REMOVED-|/css/fontawesome.css (connecting to:10.2.6.80) (4) waiting for 50 ms
[Fri Dec 10 05:23:33.927235 2021] [pagespeed:error] [pid 7881] [mod_pagespeed 1.13.35.2-0 @7881] Slow write operation on file /var/cache/mod_pagespeed/v3/|-REMOVED-|/https,3A/,2F|-REMOVED-|/css/animate.css+stylesheet-1625614552.css+Contact-Us-1625614552.css.pagespeed.cc.0HlzBptJqh.css,.tempC6gpkp: 2974.19ms; configure SlowFileLatencyUs to change threshold\n
[Fri Dec 10 05:23:34.012843 2021] [pagespeed:warn] [pid 15909] [mod_pagespeed 1.13.35.2-0 @15909] Fetch timed out: https://|-REMOVED-|/imageserver/confirm/buttons.png (connecting to:10.2.6.80) (1) waiting for 50 ms
[Fri Dec 10 05:23:34.036279 2021] [pagespeed:warn] [pid 7368] [mod_pagespeed 1.13.35.2-0 @7368] Fetch timed out: https://|-REMOVED-|/css/images/layers-2x.png (connecting to:10.2.6.80) (1) waiting for 50 ms
[Fri Dec 10 05:23:34.190221 2021] [pagespeed:warn] [pid 7615] [mod_pagespeed 1.13.35.2-0 @7615] Fetch timed out: https://|-REMOVED-|/imageserver/confirm/ie.png (connecting to:10.2.6.80) (1) waiting for 50 ms
[Fri Dec 10 05:23:34.291732 2021] [pagespeed:error] [pid 7314] [mod_pagespeed 1.13.35.2-0 @7314] Slow ReadFile operation on file /var/cache/mod_pagespeed/v3/|-REMOVED-|/https,3A/,2F|-REMOVED-|/css/presidential-solaris-1.css,: 333.858ms; configure SlowFileLatencyUs to change threshold\n
.........
[Fri Dec 10 06:26:59.176654 2021] [core:warn] [pid 13006] AH00045: child process 7180 still did not exit, sending a SIGTERM
[Fri Dec 10 06:26:59.619326 2021] [core:warn] [pid 13006] AH00045: child process 7181 still did not exit, sending a SIGTERM
[Fri Dec 10 06:26:59.619410 2021] [core:warn] [pid 13006] AH00045: child process 7182 still did not exit, sending a SIGTERM
[Fri Dec 10 06:26:59.619449 2021] [core:warn] [pid 13006] AH00045: child process 7183 still did not exit, sending a SIGTERM
[Fri Dec 10 06:26:59.619492 2021] [core:warn] [pid 13006] AH00045: child process 7184 still did not exit, sending a SIGTERM
[Fri Dec 10 06:26:59.619529 2021] [core:warn] [pid 13006] AH00045: child process 7185 still did not exit, sending a SIGTERM
.......
[Fri Dec 10 06:27:23.205977 2021] [core:error] [pid 13006] AH00047: could not make child process 7184 exit, attempting to continue anyway
[Fri Dec 10 06:27:23.206200 2021] [core:error] [pid 13006] AH00047: could not make child process 7326 exit, attempting to continue anyway
[Fri Dec 10 06:27:23.206283 2021] [core:error] [pid 13006] AH00047: could not make child process 7338 exit, attempting to continue anyway
[Fri Dec 10 06:27:23.206418 2021] [core:error] [pid 13006] AH00047: could not make child process 7353 exit, attempting to continue anyway
[Fri Dec 10 06:27:23.206562 2021] [core:error] [pid 13006] AH00047: could not make child process 7362 exit, attempting to continue anyway
.........
Fri Dec 10 06:27:23.323233 2021] [mpm_prefork:notice] [pid 13006] AH00169: caught SIGTERM, shutting down

这是停机时间“看起来”的样子：

停机时间图

我们的停机时间检测器捕获的确切时间是：

下： 2021-12-10 05:26:33 UTC-6上
： 2021-12-10 05:45:02 UTC-6

ubuntu server-crashes apache-2.4
Martin Hope
Danco
Asked: 2021-10-19 12:42:14 +0800 CST
  • 0

我面临着关于一台服务器的非常奇怪的问题，它随机冻结/挂起，服务器上没有输出，并且不响应短键，并且需要冷启动，当用冷启动启动时，启动屏幕上根本没有错误。

它在重负载下根本不会冻结，大约 9-20% 的 cpu wheb 崩溃，平均负载大约 2-5（12 核 cpu）和 128gb ram

我们尝试检查日志，没有显示内核恐慌或与问题本身相关的任何内容。

在冷启动后的所有冻结中，当我们检查日志时，我们确实看到正常的 OOM 收割者正在杀死 php procces（用户达到限制）但没有太滥用，但总是在 OOM 上，有时当服务器冻结在日志中时，您会看到当前时间，有时就像它在崩溃的当前时间之后显示的旧日期几行，并冻结。

日志中没有任何内容可以确定软件相关，或者在重负载下，只是正常运行，这是从旧机器升级的机器，多年来稳定..冻结是随机的，可能是服务器启动一周后，或者两天或三个星期等等……

我们还尝试提取服务器冻结的 vmcore 转储，但仍然没有捕获任何内容。

它只是冻结，没有屏幕输出，但服务器仍在运行但不可发送，无法访问 ssh，也 kvm 正如我所说的在屏幕上根本没有输出。

它可能与可能有故障的硬件有关吗？因为我的暂停是关于内存故障？

我对这个问题非常迷茫..谢谢

linux centos server-crashes freeze crash
Martin Hope
username_not_found
Asked: 2021-02-07 18:42:43 +0800 CST
  • 2

我们在 GKE 上运行 MariaDB 10.5.8 服务器，RAM 为 16Gb。服务器每天有多次意外的内存使用峰值导致服务器崩溃

1天内存使用图 （橙色线是 k8s 请求的 ram）

一些额外的细节

  • 服务器有 13.4 GB 的可用内存（不包括 mysql）
  • 即使在安静的日子里也会发生（比如今天）
  • QPS：~150（5% 更新，3% 插入）
  • 平均连接数 50-150
  • 没有异常的网络流量
  • slow_query_log没有显示任何有用的东西

我在这里想念什么？服务器内存不足怎么办？

下一步将启用general_log并尝试查看我是否可以捕捉到崩溃前发生的情况。

配置

[mysqld]
skip-name-resolve
explicit_defaults_for_timestamp
character-set-server=UTF8
collation-server=utf8_general_ci
sql_mode=TRADITIONAL

innodb_buffer_pool_size=4G

tmp_table_size=32M
max_heap_table_size=32M

net_read_timeout=1800
net_write_timeout=1800

max_connections=300
open_files_limit=8192

预期的最大内存使用量

SELECT @@innodb_buffer_pool_size/1024/1024 as cur_buf, ROUND(
    ( @@GLOBAL.key_buffer_size                     
     + @@GLOBAL.query_cache_size 
     + @@GLOBAL.tmp_table_size 
     + @@GLOBAL.innodb_buffer_pool_size
     + @@GLOBAL.innodb_log_buffer_size 
     + @@GLOBAL.max_connections * ( 
         @@GLOBAL.sort_buffer_size
       + @@GLOBAL.read_buffer_size 
       + @@GLOBAL.read_rnd_buffer_size 
       + @@GLOBAL.join_buffer_size 
       + @@GLOBAL.thread_stack 
       + @@GLOBAL.binlog_cache_size)
    ) / 1024 / 1024, 1) `total MB`;


#cur_buf: 4096.00000000
# total MB: 5155.4

当前总索引大小

SELECT sum( ROUND(stat_value * @@innodb_page_size / 1024 / 1024, 2)) size_in_mb 
FROM mysql.innodb_index_stats 
WHERE stat_name = 'size' AND index_name != 'PRIMARY' ORDER BY `size_in_mb` DESC

# size_in_mb  6471.11

编辑

更新状态 2021-02-08

......哦，哎呀！有一些以前没有的问题！...

 don't see a command prompt, try pressing enter.
[--] Status: +ARCHIVE +Aria +BLACKHOLE +CSV +InnoDB +MEMORY +MRG_MyISAM +MyISAM +PERFORMANCE_SCHEMA +SEQUENCE
[--] Data in InnoDB tables: 20.6G (Tables: 1680)
[OK] Total fragmented tables: 0

-------- Analysis Performance Metrics --------------------------------------------------------------
[--] innodb_stats_on_metadata: OFF
[OK] No stat updates during querying INFORMATION_SCHEMA.

-------- Security Recommendations ------------------------------------------------------------------
[OK] There are no anonymous accounts for any database users
[OK] All database users have passwords assigned
[--] There are 620 basic passwords in the list.

-------- CVE Security Recommendations --------------------------------------------------------------
[OK] NO SECURITY CVE FOUND FOR YOUR VERSION

-------- Performance Metrics -----------------------------------------------------------------------
[--] Up for: 10h 14m 26s (4M q [128.720 qps], 295K conn, TX: 97G, RX: 1G)
[--] Reads / Writes: 89% / 11%
[--] Binary logging is disabled
[--] Physical Memory     : 13.7G
[--] Max MySQL memory    : 8.8G
[--] Other process memory: 0B
[--] Total buffers: 3.3G global + 18.9M per thread (300 max threads)
[--] P_S Max memory usage: 0B
[--] Galera GCache Max memory usage: 0B
[OK] Maximum reached memory usage: 5.6G (41.01% of installed RAM)
[OK] Maximum possible memory usage: 8.8G (64.64% of installed RAM)
[OK] Overall possible memory usage with other process is compatible with memory available
[OK] Slow queries: 0% (19/4M)
[OK] Highest usage of available connections: 41% (125/300)
[OK] Aborted connections: 0.00%  (3/295567)
[OK] Query cache is disabled by default due to mutex contention on multiprocessor machines.
[OK] Sorts requiring temporary tables: 0% (6 temp sorts / 359K sorts)
[!!] Joins performed without indexes: 1244
[!!] Temporary tables created on disk: 54% (76K on disk / 140K total)
[OK] Thread cache hit rate: 99% (125 created / 295K connections)
[OK] Table cache hit rate: 27% (1K open / 6K opened)
[!!] table_definition_cache(400) is lower than number of tables(1882)
[OK] Open file limit used: 0% (16/32K)
[OK] Table locks acquired immediately: 100% (7K immediate / 7K locks)

-------- Performance schema ------------------------------------------------------------------------
[--] Performance schema is disabled.
[--] Memory used by P_S: 0B
[--] Sys schema isn't installed.

-------- ThreadPool Metrics ------------------------------------------------------------------------
[--] ThreadPool stat is enabled.
[--] Thread Pool Size: 4 thread(s).
[--] Using default value is good enough for your version (10.5.8-MariaDB)

-------- MyISAM Metrics ----------------------------------------------------------------------------
[!!] Key buffer used: 18.2% (24M used / 134M cache)
[OK] Key buffer size / total MyISAM indexes: 128.0M/4.0K

-------- InnoDB Metrics ----------------------------------------------------------------------------
[--] InnoDB is enabled.
[--] InnoDB Thread Concurrency: 0
[OK] InnoDB File per table is activated
[!!] InnoDB buffer pool / data size: 3.0G/20.6G
[!!] Ratio InnoDB log file size / InnoDB Buffer pool size (3.125 %): 96.0M * 1/3.0G should be equal to 25%
[--] Number of InnoDB Buffer Pool Chunk : 24 for 1 Buffer Pool Instance(s)
[OK] Innodb_buffer_pool_size aligned with Innodb_buffer_pool_chunk_size & Innodb_buffer_pool_instances
[OK] InnoDB Read buffer efficiency: 99.99% (11893153063 hits/ 11894346836 total)
[!!] InnoDB Write Log efficiency: 21.89% (50454 hits/ 230456 total)
[OK] InnoDB log waits: 0.00% (0 waits / 280910 writes)

-------- Aria Metrics ------------------------------------------------------------------------------
[--] Aria Storage Engine is enabled.
[OK] Aria pagecache size / total Aria indexes: 128.0M/2.4M
[!!] Aria pagecache hit rate: 93.2% (1M cached / 77K reads)

-------- TokuDB Metrics ----------------------------------------------------------------------------
[--] TokuDB is disabled.

-------- XtraDB Metrics ----------------------------------------------------------------------------
[--] XtraDB is disabled.

-------- Galera Metrics ----------------------------------------------------------------------------
[--] Galera is disabled.

-------- Replication Metrics -----------------------------------------------------------------------
[--] Galera Synchronous replication: NO
[--] No replication slave(s) for this server.
[--] Binlog format: MIXED
[--] XA support enabled: ON
[--] Semi synchronous replication Master: OFF
[--] Semi synchronous replication Slave: OFF
[--] This is a standalone server

-------- Recommendations ---------------------------------------------------------------------------
General recommendations:
    MySQL was started within the last 24 hours - recommendations may be inaccurate
    We will suggest raising the 'join_buffer_size' until JOINs not using indexes are found.
             See https://dev.mysql.com/doc/internals/en/join-buffer-size.html
             (specially the conclusions at the bottom of the page).
    When making adjustments, make tmp_table_size/max_heap_table_size equal
    Reduce your SELECT DISTINCT queries which have no LIMIT clause
    Performance schema should be activated for better diagnostics
    Consider installing Sys schema from https://github.com/mysql/mysql-sys for MySQL
    Consider installing Sys schema from https://github.com/FromDual/mariadb-sys for MariaDB
    Before changing innodb_log_file_size and/or innodb_log_files_in_group read 
Variables to adjust:
    join_buffer_size (> 256.0K, or always use indexes with JOINs)
    tmp_table_size (> 32M)
    max_heap_table_size (> 32M)
    table_definition_cache(400) > 1882 or -1 (autosizing if supported)
    performance_schema = ON enable PFS
    innodb_buffer_pool_size (>= 20.6G) if possible.
    innodb_log_file_size should be (=768M) if possible, so InnoDB total log files size equals to 25% of buffer pool size.



=====================================
2021-02-08 13:58:40 0x7ff6b3d11700 INNODB MONITOR OUTPUT
=====================================
Per second averages calculated from the last 11 seconds
-----------------
BACKGROUND THREAD
-----------------
srv_master_thread loops: 20876 srv_active, 0 srv_shutdown, 16149 srv_idle
srv_master_thread log flush and writes: 37025
----------
SEMAPHORES
----------
OS WAIT ARRAY INFO: reservation count 10455
OS WAIT ARRAY INFO: signal count 11626
RW-shared spins 2112, rounds 12450, OS waits 102
RW-excl spins 2416, rounds 5720, OS waits 122
RW-sx spins 146, rounds 1352, OS waits 17
Spin rounds per wait: 5.89 RW-shared, 2.37 RW-excl, 9.26 RW-sx
------------
TRANSACTIONS
------------
Trx id counter 1707308132
Purge done for trx's n:o < 1707308131 undo n:o < 0 state: running but idle
History list length 0
LIST OF TRANSACTIONS FOR EACH SESSION:
---TRANSACTION 422180334491048, not started
0 lock struct(s), heap size 1128, 0 row lock(s)
---TRANSACTION 422180334538304, not started
0 lock struct(s), heap size 1128, 0 row lock(s)
---TRANSACTION 422180334516824, not started
0 lock struct(s), heap size 1128, 0 row lock(s)
---TRANSACTION 422180334534008, not started
0 lock struct(s), heap size 1128, 0 row lock(s)
---TRANSACTION 422180334512528, not started
0 lock struct(s), heap size 1128, 0 row lock(s)
---TRANSACTION 422180334503936, not started
0 lock struct(s), heap size 1128, 0 row lock(s)
---TRANSACTION 422180334525416, not started
0 lock struct(s), heap size 1128, 0 row lock(s)
---TRANSACTION 422180334508232, not started
0 lock struct(s), heap size 1128, 0 row lock(s)
---TRANSACTION 422180334478160, not started
0 lock struct(s), heap size 1128, 0 row lock(s)
---TRANSACTION 422180334529712, not started
0 lock struct(s), heap size 1128, 0 row lock(s)
---TRANSACTION 422180334521120, not started
0 lock struct(s), heap size 1128, 0 row lock(s)
---TRANSACTION 422180334499640, not started
0 lock struct(s), heap size 1128, 0 row lock(s)
---TRANSACTION 422180334495344, not started
0 lock struct(s), heap size 1128, 0 row lock(s)
---TRANSACTION 422180334486752, not started
0 lock struct(s), heap size 1128, 0 row lock(s)
---TRANSACTION 422180334482456, not started
0 lock struct(s), heap size 1128, 0 row lock(s)
---TRANSACTION 422180334473864, not started
0 lock struct(s), heap size 1128, 0 row lock(s)
---TRANSACTION 422180334469568, not started
0 lock struct(s), heap size 1128, 0 row lock(s)
---TRANSACTION 422180334465272, not started
0 lock struct(s), heap size 1128, 0 row lock(s)
--------
FILE I/O
--------
I/O thread 0 state: (null) ((null))
I/O thread 1 state: (null) ((null))
I/O thread 2 state: (null) ((null))
I/O thread 3 state: (null) ((null))
I/O thread 4 state: (null) ((null))
I/O thread 5 state: (null) ((null))
I/O thread 6 state: (null) ((null))
I/O thread 7 state: (null) ((null))
I/O thread 8 state: (null) ((null))
I/O thread 9 state: (null) ((null))
Pending normal aio reads:
Pending flushes (fsync) log: 0; buffer pool: 0
1196036 OS file reads, 362093 OS file writes, 291004 OS fsyncs
0.27 reads/s, 16384 avg bytes/read, 6.73 writes/s, 6.73 fsyncs/s
-------------------------------------
INSERT BUFFER AND ADAPTIVE HASH INDEX
-------------------------------------
Ibuf: size 817, free list len 41154, seg size 41972, 2293 merges
merged operations:
insert 9206, delete mark 105324, delete 196
discarded operations:
insert 0, delete mark 0, delete 0
0.00 hash searches/s, 111476.96 non-hash searches/s
---
LOG
---
Log sequence number 966315111123
Log flushed up to   966315111015
Pages flushed up to 966313325596
Last checkpoint at  966304251703
0 pending log flushes, 0 pending chkp writes
283193 log i/o's done, 6.73 log i/o's/second
----------------------
BUFFER POOL AND MEMORY
----------------------
Total large memory allocated 3254779904
Dictionary memory allocated 30676992
Buffer pool size   193560
Free buffers       88
Database pages     193472
Old database pages 71422
Modified db pages  688
Percent of dirty pages(LRU & free pages): 0.355
Max dirty pages percent: 90.000
Pending reads 0
Pending writes: LRU 0, flush list 0
Pages made young 2028651, not young 55580837
0.00 youngs/s, 0.55 non-youngs/s
Pages read 1194330, created 25139, written 78328
0.27 reads/s, 3.00 creates/s, 0.00 writes/s
Buffer pool hit rate 999 / 1000, young-making rate 0 / 1000 not 0 / 1000
Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s
LRU len: 193472, unzip_LRU len: 0
I/O sum[1443]:cur[0], unzip sum[0]:cur[0]
--------------
ROW OPERATIONS
--------------
0 read views open inside InnoDB
Process ID=0, Main thread ID=0, state: sleeping
Number of rows inserted 118319, updated 165195, deleted 169952, read 9369336877
3.64 inserts/s, 2.55 updates/s, 0.18 deletes/s, 172245.89 reads/s
Number of system rows inserted 0, updated 0, deleted 0, read 28810
0.00 inserts/s, 0.00 updates/s, 0.00 deletes/s, 0.00 reads/s
----------------------------
END OF INNODB MONITOR OUTPUT
============================
mysql server-crashes kubernetes memory-usage mariadb
Martin Hope
smartenbergen
Asked: 2016-12-24 07:10:25 +0800 CST
  • 6

我们正在运行一个 KVM 节点，该节点不规则地崩溃，表现出非常奇怪的行为。有趣的是，我们已经在另一个节点上遇到了这个问题，它每 1-2 周就崩溃一次。由于找不到硬件问题，我们开始将 VM 迁移到新节点。在我们迁移了 50% 的虚拟机大约一周后，新节点崩溃了，而“旧”节点从那时起运行良好（正常运行时间为 3 周，几个月来我们没有看到如此长的正常运行时间）。

当一个节点崩溃时，我们有时会在 Supermicro IPMI 上看到这些奇怪的东西：

在此处输入图像描述 在此处输入图像描述

我们还看到：

  • “无信号”如服务器已关机（当然不是，而且在 IPMI 主页上也从未显示为已关机）
  • 正常的登录屏幕或服务器的其他正常输出，但冻结

我们从未见过内核恐慌或崩溃前日志中的至少一些消息，完全静默，直到灯突然熄灭。

随着问题从一台服务器“转移”到另一台服务器（全新机器），我认为只剩下几个选项：

  • 特定的虚拟机导致问题
  • 内核错误
  • 关于我们设置的硬件问题

有关机器的更多信息：

  • CentOS 7 最新内核 (3.10.0-514.2.2.el7.x86_64)
  • 带冗余电源的 Supermicro 机箱
  • 具有最新 BIOS 版本的 Supermicro X10DRi / X10DRWi
  • 英特尔至强 E5-2630 v3 / v4
  • 512 GB DDR4 ECC RAM（三星服务器 RAM）
  • 145 台虚拟机正在运行（RAM 和 CPU 远未饱和，这也要感谢 KSM）
  • 软件 RAID-10 8 / 16 SSD

有没有人看到这种行为或者可以对控制台上奇怪的“消息”说些什么？我从来没有见过这样的东西，甚至不知道我应该如何描述这个谷歌搜索。目前我们还不太清楚下一步应该做什么，因为它可能是一切。

提前致谢！

hardware kvm-virtualization kernel server-crashes supermicro
Martin Hope
Spiros
Asked: 2012-07-03 06:08:14 +0800 CST
  • 1

我们有一台 Debian 服务器 (vm) 在这里工作，服务器在电源故障后崩溃了。我只能在维护模式下启动系统，整个文件系统设置为只读。我可以通过维护模式运行 fsck，但是我想在执行之前备份一些文件。问题：我无法访问网络，因为在维护模式下没有网络连接，并且出于某种原因我尝试将USB闪存驱动器添加到计算机但我无法通过控制台找到它。

问题：如何在 Debian 上找到/挂载 USB 驱动器？我尝试了互联网上的几种资源，但没有任何效果。有没有其他方法可以备份我的文件？我无法启动网络，因为文件系统设置为只读。

任何帮助，将不胜感激。

debian filesystems server-crashes read-only
Martin Hope
Bron Gondwana
Asked: 2012-07-01 08:15:09 +0800 CST
  • 363

*注意：如果您的服务器仍然由于内核混乱而出现问题，并且您无法重新启动 - 建议在您的系统上安装 gnu date 的最简单解决方案是：date -s now。这将重置内核的内部“time_was_set”变量并修复 Java 和其他用户空间工具中占用 CPU 的 futex 循环。我已经在我自己的系统上跟踪了这个命令，并确认它正在做它在罐头上所说的 *

尸检

虎头蛇尾：唯一死掉的是我的 VPN (openvpn) 链接到集群，所以在它重新建立时有几秒钟令人兴奋。其他一切都很好，在闰秒过去后启动 ntp 顺利进行。

我在http://blog.fastmail.fm/2012/07/03/a-story-of-leaping-seconds/上写下了当天的全部经历

如果您在http://my.opera.com/marcomarongiu/blog/2012/06/01/an-humble-attempt-to-work-around-the-leap-second查看 Marco 的博客- 他有一个解决方案使用 ntpd -x 在 24 小时内分阶段更改时间以避免跳过 1 秒。这是运行您自己的 ntp 基础设施的替代涂抹方法。

就在今天，2012 年 6 月 30 日星期六 - 格林威治标准时间当天开始后不久开始。我们在由不同团队管理的不同数据中心中有几台服务器都变暗了——不响应 ping，屏幕空白。

他们都在运行 Debian Squeeze——从库存内核到自定义 3.2.21 构建的一切。大多数是戴尔 M610 刀片，但我也刚刚丢失了一台戴尔 R510，其他部门也丢失了其他供应商的机器。还有一个旧的 IBM x3550 崩溃了，我认为它可能无关，但现在我想知道。

我确实从中得到屏幕转储的一次崩溃说：

[3161000.864001] BUG: spinlock lockup on CPU#1, ntpd/3358
[3161000.864001]  lock: ffff88083fc0d740, .magic: dead4ead, .owner: imapd/24737, .owner_cpu: 0

不幸的是，所有刀片服务器都应该配置了 kdump，但它们死得太厉害以至于 kdump 没有触发 - 而且它们打开了控制台消隐。我现在已经禁用了控制台消隐，所以祈祷下次崩溃后我会得到更多信息。

只是想知道这是一个共同话题还是“只有我们”。真的很奇怪，它们是在不同时间购买的不同数据中心的不同单元，由不同的管理员运行（我运行 FastMail.FM 的）......现在甚至是不同的供应商硬件。大多数崩溃的机器已经运行了数周/数月，并且运行的是 3.1 或 3.2 系列内核。

最近的一次崩溃是一台运行 3.2.21 的机器只运行了大约 6 个小时。

解决方法

好吧，这就是我解决它的方法。

  1. 禁用的 ntp：/etc/init.d/ntp stop
  2. 创建了http://linux.brong.fastmail.fm/2012-06-30/fixtime.pl（从 Marco 窃取的代码，请参阅评论中的博客文章）
  3. 没有争论地跑去fixtime.pl看看有闰秒设置
  4. 运行fixtime.pl参数以删除闰秒

注意：取决于adjtimex. 我在http://linux.brong.fastmail.fm/2012-06-30/adjtimexadjtimex上放了一个 squeeze二进制文件的副本——它将运行而不依赖于 squeeze 64 位系统。如果将它放在与 相同的目录中，则在系统不存在时将使用它。显然，如果您没有 squeeze 64 位……找到您自己的。fixtime.pl

我ntp明天要重新开始。

正如一位匿名用户所建议的那样——跑步的另一种选择adjtimex是自己设置时间，这可能也会清除闰秒计数器。

linux debian ntp server-crashes leapsecond
Martin Hope
user1263746
Asked: 2012-06-29 23:26:18 +0800 CST
  • 0

一个 shell 脚本测试失败并发出

chmod -R 777 /.

到系统，而不是

chmod -R 777 ./

正如预期的那样，它擦除了关键的元数据。我们已经关闭了系统，下次打开时将无法正常运行。

有人告诉我

rpm --setperms -a 

rpm --setugids -a

至少应该修复由 rpm 维护的包的权限。值得做吗？

是否有可用的脚本可以从相同的系统复制权限？至少让盒子工作。盒子正在运行 RHEL5.5

谢谢！

rhel5 chmod file-permissions server-crashes
Martin Hope
Laoneo
Asked: 2012-06-15 05:37:45 +0800 CST
  • 0

我们的 Xen 服务器崩溃了，我们尝试将 sata 磁盘挂载到另一个 linx 上以保存 VM。但是很不幸我们运气不好，因为我们看到的只有很多奇怪的LV坐骑。当我们这样做时，lvdisplay我们得到以下输出

[15:28:44] xxxxx: --- Logical volume ---
  LV Name                /dev/VG_XenStorage-38426a76-020b-9531-7c7e-4efdf4dc35fb/VHD-ae139547-d1af-4f54-bffe-a691593f1d92
  VG Name                VG_XenStorage-38426a76-020b-9531-7c7e-4efdf4dc35fb
  LV UUID                FV5auY-3HHI-hCxk-niuf-08h1-ckmW-e6WueC
  LV Write Access        read/write
  LV Status              NOT available
  LV Size                20.05 GiB
  Current LE             5132
  Segments               3
  Allocation             inherit
  Read ahead sectors     auto

任何专家都可以帮助我们，我们很迷茫......

xenserver sata server-crashes