我们需要在具有 32Gb RAM 的服务器上运行一些测试。
我们可以访问的所有服务器都有 64GB,我们无法对其进行物理更改。
是否有某种方法告诉 RHEL 仅使用固定数量的 RAM,小于安装的 RAM?
我们需要在具有 32Gb RAM 的服务器上运行一些测试。
我们可以访问的所有服务器都有 64GB,我们无法对其进行物理更改。
是否有某种方法告诉 RHEL 仅使用固定数量的 RAM,小于安装的 RAM?
在配备 24 GiB 内存的 Windows 2019 服务器上,只有大约 1 GiB 是可用的。
但是当我使用Task Manager
or时Process Explorer
,没有进程使用超过一些 MiB。
如果我添加每个进程使用的内存,总和只有 3 GiB(我使用 Process Explorer 导出数据并在电子表格中进行总和)。
总内存:24 GiB已
用内存:23 GiB已
使用交换:4 GiB
每个进程内存之和:3 GiB
我总结Private Bytes
or Working Set
,每个总共大约 3 GiB。
我知道这两个指标远非最佳指标,但这些工具无法提供更好的指标。与现实相差近一个数量级!
如何更好地了解内存使用情况?
图形显示WinDirStat
会非常好,但由于这个问题是 Windows 的主要问题,我怀疑是否存在任何简单的问题。
我在 DigitalOcean 液滴(1GB RAM)上托管一个 WordPress 网站。该网站的MySQL数据库偶尔会崩溃,导致网站显示“建立数据库连接时出错”。内存使用量在凌晨 2:40 左右下降,表明这是数据库崩溃的时候。我查看了当天的 MySQL 日志文件,最早的条目是上午 10 点 47 分。这是日志文件的开头:
2021-12-06T10:47:14.800977Z 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timest$
2021-12-06T10:47:14.806192Z 0 [Note] /usr/sbin/mysqld (mysqld 5.7.36-0ubuntu0.18.04.1) starting as process 2810 ...
2021-12-06T10:47:14.819674Z 0 [Note] InnoDB: PUNCH HOLE support available
2021-12-06T10:47:14.819711Z 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2021-12-06T10:47:14.819716Z 0 [Note] InnoDB: Uses event mutexes
2021-12-06T10:47:14.819720Z 0 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier
2021-12-06T10:47:14.819723Z 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
2021-12-06T10:47:14.819727Z 0 [Note] InnoDB: Using Linux native AIO
2021-12-06T10:47:14.820551Z 0 [Note] InnoDB: Number of pools: 1
2021-12-06T10:47:14.823342Z 0 [Note] InnoDB: Using CPU crc32 instructions
2021-12-06T10:47:14.825847Z 0 [Note] InnoDB: Initializing buffer pool, total size = 128M, instances = 1, chunk size = 128M
2021-12-06T10:47:14.826246Z 0 [ERROR] InnoDB: mmap(137428992 bytes) failed; errno 12
2021-12-06T10:47:14.826258Z 0 [ERROR] InnoDB: Cannot allocate memory for the buffer pool
2021-12-06T10:47:14.826262Z 0 [ERROR] InnoDB: Plugin initialization aborted with error Generic error
2021-12-06T10:47:14.826270Z 0 [ERROR] Plugin 'InnoDB' init function returned error.
2021-12-06T10:47:14.826274Z 0 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
2021-12-06T10:47:14.826278Z 0 [ERROR] Failed to initialize builtin plugins.
2021-12-06T10:47:14.826282Z 0 [ERROR] Aborting
2021-12-06T10:47:14.832237Z 0 [Note] Binlog end
2021-12-06T10:47:14.832297Z 0 [Note] Shutting down plugin 'CSV'
2021-12-06T10:47:14.832572Z 0 [Note] /usr/sbin/mysqld: Shutdown complete
根据日志文件,MySQL 似乎内存不足。但是,Droplet 的内存使用率稳定在 73% 左右,直到凌晨 2:40 左右数据库崩溃,才下降到 32%。它似乎有足够的可用内存,那么为什么会崩溃?
编辑根据要求,这是我的 MySQL 配置文件的内容:
/etc/mysql/conf.d/mysql.cnf
[mysql]
/etc/mysql/conf.d/mysqldump.cnf
[mysqldump]
quick
quote-names
max_allowed_packet = 16M
/etc/mysql/mysql.conf.d/mysqld.cnf
#
# The MySQL database server configuration file.
#
# You can copy this to one of:
# - "/etc/mysql/my.cnf" to set global options,
# - "~/.my.cnf" to set user-specific options.
#
# One can use all long options that the program supports.
# Run program with --help to get a list of available options and with
# --print-defaults to see which it would actually understand and use.
#
# For explanations see
# http://dev.mysql.com/doc/mysql/en/server-system-variables.html
# This will be passed to all mysql clients
# It has been reported that passwords should be enclosed with ticks/quotes
# escpecially if they contain "#" chars...
# Remember to edit /etc/mysql/debian.cnf when changing the socket location.
# Here is entries for some specific programs
# The following values assume you have at least 32M ram
[mysqld_safe]
socket = /var/run/mysqld/mysqld.sock
nice = 0
[mysqld]
#
# * Basic Settings
#
user = mysql
pid-file = /var/run/mysqld/mysqld.pid
socket = /var/run/mysqld/mysqld.sock
port = 3306
basedir = /usr
datadir = /var/lib/mysql
tmpdir = /tmp
lc-messages-dir = /usr/share/mysql
skip-external-locking
#
# Instead of skip-networking the default is now to listen only on
# localhost which is more compatible and is not less secure.
bind-address = 127.0.0.1
#
# * Fine Tuning
#
key_buffer_size = 16M
max_allowed_packet = 16M
thread_stack = 192K
thread_cache_size = 8
# This replaces the startup script and checks MyISAM tables if needed
# the first time they are touched
myisam-recover-options = BACKUP
#max_connections = 100
#table_open_cache = 64
#thread_concurrency = 10
#
# * Query Cache Configuration
#
query_cache_limit = 1M
query_cache_size = 16M
#
# * Logging and Replication
#
# Both location gets rotated by the cronjob.
# Be aware that this log type is a performance killer.
# As of 5.1 you can enable the log at runtime!
#general_log_file = /var/log/mysql/mysql.log
#general_log = 1
#
# Error log - should be very few entries.
#
log_error = /var/log/mysql/error.log
#
# Here you can see queries with especially long duration
#slow_query_log = 1
#slow_query_log_file = /var/log/mysql/mysql-slow.log
#long_query_time = 2
#log-queries-not-using-indexes
#
# The following can be used as easy to replay backup logs or for replication.
# note: if you are setting up a replication slave, see README.Debian about
# other settings you may need to change.
#server-id = 1
#log_bin = /var/log/mysql/mysql-bin.log
expire_logs_days = 10
max_binlog_size = 100M
#binlog_do_db = include_database_name
#binlog_ignore_db = include_database_name
#
# * InnoDB
#
# InnoDB is enabled by default with a 10MB datafile in /var/lib/mysql/.
# Read the manual for more InnoDB related options. There are many!
#
# * Security Features
#
# Read the manual, too, if you want chroot!
# chroot = /var/lib/mysql/
#
# For generating SSL certificates I recommend the OpenSSL GUI "tinyca".
#
# ssl-ca=/etc/mysql/cacert.pem
# ssl-cert=/etc/mysql/server-cert.pem
# ssl-key=/etc/mysql/server-key.pem
/etc/mysql/mysql.conf.d/mysqld_safe_syslog.cnf
[mysqld_safe]
syslog
EC2 ECS 允许您为 ECS 任务和在这些任务中运行的容器指定memory
和。memoryReservation
我了解容器设置映射memory
到memoryReservation
底层Docker 内存设置。
但是,我不确定在 EC2 ECS 下具有和设置的任务意味着什么。这仅用于ECS的任务调度,还是有一些运行时影响?memory
memoryReservation
假设我已经在容器上设置了内存限制,那么在任务上设置额外的内存限制是什么意思?
上周我们更新了几个 wordpress 站点,这些站点通过 LXD 将 Alpine Linux 作为主机(Ubuntu 20.04)中的容器运行。
更新摘要如下:
Alpine Linux v3.8 -> 3.14
PHP 5.3.6 -> 7.4.24
Wordpress 5.0.3 -> 5.7.3
在这些更新之后,我们开始遇到服务器性能问题,我们发现更新后的容器使用的内存(常驻内存)是旧容器的 3 倍或更多(大约 150MB 对 50MB),这导致服务器开始更频繁地交换。
在旧版本中(使用 PHP 5.3),php
(进程)使用的内存会随着页面的处理(如预期的那样)增加,但在它完成后,它会恢复正常。换句话说,类似于:10MB
---> 95MB
---> 10MB
。
在更新的容器中,所使用的内存php
以相同的方式增加,但不会恢复到“正常”:10MB
---> 95MB
---> 95MB
。每次使用新进程时,都会发生同样的情况,通过可用子进程的数量(在本例中为每个站点 4 个)增加内存使用量。
7.2.x
和7.3.x
:同样的事情php 8.0.11
:同样的问题apache2
而不是lighttpd
(当前 php 作为 fcgi 运行):相同的行为它没有恢复内存的原因是什么?如何修复?
Alpine 3.14
容器并执行了“简单循环”测试。在这种情况下,常驻内存按预期减少了。但是,一旦我使用实际的 wordpress 站点进行测试,问题仍然存在。Ubuntu 20.04
容器并进行了相同的测试。结果与 clean 相同Alpine 3.14
。我有一个图像处理 shell 脚本,可以提供不同的参数集,这些参数都为我产生有效的输出。我正在尝试识别具有最低资源(CPU/RAM)使用率的参数集,因为该脚本将在云上运行。有什么工具可以用来做基准测试吗?我正在寻找可以为我提供脚本使用的总 cpu 周期或我可以比较的任何其他相关数据/指标的东西。
操作系统:Ubuntu 20.04,外壳:BASH
我正在寻找的示例:
magical-benchmark-tool -c pdftoppm -png test.pdf test
输出
time (seconds) processor cycles max memory used (MB) mean memory used (MB)
-------------- ---------------- -------------------- --------------------
19 1253 250 128
谢谢!
我们在 GKE 上运行 MariaDB 10.5.8 服务器,RAM 为 16Gb。服务器每天有多次意外的内存使用峰值导致服务器崩溃
1天内存使用图 (橙色线是 k8s 请求的 ram)
一些额外的细节
slow_query_log
没有显示任何有用的东西我在这里想念什么?服务器内存不足怎么办?
下一步将启用general_log
并尝试查看我是否可以捕捉到崩溃前发生的情况。
[mysqld]
skip-name-resolve
explicit_defaults_for_timestamp
character-set-server=UTF8
collation-server=utf8_general_ci
sql_mode=TRADITIONAL
innodb_buffer_pool_size=4G
tmp_table_size=32M
max_heap_table_size=32M
net_read_timeout=1800
net_write_timeout=1800
max_connections=300
open_files_limit=8192
预期的最大内存使用量
SELECT @@innodb_buffer_pool_size/1024/1024 as cur_buf, ROUND(
( @@GLOBAL.key_buffer_size
+ @@GLOBAL.query_cache_size
+ @@GLOBAL.tmp_table_size
+ @@GLOBAL.innodb_buffer_pool_size
+ @@GLOBAL.innodb_log_buffer_size
+ @@GLOBAL.max_connections * (
@@GLOBAL.sort_buffer_size
+ @@GLOBAL.read_buffer_size
+ @@GLOBAL.read_rnd_buffer_size
+ @@GLOBAL.join_buffer_size
+ @@GLOBAL.thread_stack
+ @@GLOBAL.binlog_cache_size)
) / 1024 / 1024, 1) `total MB`;
#cur_buf: 4096.00000000
# total MB: 5155.4
当前总索引大小
SELECT sum( ROUND(stat_value * @@innodb_page_size / 1024 / 1024, 2)) size_in_mb
FROM mysql.innodb_index_stats
WHERE stat_name = 'size' AND index_name != 'PRIMARY' ORDER BY `size_in_mb` DESC
# size_in_mb 6471.11
更新状态 2021-02-08
......哦,哎呀!有一些以前没有的问题!...
don't see a command prompt, try pressing enter.
[--] Status: +ARCHIVE +Aria +BLACKHOLE +CSV +InnoDB +MEMORY +MRG_MyISAM +MyISAM +PERFORMANCE_SCHEMA +SEQUENCE
[--] Data in InnoDB tables: 20.6G (Tables: 1680)
[OK] Total fragmented tables: 0
-------- Analysis Performance Metrics --------------------------------------------------------------
[--] innodb_stats_on_metadata: OFF
[OK] No stat updates during querying INFORMATION_SCHEMA.
-------- Security Recommendations ------------------------------------------------------------------
[OK] There are no anonymous accounts for any database users
[OK] All database users have passwords assigned
[--] There are 620 basic passwords in the list.
-------- CVE Security Recommendations --------------------------------------------------------------
[OK] NO SECURITY CVE FOUND FOR YOUR VERSION
-------- Performance Metrics -----------------------------------------------------------------------
[--] Up for: 10h 14m 26s (4M q [128.720 qps], 295K conn, TX: 97G, RX: 1G)
[--] Reads / Writes: 89% / 11%
[--] Binary logging is disabled
[--] Physical Memory : 13.7G
[--] Max MySQL memory : 8.8G
[--] Other process memory: 0B
[--] Total buffers: 3.3G global + 18.9M per thread (300 max threads)
[--] P_S Max memory usage: 0B
[--] Galera GCache Max memory usage: 0B
[OK] Maximum reached memory usage: 5.6G (41.01% of installed RAM)
[OK] Maximum possible memory usage: 8.8G (64.64% of installed RAM)
[OK] Overall possible memory usage with other process is compatible with memory available
[OK] Slow queries: 0% (19/4M)
[OK] Highest usage of available connections: 41% (125/300)
[OK] Aborted connections: 0.00% (3/295567)
[OK] Query cache is disabled by default due to mutex contention on multiprocessor machines.
[OK] Sorts requiring temporary tables: 0% (6 temp sorts / 359K sorts)
[!!] Joins performed without indexes: 1244
[!!] Temporary tables created on disk: 54% (76K on disk / 140K total)
[OK] Thread cache hit rate: 99% (125 created / 295K connections)
[OK] Table cache hit rate: 27% (1K open / 6K opened)
[!!] table_definition_cache(400) is lower than number of tables(1882)
[OK] Open file limit used: 0% (16/32K)
[OK] Table locks acquired immediately: 100% (7K immediate / 7K locks)
-------- Performance schema ------------------------------------------------------------------------
[--] Performance schema is disabled.
[--] Memory used by P_S: 0B
[--] Sys schema isn't installed.
-------- ThreadPool Metrics ------------------------------------------------------------------------
[--] ThreadPool stat is enabled.
[--] Thread Pool Size: 4 thread(s).
[--] Using default value is good enough for your version (10.5.8-MariaDB)
-------- MyISAM Metrics ----------------------------------------------------------------------------
[!!] Key buffer used: 18.2% (24M used / 134M cache)
[OK] Key buffer size / total MyISAM indexes: 128.0M/4.0K
-------- InnoDB Metrics ----------------------------------------------------------------------------
[--] InnoDB is enabled.
[--] InnoDB Thread Concurrency: 0
[OK] InnoDB File per table is activated
[!!] InnoDB buffer pool / data size: 3.0G/20.6G
[!!] Ratio InnoDB log file size / InnoDB Buffer pool size (3.125 %): 96.0M * 1/3.0G should be equal to 25%
[--] Number of InnoDB Buffer Pool Chunk : 24 for 1 Buffer Pool Instance(s)
[OK] Innodb_buffer_pool_size aligned with Innodb_buffer_pool_chunk_size & Innodb_buffer_pool_instances
[OK] InnoDB Read buffer efficiency: 99.99% (11893153063 hits/ 11894346836 total)
[!!] InnoDB Write Log efficiency: 21.89% (50454 hits/ 230456 total)
[OK] InnoDB log waits: 0.00% (0 waits / 280910 writes)
-------- Aria Metrics ------------------------------------------------------------------------------
[--] Aria Storage Engine is enabled.
[OK] Aria pagecache size / total Aria indexes: 128.0M/2.4M
[!!] Aria pagecache hit rate: 93.2% (1M cached / 77K reads)
-------- TokuDB Metrics ----------------------------------------------------------------------------
[--] TokuDB is disabled.
-------- XtraDB Metrics ----------------------------------------------------------------------------
[--] XtraDB is disabled.
-------- Galera Metrics ----------------------------------------------------------------------------
[--] Galera is disabled.
-------- Replication Metrics -----------------------------------------------------------------------
[--] Galera Synchronous replication: NO
[--] No replication slave(s) for this server.
[--] Binlog format: MIXED
[--] XA support enabled: ON
[--] Semi synchronous replication Master: OFF
[--] Semi synchronous replication Slave: OFF
[--] This is a standalone server
-------- Recommendations ---------------------------------------------------------------------------
General recommendations:
MySQL was started within the last 24 hours - recommendations may be inaccurate
We will suggest raising the 'join_buffer_size' until JOINs not using indexes are found.
See https://dev.mysql.com/doc/internals/en/join-buffer-size.html
(specially the conclusions at the bottom of the page).
When making adjustments, make tmp_table_size/max_heap_table_size equal
Reduce your SELECT DISTINCT queries which have no LIMIT clause
Performance schema should be activated for better diagnostics
Consider installing Sys schema from https://github.com/mysql/mysql-sys for MySQL
Consider installing Sys schema from https://github.com/FromDual/mariadb-sys for MariaDB
Before changing innodb_log_file_size and/or innodb_log_files_in_group read
Variables to adjust:
join_buffer_size (> 256.0K, or always use indexes with JOINs)
tmp_table_size (> 32M)
max_heap_table_size (> 32M)
table_definition_cache(400) > 1882 or -1 (autosizing if supported)
performance_schema = ON enable PFS
innodb_buffer_pool_size (>= 20.6G) if possible.
innodb_log_file_size should be (=768M) if possible, so InnoDB total log files size equals to 25% of buffer pool size.
=====================================
2021-02-08 13:58:40 0x7ff6b3d11700 INNODB MONITOR OUTPUT
=====================================
Per second averages calculated from the last 11 seconds
-----------------
BACKGROUND THREAD
-----------------
srv_master_thread loops: 20876 srv_active, 0 srv_shutdown, 16149 srv_idle
srv_master_thread log flush and writes: 37025
----------
SEMAPHORES
----------
OS WAIT ARRAY INFO: reservation count 10455
OS WAIT ARRAY INFO: signal count 11626
RW-shared spins 2112, rounds 12450, OS waits 102
RW-excl spins 2416, rounds 5720, OS waits 122
RW-sx spins 146, rounds 1352, OS waits 17
Spin rounds per wait: 5.89 RW-shared, 2.37 RW-excl, 9.26 RW-sx
------------
TRANSACTIONS
------------
Trx id counter 1707308132
Purge done for trx's n:o < 1707308131 undo n:o < 0 state: running but idle
History list length 0
LIST OF TRANSACTIONS FOR EACH SESSION:
---TRANSACTION 422180334491048, not started
0 lock struct(s), heap size 1128, 0 row lock(s)
---TRANSACTION 422180334538304, not started
0 lock struct(s), heap size 1128, 0 row lock(s)
---TRANSACTION 422180334516824, not started
0 lock struct(s), heap size 1128, 0 row lock(s)
---TRANSACTION 422180334534008, not started
0 lock struct(s), heap size 1128, 0 row lock(s)
---TRANSACTION 422180334512528, not started
0 lock struct(s), heap size 1128, 0 row lock(s)
---TRANSACTION 422180334503936, not started
0 lock struct(s), heap size 1128, 0 row lock(s)
---TRANSACTION 422180334525416, not started
0 lock struct(s), heap size 1128, 0 row lock(s)
---TRANSACTION 422180334508232, not started
0 lock struct(s), heap size 1128, 0 row lock(s)
---TRANSACTION 422180334478160, not started
0 lock struct(s), heap size 1128, 0 row lock(s)
---TRANSACTION 422180334529712, not started
0 lock struct(s), heap size 1128, 0 row lock(s)
---TRANSACTION 422180334521120, not started
0 lock struct(s), heap size 1128, 0 row lock(s)
---TRANSACTION 422180334499640, not started
0 lock struct(s), heap size 1128, 0 row lock(s)
---TRANSACTION 422180334495344, not started
0 lock struct(s), heap size 1128, 0 row lock(s)
---TRANSACTION 422180334486752, not started
0 lock struct(s), heap size 1128, 0 row lock(s)
---TRANSACTION 422180334482456, not started
0 lock struct(s), heap size 1128, 0 row lock(s)
---TRANSACTION 422180334473864, not started
0 lock struct(s), heap size 1128, 0 row lock(s)
---TRANSACTION 422180334469568, not started
0 lock struct(s), heap size 1128, 0 row lock(s)
---TRANSACTION 422180334465272, not started
0 lock struct(s), heap size 1128, 0 row lock(s)
--------
FILE I/O
--------
I/O thread 0 state: (null) ((null))
I/O thread 1 state: (null) ((null))
I/O thread 2 state: (null) ((null))
I/O thread 3 state: (null) ((null))
I/O thread 4 state: (null) ((null))
I/O thread 5 state: (null) ((null))
I/O thread 6 state: (null) ((null))
I/O thread 7 state: (null) ((null))
I/O thread 8 state: (null) ((null))
I/O thread 9 state: (null) ((null))
Pending normal aio reads:
Pending flushes (fsync) log: 0; buffer pool: 0
1196036 OS file reads, 362093 OS file writes, 291004 OS fsyncs
0.27 reads/s, 16384 avg bytes/read, 6.73 writes/s, 6.73 fsyncs/s
-------------------------------------
INSERT BUFFER AND ADAPTIVE HASH INDEX
-------------------------------------
Ibuf: size 817, free list len 41154, seg size 41972, 2293 merges
merged operations:
insert 9206, delete mark 105324, delete 196
discarded operations:
insert 0, delete mark 0, delete 0
0.00 hash searches/s, 111476.96 non-hash searches/s
---
LOG
---
Log sequence number 966315111123
Log flushed up to 966315111015
Pages flushed up to 966313325596
Last checkpoint at 966304251703
0 pending log flushes, 0 pending chkp writes
283193 log i/o's done, 6.73 log i/o's/second
----------------------
BUFFER POOL AND MEMORY
----------------------
Total large memory allocated 3254779904
Dictionary memory allocated 30676992
Buffer pool size 193560
Free buffers 88
Database pages 193472
Old database pages 71422
Modified db pages 688
Percent of dirty pages(LRU & free pages): 0.355
Max dirty pages percent: 90.000
Pending reads 0
Pending writes: LRU 0, flush list 0
Pages made young 2028651, not young 55580837
0.00 youngs/s, 0.55 non-youngs/s
Pages read 1194330, created 25139, written 78328
0.27 reads/s, 3.00 creates/s, 0.00 writes/s
Buffer pool hit rate 999 / 1000, young-making rate 0 / 1000 not 0 / 1000
Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s
LRU len: 193472, unzip_LRU len: 0
I/O sum[1443]:cur[0], unzip sum[0]:cur[0]
--------------
ROW OPERATIONS
--------------
0 read views open inside InnoDB
Process ID=0, Main thread ID=0, state: sleeping
Number of rows inserted 118319, updated 165195, deleted 169952, read 9369336877
3.64 inserts/s, 2.55 updates/s, 0.18 deletes/s, 172245.89 reads/s
Number of system rows inserted 0, updated 0, deleted 0, read 28810
0.00 inserts/s, 0.00 updates/s, 0.00 deletes/s, 0.00 reads/s
----------------------------
END OF INNODB MONITOR OUTPUT
============================
我想了解共享内存是如何在服务器上运行的不同进程之间使用的,并且得到的印象是“ipcs -m”命令可以用于此目的。所以当我尝试时,我得到了这个:
# ipcs -m
------ Shared Memory Segments --------
key shmid owner perms bytes nattch status
0x0052e2c1 327680 postgres 600 66084708352 27
这是在一个服务器上,除了 postgresql 之外,它还运行 tomcat、nginx 和其他一些程序。为什么输出中没有显示其他进程?