psql
PostgreSQL CLI 使用 readline 作为它的用户界面。Readline 允许使用 emacs- 和 vi 两种编辑模式,并且psql
默认使用 emacs 模式。另一方面,我将 vi 用于所有内容;是否有在 psql 中设置 vi 编辑模式的命令?
j4nd3r53n's questions
我有一个预配置的单主 RDS 数据库 (5.6.mysql_aurora.1.23.0),带有一个写入器和一个读取器。客户端是一堆码头应用程序,每个应用程序池最多有 250 个连接。
一段时间以来,我一直在与间歇性问题作斗争:连接尝试开始超时,主要是。它似乎与高负载相关,我已经开始information_schema.processlist
定期将整个复制到日志表(显然带有时间戳),当连接数达到 2000 时,问题似乎变得最明显。它永远不会接近max_connections
或max_user_connections
:
mysql> show variables like "%max%conn%";
+------------------------------+-------+
| Variable_name | Value |
+------------------------------+-------+
| aurora_max_connections_limit | 16000 |
| max_connect_errors | 100 |
| max_connections | 3000 |
| max_user_connections | 0 |
+------------------------------+-------+
4 rows in set (0.03 sec)
那么,为了解决这个问题,我应该关注哪些参数?我没有发现任何可能的东西,这解释了为什么 2000 个连接似乎会导致这种情况;当然,出现这样的整数可能只是巧合。
Debian 9 上的 Mysql 5.6:我在服务器 serverA 上有一个非常大的表,我已经使用 定义了从 serverB 上的数据库到 serverA 上的数据库的连接,create server connA ...
并且我在 serverB 上定义了一个表,该表通过 connA 连接到非常大的桌子。
当我在 serverB 上查询联合表时,如果查询在 serverA 上快速处理,它可以正常工作:
# on serverB:
mysql> select * from game_action where game_action_id=4;
+----------------+---------+---------+------------------+-------+--------+----------+---------------------+
| game_action_id | game_id | user_id | game_instance_id | type | amount | currency | created_timestamp |
+----------------+---------+---------+------------------+-------+--------+----------+---------------------+
| 4 | 1096 | 1 | 4 | WAGER | 1.00 | GBP | 2017-09-06 14:37:15 |
+----------------+---------+---------+------------------+-------+--------+----------+---------------------+
1 row in set (0.40 sec)
但是(仍在服务器B上):
mysql> select count(*) from game_action;
ERROR 2013 (HY000): Lost connection to MySQL server during query
表很大,所以处理需要很长时间并不奇怪,但我需要能够做需要很长时间的事情。过去我们遇到过由于超时而导致连接断开的问题,并且很难解决;在 my.cnf 中更改不同的超时没有帮助。
解决此特定问题的最佳做法是什么?
编辑
我觉得非常奇怪的是,似乎没有任何超时变量与我看到的相匹配——我设置了一个 mysql 实例只是为了测试它:
$ # On serverB:
$ time mysql -u root -pAtauseq01 gameiom -e "select count(*) from game_action;"
mysql: [Warning] Using a password on the command line interface can be insecure.
ERROR 2013 (HY000) at line 1: Lost connection to MySQL server during query
real 9m23.275s
user 0m0.014s
sys 0m0.004s
所以,如果是超时,大约 9 分钟超时。但是在远程服务器(serverA)上:
mysql> show variables like "%timeout%";
+-------------------------------------+----------+
| Variable_name | Value |
+-------------------------------------+----------+
| connect_timeout | 10 |
| delayed_insert_timeout | 300 |
| have_statement_timeout | YES |
| innodb_flush_log_at_timeout | 1 |
| innodb_lock_wait_timeout | 50 |
| innodb_print_lock_wait_timeout_info | OFF |
| innodb_rollback_on_timeout | OFF |
| interactive_timeout | 28800 |
| lock_wait_timeout | 31536000 |
| net_read_timeout | 30 |
| net_write_timeout | 60 |
| rpl_stop_slave_timeout | 31536000 |
| slave_net_timeout | 3600 |
| thread_pool_idle_timeout | 60 |
| wait_timeout | 28800 |
+-------------------------------------+----------+
15 rows in set (0.00 sec)
编辑 2
服务器A:
mysql> show variables like "flush%";
+---------------+-------+
| Variable_name | Value |
+---------------+-------+
| flush | OFF |
| flush_time | 0 |
+---------------+-------+
2 rows in set (0.00 sec)
服务器B:
mysql> show variables like "flush%";
+---------------+-------+
| Variable_name | Value |
+---------------+-------+
| flush | OFF |
| flush_time | 0 |
+---------------+-------+
2 rows in set (0.01 sec)
编辑 3
在服务器 A 上:
mysql> pager grep -v PARTITION
PAGER set to 'grep -v PARTITION'
mysql> show create table game_action\G
*************************** 1. row ***************************
Table: game_action
Create Table: CREATE TABLE `game_action` (
`game_action_id` bigint(20) unsigned NOT NULL AUTO_INCREMENT,
`game_id` int(11) NOT NULL,
`user_id` int(11) NOT NULL,
`game_instance_id` bigint(20) unsigned DEFAULT NULL,
`type` varchar(15) NOT NULL,
`amount` decimal(18,2) NOT NULL,
`currency` varchar(15) NOT NULL,
`created_timestamp` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP,
PRIMARY KEY (`game_action_id`,`created_timestamp`),
KEY `GA_IX01` (`game_id`),
KEY `GA_IX02` (`user_id`),
KEY `GA_IX03` (`game_instance_id`),
KEY `game_action_created_timestamp` (`created_timestamp`),
KEY `ga_id_cur_tstmp` (`game_id`,`currency`,`created_timestamp`)
) ENGINE=InnoDB AUTO_INCREMENT=1064199804 DEFAULT CHARSET=utf8
1 row in set (0.00 sec)
在服务器 B 上:
mysql> show create table game_action\G
*************************** 1. row ***************************
Table: game_action
Create Table: CREATE TABLE `game_action` (
`game_action_id` bigint unsigned NOT NULL AUTO_INCREMENT,
`game_id` int NOT NULL,
`user_id` int NOT NULL,
`game_instance_id` bigint unsigned DEFAULT NULL,
`type` varchar(15) NOT NULL,
`amount` decimal(18,2) NOT NULL,
`currency` varchar(15) NOT NULL,
`created_timestamp` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP,
PRIMARY KEY (`game_action_id`,`created_timestamp`),
KEY `GA_IX01` (`game_id`),
KEY `GA_IX02` (`user_id`),
KEY `GA_IX03` (`game_instance_id`),
KEY `game_action_created_timestamp` (`created_timestamp`),
KEY `ga_id_cur_tstmp` (`game_id`,`currency`,`created_timestamp`)
) ENGINE=FEDERATED DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_general_ci CONNECTION='gameiom'
1 row in set (0.00 sec)
MySQL 5.6:我正在尝试以不同的方式将数据从一个表复制到另一个表。源表位于远程服务器上,大约有 500,000 行 - 我使用联合引擎进行连接。我首先尝试了这个:
mysql > create table tgt as select * from src;
这非常快,只需要几秒钟,但它会发出警告:
...
| Warning | 1299 | Invalid TIMESTAMP value in column 'created_timestamp' at row 265975 |
| Warning | 1299 | Invalid TIMESTAMP value in column 'created_timestamp' at row 265976 |
...
64 rows in set (0.00 sec)
我试着用一个存储过程来做,打开一个游标,获取行并插入它们,但这需要很长时间;10分钟后我取消了。
那么,有没有办法找到导致问题的行?我试过select ... limit #first_row,#last_row;
了,但它似乎不起作用,我不确定它是否完全可靠。
我有一个配备硬件的 MySQL 服务器(5.6 版)(Linux,在 AWS 中,32GB RAM,56 个 CPU),根据 top 和 iotop,它甚至感觉不到温暖;然而这个查询需要大约 2 小时(实际查询,而不是解释):
mysql> explain
-> SELECT
-> DATE_FORMAT(DATE('2020-04-14'), '%m/%d/%Y') AS "Gaming Day",
-> g.name AS "Game name",
-> u.username AS "User Id",
-> ga.game_instance_id AS "Game Round Id",
-> gt.user_transaction_id AS "Transaction Id",
-> ga.type AS "Transaction Type",
-> ga.amount AS "Transaction Amount",
-> CONVERT_TZ(ga.created_timestamp, 'UTC', 'SYSTEM') AS "Transaction Date Time (EST)"
-> FROM spin.game_action ga
-> INNER JOIN spin.game_instance gi
-> ON gi.game_instance_id = ga.game_instance_id
-> INNER JOIN spin.game_transaction gt
-> ON gt.game_action_id = ga.game_action_id
-> INNER JOIN spin.user u
-> ON ga.user_id = u.user_id
-> INNER JOIN spin.organisation_site os
-> ON u.organisation_site_id = os.organisation_site_id
-> INNER JOIN spin.game g
-> ON g.game_id = ga.game_id
-> WHERE os.hostname = 'nyx'
-> AND gi.end_datetime BETWEEN CONVERT_TZ('2020-04-14 00:00:00', 'SYSTEM', 'UTC') AND CONVERT_TZ('2020-04-21 23:59:59', 'SYSTEM', 'UTC')
-> AND gi.status IN ('RESOLVED', 'AUTO_COMPLETED');
+----+-------------+-------+--------+---------------------------------+---------+---------+-----------------------------+------+-------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+-------+--------+---------------------------------+---------+---------+-----------------------------+------+-------------+
| 1 | SIMPLE | g | index | PRIMARY | BG_UK1 | 202 | NULL | 60 | Using index |
| 1 | SIMPLE | ga | ref | PRIMARY,GA_IX01,GA_IX02,GA_IX03 | GA_IX01 | 4 | spin.g.game_id | 674 | Using where |
| 1 | SIMPLE | u | eq_ref | PRIMARY,U_UK01,U_IX_04 | PRIMARY | 4 | spin.ga.user_id | 1 | NULL |
| 1 | SIMPLE | os | eq_ref | PRIMARY | PRIMARY | 4 | spin.u.organisation_site_id | 1 | Using where |
| 1 | SIMPLE | gi | ref | PRIMARY | PRIMARY | 8 | spin.ga.game_instance_id | 1 | Using where |
| 1 | SIMPLE | gt | ref | GT_IX03 | GT_IX03 | 9 | spin.ga.game_action_id | 1 | Using index |
+----+-------------+-------+--------+---------------------------------+---------+---------+-----------------------------+------+-------------+
6 rows in set (0.01 sec)
我尝试在分析下运行它,但这基本上没用:
mysql> show profile for query 2;
+----------------------+------------+
| Status | Duration |
+----------------------+------------+
| starting | 0.000160 |
| checking permissions | 0.000005 |
| checking permissions | 0.000002 |
| checking permissions | 0.000003 |
| checking permissions | 0.000003 |
| checking permissions | 0.000003 |
| checking permissions | 0.000005 |
| Opening tables | 0.000067 |
| init | 0.000133 |
| System lock | 0.000201 |
| optimizing | 0.000049 |
| statistics | 0.000416 |
| preparing | 0.000050 |
| executing | 0.000005 |
| Sending data | 999.999999 |
| end | 0.000010 |
| query end | 0.000008 |
| closing tables | 0.010543 |
| freeing items | 0.000062 |
| logging slow query | 0.000002 |
| logging slow query | 0.110968 |
| cleaning up | 0.003693 |
+----------------------+------------+
22 rows in set, 1 warning (0.01 sec)
据我了解,Sending data
涵盖了将结果实际传输给客户端和处理的主要部分。我现在正在考虑如何使用 performance_schema 进一步分析它;它可能会透露更多有用的细节吗?
编辑
输出SHOW GLOBAL STATUS\G
:https ://pastebin.com/QejRk9RA
输出SHOW GLOBAL VARIABLES\G
:https ://pastebin.com/b3B76v21
输出SHOW FULL PROCESSLIST\G
:https ://pastebin.com/N85YUwFj
MySQLtuner 报告:https ://pastebin.com/HJJsCCzL
编辑 2
$ ulimit -a
time(seconds) unlimited
file(blocks) unlimited
data(kbytes) unlimited
stack(kbytes) 8192
coredump(blocks) 0
memory(kbytes) unlimited
locked memory(kbytes) 64
process 128223
nofiles 1024
vmemory(kbytes) unlimited
locks unlimited
rtprio 0
db3 root = iostat -xm 5 3
Linux 4.9.0-11-amd64 (db3.spin-production.gamingrealms.org) 05/27/2020 _x86_64_ (56 CPU)
avg-cpu: %user %nice %system %iowait %steal %idle
0.72 0.00 0.07 0.04 0.00 99.17
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.11 11.30 20.83 17.03 3.66 0.66 233.37 0.05 1.24 1.41 1.04 0.51 1.94
dm-0 0.00 0.00 0.04 0.04 0.00 0.00 8.13 0.01 99.38 3.62 184.79 1.05 0.01
dm-1 0.00 0.00 20.88 27.29 3.66 0.66 183.42 0.05 1.05 1.42 0.76 0.40 1.94
avg-cpu: %user %nice %system %iowait %steal %idle
3.60 0.00 0.16 0.08 0.00 96.16
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 18.60 121.40 13.80 30.00 0.27 458.59 0.13 0.93 0.92 1.10 0.49 6.64
dm-0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-1 0.00 0.00 121.40 32.40 30.00 0.27 403.13 0.13 0.82 0.91 0.47 0.43 6.64
avg-cpu: %user %nice %system %iowait %steal %idle
1.59 0.00 0.16 0.03 0.00 98.23
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 15.00 49.00 17.60 12.02 0.21 375.95 0.04 0.64 0.87 0.00 0.37 2.48
dm-0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-1 0.00 0.00 49.00 32.60 12.02 0.21 306.84 0.04 0.52 0.87 0.00 0.30 2.48
db3 root = top -c
top - 04:36:25 up 167 days, 21:28, 1 user, load average: 1.97, 2.28, 2.18
Tasks: 584 total, 2 running, 582 sleeping, 0 stopped, 0 zombie
%Cpu(s): 3.4 us, 0.2 sy, 0.0 ni, 96.4 id, 0.1 wa, 0.0 hi, 0.0 si, 0.0 st
KiB Mem : 32848684 total, 5421432 free, 19970372 used, 7456880 buff/cache
KiB Swap: 66916348 total, 66844172 free, 72176 used. 12286984 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
43108 root 20 0 4440 1312 1044 R 89.2 0.0 243:45.28 /bin/gzip -
18933 root 20 0 4440 1332 1048 S 61.0 0.0 107:19.61 /bin/gzip -
43107 root 20 0 249232 40072 11788 S 9.8 0.1 24:21.13 /usr/bin/innobackupex --stream=tar --user=backupuser --password=x xxxxxx --safe-slave-backup --slave-info --databa+
18932 root 20 0 249232 40024 11740 S 7.2 0.1 10:17.08 /usr/bin/innobackupex --stream=tar --user=backupuser --password=x xxxxxx --safe-slave-backup --slave-info --databa+
18934 root 20 0 1058508 147260 9648 S 4.3 0.4 6:46.18 /usr/bin/python3 /usr/bin/aws s3 cp - s3://spin.db.backup/2020-05-27-spin.tar.gz --region us-east-1
43109 root 20 0 1069768 174488 9404 S 3.6 0.5 14:16.26 /usr/bin/python3 /usr/bin/aws s3 cp - s3://spin.db.backup/2020-05-26-spin.tar.gz --region us-east-1
54915 root 20 0 45472 4396 3240 R 1.6 0.0 0:00.29 top -c
39686 mysql 20 0 21.178g 0.018t 8980 S 1.3 58.2 2979:59 /usr/sbin/mysqld --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/lib/mysql/plugin --user=mysql --log-err+
41601 root 20 0 33196 11696 5100 S 0.7 0.0 70:42.86 /usr/share/filebeat/bin/filebeat -c /etc/filebeat/filebeat.yml -path.home /usr/share/filebeat -path.config /etc/fi+
312 root 20 0 0 0 0 S 0.3 0.0 0:36.25 [ksoftirqd/50]
1661 sensu 20 0 1594004 28004 5676 S 0.3 0.1 127:00.25 /opt/sensu/embedded/bin/ruby /opt/sensu/bin/sensu-client -c /etc/sensu/config.json -d /etc/sensu/conf.d -e /etc/se+
11120 root 20 0 109512 684 660 S 0.3 0.0 132:39.67 /var/ossec/bin/wazuh-modulesd
24552 sensu 20 0 135144 24696 0 S 0.3 0.1 217:07.55 /usr/sbin/sensu-agent start
1 root 20 0 57396 5144 3668 S 0.0 0.0 11:17.41 /sbin/init
...
当服务器似乎没有负载时(根据 top 和 show processlist)以及一些繁重的 SQL 作业将其推高到大约 1000% CPU 时,我都运行了这个查询,并且它似乎没有太大的区别,相信它或不是。〜2小时是什么都没有发生的事情。
编辑3
显示全球状态:https ://pastebin.com/5PeBEkz7
显示全局变量:https ://pastebin.com/SnGS28rD
显示完整的进程列表:https ://pastebin.com/AR2WZbnM
编辑4
db3 root = hdparm -I /dev/sda
/dev/sda:
SG_IO: bad/missing sense data, sb[]: 70 00 05 00 00 00 00 0d 00 00 00 00 20 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
ATA device, with non-removable media
Standards:
Likely used: 1
Configuration:
Logical max current
cylinders 0 0
heads 0 0
sectors/track 0 0
--
Logical/Physical Sector size: 512 bytes
device size with M = 1024*1024: 0 MBytes
device size with M = 1000*1000: 0 MBytes
cache/buffer size = unknown
Capabilities:
IORDY not likely
Cannot perform double-word IO
R/W multiple sector transfer: not supported
DMA: not supported
PIO: pio0
编辑
显示创建表并显示表状态:https ://pastebin.com/s9Y61GTb
我有一个我确定不是闻所未闻的设置(这是 MySQL 5.5,但正在升级):
我有一个快速增长的 OLTP 数据库,我正在将其复制到 OLAP 数据库。我想开始归档 OLTP 数据库中增长最快的表,但我希望 OLAP 数据库在没有归档部分的情况下积累所有内容。有没有一种简单的方法可以实现这一点,或者我已经发明了自己的程序?
我最近将 mysqld(percona 服务器)从 5.5.60-38.12 升级到 5.5.62-38.14。这是在 Debian 上,升级是这样完成的:
# apt-get update
# apt-get upgrade
我惊讶地发现内部报告的版本与我在命令行上得到的不同:
db3 root = mysql -V
mysql Ver 14.14 Distrib 5.5.62-38.14, for debian-linux-gnu (x86_64) using readline 5.1
mysql> select version();
+------------------+
| version() |
+------------------+
| 5.5.60-38.12-log |
+------------------+
1 row in set (0.00 sec)
这是正常的吗?或者我需要做些什么来解决这个问题?
编辑:我也检查了已安装的软件包:
db3 root = dpkg -l | grep percona
ii libperconaserverclient18 5.5.62-rel38.14-1.stretch amd64 Percona Server database client library
ii percona-server-client-5.5 5.5.62-rel38.14-1.stretch amd64 Percona Server database client binaries
ii percona-server-common-5.5 5.5.62-rel38.14-1.stretch amd64 Percona Server database common files
ii percona-server-server-5.5 5.5.62-rel38.14-1.stretch amd64 Percona Server database server binaries
ii percona-toolkit 3.1.0-2.stretch amd64 Advanced MySQL and system command-line tools
ii percona-xtrabackup 2.3.10-1.stretch amd64 Open source backup tool for InnoDB and XtraDB