MySQL缓慢使用内存,直到它开始使用交换


8

我正在运行1gb的RAM机架数据库服务器。由于某种原因,在大约2天的时间内,内存使用量从使用很少的交换空间变为使用100mb。如果我不重新启动sql,它将继续使用更多交换。(我的my.cnf文件如下所示,内存使用情况如下所示)

背景知识:我大约有50个活动数据库,这些数据库具有与使用INNODB的表相同的架构。我有几个使用MyISAM的流量很少的数据库。

在INNODB表上,我不使用持久连接。我还具有创建临时表的报告功能。(这可能会占用大量资源,但不会经常发生)

我正在使用CENTOS 6.3和mysql 5.5.28-log

即使我使用交换,性能仍然相当不错。我只是担心,如果我每隔几天不重新启动,就会遇到问题。

这是我大约两天的free -m日志:(第一条记录是在mysql重新启动之后)

12/26 2:08 PM EST
             total       used       free     shared    buffers     cached
Mem:           992        697        295          0         74        362
-/+ buffers/cache:        260        732
Swap:          976         15        961

12/26 4:10 PM EST
[root@php-pos-db ~]# free -m
             total       used       free     shared    buffers     cached
Mem:           992        791        201          0         97        405
-/+ buffers/cache:        287        705
Swap:          976         14        961

12/27 2:52 PM EST
[root@php-pos-db ~]# free -m
             total       used       free     shared    buffers     cached
Mem:           992        947         45          0         55        169
-/+ buffers/cache:        722        270
Swap:          976         34        942

12/28 1:41 PM EST
             total       used       free     shared    buffers     cached
Mem:           992        963         29          0         45        119
-/+ buffers/cache:        797        195
Swap:          976         48        927

12/28 7:24 PM EST
[root@php-pos-db ~]# free -m
             total       used       free     shared    buffers     cached
Mem:           992        957         35          0         41        141
-/+ buffers/cache:        774        218
Swap:          976         90        886

12/28 8:33 PM EST
[root@php-pos-db ~]# free -m
             total       used       free     shared    buffers     cached
Mem:           992        948         44          0         48        130
-/+ buffers/cache:        768        224
Swap:          976         96        880

my.cnf

# The MySQL database server configuration file.
#
# You can copy this to one of:
# - "/etc/mysql/my.cnf" to set global options,
# - "~/.my.cnf" to set user-specific options.
# 
# One can use all long options that the program supports.
# Run program with --help to get a list of available options and with
# --print-defaults to see which it would actually understand and use.
#
# For explanations see
# http://dev.mysql.com/doc/refman/5.1/en/server-system-variables.html
#
# Take care to only add/remove/change a setting if you are comfortable
# doing so! For Rackspace customers, if you have any questions or
# concerns, please contact the MySQL Database Services Team. Be aware
# that some work performed by this team can involve additional billable
# fees.
#
# This file generated for host php-pos-db please modify
# variables if the server is resized from 1016636kB

[mysqld]

### General
user                = mysql
port                = 3306
datadir                         = /var/lib/mysql
tmpdir                          = /tmp
socket                          = /var/lib/mysql/mysql.sock
skip-external-locking           = 1
log_error                       = /var/log/mysqld.log

## This prevents using host-based authentication. That means users must be
## created using an ip-address (ie 'myuser'@'192.168.100.1') or must make
## use of the % wildcard (ie 'myuser'@'%'). The benefit to not using
## host-based authentication is that DNS will not impact MySQL performance.
#skip-name-resolve

## If open-files-limit is set very low, MySQL may increase on its own. Either
## way, increase this if MySQL gives 'too many open files' errors. Setting
## this above 65535 could be unwise (MySQL may crash).
open-files-limit                = 20000

### Cache
thread-cache-size               = 16
table-open-cache                = 4096
table-definition-cache          = 512

## Generally, it is unwise to set the query cache to be larger than 64-128M 
## as the costs associated with maintaining the cache outweigh the performance
## gains. A far superior solution would be to implement memcached, though this
## required modifying the application, among other things.
query-cache-type                = 1
query-cache-size                = 32M
query-cache-limit               = 1M

### Per-thread Buffers
sort-buffer-size                = 1M
read-buffer-size                = 1M
read-rnd-buffer-size            = 2M
join-buffer-size                = 1M

### Temp Tables
tmp-table-size                  = 64M 
max-heap-table-size             = 64M

### Networking
back-log                        = 100
max-connections                 = 50
max-connect-errors              = 10000
max-allowed-packet              = 16M
interactive-timeout             = 600
wait-timeout                    = 180
net_read_timeout        = 30
net_write_timeout       = 30
# This value is the size of the listen queue for incoming TCP/IP connections.
back_log            = 128

#### Storage Engines
## Set this to force MySQL to use a particular engine / table-type
## for new tables. This setting can still be overridden by specifying
## the engine explicitly in the CREATE TABLE statement.
default-storage-engine         = InnoDB

## Makes sure MySQL does not start if InnoDB fails to start. This helps
## prevent ugly silent failures.
innodb                          = FORCE

### MyISAM
## Not sure what to set this to?
## Try running a 'du -sch /var/lib/mysql/*/*.MYI'
## This will give you a good estimate on the size of all the MyISAM indexes.
## (The buffer may not need to set that high, however)
key-buffer-size                 = 2M
## This setting controls the size of the buffer that is allocated when 
## sorting MyISAM indexes during a REPAIR TABLE or when creating indexes 
## with CREATE INDEX or ALTER TABLE.
myisam-sort-buffer-size         = 2M

### InnoDB
## Note: While most settings in MySQL can be set at run-time, many InnoDB
## variables cannot be set at runtime as require restarting MySQL
###
## These settings control how much RAM InnoDB will use. Generally, when using
## mostly InnoDB tables, the innodb-buffer-pool-size should be as large as
## is possible without swapping or starving other processes of RAM. The other 
## two settings usually do not need to be changed, but can help for very large 
## datasets.
innodb-buffer-pool-size         = 285M
innodb-log-buffer-size          = 8M

## Be careful when changing these as they require re-generating the 
## ib-logfile* files, which must be done carefully. Do not change this unless 
## you are familiar with the procedure.
innodb-log-file-size           = 128M
innodb-log-files-in-group      = 2

## This will cause each table to create its own .ibd file
innodb-file-per-table           = 1

## Setting this to 2 will decrease disk I/O but can cause up to a second of
## queries to be lost during a hard outage (i.e. power failures)
# innodb-flush-log-at-trx-commit = 2

### Replication
## Set this to the Server's instance ID in replication environments
server-id                       = 1

#log-bin                        = /var/lib/mysql/bin-log
#relay-log                      = /var/lib/mysql/relay-log
#relay-log-space-limit          = 4G
#expire-logs-days               = 5

## This should be enabled on conventional MySQL slaves
#read-only                      = 1

## This will cause replicated statements on a slave to be written to the slave's binlog
## Enable this on the middle slave of M->S->S configs
#log-slave-updates              = 1

#binlog-format                  = STATEMENT

### Logging
## This option determines the destination for general query log and slow query log output.
## The option value can be given as one or more of the words TABLE, FILE, or NONE.
## NOTE: Table logging takes away 50% of performance and thus is not recommended
##       http://bugs.mysql.com/bug.php?id=30414
## In addition, you cannot backup the contents of these tables properly
## (mysqldump skips these tables by default since they cannot be locked)
#log-output                     = FILE
slow-query-log                 = 1
slow-query-log-file            = /var/lib/mysql/slow-log
long-query-time                = 2
log-queries-not-using-indexes  = 1

[mysqld-safe]
log-error                       = /var/log/mysqld.log

[mysqldump]
max-allowed-packet      = 16M

# * IMPORTANT: Additional settings that can override those from this file!
#   The files must end with '.cnf', otherwise they'll be ignored.
#
!includedir /etc/sysconfig/mysqld-config/

Answers:


5

MySQL确实有交换愉快的讨厌习惯。杰里米·科尔(Jeremy Cole)在他的博客中对此做了最好的论述:http : //blog.jcole.us/2012/04/16/a-brief-update-on-numa-and-mysql/。从该博客中,您可以了解可以做的事情:numactl --interleave=all在/etc/init.d/mysql中添加。

建议

如果服务器专用于仅执行MySQL,请在/etc/my.cnf中更改以下内容:

[mysqld]
innodb_open_files=1000
innodb_flush_method=O_DIRECT
innodb_buffer_pool_size=768M
innodb_log_file_size=192M

如果服务器至少是双核,请添加这些

innodb_buffer_pool_instances=2
innodb_read_io_threads=16
innodb_write_io_threads=16
innodb_io_capacity=2000

接下来,登录到mysql运行 SET GLOBAL innodb_fast_shutdown = 0;

接下来,在操作系统中运行以下命令

cd /var/lib/mysql
service mysql stop
mv ib_logfile0 ib_logfile0.bak
mv ib_logfile1 ib_logfile1.bak
service mysql start

试试看 !!!

更新美国东部时间2012-12-31 08:30

根据您的最后评论

它停止在1 gb附近爬升。我删除了未使用的数据库,似乎mysql 5.5在内存中存储了很多数据,因为这在5.0中并未发生。mysql变化很大吗?

是的,MySQL发生了很大变化。实际上,在许多情况下,从MySQL 5.0升级到MySQL 5.5会导致性能下降。InnoDB 5.5现在具备执行超线程和多核参与的能力。

Percona不久前实际上对此进行了测试

请阅读我过去有关该主题的帖子

我也在ServerFault和StackOverflow中写过关于此的内容


我会尝试一下:innodb_buffer_pool_size和innodb-buffer-pool-size有什么区别
克里斯·

innodb_buffer_pool_size768M的内存可能会在只有1 GB RAM的计算机上突破极限。MySQL内核和用户空间中发生的一切仅剩256M,再加上InnoDB缓冲池之外MySQL中发生的一切……您确实需要将其设置为某些值,但老实说,我会寻找更多的内存也一样
James L

FWIW,NUMA在这里不应该成为一个因素:Rackspace上运行CentOS 6.3的1 GB机器将是一台仅提供一个NUMA节点的VM。
詹姆斯L

@James是虚拟机,所以您是对的。多核调整是不必要的,并且1GB上的75%RAM太低了。他至少需要4GB。
RolandoMySQLDBA 2012年

我已升级到2gb机架服务器,并且内存使用率仍在上升。我们对正在发生的事情感到困惑。这在mysql 5.0.96中没有发生
Chris Muench

0

除了Rolando提供的非常好的建议之外,您还可以在系统端使用sysctl激活无交换设置。我通常vm.swappiness=10/etc/sysctl.conf中的 MySQL计算机上进行设置。它提供了对交换的受限访问,但在需要时允许它。

vm.swappiness的默认值为60,这是非常允许的。


0

注意:我将此答案发布到关于stackoverflow的一个相关问题。该解决方案是特定于Linux和Systemd的,但是实际上,它可以适用于任何能够正确支持memlock调用并为不驻留根目录的进程提供支持的系统。

更新:实际上,此解决方案可能效果不佳。参见末尾的注释。

存在一类应用程序,您从不希望它们在其中进行交换。这样的类之一是数据库。数据库将使用内存作为其磁盘区域的缓存和缓冲区,并且绝对不要将它们交换。特定的存储器可能会保存一些相关数据,这些数据在一周内直到客户要求时才需要一周。如果不进行缓存/交换,数据库将只在磁盘上找到相关记录,这将非常快。但是通过交换,您的服务可能突然需要很长时间才能做出响应。

mysqld包括使用OS /系统调用的代码memlock。在Linux上,至少从2.6.9开始,此系统调用将对具有CAP_IPC_LOCK功能[1]的非root进程起作用。使用时memlock(),该过程必须仍在LimitMEMLOCK限制范围内工作。[2]。(很少)有一件好事systemd是,您mysqld无需特定的程序就可以向进程授予这些功能。如果也可以按照您的期望设置rlimits ulimit。这是一个执行必要步骤的override文件mysqld,其中包括数据库等过程可能需要的一些其他步骤:

[Service]
# Prevent mysql from swapping
CapabilityBoundingSet=CAP_IPC_LOCK

# Let mysqld lock all memory to core (don't swap)
LimitMEMLOCK=-1 

# do not kills this process if low on memory
OOMScoreAdjust=-900 

# Use higher io scheduling
IOSchedulingClass=realtime    

Type=simple    
ExecStart=
ExecStart=/usr/sbin/mysqld --memlock $MYSQLD_OPTS

注意:标准社区mysql当前随附于该选项Type=forking ,并将--daemonize该选项添加到该ExecStart行的服务中。这本质上比上述方法不稳定。

关于在systemd中覆盖文件:在中创建一个/etc/systemd/system/名为目录,并将mysqld.service.d新文件(包含上述内容)放入其中。

更新我对此解决方案不满意100%。经过几天的运行,我注意到该过程仍然有大量交换!检查时/proc/XXXX/smaps,我注意到以下几点:

  • 交换的最大贡献者是堆栈部分!最初,这似乎还不错,但是几天后,它达到了437 MB,并且在波动。这带来了明显的性能问题。它还表明基于堆栈的内存泄漏。
  • 零锁定页。这表明memlockMySQL(或Linux)中的选项已损坏。在这种情况下,这没什么大不了的,因为MySQL无法记忆锁堆栈。
By using our site, you acknowledge that you have read and understand our Cookie Policy and Privacy Policy.
Licensed under cc by-sa 3.0 with attribution required.