使用RHEL6的12G戴尔服务器上的“功率限制通知”问题

服务器：Poweredge r620
操作系统：RHEL 6.4
内核：2.6.32-358.18.1.el6.x86_64

我在生产环境中遇到应用程序警报。严重的CPU饥饿进程正在耗尽资源，并导致处理积压。在最近部署的群集中的所有第12代Dell服务器（r620s）上都发生了此问题。据我所知，这种情况的发生与CPU利用率峰值相匹配，并伴随着大量的“功率限制通知”垃圾邮件dmesg。以下事件之一的摘录：

Nov  7 10:15:15 someserver [.crit] CPU12: Core power limit notification (total events = 14)
Nov  7 10:15:15 someserver [.crit] CPU0: Core power limit notification (total events = 14)
Nov  7 10:15:15 someserver [.crit] CPU6: Core power limit notification (total events = 14)
Nov  7 10:15:15 someserver [.crit] CPU14: Core power limit notification (total events = 14)
Nov  7 10:15:15 someserver [.crit] CPU18: Core power limit notification (total events = 14)
Nov  7 10:15:15 someserver [.crit] CPU2: Core power limit notification (total events = 14)
Nov  7 10:15:15 someserver [.crit] CPU4: Core power limit notification (total events = 14)
Nov  7 10:15:15 someserver [.crit] CPU16: Core power limit notification (total events = 14)
Nov  7 10:15:15 someserver [.crit] CPU0: Package power limit notification (total events = 11)
Nov  7 10:15:15 someserver [.crit] CPU6: Package power limit notification (total events = 13)
Nov  7 10:15:15 someserver [.crit] CPU14: Package power limit notification (total events = 14)
Nov  7 10:15:15 someserver [.crit] CPU18: Package power limit notification (total events = 14)
Nov  7 10:15:15 someserver [.crit] CPU20: Core power limit notification (total events = 14)
Nov  7 10:15:15 someserver [.crit] CPU8: Core power limit notification (total events = 14)
Nov  7 10:15:15 someserver [.crit] CPU2: Package power limit notification (total events = 12)
Nov  7 10:15:15 someserver [.crit] CPU10: Core power limit notification (total events = 14)
Nov  7 10:15:15 someserver [.crit] CPU22: Core power limit notification (total events = 14)
Nov  7 10:15:15 someserver [.crit] CPU4: Package power limit notification (total events = 14)
Nov  7 10:15:15 someserver [.crit] CPU16: Package power limit notification (total events = 13)
Nov  7 10:15:15 someserver [.crit] CPU20: Package power limit notification (total events = 14)
Nov  7 10:15:15 someserver [.crit] CPU8: Package power limit notification (total events = 14)
Nov  7 10:15:15 someserver [.crit] CPU10: Package power limit notification (total events = 14)
Nov  7 10:15:15 someserver [.crit] CPU22: Package power limit notification (total events = 14)
Nov  7 10:15:15 someserver [.crit] CPU15: Core power limit notification (total events = 369)
Nov  7 10:15:15 someserver [.crit] CPU3: Core power limit notification (total events = 369)
Nov  7 10:15:15 someserver [.crit] CPU1: Core power limit notification (total events = 369)
Nov  7 10:15:15 someserver [.crit] CPU5: Core power limit notification (total events = 369)
Nov  7 10:15:15 someserver [.crit] CPU17: Core power limit notification (total events = 369)
Nov  7 10:15:15 someserver [.crit] CPU13: Core power limit notification (total events = 369)
Nov  7 10:15:15 someserver [.crit] CPU15: Package power limit notification (total events = 375)
Nov  7 10:15:15 someserver [.crit] CPU3: Package power limit notification (total events = 374)
Nov  7 10:15:15 someserver [.crit] CPU1: Package power limit notification (total events = 376)
Nov  7 10:15:15 someserver [.crit] CPU5: Package power limit notification (total events = 376)
Nov  7 10:15:15 someserver [.crit] CPU7: Core power limit notification (total events = 369)
Nov  7 10:15:15 someserver [.crit] CPU19: Core power limit notification (total events = 369)
Nov  7 10:15:15 someserver [.crit] CPU17: Package power limit notification (total events = 377)
Nov  7 10:15:15 someserver [.crit] CPU9: Core power limit notification (total events = 369)
Nov  7 10:15:15 someserver [.crit] CPU21: Core power limit notification (total events = 369)
Nov  7 10:15:15 someserver [.crit] CPU23: Core power limit notification (total events = 369)
Nov  7 10:15:15 someserver [.crit] CPU11: Core power limit notification (total events = 369)
Nov  7 10:15:15 someserver [.crit] CPU13: Package power limit notification (total events = 376)
Nov  7 10:15:15 someserver [.crit] CPU7: Package power limit notification (total events = 375)
Nov  7 10:15:15 someserver [.crit] CPU19: Package power limit notification (total events = 375)
Nov  7 10:15:15 someserver [.crit] CPU9: Package power limit notification (total events = 374)
Nov  7 10:15:15 someserver [.crit] CPU21: Package power limit notification (total events = 375)
Nov  7 10:15:15 someserver [.crit] CPU23: Package power limit notification (total events = 374)

一点Google Fu揭示出这通常与CPU运行过热或电压调节启动有关。我不认为这是正在发生的事情。群集中所有服务器的温度传感器运行良好，iDRAC中的功率上限策略已禁用，并且所有这些服务器上的“系统配置文件”均设置为“性能”：

# omreport chassis biossetup | grep -A10 'System Profile'
System Profile Settings
------------------------------------------
System Profile                                    : Performance
CPU Power Management                              : Maximum Performance
Memory Frequency                                  : Maximum Performance
Turbo Boost                                       : Enabled
C1E                                               : Disabled
C States                                          : Disabled
Monitor/Mwait                                     : Enabled
Memory Patrol Scrub                               : Standard
Memory Refresh Rate                               : 1x
Memory Operating Voltage                          : Auto
Collaborative CPU Performance Control             : Disabled

• 戴尔邮件列表中的帖子几乎完美地描述了症状。戴尔建议作者尝试使用性能配置文件，但这无济于事。他最终在Dell指南中应用了一些设置，以针对低延迟环境配置服务器，这些设置之一（或其组合）似乎已解决了该问题。
• Kernel.org错误＃36182指出，默认情况下已启用功率限制中断调试，这在启动CPU电压调节的情况下会导致性能下降。
• RHN KB文章（需要RHN登录）提到了影响未运行性能配置文件的PE r620和r720服务器的问题，并建议对两周前发布的内核进行更新。...除了我们正在运行的性能配置文件...

我在网上可以找到的所有内容都让我在这里盘旋。这到底是怎么回事？

导致性能问题的不是电压调节，而是由它触发的调试内核中断。

尽管Redhat方面存在一些错误信息，但所有链接页面都引用相同的现象。在启用或不启用性能配置文件的情况下，电压调节都会发生，这可能是由于启用了Turbo Boost功能。无论出于何种原因，这些电压波动与内核2.6.32-358.18.1.el6.x86_64中默认启用的功率限制内核中断的交互作用都很差。

已确认的解决方法：

• 升级到最新发布的Redhat内核（2.6.32-358.23.2.el6）可禁用此调试并消除性能问题。
• 添加以下内核参数grub.conf将禁用PLN：clearcpuid=229

不稳定的解决方法：

• 设置“性能”的系统配置文件。仅凭其本身不足以禁用我们服务器上的PLN。你的旅费可能会改变。

错误的解决方法：

• 将与ACPI相关的模块列入黑名单。我已经在一些论坛主题中看到了这一点。病态告知，所以不要。