Poor performance on Dell PowerEdge 12g

Follow

Problem

Eucalyptus services are running slow. System is not processing as it should.

Environment

  • Dell PowerEdge 12g Server
  • Redhat(CentOS) 6

Identify the performance bottleneck by running:

dd if=/dev/zero of=/tmp/meh bs=1M count=10

on the "bad" system, you might be getting throughput something like 20-30MB/s, whereas healthy systems you should see ~ 1GB/s or more.

You can also check /var/log/messages for possible notifications such as:

kernel: CPU16: Temperature above threshold, cpu clock throttled

or:

kernel: CPU9: Core power limit notification (total events = 1)

Resolution

  1. Upgrading to the most recently released Redhat kernel (2.6.32-358.23.2.el6) disables this debugging and eliminates the performance problem.
  2. Adding the following kernel parameters to grub.conf will disable PLNs: clearcpuid=229
  3. Update System BIOS settings to match:
    • System Profile Settings
      • System Profile Performance Per Watt = Custom
      • CPU Power Management System = Maximum Performance
      • Memory Frequency = Maximum Performance
      • Turbo Boost = Disabled
      • C1E = Disabled
      • C States = Disabled
      • Monitor/Mwait = Disabled
      • Memory Patrol Scrub = Enabled
      • Memory Refresh Rate = 1x
  4. Hard Power cycle machine

Cause

Power limit in processor conflicts with kernel causing "Core power limit" and "Package power limit" errors in the dmesg log and slows the OS.

References

http://www.dell.com/support/troubleshooting/bb/en/bbdhs1/KCS/KcsArticles/ArticleView?s=dhs&docid=622824

http://serverfault.com/questions/551538/power-limit-notification-clobbering-on-12g-dell-servers-with-rhel6

http://i.dell.com/sites/content/shared-content/data-sheets/en/Documents/configuring-low-latency-environments-on-dell-poweredge-12g-servers.pdf

https://access.redhat.com/site/solutions/134973

 

 

Have more questions? Submit a request

Comments

Powered by Zendesk