Solving 100% CPU usage of Chef beam.smp (RabbitMQ)

Search for chef 100% cpu issue and you will find a lot of sugestions ranging from reboot the server, to restart RabbitMQ and often to check the kernel max file limit.

All of those do not help! What does help is checking RabbitMQ with

rabbitmqctl report | grep -A3 file_descriptors

and have a look at the printed limits and usage. Here is an example:

 {file_descriptors,[{total_limit,8900},
                    {total_used,1028},
                    {sockets_limit,8008},
                    {sockets_used,2}]},

In my case the 100% CPU usage was caused by all of the file handles being used up which for some reason causes RabbitMQ 2.8.4 to go into a crazy endless loop rarely responding at all.

The "total_limit" value is the "nofile" limit for the maximum number of open files you can check using "ulimit -n" as RabbitMQ user. Increase it permanently by defining a RabbitMQ specific limit for example in /etc/security/limits.d/rabbitmq.conf:

rabbitmq    soft   nofile   10000

or using for example

ulimit -n 10000

from the start script or login scripts. Then restart RabbitMQ. The CPU usage should be gone.

Update: This problem only affects RabbitMQ releases up to 1.8.4 and should be fixed starting with 1.8.5.

Comments

RE: rabbit spinning when running out of fds

I'm running Ubuntus rabbitmq-server package version 2.8.4-1. I found similar reports in the Chef ticket system for example http://tickets.opscode.com/browse/CHEF-3256 which also points at this rabbitmq-general post: http://comments.gmane.org/gmane.comp.networking.rabbitmq.general/17120.

I'll post my experience on rabbitmq-discuss!

spinning rabbit -> upgrade

Ah. That issue got fixed in RabbitMQ 2.8.5. -> upgrade (ideally to the latest version)

RE: spinning rabbit -> upgrade

Thanks for the hint! I've updated the post to reflect this. I'll retest as soon Ubuntu updates to 2.8.5.

Post new comment

The content of this field is kept private and will not be shown publicly.
  • Web page addresses and e-mail addresses turn into links automatically.
  • Allowed HTML tags: <a> <em> <strong> <cite> <code> <ul> <ol> <li> <dl> <dt> <dd>
  • Lines and paragraphs break automatically.

More information about formatting options

To prevent automated spam submissions leave this field empty.
Syndicate content