"Too many open files" errors on Node Controllers

Follow

Eucalyptus Version:  All

"Too many open files" errors on Node Controllers (NC)

Problem

If the following is present:

  • Great amount of instance-related churn on a given Node Controller (NC)
  • MAX_CORES is set to overcommit the Node Controllers (NC)

the following error could be seen in the nc.log on the Node Controller (NC):

2014-02-20 16:05:36 FATAL | hooks prevented check on the hypervisor 
2014-02-20 16:05:36 ERROR | error: failed to create pipe for stdout: Too many open files
2014-02-20 16:05:36 WARN | invocation 'blkid ' failed with -1 (attempt 1)
2014-02-20 16:05:36 ERROR | error: failed to create pipe for stdout: Too many open files
2014-02-20 16:05:36 WARN | invocation 'blkid ' failed with -1 (attempt 2)
2014-02-20 16:05:36 ERROR | error: failed to create pipe for stdout: Too many open files

Resolution

This issue occurs if the ulimit [1] is not set high enough for the Node Controller(s).  To resolve this issue, do the following:

1.  Check the ulimit values for the 'eucalyptus' user on the given Node Controller:

# su - eucalyptus -c "ulimit -Hn; ulimit -Sn"
4096 (Hard Limit)
1024 (Soft Limit)

2.  Check the maximum number of file descriptors allowed on the system:

# cat /proc/sys/fs/file-max 
1208876

3.  Add an entry to /etc/security/limits.conf [2] for the 'eucalyptus' user to increase the hard and soft limits of the user.  The values of hard and soft limits will be based on the churn on the given Node Controller.  For example:

eucalyptus soft nofile 4096
eucalyptus hard nofile 10240

4. Confirm the settings have been updated for the 'eucalyptus' user:

# su - eucalyptus -c "ulimit -Hn; ulimit -Sn"
10240
4096

References

[1]   About.com - Linux - man ulimit
[2]   die.net - limits.conf - Linux man page

 

Have more questions? Submit a request

Comments

Powered by Zendesk