iSCSI Tuning for Node Controllers

Follow

Eucalyptus Versions:  3.0 - Current

Introduction

Eucalyptus uses iSCSI to provide EBS volumes from a Storage Controller (SC) or supported array to a Node Controller (NC) and its instances.  For those using a large amount of EBS in their environment and particular EBS-backed instances, it can be a useful exercise to tune the default iscsid.conf parameters on Node Controllers. Tweaking particular values can attempt to mitigate issues caused by network outages or similar disruptive events.

Scenario

You may have a scenario where you have a node with EBS-backed instances running or multiple EBS volumes attached and in-use with instances.  You experience a network outage lasting for approximately 30 seconds.  The default iscsid.conf timeouts will cause the iSCSI layer to begin timing out your connections from the NC (and instances) to the targets aggressively. As the outage continues, the SCSI layer will begin to fail I/O to these devices.  You may end up in a scenario where the EBS-backed instances are unrecoverable and EBS volumes need to be re-attached.

Tunables

There are a number of values in the iscsid.conf file which you should consider changing:

timeo.noop_out_interval

This value controls the "ping" between the initiator and the target.  This ping is used to detect problems in the network and acts as a periodic check.  You may wish to set this value higher (less aggressive), disabling is also an option. If you do disable this check, note that failures in the network will not be detected by the iSCSI layer. 

timeo.noop_out_interval

This next value is the timeout value with the associated "ping".  If the ping fails, this timeout controls how long to wait before failure at the iSCSI layer. You should set this value to be higher than the default, or consider disabling if the above value is also disabled.

node.session.timeo.replacement_timeout

This value will set the connection timeout for session re-establishment.  If the timeouts above at the iSCSI layer are hit (depending on your settings), then this is the point at which pending SCSI commands are failed.  You should set this value higher than the default considering your failure scenario, perhaps 5 mins (300) or even to a negative value, which then means I/O is queued indefinitely until the session is re-established (which would be a manual action).

Have more questions? Submit a request

Comments

Powered by Zendesk