Instance will not launch despite enough disk space in instance type

Follow
Affected Versions: [All]

A specific image terminates upon run with no user level error indications.
 

Description


If an image size is less than the disk size and the sum of the image size plus swap (512MB) is larger than the selected instance type, then the instance may terminate. One solution is to use the instance type with the next large disk size or change the intended VM type to allocate more disk.


Details

An image will have a certain size before it is bundled and uploaded to Walrus. This image size is approximately the same amount of VM disk memory that will be need. Out of this same disk space, 512MB is reserved for Swap. The remaining disk space is available as ephemeral.

This example will be using a stock centos image:
 

[root@Pod-01 euca-large-centos-2012.1.14-x86_64]# ls -l                                                                                                             

total 4735008                                                                                                                                                       

-rw-r--r-- 1 1000 1000 4831838208 Jan 14 15:54 euca-large-centos-2012.1.14-x86_64.img                                                                               

-rw-r--r-- 1 root root    9635177 Nov 12 17:42 initrd.img-2.6.32-5-amd64                                                                                            

-rw-r--r-- 1 root root    2420384 Nov 12 17:42 vmlinuz-2.6.32-5-amd64  


Note: This image may not run under the type of C1.medium as 4832MB+500MB=5332MB is slightly greater than 5GB of the c1.medium. 

[root@Pod-01 euca-centos-5.3-x86_64]# euca-describe-availability-zones verbose
AVAILABILITYZONE        Lab-01  192.168.35.13 arn:euca:eucalyptus:Lab-01:cluster:CC-Pod03/
AVAILABILITYZONE        |- vm types     free / max   cpu   ram  disk
AVAILABILITYZONE        |- m1.small     0000 / 0010   1    128     2   
AVAILABILITYZONE        |- c1.medium    0000 / 0010   1    256     5
AVAILABILITYZONE        |- m1.large     0000 / 0005   2    512    10
AVAILABILITYZONE        |- m1.xlarge    0000 / 0002   2   1024    20
AVAILABILITYZONE        |- c1.xlarge    0000 / 0002   4   2048    20

[root@Pod-01 eucalyptus]# euca-describe-image-attribute -B emi-CD6E37FD

blockDeviceMapping      emi-CD6E37FD    blockDeviceMap  /dev/sda1: root

blockDeviceMapping      emi-CD6E37FD    blockDeviceMap  sda2: ephemeral0    (4.8GB in this example)

blockDeviceMapping      emi-CD6E37FD    blockDeviceMap  sda3: swap                 Fixed 512M

 

The following is an example of the cloud-output.log.  Note, no user level error messages are generated

[root@Pod-01 eucalyptus]# grep i-ABBD42A9 cloud-output.log | sort -t ":" -k2,4

 

                                                       | :1332872858382:ClusterAllocator:6d1a4206-1087-4c4e-aeb7-7b5950d84bd2:arn:aws:euare::234161552748:user/Neil:VM_PREPARE:[/euca:RunInstancesType] ownerFullName=arn:aws:euare:234161552748:user/Neil userData=null minCount=1 maxCount=1 partition=Partition:name=Lab-01:cc-cert-serial=1106128690948253:nc-cert-serial=1106144014787338 sshKeyPair=SshKeyPair:234161552748:neil-beard:fingerPrint=ba:db:52:53:25:a3:43:fc:52:dc:67:92:ab:a3:23:90:16:45:1d:c8 bootSet=BootableSet:machine=arn:aws:euca:eucalyptus:965590394582:image/emi-CD6E37FD/:ramdisk=arn:aws:euca:eucalyptus:000000000001:image/eri-94DB3AB9/:kernel=arn:aws:euca:eucalyptus:000000000001:image/eki-E6B13926/:isLinux=true vmType=VmType c1.medium cores=1 disk=5 mem=256 primaryNetwork=NetworkRulesGroup:234161552748:default:description=default group:networkRules=[] networkGroups=[default:NetworkRulesGroup:234161552748:default:description=default group:networkRules=[]] reservationId=r-098743D9 allocationTokens=[ResourceToken:i-ABBD42A9:192.168.56.2:tag=1715:idx=17] reservationIndex=291 instanceIds=[0:i-ABBD42A9] expiration=Wed Jan 01 00:00:00 PST 3000]

Tue Mar 27 11:27:38 2012 ERROR Allocations             | Aborting resource token: ResourceToken:i-ABBD42A9:192.168.56.2:tag=1715:idx=17

Tue Mar 27 11:27:38 2012 ERROR ClusterAllocator        | com.eucalyptus.entities.TransientEntityException: VmInstance:networkConfig=VmNetworkConfig:macAddress=d0:0d:AB:BD:42:A9:privateAddress=0.0.0.0:publicAddress=0.0.0.0:privateDnsName=euca-0-0-0-0.eucalyptus.internal:publicDnsName=euca-0-0-0-0.eucalyptus.ts-lab.eucalyptus-systems.com:vmId=VmId:reservationId=r-098743D9:instanceId=i-ABBD42A9:bootRecord=VmBootRecord:machineImage=arn:aws:euca:eucalyptus:965590394582:image/emi-CD6E37FD/:kernel=arn:aws:euca:eucalyptus:000000000001:image/eki-E6B13926/:ramdisk=arn:aws:euca:eucalyptus:000000000001:image/eri-94DB3AB9/:platform=linux:persistentVolumes=[]:sshKeyPair=neil-beard:vmType=VmType c1.medium cores=1 disk=5 mem=256:usageStats=VmUsageStats:blockBytes=0:networkBytes=0:launchRecord=VmLaunchRecord:launchIndex=0:launchTime=2012-03-27 11:27:38.0:runtimeState=VmRuntimeState:reason=User terminated.:reasonDetails=[]:consoleOutput=::transientVolumeState=VmVolumeState:attachments=[]:placement=VmPlacement:clusterName=CC-Pod01:partitionName=Lab-01:privateNetwork=false:networkGroups=[NetworkRulesGroup:234161552748:default:description=default group:networkRules=[]]:networkIndex=PrivateNetworkIndex:default tag=1715 idx=17

Tue Mar 27 11:27:38 2012  INFO TerminateInstancesType  | :1332872858640:TerminateInstancesType:VM_TERMINATING:[TerminateInstancesType instancesSet=[i-ABBD42A9] correlationId=1d7749da-f937-4128-8410-40a84670af8a userId=eucalyptus effectiveUserId=eucalyptus _return=true statusMessage=null _epoch=null _services=[] _disabledServices=[] _notreadyServices=[]]

Tue Mar 27 11:27:38 2012  INFO VmRuntimeState          | i-ABBD42A9 state change: PENDING -> SHUTTING_DOWN (previously PENDING)

Tue Mar 27 12:27:39 2012 ERROR VmInstances             | java.util.NoSuchElementException: example: [VmInstance networkConfig=null vmId=null bootRecord=null usageStats=null launchRecord=null runtimeState=null transientVolumeState=null placement=null expiration=null privateNetwork=null networkGroups=[] networkIndex=null ownerUserId=null ownerUserName=null ownerAccountNumber=null ownerAccountName=null uniqueName=null ownerFullNameCached=null state=null lastState=null stateChangeStack=null displayName=i-ABBD42A9 id=null version=null creationTimestamp=null lastUpdateTimestamp=null naturalId=null]

Tue Mar 27 11:27:39 2012  INFO TerminateInstancesType  | :1332872859731:TerminateInstancesType:VM_TERMINATING:[TerminateInstancesType instancesSet=[i-ABBD42A9] correlationId=d857d0cb-4fc3-4510-9956-3debc30cb677 userId=eucalyptus effectiveUserId=eucalyptus _return=true statusMessage=null _epoch=null _services=[] _disabledServices=[] _notreadyServices=[]]

Tue Mar 27 11:27:39 2012  INFO VmRuntimeState          | i-ABBD42A9 state change: SHUTTING_DOWN -> TERMINATED (previously PENDING)

 

Solution

In this example one solution is to use the instance type with the next large disk size or change the intended type to allocate more disk.

Have more questions? Submit a request

Comments

Powered by Zendesk