Migration is broken in CentOS/RHEL 6.5

Follow

Problem

Cannot migrate instances on NCs after OS upgrade to CentOS/RHEL 6.5.

 

Description

System has been updated to latest OS. This includes an upgrade to the qemu-kvm and qemu-img packages:

qemu-img.x86_64 2:0.12.1.2-2.415.el6

qemu-kvm.x86_64 2:0.12.1.2-2.415.el6

 

This throws errors like the following:

nc.log on source machine:

2014-03-11 10:12:54 ERROR | libvirt: operation failed: migration job: unexpectedly failed (code=9)

libvirtd.log on source machine:

2014-03-11 17:12:54.667+0000: 31610: error : qemuMigrationUpdateJobStatus:1042 : operation failed: migration job: unexpectedly failed

nc.log on target machine:

2014-03-11 10:12:54 ERROR | libvirt: operation failed: migration job: unexpectedly failed (code=9)
2014-03-11 10:12:54 ERROR | [i-19DB4227] cannot migrate instance, giving up and rolling back.

i-19DB4227.log on target machine's NC, under /var/log/libvirt/qemu:

2014-03-11 17:12:02.349+0000: starting up
LC_ALL=C PATH=/sbin:/usr/sbin:/bin:/usr/bin QEMU_AUDIO_DRV=none /usr/libexec/qemu-kvm -name i-19DB4227 -S -M rhel6.4.0 -enable-kvm -m 256 -realtime mlock=off -smp 1,sockets=1,cores=1,threads=1 -uuid 2ac7c4a5-6a27-b05f-2b7d-82b9616027a1 -nodefconfig -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/i-19DB4227.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=localtime -no-shutdown -kernel /var/lib/eucalyptus/instances/work/AIDNXPPQOWWSWRMQ5L6JK/i-19DB4227/link-to-kernel -initrd /var/lib/eucalyptus/instances/work/AIDNXPPQOWWSWRMQ5L6JK/i-19DB4227/link-to-ramdisk -append root=/dev/vda1 console=ttyS0 -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive file=/var/lib/eucalyptus/instances/work/AIDNXPPQOWWSWRMQ5L6JK/i-19DB4227/link-to-sda,if=none,id=drive-virtio-disk0,format=raw,cache=none -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -netdev tap,fd=35,id=hostnet0,vhost=on,vhostfd=36 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=d0:0d:19:db:42:27,bus=pci.0,addr=0x3 -chardev file,id=charserial0,path=/var/lib/eucalyptus/instances/work/AIDNXPPQOWWSWRMQ5L6JK/i-19DB4227/console.log -device isa-serial,chardev=charserial0,id=serial0 -vnc 0.0.0.0:2 -k en-us -vga cirrus -incoming tcp:[::]:49152 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5
Receiving block device images
Completed 0 %^MCompleted 1 %^MCompleted 2 %^MCompleted 3 %^MCompleted 4 %^MCompleted 5 %^MCompleted 6 %^MCompleted 7 %^MCompleted 8 %^MCompleted 9 %^MCompleted 10 %^MCompleted 11 %^MCompleted 12 %^MCompleted 13 %^MCompleted 14 %^MCompleted 15 %^MCompleted 16 %^MCompleted 17 %^MCompleted 18 %^MCompleted 19 %^MCompleted 20 %^MCompleted 21 %^MCompleted 22 %^MCompleted 23 %^MCompleted 24 %^MCompleted 25 %^MCompleted 26 %^MCompleted 27 %^MCompleted 28 %^MCompleted 29 %^MCompleted 30 %^MCompleted 31 %^MCompleted 32 %^MCompleted 33 %^MCompleted 34 %^MCompleted 35 %^MCompleted 36 %^MCompleted 37 %^MCompleted 38 %^MCompleted 39 %^MCompleted 40 %^MCompleted 41 %^MCompleted 42 %^MCompleted 43 %^MCompleted 44 %^MCompleted 45 %^MCompleted 46 %^MCompleted 47 %^MCompleted 48 %^MCompleted 49 %^MCompleted 50 %^MCompleted 51 %^MCompleted 52 %^MCompleted 53 %^MCompleted 54 %^MCompleted 55 %^MCompleted 56 %^MCompleted 57 %^MCompleted 58 %^MCompleted 59 %^MCompleted 60 %^MCompleted 61 %^MCompleted 62 %^MCompleted 63 %^MCompleted 64 %^MCompleted 65 %^MCompleted 66 %^MCompleted 67 %^MCompleted 68 %^MCompleted 69 %^MCompleted 70 %^MCompleted 71 %^MCompleted 72 %^MCompleted 73 %^MCompleted 74 %^MCompleted 75 %^MCompleted 76 %^MCompleted 77 %^MCompleted 78 %^MCompleted 79 %^MCompleted 80 %^MCompleted 81 %^MCompleted 82 %^MCompleted 83 %^MCompleted 84 %^MCompleted 85 %^MCompleted 86 %^MCompleted 87 %^MCompleted 88 %^MCompleted 89 %^MCompleted 90 %^MCompleted 91 %^MCompleted 92 %^MCompleted 93 %^MCompleted 94 %^MCompleted 95 %^MCompleted 96 %^MCompleted 97 %^MCompleted 98 %^MCompleted 99 %^Mqemu: warning: error while loading state section id 1
load of migration failed
2014-03-11 17:12:54.619+0000: shutting down

Workaround

yum downgrade http://vault.centos.org/6.4/os/x86_64/Packages/qemu-kvm-0.12.1.2-2.355.el6.x86_64.rpm http://vault.centos.org/6.4/os/x86_64/Packages/qemu-img-0.12.1.2-2.355.el6.x86_64.rpm

service libvirtd restart

This brings the libraries that run migration back to a working state and migration can happen once more.

Have more questions? Submit a request

Comments

Powered by Zendesk