Live migration moves a running VM from one host to another, and the guest notices at most a short pause. In QEMU/KVM the whole job comes down to one problem: the guest keeps writing to memory while you copy it. This post walks through how QEMU handles that, why some VMs never finish migrating, and which libvirt options change the outcome.
What actually moves#
A QEMU VM has three kinds of state:
- Device state: vCPU registers, interrupt controllers, virtio queues, timers. It is small and goes over in the final phase.
- Guest RAM: large and constantly changing. Almost all of the migration’s time and design goes into this.
- Disks: by default, nothing moves. libvirt expects any image that isn’t network-accessed to be shared between the hosts through a network filesystem or remote block storage, at the same path unless you supply new XML. Local disks have to be copied explicitly (see below).
The destination host starts a second QEMU process with the same virtual hardware, in an “incoming” state, and the source streams state into it.
Pre-copy: copy, then copy what changed#
The default algorithm is pre-copy. The guest keeps running on the source while its memory is copied:
- Turn on dirty-page logging and send every page of guest RAM.
- While that pass ran, the guest dirtied some pages. Send those.
- Repeat. Each pass sends whatever was dirtied during the previous one.
- When what’s left is small enough to send within the allowed downtime, pause the vCPUs, send the remaining dirty pages and the device state, and resume on the destination.
Dirty tracking comes from KVM. QEMU sets KVM_MEM_LOG_DIRTY_PAGES on a guest memory slot so that KVM records writes to it, then reads a dirty bitmap per slot with KVM_GET_DIRTY_LOG. Newer kernels also offer a per-vCPU dirty ring (KVM_CAP_DIRTY_LOG_RING). Instead of fetching and scanning a whole bitmap, userspace harvests a per-vCPU ring of dirtied guest frame numbers. QEMU enables the ring with the KVM accelerator property dirty-ring-size.
Step 4 is the convergence check. QEMU estimates its bandwidth and compares the remaining dirty data against downtime-limit, the largest pause you will tolerate. QEMU’s default is 300 ms. If the pages left can be sent within that budget, it switches over. If not, it runs another pass.
Why some VMs never converge#
Pre-copy finishes only if the guest dirties memory more slowly than the link can send it. Take a VM that rewrites 2 GiB/s of memory, migrating over a 10 GbE link that moves at most about 1.1 GiB/s. Every pass ends with more dirty data than it started with, and no number of passes reaches the 300 ms budget. In-memory databases, caches with heavy churn, JVMs with large young generations and analytics jobs that sweep big arrays all behave this way.
That leaves four options: send faster, let the guest dirty less, accept a longer pause, or change the algorithm. QEMU and libvirt support all four.
The knobs, through libvirt#
The virsh migrate synopsis is long. These are the options that matter for convergence:
virsh migrate --live --persistent --undefinesource --verbose \
--auto-converge \
--parallel --parallel-connections 4 \
web01 qemu+ssh://kvm-b.example.com/systemDowntime budget. virsh migrate-setmaxdowntime web01 1000 sets the maximum tolerable downtime in milliseconds for a domain that is being migrated. Run it from another shell while the migration is running, and read the value back with migrate-getmaxdowntime. Raising it from 300 ms to one or two seconds is often the cheapest way to finish a stubborn migration, but only if the workload can tolerate the pause.
Bandwidth. --bandwidth (MiB/s) caps the pre-copy phase. migrate-setspeed changes the cap mid-flight, and --postcopy-bandwidth caps the post-copy phase. Caps protect a shared network, but a cap below the guest’s dirty rate guarantees the migration won’t converge. --available-switchover-bandwidth is a separate setting. It doesn’t limit anything; it tells QEMU how much bandwidth the final phase really has, so it stops looping when its own estimate is too pessimistic.
Auto-converge. --auto-converge lets QEMU throttle the guest’s vCPUs when the migration isn’t making progress, which slows how fast the guest dirties memory. QEMU’s defaults are a 20% initial throttle, raised by 10% each time it still isn’t converging, up to 99%. libvirt exposes the first two as auto-converge-initial and auto-converge-increment. The cost is guest performance during the migration, and at high throttle levels the guest is close to paused. A newer alternative, the dirty-limit capability (QEMU 8.1+), throttles only the vCPUs that exceed a per-vCPU dirty rate. It requires the KVM dirty ring.
Multifd. --parallel sends migration data over several connections (QEMU’s multifd, 2 channels by default), and --parallel-connections sets the count. One TCP stream and one sending thread often can’t fill a 25 or 100 GbE link, and multifd can. With --parallel, compression is opt-in: QEMU supports zlib or zstd there, chosen with --compressed --comp-methods. Compression trades host CPU for bytes on the wire, which helps on slow links and rarely on fast ones.
Timeouts. --timeout 300 tells virsh to act if the migration is still running after that many seconds. The default action, --timeout-suspend, pauses the guest and finishes the migration offline. --timeout-postcopy switches to post-copy instead.
Monitoring. virsh domjobinfo web01 reports progress, including data remaining and dirty rate. Watch it during your first migrations, because it shows whether a VM is converging or stuck looping. virsh domjobabort cancels the job. A cancelled pre-copy migration leaves the VM running on the source.
Post-copy: bounded time, real risk#
Post-copy reverses the order. After some pre-copy, the vCPUs start on the destination before all of the memory has arrived. When the guest touches a page that hasn’t arrived yet, the access faults (through Linux’s userfaultfd), QEMU asks the source for that page, and the source sends it, along with the pages that follow it. Meanwhile the source keeps pushing the remaining pages in the background. Each page crosses the wire once, so the time and traffic are bounded however hard the guest writes.
In libvirt you enable it with --postcopy. The migration still starts in pre-copy. Then you switch with virsh migrate-postcopy web01, or automatically with --postcopy-after-precopy or --timeout-postcopy.
The risk is spelled out in QEMU’s documentation: during the post-copy phase, a failure of either side loses the guest. Its memory is split between two hosts, and neither copy is complete. If the network breaks, libvirt says the domain cannot be resumed on either host and stays active on both. Recovery is possible if the network comes back: rerun the original command with --postcopy-resume, which uses QEMU’s migrate-recover path. A host crash during post-copy can’t be recovered. There is a performance cost too, since pages fetched on demand stall the vCPU that needs them.
Use post-copy when a VM must move and pre-copy can’t converge, such as when you’re evacuating a host. Avoid it on unreliable networks.
Storage: shared or mirrored#
With shared storage (Ceph RBD, NFS, iSCSI or FC LUNs with cluster-aware locking), only memory and device state move, and both hosts open the same disk. libvirt refuses a migration as unsafe if a disk’s cache mode isn’t none, unless the image is on a coherent clustered filesystem. Override it with --unsafe only if you know why.
Without shared storage, --copy-storage-all copies the disks during the migration. Underneath, libvirt has the destination QEMU export its pre-created images over NBD, and the source mirrors each disk into them while the guest runs. The mirror has the same convergence problem as memory, applied to disk writes. --copy-storage-synchronous-writes makes guest writes go to both sides synchronously, so the mirror keeps up at some cost to guest I/O. --migrate-disks vda,vdb limits which disks are copied, and --copy-storage-inc copies only the top image of a backing chain. QEMU’s older built-in block migration (the blk and inc migrate options) was removed in QEMU 9.1 in favour of this mirror-plus-NBD approach. A local-disk migration takes as long as copying the disk, and the switchover has to finish both the memory copy and every disk mirror.
CPU models decide where a VM can go#
The guest sees a CPU model when it boots, and that model can’t change during migration. Whether the destination can run it depends on the CPU mode in the domain XML:
host-passthroughexposes the host CPU as-is. libvirt’s documentation calls migrating such a guest dangerous unless the hosts are identical in hardware, QEMU version, microcode and configuration. Otherwise the guest may hang or crash on resume.host-modelcopies the host’s model and features into the XML at start. During migration the full definition travels with the guest, so it sees the same CPU on the destination even if that host is newer. The guest can only move to hosts that support everything in that definition.- A named model (
custommode), such as a baseline every host in the cluster supports, gives the widest compatibility.virsh hypervisor-cpu-baselinecomputes one from the hosts’ CPU definitions.
Mixed-generation clusters need a baseline. Without one, migrations from new hosts to old ones fail.
A practical order of operations#
- Check the CPU mode and storage first. These decide whether a migration can happen at all.
- Use a dedicated migration network, and
--parallelon fast links. - Start with the default 300 ms downtime and watch
domjobinfo. - If a VM won’t converge, raise the downtime budget, then enable auto-converge, and only then consider post-copy.
- Abort pre-copy freely, since the source keeps running. Don’t treat post-copy the same way.
Where Victor fits#
Victor’s DRS (Victor Community) compares a cluster’s hosts by their VMs’ CPU and memory demand and live-migrates VMs to rebalance them within VM/host rules, maintenance windows and host headroom. The manual migrate dialog lists candidate hosts with their CPU demand and memory, and warns when a local disk will be copied during the move. On the classic KVM runtime, Victor’s docs record that a VM on local storage pauses for about 2.5–3.5 s at switchover in Incus 7.0 (3.3–4.2 s measured from the guest), whatever QEMU’s downtime-limit. Shared storage avoids that disk-finalization path.