This issue tracks development and discussion of live migration features and related improvements. Feel free to comment, claim features you are interested in, and suggest updates as needed.

Status legend:

🔴 Core Mechanism

These are foundational correctness and reliability items - nothing is production-grade without them.

• 🔲 Re-acquire disk locks when migration fails in last step
  • If the migration fails before/during the Complete call, the VM is resumed on the sender but the disk locks are already freed. This early free however is necessary to allow the sender to start and acquire the disk locks
  • The same thing must happen if we cancel the migration at this point. Another solution would be to not allow cancellation at that moment.
• 🟩 Migration cancellation (Abort Migration API Call)
  • Working prototype: Add API Call to Cancel Migrations cyberus-technology/cloud-hypervisor#85
• 🟩 "Migration in progress": Block VMResize, AddNet, etc. with "VM in migration" error
  • Working prototype: vmm: live migration asynchronization (dedicated thread) cyberus-technology/cloud-hypervisor#24
• 🟩 "Asynchronization" / Unblock VMM lifecycle thread
  • Original upstream attempt: [WIP] vm-migration: Add support for live migration asynchronization #7038
  • Working prototype: vmm: live migration asynchronization (dedicated thread) cyberus-technology/cloud-hypervisor#24
• 🟩 Gracefully handle lost or spurious network
  • Working prototype: Gracefully handle lost network cyberus-technology/cloud-hypervisor#94
• 🟩 Adapt to new NUMA settings on destination host (VMM, not guest-visible)
  • Working prototype: Allow to modify host numa settings cyberus-technology/cloud-hypervisor#81

🟠 Transport / Wire Protocol

Ordered by priority - parallel connections and convergence matter far more than bandwidth limiting.

• 🟩 Multiple parallel TCP connections (high priority)
  • Working prototype: Multiple TCP Connections for Migration cyberus-technology/cloud-hypervisor#39
  • PR: Multiple TCP Connections for Live Migrations #7669
• ✅ Precopy iterations + downtime limit
  • PRs migration: plumbing for downtime + memory iterations + small fixes #7799, vmm: migration: add user-configurable downtime and timeout #7835
• 🔧 Smarter downtime calculation — account for VM stop and final state transfer (devices, vCPUs)
  • Note: CHV overhead after precopy appears much lower than QEMU. First measurements suggest a ~−20 ms adjustment in the precopy phase may be sufficient.
• 🔧 Optimize zero-page transmission (shortcut)
  • Working prototype: Add option to skip zero pages during VM migration cyberus-technology/cloud-hypervisor#112
• 🟩 TLS encryption of migration traffic
  • Working prototype: encrypted live migration cyberus-technology/cloud-hypervisor#35
• 🔲 Postcopy migration (probably supersedes vCPU throttling and may supersede RDMA need) (@cyberus-technology)
• 🔲 Bandwidth limit (low priority)
• 🔲 RDMA transport (Live-migration: RDMA support #7395) (evaluate only after TCP/IP is production-ready; likely unnecessary once postcopy lands)

🟡 Device Support

• 🔧 Generic mechanism for handling external FDs (@posk-io @phip1611)
  • Issue: First class file descriptor support #7704
• 🟩 Network devices with external FDs
  • Upstream effort 1: [WIP] migration: Persist Nested Virtualisation State #7074
  • Blocked by First class file descriptor support #7704
  • Fork solution by Cyberus Technology will be replaced once we have First class file descriptor support #7704
• 🔲 VFIO devices
• 🔲 Device function and state versioning (migrate from new to old Cloud Hypervisor)

🟡 Guest Lifecycle

• 🟩 Allow guest-induced shutdown and reboot during migration
  • Working prototype: Fix guest induced reboot/ shutdown migration failure cyberus-technology/cloud-hypervisor#87

🟡 Network / Post-handover

• 🟩 Send gratuitous ARP / "hello" packets after handover to populate ARP caches
  • Working prototype: virtio-net: Post Migration Announce (RARP advertisements) cyberus-technology/cloud-hypervisor#66

🟢 Observability & Insights

• 🟩 Query live migration progress & insights
  • Working prototype: Add Support for Migration Statistics API Call cyberus-technology/cloud-hypervisor#43
• 🔧 Improve dirty log aggregation (better dirty rate)
  • Issue: migration: fetch dirty pages asynchronously #7816
  • Early prototype: [Experiment] Migration: Aggregate Dirty Log in Thread, Reduce Migration Downtime cyberus-technology/cloud-hypervisor#82
• Event monitor support
  • ✅ vm.migration-starting - migration: plumbing for downtime + memory iterations + small fixes #7799
  • ✅ vm.migration-started- migration: plumbing for downtime + memory iterations + small fixes #7799
  • ✅ vm.migration-failed - migration: plumbing for downtime + memory iterations + small fixes #7799
  • ✅ vm.migration-memory-iteration {id: <num>} - migration: plumbing for downtime + memory iterations + small fixes #7799
  • 🔧 vm.migration-cancelled

🟢 CPU / Platform Compatibility

• ✅ AMX support - hypervisor: AMX state snapshot and restore support #7534
• ✅ Nested virtualization support - [WIP] migration: Persist Nested Virtualisation State #7074
• 🟩 Selectable CPU profiles (@cyberus-technology)
  • Issue: Support for CPU templates/CPU types/CPU profiles #7068
  • Working prototype: https://github.com/cyberus-technology/cloud-hypervisor/tree/gardenlinux
  • Steps: Support for CPU templates/CPU types/CPU profiles #7068 (comment)
• vCPU throttling / auto-converge — will not upstream; postcopy is clearly superior. Production-ready solution exists at @cyberus-technology: gardenlinux: PoC: vCPU throttling/auto-converge cyberus-technology/cloud-hypervisor#13

🔵 Production Readiness & Stability

• 🔧 Feature / migration protocol negotiation (@cyberus-technology)
  • Prototype: Add version negotiation for migration protocol cyberus-technology/cloud-hypervisor#114
• 🔲 Announce in README that live migration will no longer be breaking between releases

📖 Documentation

• 🔲 Overview of supported functionality, caveats, and known limitations

Collaboration process:

• Comment below to discuss implementation details, share ideas, or suggest changes.
• Assign yourself to features you want to work on.
• Leave a comment with proposed checklist changes, or send revised text directly.
• Issue will be updated as progress is made.