A KubeVirt VM’s disk is, in the end, a Kubernetes volume attached to the VM’s virt-launcher pod. Most of the work in running KubeVirt storage well is getting the right image onto the right kind of volume, quickly and repeatably. That is the job of the Containerized Data Importer (CDI), a separate project installed next to KubeVirt with its own operator (cdi-operator.yaml and cdi-cr.yaml).

Three ways to give a VM a disk#

Volume type Persistent Typical use
containerDisk No Demo and CI VMs, stateless replicas. The image comes from an OCI registry; writes are lost when the VMI stops
persistentVolumeClaim Yes An existing PVC that already holds a disk image (or is a blank raw disk)
dataVolume Yes A PVC that CDI creates and populates (import, clone, upload, blank) as part of the VM’s lifecycle

A containerDisk needs no storage backend at all, which is why it is used in tutorials. Anything with state belongs on a PVC. A DataVolume is a CDI resource (cdi.kubevirt.io/v1beta1) that owns a PVC and a population job: you say “a 20 GiB volume, filled from this URL” and CDI creates the PVC, runs an importer pod, converts the image to raw, and marks the DataVolume Succeeded. KubeVirt waits for that before starting the VM.

CDI import sources#

A DataVolume’s spec.source selects where the data comes from:

  • http: an HTTP(S) URL, with optional basic-auth secret (secretRef), custom CA (certConfigMap), extra headers and a checksum such as sha256:....
  • registry: a disk image inside an OCI image (url: docker://...), the same format as a container disk, with the image file under /disk. pullMethod: node lets the node’s container runtime pull it.
  • s3 and gcs: object storage buckets.
  • upload: an empty target that waits for data through the CDI upload proxy, usually from virtctl image-upload.
  • pvc and snapshot: clone from an existing PVC or a CSI VolumeSnapshot.
  • blank: an empty disk.
  • imageio and vddk: import from oVirt/RHV and from VMware.

CDI accepts qcow2, VMDK, VDI, VHD, VHDX and raw images, optionally gzip- or xz-compressed, and converts them all to raw.

Uploading a local image creates the DataVolume and streams the file through the upload proxy:

virtctl image-upload dv rocky9-base --size=20Gi \
  --image-path=./Rocky-9-GenericCloud.latest.x86_64.qcow2 \
  --storage-class=ceph-rbd --uploadproxy-url=https://cdi-uploadproxy.example.internal

dataVolumeTemplates: a disk that belongs to the VM#

Creating DataVolumes by hand works, but most VMs declare their disks in spec.dataVolumeTemplates. The VM then owns the DataVolume: it is created with the VM and deleted with it.

apiVersion: kubevirt.io/v1
kind: VirtualMachine
metadata:
  name: web-01
  namespace: apps
spec:
  runStrategy: Always
  dataVolumeTemplates:
    - metadata:
        name: web-01-root
      spec:
        sourceRef:
          kind: DataSource
          name: fedora
          namespace: golden-images
        storage:
          resources:
            requests:
              storage: 30Gi
  template:
    spec:
      domain:
        resources:
          requests:
            memory: 4Gi
        devices:
          disks:
            - name: root
              disk:
                bus: virtio
      volumes:
        - name: root
          dataVolume:
            name: web-01-root

Two things to notice. The template uses storage rather than pvc: with storage, CDI fills in access mode and volume mode from the storage profile (below) and adds filesystem overhead, so the manifest does not have to know the backend. With pvc, you specify everything yourself. Second, the source is a sourceRef to a DataSource instead of a URL, which is the golden image pattern.

Golden images: DataSources and DataImportCron#

Importing a 2 GiB cloud image for every VM is slow and puts load on the network. The usual pattern is to import each OS image once into a dedicated namespace and clone from it.

  • A DataSource is a named pointer to a source PVC or VolumeSnapshot. VMs refer to the DataSource, not to the volume behind it.
  • A DataImportCron imports an image on a schedule, re-imports it when the source digest changes, points its managed DataSource at the newest copy, and garbage-collects old copies (importsToKeep, 3 by default).
apiVersion: cdi.kubevirt.io/v1beta1
kind: DataImportCron
metadata:
  name: fedora-image-import-cron
  namespace: golden-images
spec:
  template:
    spec:
      source:
        registry:
          url: "docker://quay.io/containerdisks/fedora:latest"
          pullMethod: node
      storage:
        resources:
          requests:
            storage: 5Gi
  schedule: "30 1 * * 1"
  garbageCollect: Outdated
  importsToKeep: 2
  managedDataSource: fedora

New VMs that reference DataSource/fedora get last week’s image; existing VMs keep the disk they were cloned from. The sourceRef.namespace defaults to the DataVolume’s own namespace.

Cloning across namespaces is subject to an authorization check: the requester needs create on the datavolumes/source subresource in the source namespace, or permission to create pods there. For disks declared in a VM’s dataVolumeTemplates, grant it to the default service account of the VM’s namespace as well. CDI’s RBAC documentation includes a cdi-cloner ClusterRole for exactly this.

How clones are made#

CDI picks the cheapest clone method available:

  1. CSI volume clone, if the driver supports it and the storage profile’s cloneStrategy is csi-clone.
  2. Snapshot (“smart”) clone, if a VolumeSnapshotClass exists for the storage class.
  3. Host-assisted copy otherwise: a source pod and a target pod copy the bytes.

Efficient clones require the source and target to share a storage class and volume mode, and the source must not be in use. When CDI falls back to a copy, it annotates the target PVC with cdi.kubevirt.io/cloneFallbackReason and emits an event. Check for that annotation if provisioning a VM from a golden image takes minutes instead of seconds.

Storage profiles#

CDI creates one StorageProfile per StorageClass. Its status holds the recommended accessModes and volumeMode (as claimPropertySets, in preference order), the cloneStrategy, and dataImportCronSourceFormat (pvc or snapshot). CDI ships defaults for known provisioners. For example, the Ceph RBD CSI provisioners are listed as ReadWriteMany Block first, then ReadWriteOnce Block and Filesystem; CephFS and NFS CSI as ReadWriteMany Filesystem. For a provisioner CDI does not know, the profile is empty and DataVolumes using storage: without explicit access modes will not bind until you fill in spec.claimPropertySets:

apiVersion: cdi.kubevirt.io/v1beta1
kind: StorageProfile
metadata:
  name: my-storage-class
spec:
  claimPropertySets:
    - accessModes:
        - ReadWriteMany
      volumeMode: Block

You can also mark one class as the default for VM disks, separate from the cluster default, with the annotation storageclass.kubevirt.io/is-default-virt-class: "true".

Access modes and live migration#

KubeVirt live migration starts a second QEMU on the target node while the source is still running, so both nodes must be able to open the disk at the same time. The upstream rule is simple: VMs with PVCs must use ReadWriteMany to be live-migratable.

How you get RWX depends on the backend:

  • Ceph RBD (ceph-csi) supports RWX only in Block volume mode. That is the combination CDI’s storage profile prefers for RBD, and it suits VMs because KubeVirt hands the raw device straight to QEMU. RBD volumes in Filesystem mode are RWO only.
  • CephFS and NFS CSI provide RWX Filesystem volumes; the VM disk is a disk.img file on the shared file system.
  • Local storage (hostpath-provisioner, local-path, TopoLVM) is RWO. VMs on it cannot be live-migrated; they can be moved to another volume with KubeVirt’s volume migration, which copies storage between volumes on different nodes.

Block mode avoids a file system layer between QEMU and the device and is generally the better choice where the driver supports it.

Snapshots and VirtualMachineSnapshot#

KubeVirt’s snapshot API captures a whole VM: its spec plus a CSI VolumeSnapshot of each disk. It needs a CSI driver with snapshot support, a matching VolumeSnapshotClass, and the Snapshot feature gate in the KubeVirt CR.

apiVersion: snapshot.kubevirt.io/v1beta1
kind: VirtualMachineSnapshot
metadata:
  name: web-01-before-upgrade
spec:
  source:
    apiGroup: kubevirt.io
    kind: VirtualMachine
    name: web-01

Snapshots of running VMs are supported. With the QEMU guest agent installed, KubeVirt freezes the guest file systems for the snapshot; without it, the snapshot is crash-consistent, and the status says which. Restoring uses a VirtualMachineRestore that names the snapshot; stop the VM first. Remember that CSI snapshots usually live in the same storage cluster as the volume. They are a fast rollback point, not a backup, until something copies them elsewhere.

Backends at a glance#

Backend Mode for VM disks Live migration Notes
Ceph RBD via ceph-csi RWX Block Yes CSI clones and snapshots; CDI defaults to snapshot-based golden images for the Rook/ODF RBD provisioners
CephFS via ceph-csi RWX Filesystem Yes File on a shared file system
NFS via csi-driver-nfs RWX Filesystem Yes Simple; performance and locking depend on the NFS server
Local (hostpath, local-path, TopoLVM) RWO No Fast, cheap; use WaitForFirstConsumer binding

Pitfalls#

  • Scratch space. Registry imports, uploads and some HTTP imports need a temporary scratch PVC as large as the target. CDI always requests it as Filesystem RWO, so a cluster with only a Block-mode class cannot provide it. Set scratchSpaceStorageClass in the CDI config if needed.
  • WaitForFirstConsumer. With node-local storage, CDI does not start the importer until the VM is scheduled, so the disk lands on the VM’s node. Golden images can use the cdi.kubevirt.io/storage.bind.immediate.requested annotation to bind straight away.
  • Silent copy clones. A golden image on a different storage class or volume mode than the VM’s disk triggers a host-assisted copy. Keep golden images on the class you provision VMs from.
  • Cross-namespace sources. Without datavolumes/source permission in the golden image namespace, the clone is refused and the VM does not start.
  • Sizing. The target must be at least as large as the source image’s virtual size. With pvc: on a Filesystem volume, also leave room for filesystem overhead, or use storage: and let CDI add it.
  • Container disks in production. They are ephemeral by design. A VM that “loses its data on restart” is very often on a container disk.

Where Victor fits#

On KubeVirt, Victor treats Kubernetes storage classes as datastores and shows KubeVirt VMs beside KVM VMs in one inventory. For backup, Victor certifies Velero with CSI snapshots and the KubeVirt Velero plugin (tested with Velero 1.18.4, kubevirt-velero-plugin 0.9.1, KubeVirt 1.9.0 and CDI 1.66.1), reads the Velero schedules and backups, and shows on each VM page which schedule protects it. That is part of Victor Community; Velero still owns and runs the backups.

See how Victor shows VM protection