Every HA feature promises that when a host dies, its VMs start again somewhere else. The restart is the easy half. The hard half is being sure the original copy has stopped; if it hasn’t, HA has put two writers on one disk. This post covers why, the fencing methods that prevent it, what each guarantees, and how Victor applies them.

A dead host and a partitioned host look the same#

A cluster knows a host is healthy by hearing from it. When that stops, there are two explanations:

  1. The host is down. Its VMs have stopped, and restarting them elsewhere is safe and urgent.
  2. The host is up but cut off. Its management network, a switch port or a cluster link failed. Its VMs are still running, still serving clients and still writing to shared storage.

From the other side of the outage, both cases look the same: silence. If HA treats silence as death and restarts the VMs, case 2 becomes a split brain. Two QEMU processes have the same RBD image or LUN open for writing, two guest kernels interleave writes to one filesystem, and the usual result is a volume that won’t mount.

The ClusterLabs Pacemaker documentation puts it plainly: fencing protects against the split-brain scenario, “where cluster nodes have lost the ability to reliably communicate with each other but are still able to run resources. If the cluster just assumed that uncommunicative nodes were down, then multiple instances of a resource could be started on different nodes.” For a database or clustered filesystem, it notes, “the effect could be much more severe, causing data corruption or divergence.”

So the rule is: never restart on suspicion. Turn the suspicion into a fact first. That act is fencing.

Fencing methods and what they guarantee#

Pacemaker defines fencing as “the ability to make a node unable to run resources, even when that node is unresponsive to cluster commands.” The important part is the last clause: the fence must work without the failed node’s cooperation. There are three broad families.

Power fencing (STONITH)#

“Shoot The Other Node In The Head”: tell the server’s out-of-band controller (the BMC: iDRAC, iLO, XClarity, OpenBMC) to cut power, over Redfish or IPMI, or switch off its PDU outlet. When the BMC confirms the power is off, nothing on that host is running. No processes, no network addresses, no cached writes waiting to flush.

What it doesn’t guarantee: that the BMC is reachable. The failure that cut the host off can take its BMC with it, for example a switch carrying both networks. A power fence that can’t reach the BMC fails, and a well-built HA stack then refuses to restart rather than guess.

Storage fencing (fabric fencing)#

Pacemaker calls this fabric fencing: “cutting the node’s access to some capability required to run resources (such as network access or a shared disk).” Examples include SCSI-3 persistent reservations on a SAN, withdrawing an NFS export, and, for Ceph, adding the client to the OSD blocklist. Ceph’s own control-command documentation describes ceph osd blocklist add ADDRESS[:source_port][/netmask_bits] [TIME], and notes that the default time an entry stays on the blocklist is one hour.

What it guarantees: the fenced host’s writes to that storage fail. What it doesn’t: the host’s processes stop. The old VMs keep running and keep their IP addresses, even though every write fails. Storage fencing protects data; it doesn’t stop two copies answering on one IP address for a while.

Self-fencing (watchdog)#

A hardware watchdog is a timer that resets the machine unless software keeps “petting” it. The Linux watchdog API describes the behaviour: the watchdog “activates as soon as /dev/watchdog is opened and will reboot unless the watchdog is pinged within a certain time.” A daemon on the host pets it only while the host can still see the cluster. If it loses the cluster, it stops petting, and the host resets itself within the timeout. The rest of the cluster waits out that timeout (plus a margin) and then treats the host as fenced.

What it guarantees: nothing, unless you know the watchdog was armed. A watchdog nobody opened never fires, and the isolated host must know it is the side that lost quorum. The kernel’s “magic close” helps: the driver disables the watchdog on close only if V was written just before, so a crashed daemon leaves it armed and the host resets.

Summary#

Method Stops writes Stops processes Needs from the failed host Common failure
BMC power-off Yes Yes Nothing (BMC must be reachable) BMC network down with the host
Ceph blocklist Yes, to Ceph No Nothing Credentials or monitor reachability
NFS export withdrawal Yes, to that export No Nothing Server can’t narrow a subnet or wildcard export
Watchdog self-fence Yes, after reset Yes, after reset An armed watchdog and a petter Watchdog never armed; wrong side of a 50/50 split

How Victor fences KVM clusters#

Victor runs HA itself on KVM clusters managed through Incus (the runtime the console calls “Classic”). Turning HA on in Cluster › Configure › High availability sets Incus’s own cluster.healing_threshold to 0, so Incus never restarts VMs unfenced; in a lab test, Incus healing had restarted a cut-off member’s guests elsewhere while they kept writing on the original.

When a host goes offline and the healing threshold passes, Victor starts one HA failover task that tries the cluster’s methods in order until one succeeds:

  1. BMC power off. Redfish ComputerSystem.Reset with ForceOff, or IPMI chassis power down. Victor then waits until the BMC reports the power off.
  2. Ceph storage fencing. ceph osd blocklist add for each of the host’s Ceph client addresses, then rbd lock rm for its locks on VM disks. Victor reads the blocklist back with ceph osd blocklist ls and counts the host fenced only when a whole-host entry (<address>:0/0) shows for every address within 15 seconds; otherwise the fence is “unconfirmed” and fails.
  3. NFS export fencing. exportfs -u over SSH on the NFS server withdraws the exports from the host’s addresses, then exportfs -s is read back. Only a Linux kernel NFS server (knfsd) is supported; NetApp ONTAP or Pure export policies aren’t driven.
  4. Watchdog self-fence. Used only with evidence, described below.

Only after a fence succeeds does Victor restart the host’s VMs on shared storage on healthy hosts, in restart-priority order. VMs on local storage wait for their host. If no method succeeds, nothing restarts: the task fails with a message such as HA restart blocked: fencing failed (BMC: …) and the host’s alarm is raised.

Note the read-back in each method: a command that returns success is not proof the fence is in effect. Victor also checks its own work. A failover ends DONE only if its steps show a successful fence (or a manual confirmation) followed by the restart. If the failed host’s VMs appear elsewhere before Victor fenced it, the failover fails with an “HA safety” event.

The watchdog needs evidence#

Victor can’t see a host’s watchdog from outside. In an early lab run, it waited out the timeout of a watchdog nobody had armed and would have restarted VMs that were still running. So each member now runs a petter, victorctl watchdog run, which holds the device, pets it while its checks pass, and reports to Victor every 5 seconds: armed or not, driver, timeout, pets and boot ID.

VICTOR_TOKEN=… victorctl --server https://victor.example.com \
  watchdog run --host node-1 --timeout 60 --require-tcp 10.0.0.11:8443

A failing --require-tcp or --require-cmd check stops the petting, and the kernel resets the host; losing Victor does not. Victor uses the watchdog method only if a report under 90 seconds old said it was armed and petted when the host went offline, and a host that comes back must show a new boot ID or boot time. softdog is accepted only on clusters marked as labs; production needs a hardware watchdog (IPMI, iTCO), because softdog runs inside the kernel it is watching.

Quorum guards#

  • More than half the hosts offline at once: HA holds. The likelier cause is the management network, and fencing everything would power off healthy hosts. Before this guard, a lab run that cut one member’s management network had Victor fence all three members.
  • Exactly half offline (every failure on a two-member cluster): Victor can’t tell which side has quorum, so the watchdog alone doesn’t count. Two-member clusters need BMC or storage fencing, or a small third witness member, for automatic restart.

Fence held after a partition heals#

A storage-fenced host that rejoins without rebooting may still run the old VM processes. Victor keeps its blocklist entry or withdrawn export and raises the host.fence-held alarm until it reboots (or resets it through its BMC, if it has one and runs no VM Victor knows of). Ceph entries are added for 30 days, not Ceph’s one-hour default. Don’t run exportfs -ra or restart the NFS server while a fenced host may still be running: that reloads /etc/exports and lifts the fence.

How KubeVirt differs#

Victor doesn’t fence KubeVirt nodes. The Kubernetes path is node remediation: the medik8s Node Healthcheck Operator “checks each Node’s set of NodeConditions against the criteria and thresholds defined in NodeHealthCheck (NHC) custom resources” and creates a remediation from a template. Fence Agents Remediation reboots the node “using a fence agent and then evicting workloads.” NHC’s minHealthy/maxUnhealthy act as a quorum guard.

Until the node is remediated, KubeVirt keeps the VMs assigned to it, so nothing runs twice; its user guide notes a wholly unresponsive node’s pods otherwise need a force-delete before VMIs can be rescheduled. Victor observes: it reports when KubeVirt has moved the VMs on shared storage and raises an alarm if they haven’t moved within 15 minutes. Confirm host down is refused, and restart priorities don’t apply. If you run KubeVirt, set up NHC with a fence agent.

Configuring fencing in Victor#

Fence methods are set in the console or the FencingService API; there is no victorctl command for them at this commit.

  • BMC: Host › Configure › Power management. Redfish over HTTPS (recommended, with the BMC’s CA certificate) or IPMI v2.0 over LAN (UDP 623, cipher suite 17, falling back to 3 only where 17 is missing; the FIPS build offers 17 only). Keep IPMI BMCs on an isolated network. The password is sealed with Victor’s master key. Test BMC reads the power state and changes nothing.
  • Ceph: each host’s Ceph public-network addresses; on the cluster (Configure › Fencing) the RBD pool, ceph.conf and a Ceph user such as:
ceph auth get-or-create client.victor-fence \
  mon 'allow r, allow command "osd blocklist"' osd 'allow rwx pool=incus'
  • NFS: the server as user@host[:port], an SSH key restricted on the server (from="<victord>",restrict) and a strict known_hosts file. Export each member by its own address; a subnet or wildcard entry makes the fence fail.
  • Watchdog: the timeout and a margin of at least 15 seconds on the cluster, plus the petter on each member as a systemd service (not alongside the watchdog daemon: one process holds /dev/watchdog).

victor-hwcheck --bmc https://bmc-01.example.mil on a prospective host checks, among other things, that fencing can reach the BMC. Cluster › Configure › Fencing shows per-host readiness, and a cluster where any member has no usable method shows “HA restart disabled: no fence method”.

When no fence works: Confirm host down#

If every method fails and you have verified by other means that the host is off, use Confirm host down… on the host, the equivalent of vSphere’s “host is confirmed down”. HA restarts the VMs without fencing, and the event log records who confirmed it and why. If the host is in fact still running, its VMs will run twice.

Admission control: reserve room to restart into#

Fencing makes restarts safe; admission control makes them possible. It is off by default:

victorctl ha show cluster-a
victorctl ha set cluster-a --admission host-failures --host-failures 1 --validate
victorctl ha set cluster-a --admission percentage --cpu-pct 25 --mem-pct 25
victorctl ha set cluster-a --admission dedicated --failover-host hv-a06

--validate shows whether the policy would hold without saving it. Once enabled, power-on, create, migrate (including DRS moves) and reconfigure are refused if they would break the policy. HA failover, Confirm host down and maintenance mode are never refused.

The caveat: the policy checks aggregate capacity, not whether each VM fits on one surviving host. A 96 GiB VM can be “covered” by 128 GiB free across four hosts and still have nowhere to land. Size headroom for your largest VMs.

Restart order and VM monitoring#

On KVM clusters, restarts happen highest priority first (Highest, High, Medium by default, Low, Lowest), and a VM can be told to start after others:

victorctl ha vm cluster-a db-01 --priority highest
victorctl ha vm cluster-a web-01 --priority high --start-after db-01
victorctl ha vm cluster-a scratch-01 --priority disabled

start-after overrides priority, circular dependencies are refused, and a disabled VM stays down. VM monitoring (victorctl ha set cluster-a --monitoring on) resets a guest whose agent misses 3 heartbeats 30 seconds apart, at most 3 times in 60 minutes; it has no I/O check, so turn it off for VMs whose agent is stopped on purpose. For network maintenance, victorctl ha suspend cluster-a --reason 'core switch firmware' --minutes 90 disarms HA with an audited reason and re-arms it automatically.

A test plan for a maintenance window#

Don’t trust a fence you haven’t seen fire. In a window, on a cluster with non-critical VMs on shared storage:

  1. Readiness. Every member shows a usable method under Configure › Fencing; Test BMC passes on each host; victorctl ha show cluster-a reports the admission policy you expect.
  2. Watch. victorctl task watch in one terminal, victorctl events --kind host --since 1h in another.
  3. Power test. Pull a host’s power. Expect: host offline, failover task, fence step, restarts in priority order.
  4. Partition test. Cut a host’s management network only, the case fencing exists for. Expect a BMC power-off, or <address>:0/0 in ceph osd blocklist ls (or the export gone from exportfs -s), then restarts.
  5. Negative test. With no reachable method, expect the VMs to stay down with an alarm.
  6. Heal. After a storage-fence test, restore the network without rebooting. Expect host.fence-held until the host reboots.
  7. Record the task output and timings: your evidence that HA works on your hardware.

Known limits#

  • Not yet exercised on bare metal. HA has been tested on simulated, nested and cloud-VM labs; BMC fencing has not been run against real server hardware.
  • SAN LUNs have no storage fence. Use BMC or watchdog fencing there.
  • Storage fencing stops writes, not processes. “No VM runs twice” holds for writes.
  • KubeVirt: Victor observes rather than fences; Ceph CSI NetworkFence isn’t built.
  • IncusOS members show watchdog fencing as unavailable for now.

Where Victor fits#

Victor Community fences KVM hosts by BMC, Ceph, NFS or watchdog, confirms each fence before restarting anything, and refuses to restart when it can’t prove the old copy has stopped. On KubeVirt it defers to Kubernetes node remediation and reports the outcome.

Read about HA and fencing