By Hard crash failover tests are the default validation method for most high availability environments, but they skip shutdown procedures entirely. That means your application’s graceful stop sequence, shutdown ordering, and fencing transitions are never exercised before a real outage forces them. Teams discover these gaps only when recovery fails under pressure.
In this interview on TFiR, Cassius Rhue, Vice President, Customer Experience at SIOS Technology, breaks down a more complete and less invasive HA validation approach using SIOS LifeKeeper, covering switchover testing, fencing, startup ordering, and runbook inspection.
Guest: Cassius Rhue, Vice President, Customer Experience at SIOS Technology
Show: TFiR
Here is what every platform engineer and HA administrator needs to know.
Technical Deep Dive
Q: What has traditional HA failover testing focused on and where does it fall short?
Cassius Rhue, Vice President, Customer Experience at SIOS Technology, explains that traditional HA testing has focused almost entirely on whether a backup server can take over after a hard crash or hard power-off of the primary node. That approach validates crash recovery but does not exercise the application’s shutdown procedures at all. Because the node is hard-crashed, the system never goes through a graceful stop sequence, which means shutdown ordering and pre-failover fencing transitions remain untested.
“A lot of traditional testing has focused solely on whether you can perform a failover—whether you can hard-crash or power off a node—and whether the system will recover.” — Cassius Rhue, Vice President, Customer Experience, SIOS Technology
Q: What is a switchover in SIOS LifeKeeper and how does it differ from a failover?
Rhue describes a switchover as a manual, user-initiated operation in which services are gracefully stopped on the primary system in the correct order and then gracefully restarted on the backup server in the correct order. Unlike a hard failover, which bypasses shutdown entirely and goes straight to crash recovery, a switchover exercises both the shutdown path and the startup path. This makes it a more comprehensive test while also being less disruptive to the environment.
“A switchover is a much less invasive test, but it also is a more comprehensive test because it’s going to hit the shutdown scenarios as well as the startup scenarios.” — Cassius Rhue, Vice President, Customer Experience, SIOS Technology
Q: What role does fencing play during a switchover test?
Rhue explains that fencing is validated at two points during a switchover: when the primary server transitions out of its primary role and when the backup server assumes the primary role. The test confirms that fencing is correctly in place as the server changes roles, preventing split-brain or data corruption scenarios. This fencing validation is something a hard crash test does not reliably exercise because the crash itself disrupts the normal transition sequence.
“Fencing is in place as the server transitions from its role as the primary to its role as the secondary or backup. It also ensures that applications, services, and databases are started in the correct order on the new primary system, and that fencing remains in place throughout the process.” — Cassius Rhue, Vice President, Customer Experience, SIOS Technology
Q: How does SIOS LifeKeeper validate that applications started correctly on the new primary?
Rhue notes that many HA solutions, including SIOS LifeKeeper, include application validation as part of the startup sequence on the new primary. After services are brought online in the correct order, the solution checks that the application is actually running correctly, not just that the process started. This adds a layer of confirmation that goes beyond simply transferring the workload.
“For many HA solutions on the startup scenario, they’re going to validate that the application is running correctly.” — Cassius Rhue, Vice President, Customer Experience, SIOS Technology
Q: Can you test HA readiness without taking applications offline or crashing a node?
Rhue confirms that a significant portion of HA issues can be identified through runbook validation alone, without any downtime. The process involves comparing the assumptions and configuration details documented in the runbook against the actual current system configuration. If the runbook is out of sync with the live environment, that gap is a failure waiting to happen, and it can be caught through inspection before any test is run.
“A very simple and easy thing to do is to validate that the assumptions and configuration details outlined in your runbook actually match the system. This can be done without crashing a node or taking applications offline.” — Cassius Rhue, Vice President, Customer Experience, SIOS Technology
Q: What is the recommended sequence for HA validation testing?
Rhue outlines a three-stage approach. First, perform a visual inspection by validating the runbook against the actual system configuration, which requires no downtime. Second, run a switchover test, which gracefully exercises shutdown and startup sequences without a hard crash. Third, if needed, proceed to a hard failover test to validate crash recovery. Starting with inspection and switchover catches the majority of issues before introducing the risk of a hard crash.
“We can move into these scenarios of testing a switchover or even the hard failover scenario.” — Cassius Rhue, Vice President, Customer Experience, SIOS Technology
Resources & Documentation
- SIOS Technology, vendor resource for SIOS LifeKeeper high availability software and documentation
***
👇 Click to Read Full Raw Transcript
Swapnil Bhartiya: Is there a simple way to test whether an HS setup really works? And if yes, what does that process looks like? And how does it differ from traditional failover testing?
Cassius Rhue: Yeah, great, great question. So I’ll break it up and say that, you know, a lot of the traditional testing has been focused solely on can we do a failover, can we hard crash or hard power off a node, and will it recover? You know, will the backup server take over operations for the applications and databases? And there are other ways to do that validation. There are simpler ways. One with the sios lifekeeper solution. One of the easiest ways is what we call a switchover, and that’s a manual user operation where services are gracefully stopped on your primary system and then in the correct order, gracefully restarted on the backup server. The difference is in a failover scenario, you’re going to hard crash a box and you’re going to. That failover doesn’t validate that you’re able to successfully stop the application, so it doesn’t go through any shutdown procedures. You’re just doing crash recovery. A switchover is a much simpler test, and it validates both the shutdown operation, making sure that applications are shut down in the right order, that fencing is in place as the server transitions from its role as a primary to its role as a secondary or backup, and then also that applications and services or databases are started in the right order on the new primary system, and that fencing is now in place. And that’s a much less invasive test than hard crashing a node, but it also is a more comprehensive test because it’s going to hit the shutdown scenarios as well as the startup scenarios, and then also add into that for many HA solutions on the startup scenario, they’re going to validate that the application is running correctly. One other thing to add to that simple test, a lot of issues can be eliminated in just validating against your own runbooks and validating that you’re. Your configuration is accurate. So many of our customers create runbooks, and a very simple, easy thing to do is to validate that the assumptions and configuration details outlined in your runbook actually match the actual system. And so that can actually be done without crashing a node or taking applications offline. It’s an inspection, a visual inspection first. And then of course, we can move into these scenarios of testing a switchover or even the hard failover scenario.
