By High availability infrastructure that has never been validated under real conditions is not a safety net. It is a false confidence trap. After patch cycles, application upgrades, or cloud migrations, silent misconfigurations accumulate on backup servers, and no one discovers them until a hard outage forces a failover that does not complete correctly.
In this interview on TFiR, Cassius Rhue, Vice President, Customer Experience at SIOS Technology, walks through the most common gaps in HA validation strategy, from architectural sizing mismatches to client connectivity failures after failover, and why application crash recovery requires its own dedicated testing track.
Guest: Cassius Rhue, Vice President, Customer Experience at SIOS Technology
Show: TFiR
Here is what every platform engineer and IT operations team needs to know.
Technical Deep Dive
Q: Why is it risky to assume a high availability environment will work if it was only tested once at initial deployment?
Cassius Rhue, Vice President, Customer Experience at SIOS Technology, explains that every change made after initial deployment introduces new risk. Patch maintenance, application version upgrades, and configuration updates on production servers can leave the backup server in an inconsistent or incomplete state without anyone noticing. The moment of an actual outage is the worst possible time to discover that a backup service is misconfigured or that a step was skipped during a maintenance window on the secondary node.
“Assuming that it will work because it worked once is not a great strategy. It introduces a lot of risk to the business, unnecessary risk to the business.” — Cassius Rhue, Vice President, Customer Experience, SIOS Technology
Q: What happens to an organization when an untested HA environment fails during a real outage?
Rhue describes the pattern consistently: when incompatibilities surface during an actual emergency rather than a controlled test, the result is chaos and panic at the operational level, followed immediately by business stakeholders demanding explanations about what happened and what gaps existed in the plan. Teams that have not practiced failover do not have the muscle memory to execute calmly under pressure, which compounds the technical failure with an organizational one.
“When an emergency happens, that’s when you discover those incompatibilities that leads to chaos, that leads to panic, that leads to business stakeholders wanting to understand what happened and what were the gaps.” — Cassius Rhue, Vice President, Customer Experience, SIOS Technology
Q: What is the value of regular HA failover testing beyond confirming that the setup technically works?
Rhue frames regular testing as a two-part investment. The first part is technical validation: confirming that the architecture still functions correctly after each change cycle. The second part is operational: building muscle memory so that the team knows how to execute a failover under pressure without hesitation. A team that has rehearsed the procedure multiple times will not panic during an actual event, and that behavioral readiness is as important as the technical configuration being correct.
“Testing and practice are critical components of business resilience planning. It’s not just an exercise of will it work, but it’s also an exercise of do we understand how to operate in an actual outage event.” — Cassius Rhue, Vice President, Customer Experience, SIOS Technology
Q: What is the most foundational HA validation gap that organizations still miss today?
Rhue identifies the first and most foundational gap as failure to validate architectural parity between the primary and backup systems. With cloud adoption growing, many organizations provision cloud-based secondary nodes without verifying that the compute, network, and storage sizing matches the primary. If the backup system is undersized, it cannot handle the actual workload when it takes over, regardless of how well the failover software performs. Confirming that both nodes are architecturally equivalent is the prerequisite for every other validation step.
“The first gap that I notice a lot of is just the gap of not validating your actual architecture. Did they have the right sizing for compute or network or storage? Are the systems identical in size? Are they both capable of handling the actual workload?” — Cassius Rhue, Vice President, Customer Experience, SIOS Technology
Q: Why is confirming that a service started after a failover not sufficient validation that the system is actually available?
Rhue draws a clear line between a service being running and a service being reachable. Many teams limit their failover testing to confirming that the application, database, or service process starts successfully on the backup node after a simulated failure. That check misses the critical requirement: client applications must be able to connect to and use that service. A process that is running but unreachable by clients does not constitute availability, and the gap is only found during testing if client connectivity is explicitly included in the test scope.
“Having applications or services up and available on the server, that’s a great step. But if clients cannot connect to actually use that service, then you’re not really available.” — Cassius Rhue, Vice President, Customer Experience, SIOS Technology
Q: Why do organizations need to test application crash recovery separately from hard server outage scenarios?
Rhue notes that most teams structure their HA testing around hard server failure scenarios, which is appropriate but incomplete. Applications can crash independently of the underlying server, and the HA software must be validated to detect and recover from an application-level failure, not only a host-level one. If the testing program never simulates an application crash, the organization has no confirmation that the HA layer will respond correctly when that specific failure mode occurs in production.
“Customers who tend to forget to validate what happens if their application crashes focus their testing solely on hard server outages, but they forget that applications also need to be validated. If they crash, can the HA software successfully recover it.” — Cassius Rhue, Vice President, Customer Experience, SIOS Technology
Resources & Documentation
- SIOS Technology, provider of high availability and disaster recovery software for business-critical applications
***
👇 Click to Read Full Raw Transcript
Swapnil Bhartiya: I think this is also very, very important topic. It’s more or less like it’s not as same as that your car has airbag or seat belt and it will work when you need it. You need to keep testing, you need to keep checking. Most organizations don’t do that. They assume that their HA environment will work when they need it, but they don’t regularly test it. Can you talk about why is that assumption that hey, you know what, we have a very stable setup which we tested once more. Why is that assumption so risky? And what are the most common gaps that you see in HA validation today?
Cassius Rhue: Great. Yeah, those are two great questions. Let me break them apart into two. First, it’s a risky assumption when you’re making changes, you’re doing updates, even when you do the initial deployment. To think that everything’s going to work fine without doing actual testing during an actual outage or an emergency. That’s not the moment that you want to discover that a backup service is misconfigured or incomplete or that when you are doing maintenance that you forgot a step on the backup server, whereas you were able to validate things were running correctly on the production, but maybe you skipped an item. So assuming that it will work because it worked once is not a great strategy. And it introduces a lot of risk to the business. A lot of risk that’s unnecessary for the business. And when an emergency happens, that’s when you discover those incompatibilities that leads to, as you mentioned in the intro, that leads to chaos, that leads to panic, that leads to business stakeholders wanting to understand what happened and what were the gaps. So testing and practice are critical components of business resilience planning. They show that you have a well planned and thought out solution when you’re updating your servers, either through patch maintenance or you’re doing updates to the application versions on upgrades. Doing that testing reduces the risk and helps you build muscle memory. So it’s not just an exercise of will it work, but it’s also an exercise of do we understand how to operate in an actual outage event and gets you that muscle memory so that when an actual event happens, you’re not going to panic. You know, as far as the most common gaps I see, you know, one of the worst gaps we used to see early on was the gap between customers that did nothing at all and those that did just something. And that always leads to panic when you do nothing. When you have done updates and you don’t test a failover or switch over when you’ve done upgrades of your application and you’ve noticed validated that they were updated correctly on both target and source. That gap used to be a really prevalent thing and thankfully most customers realize that they have to do some validation. But I still see gaps with customers and their strategy for testing. The first gap that I notice a lot of is just the gap of not validating your actual architecture. So with the emergence of cloud and that usage growing and increasing, we see a lot of customers taking advantage of cloud resources and forgetting to validate in their architecture. Did they have the right sizing for compute or network or storage? That’s the first gap that I see is just not understanding are the systems identical in size? Are they both capable of handling the actual workload? And then once they have identified the architectural challenges and verified there are no gaps there, there’s still gaps around failover or testing to make sure that client applications can connect. Some clients that we’ve worked with in the past simply wanted to check off did their application or database or service start after a system failure test. But you have to take that a bit further. Having applications or services up and available on the server, the source or primary or target or backup, whichever language you use, that’s a great step. But if clients cannot connect to actually use that service, then you’re not really available. And then lastly, I’d say that there are customers who tend to forget to validate what happens if their application crashes. They focus their testing solely on hard server outages, but they forget that applications also need to be validated. If they crash, can the HA software successfully recover it.
