By Most high availability environments go untested until an outage forces a real failover. Without a cadence tied to patch cycles and disaster recovery exercises, IT teams have no reliable way to know whether their HA configuration will actually hold when it matters. Automation exists to close this gap, but most teams do not have a structured framework to use it.
In this interview on TFiR, Cassius Rhue, Vice President of Customer Experience at SIOS Technology, walks through how to establish an HA testing cadence aligned to business requirements and how automation through CLI and API tooling makes continuous validation practical for teams with limited capacity.
Guest: Cassius Rhue, Vice President of Customer Experience at SIOS Technology
Show: TFiR
Here is what every platform engineer and IT operations team needs to know.
Technical Deep Dive
Q: Where should a team start when building an HA testing practice from scratch?
Cassius Rhue, Vice President of Customer Experience at SIOS Technology, recommends that teams begin by ensuring their test environment is a like-for-like copy of production. From there, the starting point is the business cadence itself: how often does the organization do patching, and does it run quarterly or biannual disaster recovery exercises? Rhue advises teams to match HA testing frequency to those existing rhythms rather than creating a separate schedule.
“Start with looking at your business requirements. How often do you need to update your production nodes? Use that as the groundwork or framework for how often you do the testing.” — Cassius Rhue, Vice President of Customer Experience, SIOS Technology
Q: How should HA testing cadence align with monthly patching cycles?
Rhue notes that many organizations operate on monthly or three-week patching schedules. The project plan for each patch cycle should include dedicated time to test and validate in the QA environment before any rollout touches the production system. This means HA testing is embedded into the patch workflow, not treated as a separate exercise.
“If you’re doing patching every three weeks, make sure your test environment is set up and that your project plans include time for you to test and validate in that QA environment before you start doing the rollout on your production system.” — Cassius Rhue, Vice President of Customer Experience, SIOS Technology
Q: What role do regulatory and certification requirements play in setting DR testing frequency?
Rhue explains that many organizations run disaster recovery exercises not just for operational reasons but to satisfy regulatory requirements or industry certifications. These mandatory exercises should anchor the testing schedule. Teams should identify which requirements apply to their industry and use those intervals as a baseline for how often DR scenarios are validated.
“How often do you need to do disaster recovery scenarios either to satisfy a business requirement, a regulatory requirement, or a certification requirement for your particular industry, and use that as the framework for how often you do the testing.” — Cassius Rhue, Vice President of Customer Experience, SIOS Technology
Q: How does automation reduce the burden of continuous HA validation on stretched IT teams?
Rhue frames automation as the primary lever for making HA testing frequent and sustainable. The right framework integrates test functionality for both the application layer and the HA environment, covering hard failure simulations and recovery validation in a single automated pipeline. This reduces the manual effort required per test cycle and allows teams to increase testing frequency without adding headcount.
“Automation can play a huge role in reducing the friction, reducing the cost, and increasing the frequency of doing that testing.” — Cassius Rhue, Vice President of Customer Experience, SIOS Technology
Q: What should teams look for in an HA automation framework?
Rhue says the critical capability is a framework that can simulate node failures and activate HA solution functionality in a controlled way. The framework needs to cover both the hard failure case, where a node crashes and applications must restart, and the planned switchover and switchback scenarios that occur during maintenance. Integration with CLI or API interfaces is the practical mechanism for connecting automation tooling to the HA layer.
“You want to look for automation tools that allow you to have a framework that can simulate node failures, that can activate the HA solution functionality.” — Cassius Rhue, Vice President of Customer Experience, SIOS Technology
Q: How does SIOS support automated HA testing through its CLI and API?
Rhue explains that SIOS provides an extensive command line interface that supports both scripting-based and tool-driven automation. Teams can identify the specific commands relevant to their environment and wire them into automated workflows that cover the full patch and recovery sequence: deploying patches, restarting applications, testing failover, testing failback, and testing switchover and switchback. This applies across both Linux and Windows environments.
“SIOS has an extensive command line interface that you can use for automating either through scripting or through automated tools that simply interface with a CLI or API, and that allows you to say which particular commands are most important for your test environment.” — Cassius Rhue, Vice President of Customer Experience, SIOS Technology
Q: What does a fully automated patch maintenance and HA validation workflow look like?
Rhue outlines a complete automated sequence tied to patch maintenance: deploying the patches, restarting the application, testing failover, testing failback, testing switchover, and testing switchback. Both Linux and Windows provide OS-level commands that can crash application services or stop databases, which serve as the failure injection layer. The SIOS CLI then validates that the HA layer recovers correctly at each step.
“You can automate everything from deploying the patches, restarting the application, testing the failover, testing a failback, testing a switchover, switchback.” — Cassius Rhue, Vice President of Customer Experience, SIOS Technology
Resources and Documentation
- SIOS Technology, provider of high availability clustering software with CLI and API automation support for Linux and Windows environments
***
👇 Click to Read Full Raw Transcript
Swapnil Bhartiya: Let’s assume that there are teams who are just starting to build a regular HA testing practice. Where should they start and what does a realistic cadence look like for them?
Cassius Rhue: Yeah, that’s a good question. If you’re, if you’re just starting by building out an HA test infrastructure, right? So let’s assume that you’ve made sure you’re doing a like for, like copy of production. What you want to look at is what we tend to see with a lot of customers and recommend is what is your business cadence? Right? So there are a lot of organizations that are doing monthly patching. And so you want to make sure you’re matching your HA testing to the cadence of the business. If you’re doing patching every three weeks, then you want to make sure your test environment is set up and that your plan is. Your project plans include time for you to test and validate in that QA environment before you start doing the rollout on your production system. There are also organizations that in addition to their monthly patching or update strategy, they’re also doing like a quarterly or biannual disaster recovery exercise. And so I would say start with looking at your business requirements. How often do you need to update your production nodes? How often do you need to do disaster recovery scenarios either to satisfy a business requirement, a regulatory requirement, or a certification requirement for your particular industry and use that as the groundwork or framework for how often you do the testing?
Swapnil Bhartiya: Are there specific tooling or automation approach that makes continuous HA validation more practical without burdening already stretched IT teams? And how does SIOs help there?
Cassius Rhue: That is a really good question and great, great thing. For teams that are thinking about testing or trying to make that burden lighter, automation can play a huge role in reducing the friction, reducing the cost, and increasing the frequency of doing that testing. A lot of tools exist out there, so I won’t name specific ones. But what you want to look for is a framework that allows you to, to integrate key test functionality for your applications or services and your HA environment. We talked about tests that do hard failures and then making sure that applications and services restart. So you want to look for automation tools that allow you to have a framework that can simulate node failures that can activate the HA solution functionality. A lot of HA solutions like sios. SIOS has an extensive command line interface that you can use for automating either through scripting or through automated tools that simply interface with a CLI or API and that allows you to say which particular commands are most important for your test environment. If you’re for example, doing patch maintenance. You can automate everything from deploying the patches, restarting the application, testing the failover, testing a fail back, testing a switchover, switchback. There are lots of commands in each operating system, whether it’s Linux windows that allow you to do things, to crash application services or stop databases. And then you can use those automated HA tools or HA clis to validate that things recover properly.
