Failover Testing: How to Know Your HA/DR Really Works | Aidan Macklen, SIOS Technology | TFiR

Swapnil Bhartiya: High availability and disaster recovery are supposed to keep critical applications running, but according to 2026 SIOS survey, 76% of organizations still had an outage longer than 10 minutes in the past year, even with HA and DR in place. Most teams don’t actually know that their protection has gaps until an outage exposes them. And as applications spread across on-prem, cloud, and hybrid, those gaps are getting harder to see. So how do you know if you are actually protected from downtime? To walk us through that, I’m joined by Aidan Macklen, Customer Experience Engineer at SIOS Technology. Aidan, it’s great to have you on the show.

Aidan Macklen: Thank you for having me.

Swapnil Bhartiya: Many organizations have already invested in high availability and disaster recovery. How does an IT team tell whether that investment is actually protecting them from downtime?

Aidan Macklen: So the only way to really know if you’re protected from downtime is to simulate failure instances. So preferably you would have a QA system set up such that you can sort of predict the outage that you may experience. So for example, some of the larger issues that we see are network disconnections, whether this be the virtual IP address that is a dependency of the application you’re protecting, or possibly an overall outage of system 1. Could be an application getting hung in the process, thus not able to switch over. It’s really a various amount of outages that could occur in a highly available system. But the only way to truly know is to test and simulate these outages.

Swapnil Bhartiya: What are the most common gaps in an HA/DR strategy that only show up when an outage hits?

Aidan Macklen: I would say one of the most common gaps that we see is not accounting for every dependency that a critical application may depend on. So for example, in a SQL cluster where the application you’re truly protecting is a SQL database or a SQL instance, the SQL database needs to be accounted for, the authentication used to connect to the SQL database, virtual IP associated with that application. All of these things need to be accounted for, yet that may not become apparent until the primary application is switched over to node 2. So once node 2 takes hold of the primary application, end users may not be able to connect to the database. There may be sort of disconnections or errors in the logs that would indicate an issue. And all of these things kind of lead to some issues that we would identify as common gaps.

Swapnil Bhartiya: Once failover and recovery plans are in place, how often should teams be testing them?

Aidan Macklen: For a fixed schedule, I would recommend a quarterly test plan. However, there will be changes performed throughout the year in most organizations. So I would suggest after any change you make within the configuration, even a little change such as the virtual IP address associated with the instance, may cause an outage. So taking the proper precaution to run through your runbooks, update any documentation you have, align the QA system with the production system, all of these steps should be taken and all testing should be accounted for within this manner.

Swapnil Bhartiya: Now, applications today are spread across on-prem infrastructure, cloud, and hybrid environments. What makes it harder to keep protection consistent across all of these at once?

Aidan Macklen: The first thing that comes to mind is an organization that may have a database stored locally on a data center while their application and overall server or instance is hosted in a cloud environment. So this just serves as an additional failure path between your primary application and the data stored. So any disconnect between these environments would serve as another outage possibility. Additionally, with the introduction of cloud technology, any outage from AWS or Azure or any other cloud provider is serving an outage to you. It could be a— I believe recently there was a DNS issue that took down some AWS hosting. And you— we’re reliant on them to get that back up and running.

Swapnil Bhartiya: If an IT leader wants to improve application resilience, what are the first steps to identify vulnerabilities and build confidence that mission-critical applications can recover quickly?

Aidan Macklen: So this ties back to what I was mentioning previously, being that you should be testing. The only way to build confidence and resilience is to preferably set up a QA testing environment such that you’re not just continuously testing in a production environment. And from there, you would want to simulate as many failures as possible. That would be the best step to take in order to build resilience. Additionally, updating your documentation, making sure your architecture is organized, you have recovery points and backups in place. All of these precautions should be taken.

Swapnil Bhartiya: When teams run a failover test, what should they be checking to confirm the plan will actually work? when they need it?

Aidan Macklen: So the first thing I would recommend is to check the logging. Typically this would be within the system or application event logging if on a Windows environment, or if on Linux, you would be looking through the journal and message logging, or possibly wherever the application is routed to log at. Typically you would see some whether it be DNS issues, maybe the server’s unable to start, failover clustering’s having an issue, this would be the best place to start. From there, you could look into, is the end user able to access the application, the database? Are the resources coming online? I think those would be good places to start.

Swapnil Bhartiya: Once a team finds a gap in their production, how should they decide What to fix first?

Aidan Macklen: We are happy to help with any issue you experience that you may believe has an impact with the SIOS configuration. Typically, we would recommend that you ensure that the network is not the issue, you can— your drives are healthy and accessible. From there, we are happy to take a look. And let you know what we see throughout the logging, if we have any documentation that may assist, but we’re always happy to help.

Swapnil Bhartiya: Now, of course, there are a lot of customers who already work with you or they want to work with you, but you can do only so much. They have to make sure that their internal infrastructure architecture is actually ready for HA and DR. So what kind of preparation they should do? in order to get most out of their HA/DR strategies?

Aidan Macklen: Right. So we do have a few best practices documentation links available. However, one of the notable ones for me personally is we recommend having a 2-node cluster with a witness. Could be a 3-node cluster with a witness. The primary action here is to have a witness node. The witness node will serve to be a sort of a vote in the system such that you have node 1 go down. Now node 2 has a second point of communication. If the communication between node 1 and node 2 goes down, node 2 without a witness would see node 1 as completely offline and attempt to take resources over. This could result in a split brain. Having that witness there would serve to protect all of those instances. Additionally, we do have a pre-sales team ready to help you out with installing these best practices and configuring your environment.

Swapnil Bhartiya: Aidan, thank you so much for joining me. We covered a lot today from how to tell whether HA and DR investments are truly protecting you to the gaps that stay hidden until an outage hits and the first steps IT leaders should take to make sure that their HA and DR strategies are actually working. Thank you for your time today, and I look forward to another great conversation with you. Thank you.