Swapnil Bhartiya: Chances are that your failover is not as ready as you or your teams may think. A failover solutions being configured is not the same as knowing it will work when it’s needed. Most teams figure and find that out mid outage when downtime costs money and their customers lose trust. By that time, it’s already too late. Cybersecurity helps organizations prove high availability works before they need it. They have built a five minute checklist to validate your failover setup. And today we have with us Alexis Kaur, senior customer experience software engineer at SIOS to walk us through that. First of all, Alexis, it’s great to have you on the show.

Alexus Gore: Thanks for having me.

Swapnil Bhartiya: It’s my pleasure to have you here. As you know, as you know, most organizations, they assume that they are protected just because they have failover solutions in place. And from their perspective there is nothing wrong. That’s what they should assume. But can you talk about what is the biggest difference between having failover configured and actually knowing that it will work when they actually need it?

Alexus Gore: So the biggest difference in determining how your failover environment is going to work when you need it is largely based in the testing that is that has been completed for it. With testing, you need to kind of like check the ins and outs with how your environment is going to react in a situation like should a network outage occur and if there is like an unknown answer into whether or not you know you have your failover configured, into whether or not that failover solution you know it’s going to react appropriately in the event that a failover outage occurs, if there are any unknowns there, then that is typically not the best case scenario. You want to be able to know what’s going to happen, how your solution is going to react in the event that a network outage occurs, in the event that any type of disruptance occurs.

Swapnil Bhartiya: Ideally, and let’s assume that you have just five minutes to assess the health of a production failover environment. What are the first few things you would check?

Alexus Gore: First few things I check, I think mainly I start with the logs. Logs are generally going to tell you everything you need to know that’s happening in your failover environment. You have your failover solution logs and your failover solution logs are typically going to cover what’s going on with how your failover solution is handling your environment. And you can also check your system logs. Your system logs can usually tell you, I mean both will usually tell you if there are any like network issues or communication issues that are happening within a cluster or in your environment. But with your system logs you can also kind of determine if there are any issues with your system having any network problems or just like lack of communication or just like mishaps going on between your system and your failover solution.

Swapnil Bhartiya: You talk to a lot of customers, you talk to teams, and I’m sure you have heard a lot of stories. What are the most common failover configuration issues that you have seen? The ones that don’t show up until an actual outage happens.

Alexus Gore: So most common, I would say happens when, sometimes when customers are using, let’s say, SAP applications with Datakeeper. And a lot of the times there are, we have documentation that cover the steps that are required to configure your SAP application properly to be used with Datakeeper. But sometimes the things that are like missed in the cracks may be like small parameters that are not updated to be used with Datakeeper and are not usually caught until an outage occurs because it’s something that’s so small, you know, if this customer we haven’t done a health check with, then you’re not going to really see it until an outage occurs that there may be a problem happening. And a lot of the times it does get caught, it can get missed, but it will get caught by just checking the logs. Because we do update our product to make sure that we’re catching these things beforehand. But yeah, I think the most, like the most common issue is definitely not having your application set up in respect to the failover solution that you’re using now.

Swapnil Bhartiya: The fact is that a lot of IT teams, they hesitate to test failover because they worry about disrupting production. And that’s also a varied concern. How should organizations approach failover testing so that they build that confidence that things will not break? They get to test so they are not taking unnecessary risk.

Alexus Gore: So two things to avoid any unnecessary risks, you would, ideally you would want to create a QA cluster beforehand before you even go into production. And a QA cluster is typically going to be a one to one exact copy of a production cluster that’s primarily going to be used for testing. So that way anything that you change or that you plan to change for your production cluster, you can do it in the QA cluster first. Second thing is creating a runbook. A runbook is usually going to have the steps that you ran during your testing. What happened, what occurred, what you expected to happen, maybe why it even didn’t happen. So this is good to keep track of everything that happened up to the moment where maybe a disruption occurred that you really didn’t expect to happen there. But yeah, to avoid any unnecessary risks, create a QA cluster and create a runbook to keep track of everything that you did.

Swapnil Bhartiya: Now the fact is that organizations are modernizing. We are already in the journey of AI, but a lot of organizations are hybrid cloud kubernetes. Of course, AI driven workloads are also here. How does a failover validation also need to evolve with these changing times? What new risks are there that IT teams should be aware of and watching for?

Alexus Gore: So when it comes to failover validation, you’re now going to be testing for predictability, or at least testing more for predictability and adaptability. How well can your environment predict when an outage is going to occur and whether or not your solution is able to adapt to what your environment needs at that time? When it comes to AI, AI is largely based on learning from its failures. So when it comes to integrating with AI, a lot of more HA solutions and failover solutions, it leaves more room for error, which, you know, it’s not like the best thing to hear, but it’s bound to happen right when you’re introducing so many more products to create almost like a bulletproof, highly available solution, there’s just going to be more room for error. And the good thing about AI is that when it’s learning from its failures, it’s also succeeding and learning to improve again and again. So when you take that risk, there is a high reward at the end of it. But yeah, I mean ideally you want to just test more, cover more ground, do as much as you can. But those are the things that IT teams should be on the lookout for.

Swapnil Bhartiya: Is it possible for you to share a real world example where a company believed that its disaster recovery plan was ready only to discover a critical gap during testing or worse during an actual outage. And through that there was a lesson that was learned. And that lesson can be shared with others as well.

Alexus Gore: Yeah, so there are actually three occurrences that I can think of where this happened. Both of these customers, or all three of these customers have experience with using DataKeeper. And all three of these customers also wanted to use another failover solution or another replication solution as well alongside DataKeeper. With the systems that they provided us, with the logs that they provided us, it looked like it worked. And so everything was in place. Both solutions were working fine. There were no issues. The problem happened when a network outage occurred and the failover solution did not work. And it’s something that could have been caught if the customer had taken the steps, the precautionary steps into testing their solutions together. Technical incompatibility is one of the themes here. When I’m talking about these three customers, the lesson learned here is that it’s important to test. But I do want to mention that their configuration did look like it worked because they used both products independently. So putting them together was where they thought there would be success. But unfortunately, because there wasn’t any testing involved, it just was a catastrophe and it was missed in the beginning. Like, I think looking through the logs, you couldn’t really see where the issue lied. You would have to go way, way deep into the kernel code to really figure out what’s happening. So, yeah, making sure that you test is the biggest aspect of this and making sure that you test before you go into production as well.

Swapnil Bhartiya: How often should teams actually run through these five minute checklist and how does that frequency change as the environment becomes more dynamic? Or as we were talking about, you know, new use cases, AI and everything is entering, so it’s becoming more dynamic as well. So frequency of these tests and checks.

Alexus Gore: You know, I say you should test as often as possible. Right. When you’re introducing this new environment, you want to test as much as you can. When it comes to introducing more software in there is when you want to keep maintaining that testing, but as much as you can, as much as possible. I know it’s like if you’re running through a five minute checklist, sure, but testing is like the crucial thing here to make sure that you’re not missing anything. So as much time as you can put into testing your environment when your environment changes is the way to go.

Swapnil Bhartiya: We live in this world of AI. We talk a lot about automation. Where does automation fit into all of this? Can any of this validation happen continuously behind the scene instead of only when somebody remembers to actually check for it?

Alexus Gore: With the QA cluster that you can build before you go into production, automation can be used to automate that testing for a QA cluster continuously in the background to catch things beforehand before they’re even caught in a production cluster. So you can use automation to kind of catch what’s going to happen before it happens.

Swapnil Bhartiya: Can you also share your thoughts? What are some of the really crucial things when it comes to testing this checklist that they should not miss?

Alexus Gore: One of those, the biggest thing that you should not miss when it comes to testing is first of all with creating a runbook, which is going to have the list of steps that you ran during your testing to kind of keep track of everything that happened. You want to make sure that everything that you tested was documented in the event that a situation happens. So you can kind of trace back your steps to see what could have been in place to prevent something from happening again. Runbooks, I think alongside QA clusters, runbooks, I feel are the most crucial thing. A lot of the times we’ll have events or situations that have happened and we’re asking customers, oh, what happened when this occurred? And it’s like, I don’t really know what happened when this occurred or I don’t really remember what I did. But having those set of steps in place, having documents and information is the most crucial when it comes to failover validation and failover testing.

Swapnil Bhartiya: What kind of culture should organizations build? It could be from top down or within the engineering community that this is not seen as someone else’s problem. It should not be an afterthought, it should be the core part the moment you write your application and deploy it. High availability should be part of that. So what is your advice there?

Alexus Gore: I would say to definitely build trust with customers and making sure that there is a lot of communication that happens between you and the customers with making sure that, or at least showing the customer how you are building the trust with improvements that you are going to make to your product. Because obviously no one likes when failovers happen and you lose money. So making sure that you build trust with customers, making sure that you improve your product upon failures that have happened with your product is the biggest way to improve the culture within your organization.

Swapnil Bhartiya: Alexis, thank you so much for joining us and sharing these insights. And for anyone who is watching, to make sure their failover actually works when it really matters, please check out SIOS Technology and the solution that they’re offering. Once again, Alexis, thanks for joining me and I look forward to chatting with you again.

Alexus Gore: All right, thank you for having me.