Swapnil Bhartiya: Is it possible for you to share a real world example where routine HA testing uncovered a hidden issue that would have only surfaced during an actual outage and would have actually caused downtime and chaos?

Cassius Rhue: Yeah, we actually have. I have two in fact. One is very recent, as recent as earlier this week. So in the first one, we were actually working with a partner who was setting up a system for an end customer. And they engaged with us to do some testing to make sure that after their installation and updates that the system was working correctly. And our very first validation test, we discovered that one of the protected services was misconfigured. In this particular case, the service was set to what’s called automatic start. So it actually would start outside of the HA solution, and when that occurs, the data that it would serve up would be stale data because it was not a part of the HA monitoring and recovery. By detecting that early, we were able to go in, change the configuration so that it would not start outside of the HA solutions control, and make sure that it was always going to be pointing to the latest set of data if that had been found during a real world scenario. Just think of how long it may have been before the end customer discovered that they were receiving stale data. Or in some examples related to this particular environment, the client application can start in a way in which it no longer allows anyone else to access the data. And so finding that prevents an outage, it prevents an error, and it prevents data integrity issues because you’ve detected it early, you’ve corrected the configuration, and then you went on to further validate that things are working properly with the HA solution. You know, a second one related to that, to a client realizing that they had configured all of their client software. So our end customer had an HA solution in place. Using our sios lifekeeper high Availability solution, they were protecting several databases and applications. Their clients, however, were configured originally to point to the physical server IP address. And so once the application services and databases failed over to the backup, those clients could not connect because they had been hard coded. Testing and validating in the customer’s environment helped us find that early, make that a change and adjustment, and then get things corrected and retested in a way in which the failover and client reconnectivity became seamless. And that’s something you don’t want to find during an actual outage, because of course, when there’s an outage, you’re having to troubleshoot, you’re having to do a lot more root cause analysis, you’re opening tickets with multiple Vendors, you’re trying to coordinate understanding and what are the symptoms and signs and problems. And one thing that often gets overlooked, if the applications or databases are up and running, then you’re having to go look and see, okay, what happened with clients. And that and that panic and chaos and that moment where you are scrambling is something that can be missed. So testing in real world scenarios helped us find that before it became an issue in an actual disaster. And then one. Recently we had a client that did an upgrade of their primary system, but forgot to actually do the upgrade on the backup system. And they called us in because they wanted to do some testing. And what we found is that there was a configuration file that needed to be updated on both servers. And that was something that we needed to make sure that they were able to run through. That allowed us to find the issue, update the configuration file, and also update their runbook. So now going forward, they know the procedures and steps they have to do for updates of the operating system, updates of their critical database, updates of their client application, and they know how to test it in a proper manner and where to validate that things were done correctly. And so that saves them a lot of time. Of course, if you’re in an actual disaster, you’re looking at okay, well why didn’t this work? And you’re starting with troubleshooting steps. And often these are the little nuances that you might miss. While there are 21 executives wondering why their business is losing money by the second.

Swapnil Bhartiya: Now let’s flip this coin. Can you walk us through a scenario where an HA environment look perfectly healthy, but the absence of routine testing led to real problems when a failover event actually happened.

Cassius Rhue: That’s a great question, and it’s a very vivid example in my mind because of how many different organizations were involved in this particular incident and how long it took to resolve and remediate the problem. And the fact that it was a business critical system in a live production down eventually. So we did have a client who set up their system with our software and they went through some what they considered to be routine maintenance. It was actually turned out to be more than routine maintenance, but for time constraint pressures or business pressures or stakeholder involved pressure, this particular client opted first to decline services that we offer for validation and and testing. And then second, they also declined to do that testing themselves to make sure during their maintenance window that the failover would work or that the applications would run on the backup. A third thing that we’ve kind of touched on earlier, they also did not have a test environment where they could run these updates prior to rolling them out to production. And so all of the updates were made on their production system. Things appeared to be healthy on the production node. They had an outage, a power outage in their data center. Things attempted to fail over to the backup server. And that’s when they discovered that there were, during the upgrade of the primary, there were some configuration changes that were made. There were also some libraries and packages that had been updated to a newer version that did not exist on their target system or their secondary. And so when it came time to start the database, the database failed with some errors related to those updated packages that were missing the application. When they were able to troubleshoot the database and get it to a running and working state, the actual application failed because there were some configuration parameters that had been made on the primary and stored locally in a config file had not been replicated or synced to the backup server. And so in a scramble, you have a team member from their organization, from the application team trying to troubleshoot what’s going on, why the application won’t start. A lot of times you have in an emergency like that, you may not have the architect of the application available. And so you’re dealing with an on call engineer from the app team, an on call engineer from the database team. You may be dealing with folks that are not as familiar with the system. And in that emergency situation, this particular customer was trying to figure all of these things out at one time. Ultimately, their resolution was to update the packages on that target system, update the configuration file, and update the database so that it would all start and run that outage. Instead of it being a simple failover. Power failed in the primary data center. Everything starts up seamlessly on the backup without clients noticing it. Instead, that turned into a major outage event and cost them significantly. Whereas a routine test would have identified all of these issues and would have saved the business a lot of money.