By Two replication or failover tools can each pass every internal check and still fail together the moment a real network outage hits. The incompatibility lives at the kernel level, below the visibility of standard log analysis, and it stays hidden until production exposes it at the worst possible time.
In this interview on TFiR, Alexus Gore, Customer Experience Software Engineer at SIOS Technology, walks through three real customer cases where a combined DR configuration appeared sound on paper but collapsed during a live outage, and explains the pre-production testing discipline that would have caught the gap.
Guest: Alexus Gore, Customer Experience Software Engineer at SIOS Technology
Show: TFiR
Here is what every platform engineer and DR architect needs to know.
Technical Deep Dive
Q: Why do disaster recovery plans that pass all log checks still fail during a real outage?
Alexus Gore, Customer Experience Software Engineer at SIOS Technology, explains that each tool in a combined DR stack can operate correctly in isolation while harboring a kernel-level incompatibility with the other. Standard log review does not surface this conflict because both products appear functional when tested independently. The incompatibility only becomes visible under live failure conditions, specifically when a network outage forces both tools to operate simultaneously in a failover scenario.
“Looking through the logs, you couldn’t really see where the issue lied. You would have to go way deep into the kernel code to really figure out what’s happening.”
Alexus Gore, Customer Experience Software Engineer, SIOS Technology
Q: What actually happened in the three customer cases where the combined DR configuration failed?
In all three cases, customers were running SIOS DataKeeper alongside a second failover or replication solution. The combined configuration was reviewed and the logs showed both products working without errors. The failure occurred when a network outage triggered the failover path: the secondary solution did not execute, exposing the incompatibility that had been invisible during routine checks. Gore notes that each customer had tested their tools, but always independently, never together under failure conditions.
“Both solutions were working fine. There were no issues. The problem happened when a network outage occurred and the failover solution did not work.”
Alexus Gore, Customer Experience Software Engineer, SIOS Technology
Q: How does running two replication or failover products together create a compatibility risk?
Technical incompatibility between tools that interact at the kernel level is the core risk. Each product is engineered and validated independently by its vendor. When a second replication or failover solution is added to a stack that already contains DataKeeper, the interaction between the two at kernel level may produce conflicts that neither vendor’s standard testing surface. Gore identifies this pattern across all three customer cases as the shared root cause.
“Technical incompatibility is one of the themes here when I’m talking about these three customers.”
Alexus Gore, Customer Experience Software Engineer, SIOS Technology
Q: What specific testing steps would have prevented these DR failures before production?
Gore is direct: the configurations needed to be tested together, not separately, before going into production. This means simulating the actual failure scenario, including a network outage, with both solutions running simultaneously in the same environment used in production. Testing each product in isolation and reviewing logs for errors is not sufficient to validate a combined stack. The precautionary step is to force the failover path to execute under controlled conditions before any live workload depends on it.
“Making sure that you test before you go into production is the biggest aspect of this.”
Alexus Gore, Customer Experience Software Engineer, SIOS Technology
Resources & Documentation
- SIOS DataKeeper, block-level replication and high-availability clustering software for Windows environments
***
👇 Click to Read Full Raw Transcript
Swapnil Bhartiya: Is it possible for you to share a real world example where a company believe, we don’t have to name the company, that its disaster recovery plan was ready only to discover a critical gap during testing or worse, during an actual outage? And through that, there was a lesson that was learned and that lesson can be shared with others as well.
Alexus Gore: Yeah. So there are actually three occurrences that I can think of where this happened. Both of these customers, or all three of these customers, have experience with using DataKeeper. And all three of these customers also wanted to use another failover solution or another replication solution as well, alongside Datakeeper and with the systems that they provided us, with the logs that they provided us, it looked like it worked. And so everything was in place. Both solutions were working fine. There were no issues. The problem happened when a network outage occurred and the failover solution did not work. And it’s something that could have been caught if the customer had taken the steps, the precautionary steps into testing their solutions together. Technical incompatibility is one of the themes here when I’m talking about these three customers. And the lesson learned here is that it’s important to test. But I do want to mention that, oh man, it’s like their configuration did look like it worked because they used both products independently, so putting them together was where they thought there would be success. But unfortunately, because there wasn’t any testing involved, it just was a catastrophe and it was missed in the beginning. Like, I think looking through the logs, you couldn’t really see where the issue lied. You would have to go way deep into the kernel code to really figure out what’s happening. So, yeah, making sure that you test is the, is the biggest aspect of this, and making sure that you test before you go into production as well.
