Guest: Neil Palmer (LinkedIn)
Company: Qarik Group (Twitter)
Show: TFiR: T3M

There has been an explosive growth in the amount of data that is being generated and accessible to organizations. Companies are still struggling with cataloging, accessing, and actually proving the value out of data. In this episode of TFiR: T3M, Neil Palmer, Partner at Qarik Group, shares his insights on the current data trends in the market, how companies are evolving in the way they store and consume data, and how Qarik is helping them through it.

Current trends in the market:

• The consumption of data across any given organization and through to its third-party providers or its third-party clients has become more and more critical.
• There are initiatives about being able to share datasets across large organizations, controlling both accessibility and security around the datasets that are being consumed within an organization.
• The economic value of the information that you’re trying to either produce or consume from the system you’re building is emerging as a new dimension to systems engineering and systems design. Having organizations being able to think about what the economic benefit of that information is to them across the board, allows them to put a cost benefit analysis over both storing and consuming that data.
• There is a pivot away from traditional backup and data recovery to a viewpoint around resiliency and scalability within the applications.
• A lot of the cloud-native applications bake in or improve on the need for DR by using multi-regional storage for their data. Instead of just doing backup, they gain the benefit of scalability and resiliency within their applications.
• Application developer teams are now conscious of data — where they’re consuming it from and where they’re providing it to — from a security and availability perspective.
• The Snowflake dataset sharing solved a lot of problems by providing essentially access controls around what was previously a series of fairly complex batch jobs that would run in obscure times of the day and the scripts were understood by one or two people. From a security perspective, you’ve got easily controllable access to the data that your clients or your providers need.
• There is far less resistance to the idea of having multi-regional datasets within systems. For production workloads, multi-region is the de facto requirement, whether it’s within the US or on a global basis, which brings in broader questions around data protection and legality.
• Testing and making sure that the systems work on the multi-regional capability is a different matter. The advent of chaos engineering has helped organizations embrace that concept of failure from the get-go.
• The need to test for resiliency from the very beginning, rather than using a traditional disaster recovery mode where you unplug stuff once a year or once every six months and hope that it still worked.
• There is a lot of pressure within the current economic environment to be constantly delivering business value, which tends to mean new features and new capabilities. Addressing the organization’s technical debt gets lower priority. More organizations are seeing the need to consistently invest in resiliency engineering. Some even have initiatives such as 20% of every sprint is invested in resiliency engineering.
• Qarik is focused on app modernization and platform engineering to help organizations move from data-center-native to cloud native. It creates resilient platforms and pipelines that developers can use to build out their systems and applications with accessibility to Kubernetes clusters that have the appropriate level of scalability, failover, CI/CD pipelines, data tagging, and audit trails.
• Making these tool sets and putting them in the hands of the developers help them to focus on business capabilities as well as the capabilities they need around resiliency and auditability, particularly in a highly regulated environment.

This summary was written by Camille Gregory.