Hydrolix is a real-time streaming data lake designed to manage massive log volumes efficiently. By leveraging decoupled storage and compute, the platform enables organizations to process logs at scale while optimizing both cost and performance. In this show, Todd Persen, CTO of Hydrolix, talks about the company’s approach to handling large-scale log data, the challenges of managing its rapid growth, emerging AI-driven use cases, and Hydrolix’s vision for the future of big data management.

Observability data volumes are increasing rapidly, creating cost and storage challenges for organizations. Some customers who are now managing hundreds of terabytes to petabytes of logs daily are forced to decide whether to retain this valuable data. Persen says, “One area we’re looking for innovation in this space is how we can handle larger data volumes more efficiently.” Hydrolix aims to address this issue by improving storage efficiency and ensuring businesses can extract value from logs in areas like business intelligence (BI), security, and AI training.

Log volumes are expanding due to increasing data from content delivery network (CDN) logs and infrastructure logs in containerized environments like Kubernetes. Persen notes that while AI and generative AI (GenAI) rely on these logs for training and analytics, the primary challenge is managing their scale efficiently. While tools like Elasticsearch have historically been useful, their complexity and high operational costs have led many organizations to seek alternatives. Hydrolix aims to provide a more scalable and cost-effective solution, allowing businesses to handle real-time log processing without the overhead of maintaining legacy infrastructure.

To address the future of big data management, Persen discusses Hydrolix’s innovative use of object storage as a primary data store. Persen explains how this eliminates the need for separate hot and cold storage tiers, providing on-demand access to vast amounts of log data while optimizing compression and network access. The shift towards object storage also allows organizations to scale seamlessly, reducing both cost and complexity in managing growing data volumes.

Hydrolix’s plans for the future will focus on expansion, including partnerships with Amazon Web Services (AWS), Google Cloud Platform (GCP), and other cloud providers. The company is also exploring deployments in air-gapped environments for security-sensitive organizations and enhancing integrations with analytics platforms like Apache Spark and Splunk. By continuing to push technical innovation, Hydrolix aims to provide enterprises with a flexible, scalable, and cost-efficient log management solution.

Guest: Todd Persen
Company:  Hydrolix
Show: Let’s Talk

This summary was written by Emily Nicholls.