Author: Kendall Miller, Chief Extrovert at Axiom
Bio: Kendall is a bubbly extrovert in a technical world preaching a life of living, laughing, and logging at Axiom.

We live in a world today where we’re more capable of collecting data on just about everything our products do than we could have ever imagined. Manufacturers of hammers could never have imagined being able to measure how many people stood in front of the hammer aisle and looked at their hammer before purchasing, let alone how many reached out their hand for their hammer but bought a competitor’s. They couldn’t imagine knowing every detail of every hammer manufactured, with detailed logs pointing out flaws in the handles or the metal. And they never would have had the ability to A/B test different hammer models with a few clicks of a button.

Modern SaaS applications, e-commerce and just about everything done on the internet is capable of collecting and observing all of this. We can know where a user’s cursor was the moment they arrived on our website, and optimize our call to action buttons to be 2% easier to click. We can follow every bit. Know every single time a customer has a bad interaction with our chat bot, every single time a credit card fails, every time our app takes a few seconds too long for a customer to open and they abandon it as a result. The mountain of data is growing, and fast. Infrastructure data is likewise proliferating with microservice architectures and cloud native tooling.

Missing out on the opportunity

But companies are missing it. Instead of collecting this data today so they can draw conclusions about it tomorrow, they throw it away.

That’s because they mistakenly think of most data as ephemeral and lacking value. They may have thought the same of Reddit posts and microblog rants. Yet in retrospect, that authentic behavior of humans at massive scale was invaluable to the development of today’s AI models. Likewise, today’s proliferation of event data is absolutely essential to every company that hopes to build a better version of itself tomorrow.

Most companies that operate at scale today throw away huge amounts of their event data — it’s an industry anti-pattern — because they still rely on outdated architecture and generation-old tooling to capture, store and query it, thus making it prohibitively expensive. But this is almost like Wikipedia throwing away nine out of ten words to keep costs down and then hoping to be readable as a source of human knowledge.

Sampling

Thankfully Wikipedia went the other way and kept everything for transparency. As a result, AI models can now answer a shocking amount of obscure history and science questions because they’ve memorized the online encyclopedia. In a few years it will be just another day at work to feed your event data into an AI model and get insights like, “Move this button to the left and conversions will increase by 5%.” Or, “Customers often sign up for your software, then cancel after they attempt to use this one feature.”

With AI digging through our event data we wont have to run the queries ourselves to find meaningful insights— always a problem, because it’s hard to know what to ask. AI will expose things we never imagined about user attention, sales, retention, etc…

AI trained at scale on sales data, marketing data, user behavior and even cloud infrastructure data will tie sales contract numbers to the number of touches in marketing. It will tie retention rates to rolled-out features and even specific deployments. You’ll know which users stopped using your product when you updated its font. Or that you lost exactly 1.2734 million in ARR because of your logo refresh. You’ll know that ten lines of code no one thought important directly contributed to this year’s company-wide bonuses.

Scale

Event data is the most important data your company collects, because at scale it tells the whole story of the entire organization.

When you were running your application on three physical servers in your closet rack, you didn’t collect enough information to make meaningful insights. With hundreds or thousands of events per day, you lived in a world without enough data to feed to an AI engine . But today’s infrastructure runs microservices at a scale producing hundreds or thousands of events every second, where every communication between API can be logged, traced, and recorded. Today every company is plugged in to marketing and sales analytics tooling that know every single touch a user made before a buy, and every single time they used the product after. Every customer is sending enough data to make it feasible for solutions customizable to a single user.

The decisions your c-suite needs to make tomorrow absolutely depend on keeping every bit of your data today.

The future of business intelligence is event data. The future of SaaS, e-commerce, social media, entertainment and everything else that touches the internet is event data. In the next three to five years every company will be feeding this data into AI systems to leverage insights previously impossible to find, pushing optimization to new extremes.

The companies that keep the most the data will have the biggest lead. Stop throwing away your most valuable asset.

To learn more about Kubernetes and the cloud native ecosystem, join us at KubeCon + CloudNativeCon North America, in Salt Lake City, Utah, on November 12-15, 2024.