AI Infrastructure

How to Fix Stale Warehouse Data in AI Agent Answers | Ian Alton, Airbyte | TFiR

0

A data warehouse that syncs every 15 minutes is always a little behind. When an agent has to answer a question about something that happened a minute or a second ago, the warehouse alone cannot give a current answer. The fix is to let the agent reach the source systems too, not only the synced copy.

In this interview on TFiR, Ian Alton, Product Manager at Airbyte, breaks down how an agent can query a warehouse in plain language and combine it with source data for answers that are current at the second you ask.

Guest: Ian Alton, Product Manager at Airbyte
Show: The Agentic Enterprise
LinkedIn: Ian Alton
Watch the discussion on YouTube

Here is what every data engineer and platform engineer needs to know.

Technical Deep Dive

Q: How do I fix stale warehouse data in AI agent answers?

Ian Alton, Product Manager at Airbyte, explains that a warehouse holds the majority of the data an agent needs, but time-based questions can need an up-to-the-minute or up-to-the-second answer. Even a warehouse synced every 15 minutes can miss something that happened a minute ago. Because the Airbyte MCP holds the credentials and access to the source, Alton says the agent can stitch together a complete answer that is current at the moment of the question.

“Airbyte has a lot of advantages in this case because we hold the credentials and have access to the source, allowing us to stitch together a very complete answer that is perfectly up to date at the exact moment you ask the question.” Ian Alton, Product Manager, Airbyte

Q: Can an AI agent combine data warehouse data and source system data in one answer?

Yes, and Alton says that is often what the agent will want to do. Airbyte securely stores credentials for the source data, meaning the systems where data comes from before it reaches the warehouse. Queries are therefore not limited to the warehouse, and an answer can be complete even when the latest sync has not caught up.

“In many cases, what the agent will want to do is stitch together an answer that consists of both warehouse data and source data.” Ian Alton, Product Manager, Airbyte

Q: How does an AI agent query a data warehouse in plain language?

Alton describes a capability delivered through an evolving Airbyte feature called the context layer. A user can ask a question such as how many orders came in through the POS system last week. The agent uses the MCP to write the SQL, runs it on the Snowflake destination, gets the data it needs, and returns the answer in plain language.

“The agent can then use the MCP to write the SQL, run that SQL on your Snowflake destination, get all the data that it needs, and then return an answer to you in plain language.” Ian Alton, Product Manager, Airbyte

Q: Is it complicated to set up an MCP for a data warehouse?

It does not have to be. Alton notes that a lot of people have complained that setting up the BigQuery MCP is super complicated. Because Airbyte already stores credentials securely, no additional configuration is needed, and a user only enters an email address and password.

“With Airbyte, it’s a simple one-step thing. Put in your email address and password, and you’re good to go.” Ian Alton, Product Manager, Airbyte

Resources & Documentation

  • Airbyte MCP, Airbyte’s Model Context Protocol offering for connecting AI agents to data

***

👇 Click to Read Full Raw Transcript

Swapnil Bhartiya: Can you also talk about the kind of blocker when it comes to query a data warehouse in natural language and how Airbyte is addressing that irrespective of whether you’re Claude, ChatGPT? Talk about that aspect and we can also get into weeds later on what happens when a user actually asks a question there.

Ian Alton: Yeah, definitely. So this is a capability that the MCP has through an evolving feature that Airbyte calls the context layer. You can ask a question like, how many orders came in in my POS system last week? The agent can then use the MCP to write the SQL, run that SQL on your Snowflake destination, get all the data that it needs, and then return an answer to you in plain language. Now, the agent, you know, we— Airbyte already stores securely your Snowflake destination, get all the data that it needs, and then return an answer to you in plain language. Now, the agent, you know, we— Airbyte already stores securely your credentials. So this is really easy to set up. You don’t need any additional configuration. So if you’ve ever tried to set up, for example, The BigQuery MCP, I know a lot of people have complained that that’s super complicated to do. With Airbyte, it’s a simple one-step thing. Put in your email address and password, and you’re good to go. Now, querying the data warehouse is great because in a lot of cases, it does have the majority of the data that you need. But what happens when you ask a question that’s time-based where you need an up-to-the-minute or up-to-the-second answer? Well, Airbyte also securely stores credentials for your source data. So where that data is coming from to get into the warehouse in the first place. So these queries aren’t limited to the warehouse. In many cases, what the agent’s going to want to do is to stitch together an answer that consists of both warehouse data and source data. So if you’re syncing, you know, even if you’re syncing your data 15 minutes, every 15 minutes, and it’s super fresh, What happens when you need something that happened a minute ago or a second ago? So Airbyte has a lot of advantages in this case because we hold the credentials and we hold access to the source, so we can stitch together a very complete answer that is perfectly up to date at the second that you ask the question.

How to Find a Clean Recovery Point After Ransomware | Jim McGann, Index Engines | TFiR

Previous article