AI Agents for Data Pipeline Automation | Ian Alton, Airbyte | TFiR

Swapnil Bhartiya: As it is quite obvious that today, every Data team has AI assistants like Cloud or ChatGPT sitting on their desk. But those assistants still cannot see what’s actually inside your warehouse, because that connection becomes the problem. Meanwhile, if you look at the other side, the same team runs hundreds of pipelines on Airbyte, which syncs data from more than 700 connectors. And the problem is that when one of those syncs break, someone still has to go read the logs by hand. The pattern you’re seeing is that even if AI is there, it can do a lot of things autonomously. There are things that break and human get involved there. Now, AirByte, of course, has MCP servers. They have a new MCP server that changes both of these things at once. It lets agents quickly query live data in plain language, and then it lets them monitor, diagnose and fix the pipeline themselves. And to walk through how that works, today we have with us Ian Elton, product manager at AirByte. Ian, first of all, it’s great to have you on the show.

Ian Alton: Thanks for having me, Swapnil.

Swapnil Bhartiya: It’s my pleasure. Of course, everybody knows what is Model Context Protocol. Our audience, they also know the work that you have done in space. But just to do a quick, quick recap, if you look at how AirByte started, how it’s helping, let’s talk about how is AirByte leveraging MCP? And then we’ll talk about this new announcement.

Ian Alton: So, the MCP is the Model Context Protocol, and basically it gives AI agents access to external tools beyond what those models are capable of. You can think of it like a USB for AI. So AirByte has always been about moving data, and specifically about breaking down the sort of brittle data pipelines that tend to originate with homegrown solutions and providing more robust and standardized infrastructure for moving your data. So the MCP that we’ve released is essentially the means by which you can turn something like Claude or ChatGPT into the interface that you use to run Airbyte. So it connects your AI agents to your Airbyte organization. It’s very easy to sign in. You sign in the same way that you sign into AirByte today, and your assistant can do all that work for you. Why this matters to Airbyte, essentially, almost everything that you can do in Airbyte today, you can now do with the MCP. It gives you access to those same 700 connectors that we already provide. And the agent can use those, connect can use those connectors and build those data pipelines directly. It can build them, it can run them, it can fix them, and it can read data all from a chat, all with the benefit of a nice natural language interface.

Swapnil Bhartiya: Can you also talk about the kind of blocker when it comes to query a data warehouse in natural language and how Airbyte is addressing that irrespective of your cloud chatgpt? Talk about that aspect. And we can also get into weeds later on. What happens when a user actually asks a question there?

Ian Alton: Yeah, definitely. So this is a capability that the MCP has through an evolving feature that Airbyte calls the context layer. Once an admin turns on the context layer, the Airbyte MCP, the same one that’s building your data pipelines, can also make queries into your data warehouse. A great example of this would be with something like Snowflake, you can ask a question like how many orders came in in my POS system last week? The agent can then use the MCP to write the SQL, run that SQL on your Snowflake destination, get all the data that it needs, and then return an answer to you in plain language. Now, the agent, Airbyte already stores securely your credentials, so this is really easy to set up. You don’t need any additional configuration. So if you’ve ever tried to set up, for example, the BigQuery MCP, I know a lot of people have complained that that’s super complicated to do with Airbyte. It’s a simple one step thing. Put in your email address and password and you’re good to go. Now, querying the data warehouse is great because in a lot of cases it does have the majority of the data that you need. But what happens when you ask a question that’s time based, where you need an up to the minute or up to the second answer? Well, Airbyte also securely stores credentials for your source data, so where that data is coming from to get into the warehouse in the first place. So these queries aren’t limited to the warehouse. In many cases, what the agent’s going to want to do is to stitch together an answer that consists of both warehouse data and source data. So if you’re syncing, even if you’re syncing your data 15 minutes every 15 minutes and it’s super fresh. What happens when you need something that happened a minute ago or a second ago? So Airbyte has a lot of advantages in this case because we hold the credentials and we hold access to the source so we can stitch together a very complete answer that is perfectly up to date. At the second that you ask the question.

Swapnil Bhartiya: And can you also talk about. You also have its own context layer, which lets agents search by meaning, just not by keyword, to give more context in real time. How does that work technically? And how is it different from regular text search?

Ian Alton: Keyword searches are great, and I’m reminded of all the times in my career when knowing how to look for something on Google made me a lot better at my job. But what happens when you need to search and you don’t know the right keywords to use? So a good example of this is you’re syncing your support data and you want to know, tell me about customers who are unhappy with billing. Well, what happens when the text in the ticket says, I was charged twice? The keywords don’t match, but the meaning is the same. So when we turn indexing on for a connector, Airbyte will index those records as they sync, and we’ll build a keyword index, and we’ll also build a semantic or vector index. The agent can then search based by keywords, but it can also search based on meaning. So we will embed and chunk the source data as it moves through Airbyte’s system, and then at query time, we’ll use the same method to embed your query. We’ll compare it with what’s in the semantic index, and Airbyte will return to your agent a record that is both the actual text that it needs to return, as well as a ranking score so that the agent knows how relevant that result is.

Swapnil Bhartiya: Excellent. And if I’m not wrong or listening to you, and of course, based on previous conversations, it’s not just reading data. You know, it can actually monitor things, diagnose failures, and even help resolve issues. So it’s not just monitoring. Tell us how that works and how far the agent can actually go on its own. And when human intervention is necessary or needed.

Ian Alton: This is the other half of the Airbyte MCP. It works today across both open source and our cloud product. You don’t need the context layer to use this, but when we asked our users, many of them told us, when I have a problem with syncs, when my data pipelines break, I’m losing anywhere from hours to an entire day trying to diagnose and fix this problem. So you can imagine how quickly this kind of thing can really stall a project. So take for an example a question like, which of my connections failed in the last day? Well, the Airbyte MCP can tell you exactly what’s failed and not just can it tell you what failed? But it can tell you why it’s failing. So it’s reading the logs, it’s looking at the sync history and it’s coming up with an answer that you can action upon. And in many cases it can actually resolve that for you if it’s something as simple as a settings changed. If it’s an authentication problem, you should not put your sensitive keys into an LLM, but the agent will point you toward a place in Airbyte where you can just go and re authenticate and get that data pipeline running again. Now one of the things people also told us is occasionally when their data pipelines broke, they got paged at 2 in the morning. This is not an ideal situation in any case, but one of the great things about the AirByte MCP is of course you can run agents autonomously. So when you’re in this on call situation like this, the ideal situation is not that you sit down at your laptop and start reading logs and trying to figure out what’s going on. You don’t want to be doing that at 2 in the morning. But with the AirByte MCP, an autonomous agent can actually take that action and find the problem right away. So by the time you get to your laptop, it’s already saying, hey, here’s the issue, I found it. This is what you need to do to fix it. Five minutes later you’re back in bed asleep where you actually want to be.

Swapnil Bhartiya: While we do like to give agents more autonomy, but I mean you are have user, you yourself know they hallucinate, they make mistakes. So people do worry about agents going rogue, deleting or changing things they shouldn’t touch. What guardrails does airbyte put around, what an agent can do and who is actually held accountable when an agent takes an action. Because at the bottom it may say, I can make mistakes, you know, I’m just an AI. But that is not enough.

Ian Alton: I think we should differentiate my answer into two different areas of focus. So the first, let’s talk about hallucination. When I think of an agent that’s hallucinating, I think of an agent that’s starving for context, so it’s unable to see the data that it needs to give you a proper answer. Most frontier models today are very capable of reasoning together a complex answer. But they can only do that if they have the right access to data. AirByte has always been about data access and making that data available to anyone and any place that needs it. AI agents are just one more of those audiences today. On the other half, we have the question of like, what happens when an agent is doing something bad and how do you deal with accountability for this? We designed the MCP from the beginning to respect your permissions in Airbyte. So when you authenticate with the MCP, you are essentially authenticating as your Airbyte self. So that MCP is taking actions on your behalf using the permissions that you have in Airbyte. Your admins in your organization control what you have access to. If you’re a read only user, you can only do read only actions with the MCP. If you have write access only then can the MCP take any kind of write action. We also allow you to decide what agents can actually reach. So even when you turn on the context layer, you’re not turning it on for everything. You’re turning it on on a source by source basis. So you may have systems where it’s totally fine for the agent to see the record. You may have systems with a little more sensitive data, and you actually don’t want the agent to be exposed to that data. Airbyte fully supports that. You can turn this on and off on a per connector basis. And within workspaces, you can also turn this off and on on a per connector basis. So you can have different levels of permission for say, a production environment and a staging environment. In Airbyte, data access is also read only. So we don’t allow agents to write to your warehouse or to update your source records, for example. And of course all credentials are securely stored in Airbyte. They are never exposed to the agent. There’s no worry about third party secrets leaking into an LLM model or anything like that. We also place some basic guardrails on destructive actions. So agents can’t just delete sources or destinations or connections unless it’s been specifically marked. Delete me. So every tool has a label labeling it as a read only tool or a destructive tool. So the agent can take the appropriate action based on what it needs to do. We also offer the ability to lock the MCP down further. We have a number of advanced config options which you can send to your agent through headers in the MCP. And those provide robust control over what the agent is allowed to do. Essentially, we treat the agent like a new team member, least privileged by default.

Swapnil Bhartiya: The MCP service, I’m not wrong, is an open beta, free for all Airbyte users to try out what are still missing and what’s on your roadmap. From here onwards.

Ian Alton: At the time we’re recording this, it’s an open beta. I think by the time it airs it will be in general availability. Everyone can use it today. So all full pipeline management, creating, managing pipelines, creating custom connections, doing everything you would normally do in Airbyte that is available today. Direct data access and queries to your warehouse and over 50 sources are available today. Search should be rolling out very soon. We’re very interested in the context layer. We have a number of ideas going forward and I think you’ll be seeing some very interesting stuff from us in the months to come.

Swapnil Bhartiya: You folks have been running this beta for a while, what kind of feedback you received, which can tell two things, what potential customers or users want to see at the same time. The feedback was, this is where you’re like, this is. I mean, somebody came back and this is the problem that it solved for us. And we were struggling for a bit and you’re like, that’s exactly what we wanted to do.

Ian Alton: Funny thing, the MCP was actually the most requested feature in Airbyte’s history that I’m aware of. So for us it was kind of a natural thing to build it. Generally, the feedback has been extremely positive. We’ve been seeing on a day to day basis almost a doubling every day of the number of tool calls people are making as the MCP makes its way through different organizations. So I think people have pretty quickly adopted it and once they adopt it, they continue to use it and to integrate it more into their workflows. So already we’ve seen a lot of requests for some of the more edge cases in Airbyte. So features that are less commonly used, but maybe people want to do so. This could be, for example, orchestrating a full data refresh in your destination. The MCP doesn’t do this today, but the fact that people are asking for these things is a good sign. It means they want to use it and they want to use it for more and more advanced tasks.

Swapnil Bhartiya: And thank you so much for walking us through this and of course, those who are watching, please, if you want to see the MCP server in action, check out airbyte.com try their open beta. And Ian, once again, thank you so much and I look forward to talking to you again. Thank you.

Ian Alton: Thanks for walking. Take care.