AI Infrastructure

When to Remove Humans from AI Agent Loops | Jean Lafleur, Airbyte | TFiR

0

Enterprises deploying AI agents face a threshold problem: human review adds latency and cost, but removing it too early introduces operational risk. Most teams lack a principled framework for deciding when a specific agent action has proven reliable enough to run without oversight. The result is either perpetual human bottlenecks or premature autonomy that leads to silent failures in production pipelines.

In this interview on TFiR, Jean Lafleur, Co-founder and Chief Operating Officer at Airbyte, walks through how Airbyte built and operates a staged confidence model for AI agent autonomy in its own connector maintenance pipeline.

Guest: Jean Lafleur, Co-founder and Chief Operating Officer at Airbyte
Show: The Agentic Enterprise
LinkedIn: Jean Lafleur
Watch the discussion on YouTube

Here is what every platform engineer and AI automation team needs to know.

Technical Deep Dive

Q: When should enterprises start removing the human in the loop from AI agent workflows?

Jean Lafleur, Co-founder and Chief Operating Officer at Airbyte, says the default starting position for any agentic deployment is mandatory human review. Only after an AI has demonstrated consistent, error-free performance on a specific category of task should the human gate be removed. The key is narrowing the scope: rather than granting broad autonomy, teams should identify the precise issue types where the AI always works correctly and remove oversight there first.

“At the beginning, you always want a human in the loop. Once you start building confidence, you can remove the human from the loop.”

Jean Lafleur, Co-founder and Chief Operating Officer, Airbyte

Q: How does Airbyte apply the human-in-the-loop model to AI-driven connector maintenance?

Airbyte built an internal AI factory to handle connector maintenance for its data pipelines. In the early phase, every AI-generated pull request required a human reviewer before merge. Over time, as the team accumulated performance data on specific issue categories, they were able to identify cases where the AI consistently produced correct results and began removing human review for those specific cases. Lafleur notes that maintaining data pipelines is hard, which is why the confidence-building phase takes time.

“We’ve built an AI factory for that. Initially, the AI would submit a PR, but a human would always be in the loop to review it. Over time, as we build confidence, we continue to improve the process.”

Jean Lafleur, Co-founder and Chief Operating Officer, Airbyte

Q: Is the staged autonomy model specific to Airbyte or representative of how most enterprises are approaching agentic AI?

Lafleur frames the Airbyte connector maintenance workflow as representative of a broader enterprise pattern rather than a company-specific approach. The staged model, starting with full human oversight and progressively removing it as confidence accumulates, reflects how most organizations are navigating the trust problem with AI agents in production. The implication is that teams should expect this process to take time and should not treat the initial human-review phase as a failure of automation.

“I think that’s pretty representative of what’s happening in every enterprise. In the beginning, you always want a human in the loop.”

Jean Lafleur, Co-founder and Chief Operating Officer, Airbyte

Resources & Documentation

  • Airbyte, open-source data movement platform for building and maintaining ELT pipelines at scale
  • Airbyte on GitHub, source code, connectors, and contribution docs

***

👇 Click to Read Full Raw Transcript

Swapnil Bhartiya: Can you also talk about, of course, human oversight? How are you thinking about the balance between autonomy that we want to give to agents? Because that’s the whole point of automation. At the same time, keep humanly loop. Human oversight is also there because this is a complex as we are giving them more access. More Right, access. How do you balance that human versus autonomy?

Jean Lafleur: I would say let’s take an example. At Airbyte for instance, a lot of our connector maintenance issues right now is done through AI and we’ve built a AI factory for that. And it’s taking a long time because maintaining pipelines data plan is hard. But what we do is at the beginning this AI would submit a pr, but there’ll be a human in the loop to check the pr always. Now then with time we build confidence, we make it better. We see that for certain issues it always works perfectly. At that point we can remove the human in the loop. And I think that’s pretty representative of what is happening in every enterprises. At the beginning you always want a human in the loop. And once you start building confidence, then you can remove that human in the loop. But most likely you will start with a human in the loop.

How to Run Payee Verification Before a Wire Is Sent | Damien Dugauquier, IPID | TFiR

Previous article