By Last year in October, Bloomberg and Tetrate joined forces to create a community-led set of core AI gateway features for enterprise AI integration. This collaboration has now resulted in the first release of Envoy AI Gateway, an open source project that aims to improve enterprise AI scalability and governance. The gateway provides a standardized approach to managing AI infrastructure, ensuring compliance while simplifying developer workflows.
The initial release (v0.1) introduces a unified application programming interface (API), upstream authorization for centralized credential management, and usage rate limiting to enforce governance policies. By addressing scalability and security concerns, Envoy AI Gateway helps organizations manage AI workloads more efficiently. Future enhancements will expand API integrations, refine model fallback mechanisms, and introduce semantic caching for cost efficiency.
📹 Going on record for 2026? We're recording the TFiR Prediction Series through mid-February. If you have a bold take on where AI Infrastructure, Cloud Native, or Enterprise IT is heading—we want to hear it. [Reserve your slot
Tetrate is a major player in the CNCF ecosystem making open source projects like Envoy ready for enterprise production use. The release of Envoy AI Gateway in collaboration with Bloomberg is a testament to that commitment to open source and Envoy.
Why do we need Envoy AI Gateway?
Key enterprise AI requirements that shaped the project included scalability, multi-tenancy, and developer experience improvements. David Wang, Head of Product Management and Product Marketing at Tetrate, tells us that the ability to manage shared infrastructure across teams, simplify access through a unified API, and ensure visibility into usage were central to Bloomberg’s needs. Additionally, governance features, such as upstream authorization and rate limiting, were designed to address compliance concerns and enable centralized oversight of AI workloads. As Wang notes, “These three things overall improve developer experience and enhance the ability to centrally govern AI usage.”
The initial release of Envoy AI Gateway delivers three critical functionalities: a unified API that integrates with AWS Bedrock and OpenAI, upstream authorization to centralize credential management, and usage rate limiting for governance and cost control. These capabilities aim to allow enterprises to scale AI services while maintaining control over security and resource allocation.
Both companies helped shape these features by contributing to the development and working closely through code reviews and design discussions. The project also encourages open source community participation, with vendors and end users actively involved in roadmap planning and implementation.
Tetrate plans to extend the unified API to additional AI models such as Google Gemini. The company is also working to introduce advanced fallback logic for model availability and implement semantic caching to optimize resource usage and cost management. Wang explains that these enhancements will further refine Envoy AI Gateway as a scalable, enterprise-grade solution for AI infrastructure.
Wang believes that as AI adoption grows, governance and compliance challenges will further intensify. While Envoy AI Gateway focuses on traffic management and security, broader concerns such as tracking AI usage, ensuring regulatory compliance, and forecasting ROI remain key industry challenges. Wang underscores the need for open source collaboration and ongoing contributions to address these complexities in the evolving AI landscape.
Guest: David Wang (LinkedIn)
Company: Tetrate
Show: An Eye on AI
This summary was written by Emily Nicholls.
