Guest: Marco Palladino
Company: Kong
Show: Let’s Talk

Sometimes a solution doesn’t come from a drawn-out plan, but from what a company views its customers doing in production. Such is the case with Kong and ZeroLB. The company observed how its customers were using their products and the most common use case was with a service mesh to get rid of centralized load balancers. Kong looked at the pattern and realized it was quite monolithic and legacy. They had a centralized load balancer in front of a decentralized architecture, which didn’t seem right on paper or in production.

Because of this, Kong wanted to codify a new way of doing things, which led to ZeroLB.

The ZeroLB pattern came about because Kong saw a single point of failure, an extra hub in the network, which meant they couldn’t introduce enhanced service level security or observability. By using a decentralized pattern, Kong could deploy decentralized load balancers that are close to the actual services. Because these are built on top of open-source implementations, they can be carried across both of their data centers and the cloud (which meant they could work across vendors).

According to Marco Palladino, CTO and Co-Founder of Kong, this new pattern could improve performance by 2x by removing extra hubs in the network. Without those extra hubs, the service request goes directly from one service to another. And since they are built-in with the Kong service infrastructure, they don’t have to use cloud load balancers or enterprise load balancers to perform the functions. Palladino says, “I was speaking with one of our enterprise customers that was able to remove 16,000 elastic load balancers from one of their cloud vendors by leveraging ZeroLB. This gives them more performance at a lesser cost. It’s a no-brainer.”

The Kong load balancer is decentralized, whereas its previous iteration wasn’t cloud native. With cloud-native load balancing, Kong can deploy load balancing features and capabilities on the same sidecar proxies and data plane proxies they run in a service mesh. This type of deployment makes it possible to provide all of the other functionality for their underlying API infrastructure. On top of this, thanks to other products Kong has developed, they can provide an automated method of injecting these sidecars across Kubernetes, containers, and virtual machines. With the help of these sidecars, those services gain enhanced load balancing features Kong calls ZeroLB for zero-trust observability for routing across different algorithms.

This new pattern, according to Palladino, makes it possible for this to be “dynamically propagated now because the requests are going straight from one sidecar to another sidecar. There is no extra hub in the network. This means that we can intelligently route our software, but also in a more performant way. With microservices, speed is a must-have. If microservices are slow, our applications are down in a monolith.” He adds that if the app was down, the monolith was also down. But with microservices if they’re slow, they’re down. So, in Palladino’s view, “being slow is the new down.”

The ZeroLB pattern also allows Kong to build applications in a much more predictable way. With ZeroLB, deployments are lightweight. And because it runs locally as they develop the software, they can better manage it, instruct it, deploy it, and run it. To that, Palladino says, “So today, if I’m deploying on Amazon and I’m using the elastic load balancer that Amazon provides, but then I want to also deploy software on Azure in every major enterprise in the world is multi-cloud, we’re going to have different load balancing behavior across the board because the load balancer that Amazon provides and the ones that Azure provides, or the ones that we have deployed in our data center, they’re not portable across different environments.”

ZeroLB offers predictable behavior because the underlying service mesh supports every cloud, every container, and Kubernetes. And everything works the same way across the board.

Kong has released Kong Mesh 1.4, which has ZeroLB built-in. This new release introduces more load balancing capabilities. There are five different load balancing algorithms that users can enable, based on different traffic paths. Of this new release, Palladino says, “So we may say within this region or this cloud, I want to use Round-robin (RR), but these other traffic requests going to our database should use a different algorithm. And then on top of that, we have introduced a service map, topology view, which is a GUI that allows the user to visualize all the traffic going from one service to another, including the rate of requests, the error rate, and the latency of these requests in such a way that howsoever we decide to connect our services together, we always have a bird’s eye view to make sure that the status is always up and running.”

The summary of the show is written by Jack Wallen

Here is the unedited transcript of the show…

Swapnil Bhartiya: Hi, this is Swapnil Bhartiya and welcome to TFiR Newsroom. Kong recently announced ZeroLB, a modern decentralized load balancing pattern that aims at simplifying load balancing by removing every load balancer that is being deployed in front of individual services and applications. But what exactly is ZeroLB pattern, and why is Kong tackling load balancing at this time? Let’s talk about that. And today, we have with us, once again, Marco Palladino, CTO and co-founder of Kong. Marco, it’s great to have you on the show.

Marco Palladino: Thanks for having me here.

Swapnil Bhartiya: Why suddenly you felt, not suddenly, but that load balancing need to address some of the challenges. Talk about what challenges that were there that led to kind of creation of ZeroLB pattern.

Marco Palladino: Yeah. So ZeroLB really is something that we’ve seen our customers doing in production. It’s not something that we planned ourselves. So we were observing how the customers are using our products, and one of the most common use cases was using the service mesh to get rid of the centralized load balancers. Centralized load balancers, with a [inaudible 00:01:16] load balancer in front of our services, and for each service, there is a load balancer. When we look at the pattern, it is quite a monolithic and legacy pattern. We have a centralized load balancer in front of a decentralized architecture, which doesn’t sound right, and in production also doesn’t look right. There is a few problems that centralized load balancers introduce, and so that’s why we wanted to codify this new way of doing things with the terminology ZeroLB.

Swapnil Bhartiya: Can you go a bit deeper into explaining this pattern, ZeroLB pattern. You did allude to it on how you look at it, but if I want you to just go in a bit detail.

Marco Palladino: We look at the traditional centralized load balancer. And when we look at that, what we’re seeing is a single point of failure, an extra hub in the network, something that, for the most part, it is not very smart, therefore, they cannot introduce enhanced service level security or observability, because they have a very flat view of our services. And also, we typically look at something that’s expensive and something that is not portable. Why do we have it?

With a decentralized pattern like service mesh, we can deploy decentralized load balancers that are close to the services. They are portable because primarily, they’re built on top of open source implementations so they can be carried across both our data centers as well as the cloud. And in the cloud, they can work across every vendor. They are smart because they provide smart security, and intelligence, and observability to every service that we have.

We can improve the performance by 2X by removing extra hubs in the network, so the service request goes directly from one service to another. And they are built in with our service infrastructure, which means that we do not have to use all these cloud load balancers or enterprise load balancers to perform these functions. I was speaking with one of our enterprise customers that were able to remove 16,000 elastic load balancers from one of their cloud vendors by leveraging ZeroLB. This gives them more performance at a lesser cost. It’s a no-brainer.

Swapnil Bhartiya: Can you also talk about what kind of architectural changes they have to make? How does it integrate with service mesh solutions customers are using, and of course Kongs are obviously there, so talk about that as well.

Marco Palladino: You will be is what I call Cloud Native load balancing, our load balancing the one that is the centralized, the one that we’re used to deploying it is not Cloud Native. It really belongs to a different era of software. So with Cloud Native load balancing, we can deploy these load balancing features and capabilities on the same sidecar proxies the data plane proxies that we run in a service mesh to provide all the other functionality for our underlying API infrastructure. These load balancers are decentralized in the case of the products that Kong is working on to [Mount 00:04:31] , which is a CNCF project where one of them Containers as well as [comesh 00:04:35] she enterprise service mesh. It’s built on top of Envoy proxy, and we provide the automated way to inject these sidecars across both Kubernetes, Containers and virtual machines and out of the box, all of the services that have these sidecar next to them, they get these enhanced load balancing features that we call ZeroLB for zero trust security for observability, for routing across different algorithms.

Marco Palladino: Like Round-robin, least connect Maglev and so on and so forth. And all of these is dynamically propagated now because the requests are going straight from the sidecar to another sidecar. There is no extra hub in the network. This means that we can intelligently route our software, but also in a more performance way. In microservices speed is not a nice to have, it’s a must have, if microservices are slow, our applications are down in a monolith. If the app was down, the monolith was down, the application was down, but with microservices, if they’re slow, they are down. So being slow is the new down being slow it’s like being down. So we have to improve that performance. And ZeroLB allows us to do just that.

Swapnil Bhartiya: What are the benefits that ZeroLB provide in addition to these two or three things that you mentioned?

Marco Palladino: Well, it also allows Us to build applications in a much more predictable way. Traditional load balancers, especially the ones that are either enterprise or in the cloud. They’re very hard to provision when we develop our software. Therefore, the load balancing behavior it’s always an unknown until we finally deploy it in our production environment. Whereas, with ZeroLB it’s lightweight, it runs locally as we develop the software, we can manage it, instructed it, deploy it, run it as we build our software. And we can do that in a very consistent way across every infrastructure. So today, if I’m deploying, let’s say on Amazon and I’m using the elastic load balancer that Amazon provides, but then I want to also deploy software and Azure in every major enterprise in the world is multi-cloud, we’re going to be having different load balancing behavior across the board because the load balancer that Amazon provides and the ones that Azure provides, or the ones that we have deployed in our data center, they’re not portable across different environments.

The load balancing behavior, which is on every single execution path of every single API or service traffic requests. It is not predictable is not portable, which is madness. So we think about where the industry is going. And so with ZeroLB, we can have predictable behavior because the underlying service mesh does support every cloud. It supports the ends, it supports Containers, it supports Kubernetes, supports pretty much everything and everything works the same way across the board. And as we’re building our software, we can also provision it locally. So there is no surprises when we run our application in production, which in turn will accelerate time to market and delivery on our applications.

Swapnil Bhartiya: Are there Caveats also that companies who do want to leverage ZeroLB they should be worried about? Though as you mentioned earlier when you embrace data architectures, you’re moving from model it to basically more Cloud Native. But if there are anything that they should be aware of?

Marco Palladino: Well, they should be aware of the existence of a better way of doing the load balancing, which is why ZeroLB something that we are promoting. And again, this is something we’ve been seeing from the actual users and customers. It’s not something that we initially even thought of and then it makes sense reconnecting the dots. The challenges are obviously to enable ZeroLB via a service manager and service manager, by my means, it’s not the only way that ZeroLB we can be implemented, smart clients could be another way, but with a service manager, it’s important that we choose a technology that can allow us to implement that load balancing across both the Greenfield applications, which are going to be running on Containers and Kubernetes and the cloud and the brownfield ones, because we want to one point migrate some of them to the cloud and the load balancing the ZeroLB pattern, which is a smarter way to also implement the blue-green deployments and cannot we release this allows us to do just that.

Marco Palladino: It allows us to more easily transition lifting, shift all their workloads to the cloud and to Kubernetes. Choosing the technology that can allow us to support these in a uniform way. It’s probably one of the biggest choices an architect can make and obviously being the CTO of Kong, I deeply empathize with this problem. And that’s why we’re trying to provide a product that allows them to run ZeroLB pretty much everywhere. And that’s why Kuma was the first service manager, which is now a CNCF project that came out with a native first-class VN support. In addition to Containers, obviously there’s, first-class support for Containers as well, and it can run pretty much everywhere. There was no service mesh that matches that level of support for both Containers, Kubernetes, and virtual machines and hybrid.

Swapnil Bhartiya: You folks have also released… You also mentioned that earlier Kong Mesh 1.4, And it has ZeroLB support built into that. Can you talk a bit more about the interpretation there?

Marco Palladino: Yeah, so we have introduced the many more load balancing capabilities are there is a five different load balancing algorithms that the users can decide to enable, and they can choose different algorithms based on different traffic paths. So we may say within this region or this cloud, I want to use Round-robin, but these other traffic requests going to our database should use a different algorithm. And then on top of that, we have introduced a service map, topology view, which is a gooey that allows the user to visualize all the traffic going from one service to another, including the rate of requests, the error rate, and the latency of these requests in such a way that however, we decided to connect our services together, we always have a bird’s eye view to make sure that the status is always up and running. There is a 70 plus charts and observability charts that Kong Mesh and Kuma already provide out of the box.

So this is going to be in addition to all the observability infrastructure that already is there when using the service mesh, as well as we have introduced capabilities. So being able to apply these load balancing routes across different subset of the traffic paths. So it’s not just traffic between one service and another service that is a subset of requests generated by that service should have these or that different load balancing algorithm. So it’s a very fine grain in the way we can control load balancing across the board. Centralize load balancers they do not allow a fraction of this while costing us more because we have to pay for an extra component, having more bandwidth and having an extra hopping, the network, making everything slower. So it’s really a no-brainer.

Swapnil Bhartiya: Awesome, Marco, thank you so much for taking time again, out from a schedule and talk about knowledge ZeroLB but also how you are kind of helping customers. JD is the complexity that is already there in the cloud any way, thanks for your time today. And as usual, I love to have you back on the show. Thank you.

Marco Palladino: Thank you for the opportunity.