How to Run Self-Service Databases Without Losing Control | Oliver Wolf, anynines | TFiR

Swapnil Bhartiya: As a developer, you know that developers want a database the moment they need one without opening a ticket and waiting for someone to set it up for them. But self-service raises a harder question, which is, who owns access control, auditing, and cost allocation for every data service on the platform? That’s what Klutch 2.0 is built to solve, and in this demo, we will see it publish a container-based Postgres and bind it to an app running on Cloud Foundry. And today we have for this demo, Oliver Wolf, CTO at anynines. Oliver, it’s great to have you on the show.

Oliver Wolf: Thank you very much. Great to be here.

Swapnil Bhartiya: It’s my pleasure. I mean, as you already know, we have had Julian on the show a few times to talk about Klutch. Everybody knows what Klutch is, and who don’t know, it’s a Kubernetes-based orchestration layer for data services. But If I’m not wrong, you are the one of the people who have been working on it, who are building it. So I want to understand what is your role on this team?

Oliver Wolf: So while Julian is on the product strategy level and on the vision level, I’m responsible for the technical part at Klutch and at anynines. So right now I’m working very hands-on in the Klutch team and that’s Quite fun because right now we’re kind of taking all the learnings which we had from Klutch 1.0 and we are reshaping Klutch into Klutch 2.0 and basically come up with everything we learned and try to improve the system and solve the requirements which we learned on the way, like what our customer problems are and to be more on point there.

Swapnil Bhartiya: And if you look back at the first version of Klutch, what did you change for version 2.0 and what turned out to be right from the start versus how it evolved?

Oliver Wolf: Maybe let’s have a first introduction, like what is still the same before we talk what changed. So what is still the same is the actual problem Klutch tries to solve, which is like If you are like a company which tries to build an, or which is building an internal developer platform, you need to offer your developer team some kind of database services, message queues, key-value caches, and so on and so forth. And Klutch does that very good. It provides you a building block which you can use to build that part of your internal developer platform to solve this data service problem at scale. And the core idea is that you have a small operation team which allows you to operate thousands of service instances without scaling the actual operations team. And the principle how we achieve that in Klutch 2.0, it is basically still the same. The secret sauce is a separation of concern and a self-service contract. So you have this self-service contract which describes how your developer interact with the platform and what functionalities they can consume from your platform and which responsibilities the operation teams has. So this is a clear separation of concern, like, and how this interface looks like, like a developer team would still go and provision service instances. It would still go and create service bindings, which are like credentials and connection information. And a developer team can also like in a self-service manner create backups and create restores. So this part and this interface is still the same. And as I said, it’s about separation of concern. So you have these operation teams in the backside which operate the databases care about lifecycle management, like monitoring and disaster recovery and all these things. And you have the developer which can consume these services without opening tickets and so on and so forth. And what we learned now along the way is that within the platform team, you might also have another layer of separation of concerns. Like when you look how companies today build internal developer platforms, which are based on Kubernetes, then you see some common patterns, how they all try to do kind of the same thing. They have like one centralized control plane, which acts kind of like a marketplace where the developers go to, to see what offerings exist. And underneath that, they again have different operations team. So they might have like a specialized operations team which is focused on Postgres, a different team which is focused on, let’s say, some kind of message queue. And then they specialize in that. And now you can envision Klutch 2.0 as a platform to both of those sides, like to the developers and to the operation teams, to different operations teams which work decoupled from each other. And then Klutch 2.0 solves all these cross-cutting concerns like access control, like auditing, and so on and so forth. Also like cost allocation and all these things. And we see now that this kind of cross-cutting concerns for many, many companies, it’s still unclear which team should solve it. Is it like the the team which operates like this marketplace or this control plane? Or is it like each team should solve it separately for its own? And Klutch 2.0 provides mechanism A, if you, for example, don’t have the centralized control plane yet, then Klutch can act as a centralized control plane. Or if you already have one, you can use Klutch or like Every local team which operates a specific data service can use Klutch to solve this kind of concerns for itself. I think this is kind of the new insights which we had to, yeah, which led us kind of rebuild Klutch 2.0 among other many, many things.

Swapnil Bhartiya: In past, when I talked to Julian, the conversation focused a lot on VM-based data services and AWS integrations. Today you are showing Postgres, which runs in containers. Does Klutch actually care what’s behind a service and how would a platform team decide between VMs and containers, or it doesn’t matter?

Oliver Wolf: It actually doesn’t matter. So it’s actually the, we call it in Klutch, the service integration. Like you have Klutch, the control plane, and then you can have different service integrations. And this service integration then decides what does it mean to have, for example, a service instance? Is it something that runs on a container? Is it something that runs on a virtual machine? Or is it something maybe that runs, is provided by a hypervisor? Like for example, AWS RDS can be integrated into Klutch as well, or any other hypervisor cloud. So for Klutch itself, it doesn’t matter. It provides all these cross-cutting concern features like access control and auditing and so on. And it’s then up to the service integration to implement a contract between, a simple contract between Klutch, the control plane, and like the service automation. And then the Klutch itself doesn’t really care. how a service instance is represented or implemented.

Swapnil Bhartiya: Now we are of course going to see the demo, but before we jump in, can you just quickly summarize what are we going to see today?

Oliver Wolf: So I prepared a small demo which shows Klutch 2.0 as a control plane, and as a service integration, we have installed the any8s Postgres, which is a container-based Postgres implementation. Now this container-based Postgres implementation has the benefit that it spins up faster and of course it’s cheaper because it has a better resource utilization. And now with Klutch, you can consume Klutch from different systems. Like for example, a developer team might use Kubernetes to deploy their workload and then they can use or can connect their Kubernetes cluster which they own and they deploy their workload to, they can connect it to the Klutch control plane. And then they can consume those data services, can discover those data services, and can provision them, back up them, or make plan updates and so on and so forth. And in the same time, you can also connect Klutch to a Cloud Foundry system. And that’s what I’ve prepared today. I’ve connected this Klutch control plane, which I’ve set up to Cloud Foundry system, and now We’re going to see how you can first connect the Klutch control plane to the Cloud Foundry system, and second, how you can consume these services, how you can, we call it, bind the service to an application, which basically means create the credential set. Yeah, that’s what we’re going to see.

Swapnil Bhartiya: Perfect. Let’s start the demo then.

Oliver Wolf: Okay. Now you see here on my screen, 2 terminal windows, and those terminal windows, they basically show Klutch from two different perspectives. On the right side here, you see Klutch from an operations point of view. This would be like the operations team which operates the, for example, the Postgres service using Klutch. On the left side, you see the system how the developer would consume the system. Like this basically here, I’m logged in in a Cloud Foundry environment and if I say CF Marketplace. Let me make this a little bit bigger. You see that we already have some service offerings here in this Cloud Foundry Marketplace, and those are all the anynines data services, and those are the ones which are based on virtual machines. And now we want to extend this marketplace by providing the container-based Postgres automation using Klutch. And on the left side, you see here I’m logged in with kubectl. And this is basically the platform operator side where the people who operate this Postgres service interact with. And the first thing when you notice when you use Klutch 2.0 as an operator is that in Klutch 2.0, the tenant concept is now a first-class principle. So everything is like, the tenant is used to group a different amount of namespaces into one tenant. And this is basically good for 3 different things, like the building block for 3 different things. One is like access control. The other one is like authorization and the other one cost allocation and the other one is isolation. Isolation, for example, if you’re using Klutch to provide AWS-based or Google Cloud-based services to your developer, you could say that each tenant is provisioned into a different AWS account. That’s what I mean with isolation. So the workload is actually isolated in different accounts and so on and so forth. Like the tenant concept is also a very important thing which you immediately see. And now in Klutch 2.0, it is like a major building block and you immediately see it. Like there’s no way around not to see it. Why? In Klutch 1.0, it was primarily used for isolation, but now also for access control. Yeah. So you can list all the tenants you have available on your control plane. Again, this is like the view from the platform operator’s perspective. This is not something the developer would do. But if you say kubectl tenants, you see that we have 2 tenants already here. Like you always have a default tenant and then there was already a tenant created for the Cloud Foundry environment we already have. And that tenant actually maps to a Cloud Foundry organization. That is a feature of Klutch 2.0 that you can configure how the Cloud Foundry organization or how your, for example, Kubernetes namespaces are mapped to different tenants. And in this setup, it is configured that whenever you create a new organization in Cloud Foundry and you order a service instance there, this will basically be corresponding to a Klutch tenant. So on demand, a new Klutch tenant will be created when you for the first time create a new service instance in the new Cloud Foundry organization. And it works the same way in Kubernetes. When you would consume Klutch from a Kubernetes cluster, and for example, you decide that for each Kubernetes namespace you want to have a separate tenant, or for each Kubernetes cluster you want to have a separate tenant, then whenever you create the first instance, a tenant into the Klutch control plane will be created automatically for you. Now, yeah, this is about the tenants. And another entity which you can see in Klutch is Klutch clients. And the Klutch clients, you can think of it as a user, but it’s primarily used for machine-to-machine communication. In particular, now we are basically coupling the Cloud Foundry which we have on the left. We want to couple it to the to the Klutch control plane on the right. And to do that, we would need to have a Klutch client. So I already created a Klutch client for that, which is the Cloud Foundry path here. If, for example, I don’t want to couple a Cloud Foundry, but I want to couple a Kubernetes cluster to the control plane, I would create another Klutch client for that. For example, this one was created and I called it Klutch Operator. And the Klutch Operator is basically a component which you would install in the Kubernetes cluster of the developers. So in the cluster which the developer use, you would install the Klutch operator, you would configure it with the Klutch client secret, and then from there on, the Klutch operator would reconcile all service instances you create on the app cluster, like the developers create on the app cluster, and it would basically synchronize those or like create those services into the control plane. And from the control plane, they can then be picked up and reconciled by the service integration. So let’s go on and use this CF Pass client and create a service broker on the Cloud Foundry end. So to do that, Cloud Foundry has a command in which you basically say CF create service broker. The service broker concept in Cloud Foundry is something how you extend the Cloud Foundry marketplace, and Klutch happens to be a comfort with that interface. So to do that, you basically give this broker a name, so you can choose whatever you want. You have to name the Klutch client, which basically corresponds to this name here. And then you provide the Klutch password, which basically is a token you can create here on the Klutch control plane. Then you provide the endpoint where your Klutch control plane is running. And here in this demo, I don’t have admin privileges in this Cloud Foundry system. So there are 2 ways to extend this marketplace. Usually you would be an administrator and you create the service broker. And the service offering would be available to all of the users in this path. Or you use this kind of space-scoped broker, then this service offering would only be available in this particular Cloud Foundry space. So let’s go with that one first. That takes a while. So you see the service broker has been created now. And when we look at the marketplace again, You see now that we have the any8s-postgres here available, which we can provision. Let’s have a closer look at that service offering by saying let’s describe all the offerings provided by that broker. And you see that we also have like one plan available. Like, yeah, in this Klutch control plane, I only configured one service and one plan. And that is also something that changed with Klutch 2.0 is how the platform team— now we switch back on the right side here— is how the platform team configures custom service plans and custom services. To do that, let’s have a look at at the configuration, how now the platform team would decide which kind of services and which kind of service plans are available in this Klutch control plane. So this is basically a simple YAML file which describes all the services and which describes which service plans are provided by Klutch. And you basically see Every plan maps to a Kubernetes resource. That means whenever a user, a developer orders a service instance, then the Klutch control plane will create a Kubernetes resource of that kind in the control plane or in any other Kubernetes cluster, which is used to reconcile and to provide the actual service instances. So this is freely configurable, which kind of resources are created. And it’s also configurable which kind of configuration these resources receive. So you see those values are configured by the platform administrator and they cannot be changed later by the user. But still you want some configurations to be changeable by the user. And this is where the parameter policy comes in. Like for example, if you look at some Kubernetes operators, they usually have a wide, very big list of parameters that can be configured. And those parameters, they usually are for database administrators, not for the developer who uses them. And you usually don’t want your developers to be bothered by that. So this is where this parameter policy comes in. You can say, which parameters are actually— should actually be exposed to the developers and which parameters you want to freeze here and don’t expose it to the developers. So this is again the separation of concerns, which is important. Which kind of parameters should be changeable and should reach the developer and which kind of parameters should be decided by the platform operator. It’s basically You can also call it like a plan corresponds to a golden path in your company, which assembles the most common use cases and which you know your user might need. And it’s also common practice, which makes the cost allocation easy to say that the plan corresponds to a certain price. It doesn’t have to, but it will make the cost allocation easier if you, for example, say that everything that influences the price is like fixed or not changeable within a plan. But again, you don’t have to do that. You could also just move these limits here into the parameter policy, and then the user could decide how many replicas and how many CPUs this should be. But as a recommendation, I would first start with having everything that impacts the price fixed and not be changeable by the user. So this is basically how a team who operates a Postgres service configures its service offering, and then it will become available here on the consumer side where the developers can actually order it. And now how would you order it? You would just say CF create service, You would say which service you want. You would choose a service plan. So that’s an easy choice in this table because we only have one plan and you give it a name. So the third parameter is basically the name of your instance so you can interact with it later on. So we send this creation request and now the creation of this container-based Postgres usually goes very fast. What is special in this plan is that it is exposed by a load balancer. That means for this particular plan, I have decided when I configured my catalog that each Postgres instance should be exposed by a single load balancer to make it available to the other environments. For example, if there are network boundaries in between. To overcome this hurdle, and let’s say for example, you don’t wanna use one load balancer for each service instance, you could also install a component which we call the Klutch Network Connector. And it basically is responsible for taking care of the network connectivity between the provider side where the service is running and the consumer side where actually the application is running that needs to connect to this database. So the reason why the creation of this instance, service instance might take a little is because now it created a load balancer on AWS because that’s how I configured that service plan here. But you see the test-pg succeeded already, so that part is running. And now if we look at the applications, so to this Cloud Foundry instance, I already deployed one application. And if I open it in the browser, it is like a simple blog post. If I create a blog post and I save it, like this application, it is running on Cloud Foundry. And we see that when we create a blog post, it’s basically not saved because it’s not connected to a database where it can save the data in. So let’s take our container-based Postgres, which created here, and bind it to the application. So we call this service binding. It’s a terminology that actually originates from Cloud Foundry and— oh, it’s just bind. So we specify the app name and we specify. So in the bind command, it’s called bind service. We specify the application and we specify the the the service instance we want to bind to each other. Now what happens? This request is also sent to to the clutch control plane, and then from the clutch control plane, it’s like. Forwarded to the service integration, which actually takes care of automating the Postgres. And then it’s the responsibility of the service integration to create a credential set, which can be used by this application to connect to the service instance. And when we restart the application now, And create another test block, we see that now it is saved and it will also survive another restart, meaning it’s like served, saved persistently in the database instance. Yeah, that’s about it. We saw how we can configure in the Klutch service offering in the Klutch control plane and how we can publish it and how we then can connect it to a Cloud Foundry and consume it from there, bind it to an application, let Klutch create a credential set, and then use the credential set within the application and write data into it.

Swapnil Bhartiya: Excellent. Thank you for that great demo. Now, if we look back at it, those credentials were created per binding. What happens when the app is unbound or deleted?

Oliver Wolf: So when the app is unbound from the service instance, then that request will also be sent or forwarded to the Klutch control plane, and the Klutch control plane will forward it to the service integration. And then it’s the service integration responsibility to delete this or like make this credential set unusable, like delete the user or like, yeah, make sure that it’s not usable anymore. And this way you can do like credential rotation. If you like think like one credential set might have been exposed, you can just delete it. And for example, let’s think of a message queue. A message queue, you would usually bind multiple applications to it which consume, which talk over this message queue. this message queue. And let’s say one of those applications leaked the credentials, then you can just delete this one binding and the other binding of the other applications still are usable. So yeah, it’s the responsibility of the service integration to make sure that whenever you delete a service binding, these credentials are not usable anymore.

Swapnil Bhartiya: You also said that consumer could also be another Kubernetes cluster, how different would that look like from what we just saw?

Oliver Wolf: Right. The main step would basically be the same. You would need a Klutch client in the control plane, which you can use to configure in your Kubernetes cluster. Now we have to distinguish a little bit. It’s important to understand that the Klutch control plane runs on a Kubernetes cluster, But the application developer who wants to consume a database usually in most cases has its own Kubernetes cluster. And in this cluster, which he owns, he would install the Klutch operator and he would configure it with the Klutch token of a Klutch client. And then we are using, yeah, the developer can discover the service offering of the Klutch control plane by using kubectl api resources, then it will be listed. He will see, ah, I can here create a Postgres instance or whatever service is offered by the Klutch control plane. And he can also see which plans are provided. And also with the kubectl explain, he can have an idea which kind of parameters are available for this plan or this service. So it’s pretty much the same. And then once he discovered the service offering, he has an understanding what kind of services are available and provided, which plans are provided, which parameters he can configure. He can just write a normal Kubernetes YAML manifest and use it with his standard GitOps pipeline to just provide it in a declarative fashion and then let the GitOps pipeline apply it to its Kubernetes cluster. And that also works if he wants to change parameter later on, or if he wants to change the plan. Let’s say the Postgres database is now growing and it’s running out of disk space. He could also change the plan of that in a GitOps fashion like he would be used to when he’s using Kubernetes.

Swapnil Bhartiya: Excellent. Thank you for this excellent demo. Now, if I ask you, where is Klutch 2.0 today and what kind of feedback are you looking for and from whom and how they can get involved?

Oliver Wolf: So Klutch 2.0 is now, we are still working on a few last edges to straighten them out. I expect this to be like done in not very long time. And we are looking for all kind of feedback. Like if you are a mid-sized company or a large company which started started out or already like are a little bit further in this journey of building internal developer platforms for your developers, then most certainly you have some insights that, that can help us to, to match all kinds of requirements which are out there and, and evaluate it against different use cases.

Swapnil Bhartiya: Oliver, thank you so much for joining me today, sharing this demo, and of course sharing great insights. And I would love to have you back on the show. Thank you.

Oliver Wolf: Thank you very much.