Swapnil Bhartiya: Hi, this is your Swapnil Bhartiya and we are here at Open Source Summit in Minneapolis and we have with us once again Madelyn Olson, Valkey project maintainer and principal engineer at AWS. Good to see you again.

Madelyn Olson: Yeah, great to see you again.

Swapnil Bhartiya: And this time we are, of course we’ll talk about Valkey. But this time I think the biggest discussion and I think you folks are also delivering a keynote is about AI and open source AI in general when it comes to software. Either way, it’s kind of becoming a point of friction, but also it’s bringing a lot of value also because the fact is that because of AI, a lot of code is being written, vibe coding is happening and a lot of developers are using it just to assist them as well. But which also means that a lot of code is coming. And now how do you maintain the balance between the quantity of code versus quality of code? At the same time you may also want to use it. And once again you have to ensure that you are able to. So let’s first of all talk about your favorite project because you said, let’s talk about Valkey. So let’s start with Valkey. How are you folks using AI internally for Valkey?

Madelyn Olson: So yes, the main project that I’m a maintainer of is Valkey. And I think it’s great to add some numbers to everything we’re talking about.

Swapnil Bhartiya: No, you’re talking about that.

Madelyn Olson: So since about December, we’ve seen about a 30% increase in the number of PRs getting opened to the project. But more importantly, we’ve seen about a 500% increase in the number of lines of code changed across those PRs. So the big difference is not only are people making more changes, those changes tend to be significantly bigger, they have way more changes involved. And the bottleneck that we’ve actually seen inside the project for a long time is actually reviewing that code.

Swapnil Bhartiya: Right?

Madelyn Olson: And when we review code, we look at it more or less in two different ways. We look both at whether this code is functionally correct and whether this code is the right code for the project. Is this the right architecture? And from a maintainer perspective, I think AI is kind of okay at that first one. It’s like, hey, is this code going to crash? What bugs are there? But it’s not very good at that second one. A lot of the foundational models, they have a very strong bias for generating new code. They have a very strong bias for building stuff a specific way. And they often are not good at abstracting code in a good way to incorporate new functionality. So we definitely haven’t been able to scale maintainers very well to be able to incorporate this sudden increase in code. And we’ve done some stuff to optimize that. I, for example, have an agent that I use while reviewing code and it does the thing that AI is pretty good at. It’s like, hey, summarize this code for me. Tell me what the key changes are. I used an agent not too long ago to find a lot of bugs inside the Valkey project. We actually found a CVE that’s currently under embargo, which I can talk about a little bit later, not today under embargo. But the methodology that we use to find these bugs works very well to just apply on code PRs, which is basically adversarial testing, which is, hey, go take this change and just go try to crash it, go try to find bugs. And then if you do find a bug, report back, which builds a lot of confidence that that code actually isn’t going to break in production, which is very important for Valkey. Valkey is a runs-in-memory database used for caching and you really don’t want that system crashing in production. So we’ve definitely been using AI for that. Is this functionally correct? But the architecture half is a little bit harder. You can’t necessarily ask the AI, is this code good for the project? Because it’ll be like, oh, of course. And you’ll probe it and it’ll be like, oh, you’re totally right, I misunderstood. As we’ve all had, AI agents love to double back on what they’ve said. And so we’re trying inside the Valkey project to automate as much of that first half as possible so that maintainers can focus on that second half, that maintainership half. I’m going to be doing a keynote tomorrow talking about AI, how we’re using AI agents inside the project. And one of the key things we’re going to be highlighting is basically how we use agents to automate some of the backporting process within the project. So, unrelated to this conference, we actually launched Valkey 9.1 this morning. And part of that process involves taking commits from the development branch into the actual release branch, because not everything that goes in development is going to go for a specific release. And so we built an agent that’s responsible for taking the commits, applying them from the development branch to the release branch and then if any tests fail, it can go and try to resolve them itself. And if all of that fails, we still have the backstop of maintainers to go and look. But the ideal case is the maintainers don’t get involved at all. The maintainers are just there to say like, hey, this is correct. These are the correct set of commits to be backporting. So we’re trying to take that load off of the maintainers.

Swapnil Bhartiya: This is actually very important because some projects, and I knew that the pitch will come for a lot of maintainers, they are overwhelmed in the past because they could not keep up. So if you look at Valkey, there are a lot of other projects. What percentage of work is being done by agents versus humans? And how is it freeing your time to focus on the things that you wanted to focus earlier, which you could not, because the grind that happens when you have to review the code was just too much.

Madelyn Olson: And so I kind of highlight like, the core decision making inside the project is still done by the maintainers. We’re not handing that over.

Swapnil Bhartiya: So what percentage? If you talk about like 90% AI, 10% maintainers, too much.

Madelyn Olson: Trying to come up with a number is probably going to be a little difficult. Like we’ve definitely with some of the automation we’ve built, that kind of offloads probably, like 20 to 50% of the day to day work of maintainers, we’ve been able to offload to AI. And that also obviously changes week to week. So like a week like this one where we’re doing a release, a lot more of the work has been offloaded because a lot of that backporting stuff’s been automated. But on a typical week, I might be doing almost all of my time doing this deep work, because it’s fun also.

Swapnil Bhartiya: Right? Sometimes it’s fun, yeah. To just dive into those. We hear the word slop a lot these days in terms of when it comes to a maintainer’s job. How would you define it and how do you catch it? Because that’s the tricky part. And actually when I was working on your interview, I just wanted to do some research. So Claude told me that you are founder of Valkey. I said no, I interviewed her. This is my interview. Oh, I’m sorry, you’re correct. So you have to keep an eye. It’s like dealing with a toddler which will lie all the time. So how do you define slop and how do you catch it?

Madelyn Olson: That’s a good question. So the way I think about AI slop is there’s a couple different parts. There’s the code that’s generated is just wrong, or documentation that’s generated is just wrong. It’s basically pulling from incorrect sources. And that’s mostly because you’re not providing enough context into the LLMs themselves. What we see is a lot of new contributors will often say, like, hey, Claude, go solve this issue. And they’ll point to some issue in Valkey and Claude will give its best guess. It will try to figure it out. What you need to do is give it a lot more context. If you just point at the issue, it’s going to probably come up with code that looks correct because that’s what they’re very good at doing. But it’s probably missing edge cases. It’s probably not well integrated inside the code. And when we see those as maintainers, the first thing is either to say if it’s the type of person who’s probably never going to be good at contributing code, we might just say, please don’t contribute. One of the things we’ve noticed is because it’s so easy to point an LLM at an issue that the bar for who’s contributing has come down a little bit. And so those are people who either, if they are excited but don’t have the understanding yet, are excited to be involved. And so we still want to encourage those people. One of the things we’ve changed in the project is we used to keep a list of issues inside the Valkey project which we labeled as good first task. So if you wanted to get involved, you could go pick these up. And one of the things we noticed is a lot of LLMs like various tools were going and picking these issues and just trying to solve them themselves, which doesn’t really solve the problem. These issues aren’t strategically important. They’re designed in such a way to help get people involved. So we still have another mechanism like you join our Slack and we will hand that individually to people. And that’s how we get people to actually do these first issues. So if you’re just showing up and generating a bunch of code because you just pointed the LLM at the project, we’re probably just going to gently push you away. And so that’s kind of like the first level of slop. The second level of slop is someone who genuinely does kind of understand what’s going on. The code, at least more or less looks correct, but it’s kind of subtly not production ready, maybe a subtle bug. That’s one of those cases where it’s actually just not too hard to just gently poke their agent to be like, hey, you’re missing these bugs. Think about these cases. So just because the code is not necessarily as good as a human would write, that doesn’t mean you should just throw it out. You should still work with them, help them grow. Because as long as there’s a human on the other side looking at these inputs and being like, hey, I understand, I missed these cases, next time, hopefully they’ll produce better code and continue to be involved in the project. So it’s kind of like how the slop was generated that’s more important, not necessarily that it was bad.

Swapnil Bhartiya: You said the bar is kind of low, but have you also changed some expectations which would not even be realistic two years ago, before agents, where you’re like, okay, these are the compromises we’re making. We are lowering the expectations of the bar so that AI agents can also contribute versus humans who are like, oh, that’s not even something that you can ask me to do.

Madelyn Olson: Yeah. And so just to clarify, when I say the bar for someone to be able to contribute is lower, our bar is definitely not lower. And we’ve, if anything, been trying to use agentic AI and code generation to raise the bar. We want to add more verification, more validation.

Swapnil Bhartiya: So the change of the bar is that you have increased the height of your bar, but lowered the bar for people to be able to contribute.

Madelyn Olson: Okay, perfect. That’s what’s important. If someone shows up and they have a simple feature, there are some simple features which you can just contribute and AI agents are helping with that. We have an AGENTS.md file inside the codebase so that you can basically point your LLM to it, your Claude Code to it and be like, hey, use this file as an understanding for the codebase. And the whole goal of that is to raise the bar of what a typical agent will be able to do.

Swapnil Bhartiya: Can you talk about what the provenance guard function does? What does it do that typical traditional review would miss?

Madelyn Olson: Yeah, so what you’re talking about is one of our goals recently in the project is to build more automation and verification so we can offload work from the maintainers. So one of the unique constraints that Valkey has is that Valkey is a fork of another project called Redis and they have different but incompatible licenses. So Redis is tri-licensed, one of which is AGPL, which is an open source license. Ours is BSD. So we can’t take commits from AGPL and apply them onto BSD licenses. So one of our concerns has been that someone might unintentionally think they are allowed to, since they’re both open source. Not everyone is deeply familiar with the nuances of those licenses. And one of our other concerns is, what if an agent does this? What if an agent goes and tries to pick a Redis commit and apply it to Valkey, not knowing that they aren’t allowed to do this? So one of the things we built is a tool that basically keeps a large list of hashes of commits from Redis. And we’re also basically able to quickly check, see, hey, is there a match here? And then on top of that, we can use LLMs to do a secondary level check to be like, hey, is this okay? And the goal is to add confidence that this edge case, which we’re worried about, we don’t spend so much time as maintainers worrying about. Because it’s possible they might not declare where the provenance came from of the code. And there’s a DCO check which we use, basically says you’re allowed to do this, but that’s still not a sufficient check. We want to make sure that we’re not accepting this code. So that’s been our goal as a project, to continue to try to build this automation. And I think that’s a great use of AI. As a maintainer, I can write a lot of code more quickly that’s very tailored to what the Valkey project needs. And so we’ve been trying to do a lot in the project.

Swapnil Bhartiya: You did talk about some of the tedious tasks, but if you look at things like backporting, CI testing, how much are you handing over to AI and AI agents?

Madelyn Olson: I’ll give another example, which is one of the features that we’re thinking a lot about inside Valkey is being able to support SSDs as an alternative storage of data and not just RAM. Obviously RAM is very expensive these days, there’s a RAM shortage. And so one of the things we can do very quickly is evaluate a bunch of different trade-offs with prototypes. Basically we can go and say, hey, here’s the high level interface we want to build. Now go try to store both the keys and the values on disk. Just store the values on disk, keep the index in memory, maybe store the keys in memory and the values on an external network attached drive, like an EBS volume or maybe S3 or maybe something like Dynamo. And so agents can very quickly go and validate all of those. Because as I said, they’re very good at generating lots of code and they’re very good if something doesn’t work. They can try new things, especially if you give them sort of guardrails to think through. And that helps us prototype a lot more quickly. And so when we actually get to the time where it’s like, okay, we’ve evaluated a bunch of prototypes, now which is the actual choice we want to make, we have a lot more information than we did before. So AI is great in that sense. But then when I actually go to write code myself and be like, hey, this is the actual production quality code that I want to be writing, I’ll typically do much more like a pair programming type of thing with the AI. We’ll typically write all the tests together to make sure everything’s constrained correctly. Then I’ll maybe let go of the agent and let it go try to implement everything to those specs. And then there’s always going to be that interface because we need to get that code generation pointed more towards the right abstractions and the right architecture. So AI is both good in the unbounded exploration phase and then also when you want to write the code, it’s good at helping accelerate the actual speed at which you can write well-constrained and well-focused code.

Swapnil Bhartiya: AI is very good at what you explained, but sometimes AI is not very good at coming up with new ideas because it’s always someone else’s idea that it learned from. How do you also ensure that when we look at the Valkey project, the trajectory, the vision, the evolution of Valkey is guided by humans, not by AI?

Madelyn Olson: I think the real mitigation we have for that is just making sure humans are in the loop for all of the key decisions that are being made. Valkey is actually in a unique position. It’s not really on the cutting edge of technology. It’s kind of at the edge of the application of it, the engineering side. So we do rely a lot on papers. We do read a lot of papers to figure out what we should be building. And that’s where a lot of the actual ideas come from. We’re not researchers. Most Valkey maintainers are engineers. We go and try to figure out how do we apply this to make it practical for production workloads. And so we’re just deeply involved in the entire process. We come up with all the ideas and then we work with AI agents to go build them. But yeah, it’s mostly us that are the ones coming up with all the ideas.

Swapnil Bhartiya: From your perspective, ever since Valkey came out and then agentic AI and now we have a foundation as well for that. How has AI impacted open source? I mean we talked about the toil and everything, but in general, not just for Valkey, but in open source because we hear mixed stories, we hear a lot of horror stories, but it can find bugs that humans will never find because there are a lot of things it can check, it can also help fix those bugs. So how have you seen that because of AI, Valkey itself grew very fast, and if it is done right, the way you focus on it, it can help other open source projects grow faster?

Madelyn Olson: At the end of the day, you really have to accept the fact that AI is just another tool. I’m not the biggest fan of AI for a bunch of different reasons, but the truth is it’s not going away and it’s not going away anytime soon. And so I think you do have to have this optimistic view of trying to figure out what’s the best way to use your tool to solve your problem. And you kind of already said everything that I was going to say, like go find the task inside the project that you don’t like doing that the AI is well situated to offload. I don’t really think you should be giving AI high level decisions of the project, but it’s pretty good at doing things where it can exploit its non-determinism. It can go try lots of things. As I said, we did a bunch of security reviews and basically the harness we set up was, hey, here’s a deterministic test and it looks for crashes, it looks for corrupted streaming of data. If you can find any of these cases, let us know. And then we just said, go try 10,000 different things, come up with new things, and let us know what you find. And it found a couple, it found the CVE we talked about. It found a bunch of other esoteric bugs. And it’s better for the AI to have found that proactively than some end user finding it or some security researcher who has an embargo date they want to release it on. So you need to at least be thinking about that as a maintainer. There’s a lot of stuff that we sort of internalize. We have to do this work and we have to say, no, these AI agents are tools. We should be letting the tools do what they’re good at. And that doesn’t mean handing over everything. There’s a phrase I hear a lot, which was from product managers at Amazon. They’re like, well, why can’t agents do that? And the answer in many cases is no, agents can’t do that. The risk of hallucination is too high. At some point, we really want humans to be thinking about the direction of the project. If we offload all of the reasoning and thinking to the agents, then the humans aren’t doing the part they enjoy. Like, you’re not required to use agents. If there’s a part of the job you enjoy, keep doing it. Don’t offload it. That’s not keeping your sanity. That’s not maintaining your mental health. There’s a long history of maintainers feeling burnout. I myself was very burned out back when I worked on Redis open source for a while. And I came out of it just because Valkey happened and I was, it was very fun to work on Valkey. I got to focus on all the stuff that I like to do. We got to build lots of things. And so it’s really, I think AI is at an interesting place where a lot of people are experimenting. A lot of the stuff I mentioned is not super novel, not super unique. But the joy of learning and trying to figure out how you can use these tools to solve some of your problems will get you a long way.

Swapnil Bhartiya: You actually hit on a very important point, which is burnout. I am also a gamer and a woodworker. I use power tools, but I am the operator. They are just a tool. A lot of games have a lot of grinding so I lose interest in the game. The same thing with a lot of projects, people burn out because it’s repeated, it’s grinding. So they don’t enjoy it. So I flipped it. What advice do you have for maintainers or developers, if not a total playbook, on how they should use AI? Because some are hesitant, some are totally reluctant, some don’t like AI at all. That will actually help them deal with burnout because then they will get back to the exciting part of the project they’re involved with, which is the exciting new things. Let AI do the toil. So your advice to maintainers and developers, how they should use AI and agents?

Madelyn Olson: I would say try to figure out what makes you, what drives you to work on open source. So the original thing for me was always that I like to help people. I originally got involved in Redis open source when there were bugs that I found inside the managed ElastiCache service that I would rather give out so that other people didn’t face the same issues that we did inside the ElastiCache service. And I enjoyed also just going through issues, finding other unrelated bugs and going to help solve their problems. That’s what I truly enjoyed from the project. And over time as I got more leadership in Redis Open Source and then in Valkey, I sort of lost a lot of that connectivity to the actual end users’ problems that we were solving. And the thing that I’ve been trying very hard to do is if I automate something with AI, I’m trying to not go do more of the things that burn me out, but basically take that time and spend it on stuff that I actually want to be spending time on, which is going and solving the little things, the little issues people have. I love being on the Valkey Slack when someone’s like, hey, I have this little weird edge case. Then maybe I can’t easily solve it, but I like going and talking to them and understanding why they have it. And someone in my team literally said, why don’t you offload that to an agent? You could install a knowledge base and they could go talk to the person. And I said, but that defeats the entire purpose. That’s not what I want to do. And that’s what I see with some folks, not all other maintainers, is they see this pressure to automate everything with AI and they’re like, this is what I love, why would I automate everything? And it’s really just be curious about the tool and go and try to automate the things that you don’t want to do. Don’t try to automate everything.

Swapnil Bhartiya: Madelyn, thank you so much for joining me. And of course, talking about the things that keep you excited and how AI is helping you in that excitement. And also thanks for sharing the way you folks are using AI because that also helps other projects to learn from it because the growth momentum is evidence of itself. So once again, thank you for your time today, and I look forward to chatting with you again.

Madelyn Olson: Yeah, so great talking. Thank you. Thank you.