How to Run Local AWS Testing Without Live Credentials | Tom Calloway, Amazon Web Services | TFiR

Swapnil Bhartiya: When it comes to testing against real AWS account, it is slow, costly, and risky, and it puts live credentials on developers’ machines. AWS is now bringing that experience local with an emulator built, maintained, and improved by AI mapping to more than 400 AWS API models. And to unpack what that means for developers, today we have with us Tom Callaway, Principal Open Source Strategist at AWS. Tom, it’s great to have you on the show.

Tom Callaway: Thanks for having me.

Swapnil Bhartiya: Let’s take a step back and look at the beginning. What was the problem the team set out to solve and how did Emulator came out of it?

Tom Callaway: So yeah, so the problem that the team set out to solve was that code moves a lot faster now in the AI era. You know, before you would have long windows of time, you know, a lot more budget allocated for testing, and now you just sit down with an LLM and you say, You know, I want to build this application, go ahead and do it. And what we were finding is that customers were getting to a point where the application was built, it was ready to deploy, it was ready to go on top of the AWS cloud, but the LLM would have to stop there and say, you know, I really can’t test this any further. I can’t confirm that this is going to work the way that I think it is on AWS. I’ve read all the documentation, but if you want me to proceed further to actually test this, you’re going to have to give me your AWS account information so that I can actually go and test this, you know, solution. And lots of folks were really uncomfortable about that, because we know that the LLM sometimes gets things wrong. Sometimes it hallucinates services that don’t exist or APIs that don’t quite exist. And so what we wanted to do was we wanted to create a local mock setup so that we could test whether an application is going to work on AWS the way that someone would expect it to without having to connect to the cloud at all, without even having to connect to the network entirely locally, so that you can have the breadth of AWS service offerings at your disposal to confirm that, hey, it is going to work the way that we say it’s going to work.

Swapnil Bhartiya: If I’m not wrong, the service is described as 100% AI-generated, AI-maintained, and AI-improved. What does that actually mean day to day, and where do humans sit in that loop?

Tom Callaway: Yeah, so that’s a great question. So we’re actually building this different than we’ve ever really built anything before. There’s no human-written code in this. There’s no human-edited code inside of this. This is all being built by AI agents. Now, we did write the AI agents that wrote the code. So there are humans involved in the process, but in the actual creation of the code, none of it is done. We were inspired by some other players in the market who started to do some really interesting things like Anthropic building a compiler entirely through AI and not having it be hand— human-generated at all. So we said, what would it look like if we were able to sort of take this problem space, which is remarkably well bounded because fundamentally the APIs for all of our services are documented very clearly. So if we take a problem space like that where we should be able to, you know, have the AI agents work within very tightly constrained spaces and in a very well locked down sandbox to build this code from scratch. And that’s what we’ve done here is we really have set up a model where we have 3 agent teams. Mm-hmm. One of them is managing fidelity. One of them is finding issues in the emulator itself, and one of them is fixing. So each of these teams is operating independently, and they’re just constantly iterating across this. The nice thing about this from my perspective is that it means that as we add new features or new services, this tooling setup automatically picks that up and starts building for those new services. So there really isn’t much of a delay. in what AWS Emulator can emulate from our product portfolio.

Swapnil Bhartiya: A developer may want to know why they should trust code that has no human review. What would you say to build that confidence and trust in those developers?

Tom Callaway: So one of the ways that we’re earning trust with this is twofold. You know, the first one is we’re open sourcing it. So there’s no secrets in this. It’s not some magical binary that AI built that you have to trust. All of the source code is there. It is written in Rust. You can download it and build it yourself if you want to. Of course, we provide prebuilt binaries and container images for it, but you don’t have to use those. You can just get all of that and build it from source and hand audit it or have your AI help you audit it. You have that transparency at your disposal. The other thing is that we, we’ve released all of the benchmarking that we use to test the accuracy of the emulator so that you can run the benchmark against the code itself and confirm that it does pass the same test that we say it passes. We know how important it is for people to be able to have confidence in a tool like this that it does what it says it does. So we provide the tooling to validate it, and we also provide the source code so that you can internally look at it.

Swapnil Bhartiya: And if I’m not wrong, it’s also released under Creative Commons CC0 license. Is that correct? So how different is, I mean, when it comes to open source licenses and what does that license mean for the community and people?

Tom Callaway: I’m not a lawyer and I wanna be careful how I speak on this one, but, you know, this code was not written by people. This code was written by AI. And so, you know, the copyright of that is a little unclear. And so we chose a license that allowed for very permissive use, modification, distribution, all the things that open source requires, but doesn’t depend on copyright. And so that’s why we chose the license we did.

Swapnil Bhartiya: Now, can you talk about how does this emulator actually work when a developer runs simulation locally, what’s happening under the hood and how does it map to those 400+ AWS API models that you folks are supporting?

Tom Callaway: Yeah, so as you mentioned, we do use these AWS API models. They’re part of our Smithy infrastructure, which is also open source. These are the fundamental building blocks that we use to build all of our SDKs. So these models describe all of the API calls that our services accept. And we told the emulator basically, hey, you know, the agentic toolkit that built the emulator is looking at those models and saying, these are the calls I can make. And then we have a set of traces where we’ve actually taken real-world scenarios and shown what it looks like across the entire transaction. So when you start in doing a thing, we can do some really complicated tasks on AWS combining multiple services together. And these traces reflect a lot of those scenarios. And we know from these traces what the end result should look like. And so the emulator has, you know, a set of these traces to check against. That’s our fidelity checking part of it. So as the model is iterating through with these 3 teams of agents, the fidelity traces are there to be able to make sure that we’re not stepping backwards in performance or we’re not stepping backwards in actually meeting the goals that it’s supposed to meet. The traces say, Input, output. If it doesn’t match that, it doesn’t get committed, it gets rolled back. And so it’s constantly iterating through this to get better coverage of the APIs and to get better results on the traces that it has. And then one of the things we’re actually excited about having customers help us with is to get new traces so that as we get new scenarios in here, we can improve the accuracy of the emulator.

Swapnil Bhartiya: And is that also how you know that the emulator actually behaves like real AWS and To ensure that, you know, the agents won’t ship changes that would break that experience.

Tom Callaway: So one of the nice things about our internal build pool for this is that they have access directly to the AWS cloud. So one of the steps they actually do beyond using the trace rollback is they can access AWS, spin up an instance, confirm that it does behave the way it’s supposed to and matches the behavior of the emulator. And if it doesn’t match, it logs that match. And if it goes backwards in fidelity, It says, hey, no, we can’t, we can’t make this change. We’ve gotten worse, not better.

Swapnil Bhartiya: Of course, it’s released today, it’s open source, but you folks must have done a lot of testing, beta testing. But are there any areas where this emulator still falls kind of short? And how should a developer decide whether to use it for their testing or not?

Tom Callaway: Yeah, so this is something that we’ve been using internally for a couple of months now. And because we have the same sorts of needs that our customers do in this regard, we want to be able to do fast, simple testing to to confirm validity of AWS solution use cases. We anticipate that this is something that we think a lot of customers are going to be really interested in putting into their continuous integration, continuous development workflows because we’ve done it ourselves. And we’ve seen that the agents have been very successful as we’ve added new services into the mix because it didn’t start out covering all of AWS. When we first started building it, it covered a narrow subsection of AWS to prove the concept. But very quickly early on, we realized that what we had built was very effective at implementing this because it’s a nicely well-bounded problem. It doesn’t require a lot of interpretation. It basically is just to make sure that we get the same results that we would expect when we do an API call. So that’s something that, you know, with enough tokens, that is a solvable problem.

Swapnil Bhartiya: Just continue on that, you know, the full history of the agent’s issues and commit is also public on GitHub, which also means that developers can go and check it out. What will you tell developers developers, since the way we use open source is different from, you know, the, of course, you know, bundle a product, so that they should actually go read it, get involved. What would be the right approach to take the, take most out of this emulator for developers versus just, you know, using it?

Tom Callaway: Yeah, I mean, so the default open source answer to this is accepting patches, but this project’s a little weird in that we’re not. We actually have disabled pull requests on this because of the way we build it. You know, we don’t have humans in the loop in the direct development process, and so We’re not taking pull request changes on this. Now, we do anticipate that people will find bugs in the code, but we think those bugs are really bugs in our agents. And so we’ve left issues open so that if you see something that you think is wrong, we’d love to have you report it so that we can fix the agents so that they generate the right code to get the right result. This model is a very different way of thinking about developing software. It is about taking a problem that doesn’t require a lot of creativity, that is very well bounded by requirements and then leaving it to an agentic team set to address it. So from that perspective, what we really want from customers is to let us know when you see that the agents have done poorly so that we can make the agents better. But what we also want is, as I mentioned before, we’d really love to get traces from customers. We’d love to have them describe scenarios where the emulator doesn’t behave in the same way as it does in the real world so that we can work with them to generate traces on our end that we can then hand to the agentic AI team and let them test, iterate, improve, and then come back and we’ll have a new commit for you from them that hopefully has that addressed.

Swapnil Bhartiya: And when it comes to those agents, those agents need real services to call and test against. What does that local AWS emulator change For agentic development, and of course, you know, when it comes to open source software, how much does it change you up?

Tom Callaway: I mean, that’s a great question. I mean, you know, right now, as I mentioned before, you know, the LLM gets to a certain point in your development lifecycle and says, hey, give me your AWS account credentials, I’ll go see if this works. You don’t want to do that. So what we wanted to do is we wanted to be able to have, you know, developers have an agent at their disposal that is connected into AWS Emulator running locally. So that instead of saying, hey, give me your AWS credentials so I can go in and spin up some stuff in the cloud, that it just calls the AWS emulator agent and says, validate that my configuration is correct, and then gets that source of truth from the local running AWS emulator. So you can run that without an AWS account, you can know whether it’s going to work before you even have an AWS account. And that gives an additional step in that agentic workflow that would require you to disconnect or to trust your LLM. And we’ve made it so that you can keep in that workflow and you can say, okay, the agents are doing what their strengths are. We’ve just added a tool for another agent to be able to validate your AWS configuration.

Swapnil Bhartiya: And for teams running tests today against real AWS, what changes when they move to this emulator? Can you walk us through what they notice first? Good things, bad things, the whole experience, or they’re like, hey, this is what we should be Using?

Tom Callaway: Yeah, I mean, it really depends on how they have it set up today. You know, certainly, you know, if your intern is going and validating all of this manually, then it changes a lot, you know, because now you can automate that process through your CI/CD. You can make it so that every commit that you make to an application that you intend on rolling out to AWS will run a test against AWS Emulator and confirm that, hey, this, this will still work. You know, I mean, one of the things that we’re trying to protect against is, you know, when your LLM you know, hallucinates an API that doesn’t exist, you know, which it does sometimes. You know, it’s trying to please you. It’s trying to say, hey, wouldn’t it be nice if we were just able to use this one magical API? And it says, sure, I just invented it out of thin air. AWS Emulator will be that source of truth that will come back and say, no, that’s, that’s not a real API. That’s not something that you can use. That’s not something that exists. It would be nice if it existed, but it doesn’t exist. And catch that and immediately bounce back so you can say, oh, that commit needs to get reverted. It did not. Yeah. It does not match reality. AWS Emulator is intended to be that reality check for your cloud-based applications.

Swapnil Bhartiya: And as you said, you know, you have been using it internally, so I’m assuming that it is mature. But if you look at the maintenance being done by agents, the changes that are landing every day, how stable is each release of AWS Emulator? What should developers be pinning to?

Tom Callaway: One of the guidances that we’ve given these teams is that they cannot go backwards in fidelity. So as it’s iterating through, it may be doing lots of internal work that never makes it to GitHub. So it only is going to be pushing to GitHub when it makes an improvement and set against its own fidelity check. So it’s got that fidelity gate that says, I can’t get worse. If I ever get better, okay, push. But if I get worse, stop, roll back, this isn’t right. So we have that internal build mechanism that Ensures that what developers are seeing and what they’re consuming is always at least as good as it was before, with a goal of making it better. The hope is that we can earn trust with developers who are depending on this tool to do so, but it’s still just traditional Rust code. Like, it’s not magic, you know. If you do a checkout and you’re like, I like the way this behaves, this reflects my expectations in all the ways that we’ve tested, use that commit for as long as you’d like. You know, you can always stick on that one build that you’ve got. We’ll keep building, we’ll keep committing, we’ll keep trying to make it better, but it’s not going to change on you in mid-flight.

Swapnil Bhartiya: When it comes to open source, I’m a huge, you know, of course, supporter of open source, but open source solves only day one problem. You know, you need features, you need functionality, you need a throat to choke as well. And sometimes some organizations, they don’t have internal resources to work on because open source may be free in cost, but it takes a lot of resources to actually run and manage it. So is there any potential commercial offering also in the pipelines that, you know, so somebody wants turnkey solution, they can just go and buy it? Yeah, just, you can do that also, or it’s going to remain purely open source project?

Tom Callaway: Yeah, I mean, we released this under an open license. You know, we encourage people to, you know, use it to its full extent. We don’t know all the things that people are gonna do with AWS Emulator. I mean, we certainly have some specific use cases that we’ve done for it that we know that customers have asked us for. We have done emulation and mock work in the past in a number of areas. So we have a pretty good idea of what people want to do with this. But one of the fun things about open source is you put it out there into the world and someone comes up and says, hey, I turned this into a toaster. And you’re like, wow, that was awesome. It’s not what I built it for, but that’s really cool. And so we are excited whenever our customers find something that we put out there and they grow it and they evolve it. And we want to encourage that here as well.

Swapnil Bhartiya: These days, a lot of discussions going on in terms of AI capability, AI-written code. And this is a great example of what AI, AI-written code can do, as you rightly mentioned, the focus is more on agents. If you get the agent right, the code it generates or any content, any work that it generates, you can trust that. So talk a bit about what does this emulator prove or show us that we can rely on AI if we do it right?

Tom Callaway: Well, I think that there’s a couple different thoughts here. I think the first thing is that, you know, we are as skeptical of this code as anyone because we depend on it. You know, internally, we have built a rigorous test suite and a rigorous benchmark suite to confirm that it does what it should do. And when it doesn’t do that, we revert, we, you know, we pull back, like this is an automated mechanism that has multiple checks and balances built into it. And we’re open sourcing it. So if you don’t trust us, you can always validate it on your own side. But I think that it’s interesting in that Some people might have tried to solve this problem by injecting a model into this where you would have an LLM or a small language model trying to determine the validity of an AWS configuration, training it intensively on a dataset. And we chose to go with something where, yes, the code is written by AI, but it is an artifact that remains static. It is not going to hallucinate the value on the other end of this. This is not an AI trying to guess the right solution. This is something where we can confirm and validate the fidelity of the solutions and make sure that when we say it works, we know it does.

Swapnil Bhartiya: Tom, thank you so much for walking us through this. Thanks for talking about AWS Emulator. And of course, those who are watching, you can learn more about the emulator at aws.amazon.com. And Tom, back to you. Thank you so much for your time today, and I look forward to chatting with you again. Thank you.

Tom Callaway: Thank you for having me.