AI Code Review at Scale: What Actually Works | Neeraj Deshmukh, Blitzy | TFiR

Swapnil Bhartiya: Now, as we all know that today AI agents can generate tens of thousands of lines of code in a single run. Which means that the bottleneck in software development is no longer writing code, it is reviewing and validating it. If a pull request arrives with half a million change lines and the engineer reviewing it only has a couple of hours, then the old review process is not going to work. So the question becomes what signals or reasoning needs to sit alongside each pull request so human can still decide whether it is safe to merge or not. And to talk through this, I am joined by Neeraj Deshmukh, Field CTO at Blitzi. Neeraj, it’s great to have you on the show.

Neeraj Deshmukh: Thank you very much Sapnikal. I’m very happy to be here.

Swapnil Bhartiya: It’s my pleasure. As this problem is becoming big and even people have started question, is there any job left of software development? Well, it depends on what side of aisle you are. But the fact is that a lot of, you know, labor industry manual job is being done by AI, which means that’s even more important for so software development job is more like review and audit the code that has been written to make sure it’s clean. You don’t just push things with all those AI slop. If I ask you. Actually, you know what, let’s just take a pause here. Before I get into this problem area, I would love to just know a bit about the Blitz yourself. How old is the company when it was founded, what problem you started the company to solve? And we are now because things are changing so fast in this space because of AI.

Neeraj Deshmukh: So Blitzy is about a little under 3 years old now. At that time, Brian Elliott and Sid Pardeshi, our now CEO and CTO had this vision to use AI to build code at the enterprise scale. So the fact that AI can write code was already well established with the advent of GPT and similar models by then. But the goal was how do we bring this to large code bases, to enterprise scale? Because those are the problems that are more difficult. Because that’s where the context of the surrounding code base, which could be like millions or even several millions of lines of code, comes into play. And that’s the problem that they wanted to solve and that’s how Blitzy was born. So Blitzy is effectively an autonomous software development platform that is designed from the ground up for enterprise for large code basis.

Swapnil Bhartiya: Excellent, thank you. And now let’s talk about this problem that as AI generated code grows in volume and complexity, why is code review becoming the new bottleneck in Software development, not writing code itself.

Neeraj Deshmukh: Absolutely. So definitely, I mean, generation of code has scaled with AI review still hasn’t, because the review process hasn’t really evolved. There is a working paper out of Harvard that’s coming up in a few weeks that talks about how there is like 30% more code being generated, but correspondingly, the PR to merge time is up by almost 50%. And the issue is that the reviewers, they lack the context and the evidence to go through that large volume of code. And that has become the bottleneck now. So just as generation has scaled, the review process also needs to scale with that. And Blitzy is actually providing a solution to do just that through a feature that we call PR review.

Swapnil Bhartiya: We will of course talk about PR review, we’ll talk about the solution part. Let’s just focus a bit on the problem area and how it’s actually redefining the industry itself, as you gave some examples, and refer to some of the studies. When AI agents are generating and kind of even validating increasingly large amount of code, what does it mean for the role of a software engineer itself?

Neeraj Deshmukh: I mean, AI can write the syntax, the code really well. So what that means for the software engineer now is the syntax writing part is no longer the challenge or the task. It’s basically telling AI what to write, telling AI how to understand the system, not just the specific part of the code that the AI is supposed to write, and then also telling AI how that code should be validated, which is equally important because writing code is one thing, but writing acceptable code, that is production grade, that is going to be actually put into your working services and applications. And how do you decide? How do you determine that this is indeed of that good quality? So doing all of that is basically the role of the human software engineer going forward. It’s to determine and define and then describe that target state, provide the constraints to the AI in terms of what must happen, what must not happen, and then provide success criteria and validation criteria to the AI and then let AI do the grunt work of carrying out all of those actions, but then making those decisions about whether the outcome provided by the AI indeed satisfy all of those criteria. That’s the role of the human developer. Now,

Swapnil Bhartiya: if you look at this human developer, what validation signals does that developer need to see when the merge is coming, which is written through AI before they can confidently review it and merge that AI generated code, as you said, because this code is going into production and we actually don’t have that luxury to just send AI sloppy code into production itself.

Neeraj Deshmukh: So I mean, I don’t like to use the term AI slop because at Blitzy we actually have an army of agents writing code, but they are all writing code to a spec. And that spec has been previously reviewed and approved by a human. And so that adherence to spec means that the code that the AIs are generating is actually kind of guardrailed by those constraints and those are essentially decided by the human in the lead. Right. And driving all of this is a knowledge layer that we have extracted from the entire software estate in the form of a knowledge graph that basically provides full context to every agent as part of that code generation. And so the way we do things in Blitzy is we have that system wide context through the knowledge layer, using that context and the task description that is provided by the human in the lead. And again, like I said, we want the intent, which has to be prescriptive, we want the constraints. Now those could be like regulatory compliance wide. They could be simply related to standards or technological constraints. And then the success criteria based on those, our agents will produce a spec. We call that the agent action plan. And this is a very detailed kind of roadmap of how that intent is going to be actually implemented that has to be reviewed and approved by the human. And at that point Blitzy will kind of unleash its agent swarm to do the actual code. But as the code is getting generated, we are also pairing that with extreme validation. So every line of code that gets written gets tested. The code is confirmed to be compiled, running in our kind of sandbox environment, we create and run tests, both unit tests, but also functional tests we can plug into the customers CI CD pipelines, integration test pipelines, end to end test pipeline. So basically your entire QA cycle has now shifted left and become part of that development process itself. Right. And we also do security analysis, dependency scans, we use cross model review. So like AI using one model will be writing code, but a different model will be used to kind of validate that work. And so what is coming out at the end is basically code that has been thoroughly tested, validated already, kind of passing all of your QA gates and that much closer to production as a result of that.

Swapnil Bhartiya: Excellent. I think. Now let’s just go a bit, start talking about Blitzy itself. I mean the PR review you have touched on all the aspects, but I want to just focus purely on how does it work and all the things that developers, engineers need to confidently merge code that has been written by AI. So give us kind of a technical overview of PR review.

Neeraj Deshmukh: Absolutely. So the motivation here is essentially what you mentioned at the beginning of our conversation, which is like PR review is the bottleneck. Now the code review is the bottleneck because when you generate code at scale, and Blitzy’s PR can be like hundreds of thousands of lines, sometimes even million plus lines of code, the line by line review is obviously not going to scale. And so we essentially split that review process into multiple stages. That plan approval that I talked about, the spec that we create that is based on your requirements, that is kind of step number one. So you are essentially making sure that what the AI is going to create is already aligned with your requirements. So that’s kind of one level of review, but then the actual code that gets generated. The motivation for review is threefold, right? You want to know what changed, you want to know why it changed, and you want to know how it was validated. So the PR review feature in Blitzy essentially provides you a group of these kind of key changes that are prioritized based on their importance, instead of just giving you kind of one rod there. And so for each of those groups, for each of those kind of key change sections, we are giving you that rationale of what exactly changed here and why it changed and how we evaluate, validated that it was working as designed. And the reason this is different is because we are not doing this as an afterthought, right? This is basically putting the evidence right on the pr. The logic or the decision making that went into the planning and the execution and the implementation is basically what you are seeing as the rationale for that particular section in the PR review. So you’re not coming after the fact and then trying to kind of reverse construct, like what may have happened here or why did this change happen? As the AI agents are making that change, they are essentially telling you, hey, I’m building this evidence for you, this documentation for you to tell you why I’m doing this. And then when you come to kind of the human review stage of that, you have all of that information as part of how it happened during development, not as reconstruction. So you have access to that reasoning as opposed to trying to figure things out after the fact. So this essentially allows you to focus your energy on just the things that matter. And there is a lot of additional code in there which is effectively kind of connecting all of those important sections together. But that’s essentially process flow and the validation that we do through the building and testing that takes care of making sure that those parts, the glue code, is working correctly. But the decision points are basically what we are showing you through this PR review feature as a step by step guide.

Swapnil Bhartiya: You use Blitzy to build Blitzy. That means you are in a unique position to actually dog food yourself. What your customers go through. Can you talk about what does that process look like and what are the areas where you feel that these are still the areas we need to improve on.

Neeraj Deshmukh: So certainly using Blitzy to develop Blitzy has been a transformational experience for us, not only because it has kind of put us in our customer shoes to kind of figure out like what works and what doesn’t work for us, and therefore by extension for them as well. It’s also given us insights into how, how our mindset about what software development means in the age of AI has changed. And so that mindset change is an extremely critical part of this whole transformation is probably the biggest takeaway that I would have from this process. Because we no longer have the kind of sequential nature of development where you have product requirements, then kind of a back and forth with engineering to determine the scope, then doing the actual engineering where you’re breaking it down into smaller pieces of work, like an EPIC gets broken down into user stories and tasks, and then those tasks kind of get assigned to different engineers to work in parallel and in sequence through different sprints. All of that has been completely appended now by this kind of all pervasive knowledge layer that gives us the full context across the board. So this kind of siloed and sequential nature of the SDLC has been changed to essentially be driven by this kind of a unified team, where we have representatives of the product requirements, engineers, qa, all of them working together to define these tasks at a kind of higher scale. So we are dealing with features at the EPIC level, not at the user story or task level. And what matters there is to essentially define the target state, what you want to see the outcome in the end, in a very clear and precise fashion. And then the platform basically works with you. We have a feature called Chat to Prompt. So essentially you can talk to the code using that knowledge graph and work with the code base to say, hey, I want to make this change. What’s the best way to make it? How do I go about to achieve this outcome? And the code works with you through this Chat to Prompt feature to essentially give you that clarity in terms of what are the things you need to consider, what are the judgment calls that you need to make. And then at a push of a button that gets transformed into a prompt that now you can feed Blitzy as essentially the task description. And Blitzy will give you a score associated with the quality of the prompt. It’ll ask you like, hey, here are certain things that I need you to give me more information because it’s not clear yet completely, it’s ambiguous. And so you work through that and you kind of get to that point of like, I know exactly what I want, this is how I want to describe it. And now you go through that agent action plan generation and continue that. And by folding in your QA cycle as part of the development, you have effectively achieved a huge amount of speed up in your ability to deliver these kind of finished product ready features at the end. Now we don’t do 100% of the work. No AI system can deliver 100%. And for a variety of reasons, sometimes the task description is not clear enough, sometimes you do need subject matter expertise that’s outside of what we can glean from the code base instance itself. And so the human engineers will come in after Blitzy has returned its pr. And Blitzy will also give you an evidence document that we call the Project Guide, which tells you everything that we have done, but also everything that needs to be done and what effort, how much time it’s going to take in terms of estimate. So the human engineers can come in, they can finish that remaining task. We typically call it the 8020 split, but quite often it’s more than 80% work that is done autonomously. But once that is done, basically you’re ready to go. And again, nothing stops you from doing multiple such projects in parallel, because Blitzy’s agents are running continuously while they’re generating code for days, sometimes even weeks, depending on the complexity of the project. So your engineers can basically move on to do another project, go through the same kind of requirements to prompt to AAP and then kind of set up the code generation. And as projects get finished, you can come back to do the PR review for them and take the following steps towards the remaining 20% finish up work and then the deployment. So essentially running projects in a staggered fashion, achieving this kind of accelerated and fully agentic SDLC, those have been truly transformational for us. And it has allowed us to essentially share that knowledge, share that working mode with our customers, and help them with their own agentic transformations in a similar fashion.

Swapnil Bhartiya: Now let’s look at maybe an edge case where generate PR still fails review in those cases. And they may be happening where it will not make into production what is most of the cases. The root cause is it the Code? Is it the context around the code? If you have come across. What are those reasons?

Neeraj Deshmukh: Yeah. So typically the reason the generated code is not up to the expected quality is usually to do with ambiguous requirements or lack of testing. So like I said, we plug into CI CD integration end to end tests. If we don’t have access to those test pipelines, then all we have to do, all we have to fall back on, is unit test and functional tests. And so the more testing we do, the more we are confident of the quality of the outcomes, the less testing that happens, there are chances of things not being as correct. And then of course, if there is like additional subject matter expertise required, which is not accessible through the code context and that was not provided as additional knowledge through the prompt when the task was specified, then that can also be a concern. The good thing is when something like this happens, Blitzy has a feature called Refine pr. So you can kind of review the pr. You say, oh, some of this is looking quite good, but here is something that is not quite right. And so I can now say, okay, well let’s define this pr. Let’s build on top of the work that we’ve already done with these additional instructions. And so Blitzy will go and then kind of do that work and give you a new PR which is built on top of the previous pr. There are a lot of times when customers see that and they say, oh, I forgot to do this thing, or I wasn’t quite clear enough about this thing. I meant X. When the AI interpreted that as Y, you can kind of go back and say, no, no, I meant X. Change that implementation for X. And that’s basically what Blitzy will do and generate that kind of much closer to your requirements code as a result.

Swapnil Bhartiya: Where does Blitzy fit in? How does it fit into a developer’s workflow pipeline? And second is if they want to try it out, how they can try it out.

Neeraj Deshmukh: Blitzy is not a tool that is meant for making an individual developer faster. Like you see with a lot of the kind of ID based coding agents, right? It’s supposed to make the enterprise level large scale software development faster. Right. And by like order of magnitude, like so the way we do that, like I said, is number one, kind of creating that knowledge layer through reverse engineering the code and then combining agent orchestration at scale with massive and extreme validation. That requires a certain mindset change in terms of how you do software. And so for a lot of our customers that change management is equally important as kind of integrating Blitzy into their workflows. And so we as a company make sure to be very closely involved with our customers, provide them support through forward deployed engineers or AI solution consultants, like work with them hand in hand to get them up to speed and to kind of help them understand how to best use Blitzy. And one such thing for our kind of prospective customers to kind of try out, because this is a very different way of doing software, is to use a feature called Blitzy Sandbox. It essentially allows any kind of qualified company to try this out on their own code. Up to a million lines of code base can be ingested and I think about 100,000 lines of code can be generated for free so they can try this out for themselves.

Swapnil Bhartiya: Neeraj, thank you so much for this really deep conversation and of course for viewers, of course you can go and check it out. As Neeraj mentioned, check it out on blitzy.com Neeraj, once again, thank you for your time and I look forward to another great discussion. Thank you.

Neeraj Deshmukh: Thank you very much. Pleasure to be here today and indeed I look forward to talking to you again soon.