GitHub App Keys That Never Expire | Gaetan Ferry, GitGuardian | TFiR

Swapnil Bhartiya: When it comes to credentials, a leaked password gets reset and a leaked API key gets rotated. But leaked GitHub app private key just keeps working because it never expires and whoever holds it can act as that app with every permission it was ever granted. So a key committed by mistake years ago is still a working door into your organization today. GitGuardian just tested thousands of leaked GitHub app keys pulled from public commits to find out how many are still working and live. To walk us through the findings we have with us today, Gaton Farish, security researcher at the gitguardian Geeta, it’s great to have you on the show.

Gaetan Ferry: Thank you for the invitation.

Swapnil Bhartiya: It is my pleasure. Of course we will talk about these findings, but before that, just to give our viewers a bit of a context, quickly tell them about GitGuardian and the role that you folks play in this space, especially when we live in an API AI driven world.

Gaetan Ferry: So GitGuardian is a French company that specializes, specializes or specialized in detecting and remediating leaked secrets at scale. The very core of the company as it was founded years ago was to detect leaked credentials in public GitHub commits. Obviously this evolved a lot and now we can monitor Docker Hub internal systems too. That’s really where we sit. And so we worked a lot with leaked credentials, which is a topic of today’s discussion.

Swapnil Bhartiya: When you folks are working on this project, you start with some roughly like 500,000 exposed RSA private keys and then you narrowed it down to about roughly 4800 which were tied to GitHub apps and then found that out of these 4800, 474 were still working, they were actively being used. So if possible, can you walk us through this process so as to ensure that the pain that you folks took to go through it and then pinpoint the problem?

Gaetan Ferry: So as you said, we started with a data set of half a million private keys. Those we collected over the years since the very start of the GigGuardian company in 2017 up to basically today or yesterday. And those keys mainly come from public GitHub commits. Some of them also come from public Docker Hub images and other public data sets. And so as you said, we narrowed down this data set to a set of a little bit less than 5,000 private keys that were tied to GitHub app context. So we basically performed some contextual filtering, looking for specific keywords or specific client IDs near the private keys in their lib context. And from those 5,000 keys using them we started to create authentication tokens. I don’t know if we want to go into too many details about how those keys are used to authenticate on GitHub, but basically we can sign an authentication token using this key and then just use it on the API to determine if the key is actually working, if it’s tied to an existing application, and also to have an idea of the ownership of the key. So basically which application it’s bound to and who is the owner of this key. And Yep, we found 474 live keys at the time, which was kind of a surprise because it’s a lot of live keys. That’s more than 10% or about 10% of the keys we tested. And that’s three to four times the average validity rate that we observe on the public item. So that’s really a lot of the private keys.

Swapnil Bhartiya: Yeah, we’ll get into that in a bit. What I also want to understand is that, you know, if I’m not wrong, GitHub app private keys never expire. That’s how they’re designed for. So app is working. But why does that make them different from API keys or service account credentials that a CIO’s team already rotates, you know, on the regular basis?

Gaetan Ferry: The main difference, as you say it, is expiration time. GitHub app keys are a little bit like SSH keys. You create it once, then you never really touch it anymore and keep on using it. So it’s basically the same. And with this lack of exploration comes a lack of automation. You can’t really easily revoke a key, create a new one and rotate the keys on GitHub. So that really makes them very long lived credentials as opposed to short lived credentials like we can have in host protocols or IDC or even shorter lived API keys, even if those are not really short lived credentials but that have an expiration date like, I don’t know, six months, a month, a year. That’s really the main difference, the lack of automation. That means that when you have a key it will just keep on working until someone just goes on the GitHub UI and clicks the buttons to disable the key and create a new one. Which is a pain and basically the reason why nobody rotates their keys.

Swapnil Bhartiya: What is also interesting about most of the compromised apps that you found has a single installation, unlike, you know, the whole marketplace where they are installed on customer submitter. So the attack surface was mostly internal bots and one of automation rather than the whole market place tool. What does that say? About where the risk really sits inside an enterprise. What are they exposing themselves through these keys?

Gaetan Ferry: As you said, there are really two cases. The app keys we found sit in two categories. The keys that are bound to applications that have, let’s say hundreds of installation, meaning that they are kind of supply chain components that you will install on your GitHub organization. And the other category, which is the vast majority that are installed on 1:2 organization, meaning they are bound to applications that are more one of tools that are only used inside an organization. That means first only the organization itself is responsible for the security of the key and of the app. So that’s already something. And if we look at the reaction and the responses after our responsible disclosure campaign, what we have observed is that most of the apps were disabled after we reported the leak. The keys were not rotated, companies just disabled the app, which means in most cases the apps they were not useful anymore. The keys had been there for years, the apps had been live for years. In most cases they were just forgotten tools, tools someone set up for short tests and just nobody removed them. And that’s really where the vast majority of the keys lie and that’s really where the risk is. Forgotten applications that are still open doors for organizations, that’s really the main risk. And obviously when the permissions bound to those apps are very high, which is again a vast majority of the keys we’ve observed, the risk is even higher.

Swapnil Bhartiya: The difference, as you very well explained, between Regular API Keys vs Apps is also critical. Does this finding change anything? The way organizations are of course creating these app keys follow better practices or those practices are in place already is just the way most of the time we come across a lot of zombie code, zombie APIs sitting there, somebody created, they never bothered to delete, people moved, they leave the job, they got promoted. So what is the cause? Root cause? Did you try to also find out what are the root cause that these live keys are still there?

Gaetan Ferry: That’s a very good question. First off, what we’ve observed is that in a lot of cases the keys they had leaked years ago or at least months ago and nobody noticed. So first the key, when it leaks, it’s not detected. It can just stay there for years. If no one notices, then the applications, it’s a little bit more difficult. We don’t really have an idea of the root cause. We have observed examples where for example a given open source project add a key leaked and the project wasn’t even live anymore. So it had received its last commit like in 2022. So it’s four years from today and the application was still there. The thing is, we even didn’t manage to have proper contact with the former open source project. So that means the key is still live today and there’s just no one to revoke it. So it’s a symptom, as we could say. I don’t have a precise idea why those applications just keep on being there. It’s really, I don’t know, a mystery.

Swapnil Bhartiya: Now, your report on the CDC’s leaked app key took like two weeks from disclosure all the way to revocation, and it went through the HHS disclosure portal. What does the timeline tell CIOs about their own ability to receive and act on an outside report? Because sometimes that time, depending on who you ask too much for bad actors, they’re like, hey, that’s a golden opportunity for me.

Gaetan Ferry: There are really two topics here. On one side, there’s how easy it is to contact an affected organization, and on the other side, there’s how easy it is to actually act after you receive the report. The CDC history is pretty interesting because it’s been very easy to identify the proper security contact. They have the information publicly available, so you can just browse the CDC’s website and you are redirected to the HHS responsible disclosure portal. So this is very easy. It’s not always the case. And in a lot of cases companies don’t have an easily identifiable security contact, in which case it can be very difficult to report the issue in the first place. So that’s really a first attention point. A company needs to have a public security contact exposed somewhere, either on their website or in a security txt file, for example. So for the CDC case it was very easy. The contact was easy to identify. Then once we reported the issue, it actually took two weeks for the key to be properly rotated. And that shows one thing, which is rotating credentials isn’t an easy problem, especially if you don’t have enough visibility on your credentials, where they are used, where they are plugged with. You can’t just rotate a key and break your production systems. The key we found for the CDC was production critical. If you had rotated the key automatically without thinking about it, you would have broken the whole software pipeline of the CDC mostly, which is an issue in terms of availability. And so I guess we don’t have any inside of you inside this revocation process on the CDC side. But I guess that it took the CDC a long time to evaluate what it was necessary to change in their process for the key to be safely rotated. That’s most probably the reason.

Swapnil Bhartiya: Now what does that also tell CIO or CISO who is watching this and who wants to have a better control over the whole secure departure within their own organization? And they want to know whether their organization has this exposure. What do they ask their team? Who in their team do they ask? And should they also reach out to gitguardian or if there are other players as well, how much work they should do themselves versus reaching out to experts?

Gaetan Ferry: Well, I think there are multiple things to look at regarding this GitHub app issue in general. First, I think it’s very important for every organization to have a precise view and understanding of what app is installed on their organizations, because as we’ve seen in our campaign, in most cases applications they are not useful anymore and so they should have been uninstalled months if not years ago. But the first thing to do is probably audit your GitHub organizations and find out which applications are installed and which one can be disabled. A second interesting point is related to permissions and scopes. If some apps are absolutely needed, a good step is to audit the permissions this app has and decide if those permissions are safe. If they are not over privileged, and if they are over privileged, then downgrade those privileges and similarly reduce the scope the app has access to. So if the application is necessary, is useful only on one repository, let’s say, then just downgrade the scope to that single repository. That doesn’t really solve the key issue, but at least if the app gets compromised either via a key leak or through a supply chain attack, if it’s a marketplace application, then the impact will be less severe and the blast radius will be lower. So I think that that’s really part of the hardening strategy, which is often very important. And obviously on the key leak issue, the best you can do is monitor your perimeter for leaked secrets. That’s obviously something GitGuardian can help with and probably other companies can do too, but that’s really an important part of it because again, what we have observed is that the keys often had been leaked years before we performed our research and disclosure, which means no one was aware. And not being aware is definitely an issue. And maybe the last point would be to rotate the keys on a regular basis, which is an issue right now because GitHub doesn’t offer this rotation mechanism, or at least not in an automated way. But if companies organizations are conscious that those keys never expire, then they can take the necessary steps to actually manually revoke them, even if it’s only once a year, that’s already better than nothing.

Swapnil Bhartiya: And if I’m not wrong, 44 of the apps behind those keys had organizational administration privileges. What does that mean for an attacker? What can they do with that level of access?

Gaetan Ferry: Well, the most obvious risk in that case is a supply chain risk. If you have organization administration permission, you can basically change whatever settings are set at the organization level, including changing the permissions on repositories. And so that means in a way or another, the threat actor would be able to change the source code of any software that is stored on the organization’s GitHub. They could change the workflows, they could change the CI CD systems, they could most likely extract secrets from the CI CD. So in that very specific case, the risk is very high. And everything that we’ve observed recently with the open source supply chain attacks, this can absolutely happen if an application with organization administration privileges gets compromised on a smaller level at a lower scope of only one organization. But that’s basically what can happen and it can be very severe and very, very bad. But even if this level of permission isn’t reached, if an app leaks and it has slightly lower permission, let’s say only content write permission, this app can still change the code that is stored on the organization’s repository. And that means it can already poison the source code, deploy malware, and that’s already pretty bad. So in most cases, and that’s basically what we’ve observed with the leaked apps we have studied, in most cases the permissions are sufficient to compromise a whole company just through this leaked key and compromised application.

Swapnil Bhartiya: Now can we also talk about the tokens that keys generate, when do they expire and the lifespan of this token? What does that mean for safety?

Gaetan Ferry: The keys that are used by applications, they are used to generate ephemeral tokens which we call installation tokens, which are used to impersonate the organization the app is installed on. So those tokens specifically, they expire very, very fast. It’s a maximum of 15 minutes if I’m correct. So those are very short lived. Leaking such a token is not a good idea, obviously. But if you leak such a token, it will be live only for 10, 15 minutes. So the window of opportunity for a threat actor is actually pretty short, pretty small. So it’s bad, but it’s not that severe. The thing is, once you have the application private key, you can sign those installation tokens at will. So when one expires you can just generate a new one. And so the fact that they are short lived doesn’t change anything. Once the long lived credential has leaked.

Swapnil Bhartiya: Or has been compromised, what impact is AI having? Is it making things better, is it making things worse? Or also, did AI also play any role in this research and report of yours?

Gaetan Ferry: That’s actually a very good question. For the easy part, we absolutely used AI as part of this research, as I guess everyone does by now, mostly for code writing and fast exploration, prototyping, nothing too advanced. On a more global level, what we have observed is that AI is making threat actors faster and more efficient. We have already observed attacks performed by completely autonomous AI agents even back in March of this year or something like this. At the time of the IKEL campaign where AquaSec was compromised. So it’s definitely making things worse from a defender point of view at least, if you only consider the threat actor part of the question. And we can expect that an AI agent will be able to find the keys and exploit them in the future if they are not already able to do it. I mean, I’m pretty sure they are able to do it. I don’t know if threat actors are actually using AI agents to do it, but AI agents will be able to do it. So having leaked credentials is going to be more and more of a problem in the future in my opinion, because basically AI is going to make it very easy and very fast to detect, even in the high volume of data that a platform like GitHub can host.

Swapnil Bhartiya: Gaetan, thank you so much for talking us and taking us through this research and the findings and also kind of telling organizations how they can prepare themselves for these kinds of issues. This is not a technology problem, this is more of a human problem. And we always believe that human culture and people challenges are more challenging than technology because this is not something you can solve with just one key. So thank you for sharing. And those who are watching, please check out gitguardian.com to learn more about how you can protect yourself with leaked credentials. And once again, thank you so much and I would love to have you back on the show.

Gaetan Ferry: Thank you very much.