Swapnil Bhartiya: Hi, this is your Swapnil Bhartiya and we are here at Open Source Summit in Minneapolis. And today we have with us once again Kate Stewart, VP of Dependable Embedded Systems at the Linux Foundation. Kate, it’s great to have you back on the show.

Kate Stewart: Lovely to be back here. Thank you very much for your interest.

Swapnil Bhartiya: It’s very exciting time. It is ten years of Zephyr.

Kate Stewart: Exactly.

Swapnil Bhartiya: And we have.

Kate Stewart: Not so fast. Yeah.

Swapnil Bhartiya: And you know, I have been part in one way.

Kate Stewart: Part of.

Swapnil Bhartiya: Yeah, yes. And so it’s. There are a couple of things here. First of all, it’s exciting and second is that because of your own background and security, of course SPDX is there and then other project. Zephyr is also one of the most secure. I think it’s only one which is ready for CRA and all of this. And then of course, even if it’s not as big as Linux kernel, no pun intended, whether it comes to size, but the kind of standards it has set, the kind of community it has built, there’s so much to learn from it. So the point is there is so much to talk about today, I hope. Exactly. I will start with the very basics for those who may not know what Zephyr is all about. Let’s start from there.

Kate Stewart: Sure. So Zephyr is an open source RTOS real time operating system. And it’s more than just a kernel though. It actually has a lot of peripherals and communication stacks and information integrated into this, into the code base. So you can basically just configure in what you need and build what you need at any point in time, as opposed to having to go find extra libraries to make a USB stack work or make your Bluetooth work. It’s all integrated and that has made it an easy solution to a large extent for people who are actually trying to use the technologies, especially if they’ve come from Linux, embedded Linux. There’s a lot of things that are in common with it. And at this point now there’s over a thousand boards in the repo and you know, we’ve got like 9,000 forks of the repo, so there’s a pretty good chance. Do you know which processor’s on your laptop? Inside your laptop? Because it looks like a pretty new laptop. Right. There’s firmware in here before Linux gets brought up. Right. The firmware now is likely to be a Zephyr. Okay. The firmware and the Google Chromebooks are running Zephyr. Zephyr is turning out to be a pretty good solution beyond just IoT at this point now, however, its heart and sweet spot is in the IoT realm and we’ve got lots of more and more products showing up, which is my fun part to find. As you know from our prior discussions. So we’re seeing this, but things like trackers, things in the industrial space, if you’re familiar with keychrons keyboards, they’re running Zephyr now, and actually that got them a lot better polling rate and battery life. It’s a tremendous improvement. So we just finished putting out a 10 year report and we interviewed a bunch of. As part of it, we did a survey and got some data about what’s working, what’s not working. And we didn’t just survey the Zephyr people, we actually surveyed anyone who worked with rtos that we could find. We got a good response rate from non Zephyr as well. We were comparing with what between Zephyr and non Zephyr, what is the numbers, what do things look like? And so it’s given us some good insight as to where we should aim for next. And it also is sort of telling us what we’re doing well and where. Mostly most people are using Zephyr. Sorry, most people that are using Zephyr are using it in the 32 bit space still and ARM M cores. Okay. RISC V is probably the next most popular portable from last. Yeah, indeed. And so we’ve got. So we’re sort of seeing those two platforms and then there’s a whole range, you know, beyond it, for the ones who are responding to the survey. So we’re sort of seeing this stuff and we wanted to figure out, okay, we’ve been here for 10 years. Before we started the project, we did some focus groups and a bit of research to figure out why do we need another rtos. You know, there was a whole bunch of options out there, why don’t we just get onto one and start doing it? And we really decided what the open source developers told us at the time is we kind of needed to have some security. And we kind of want to be able to have a safe rtos, to be able to use the safety systems, because a lot of the embedded stuff goes into safety. And so that’s why when we formed up the project, that was the starting point right from the start, was we were gonna go for safety and security as well as, you know, getting all the connectivity integrated and make it easy for people to use us. And that’s kind of how we started. And we’ve been changing things, listening to developers ever since, and it’s worked out well for the project over time.

Swapnil Bhartiya: Excellent. Thank you for sharing that journey. And I’m also kind of curious. If you look at these 10 years, you folks have achieved a lot. So I’m not going to ask you to just talk about everything, but talk about some of the milestones that even when you look back and you’re like, those are like game changer, not just for the Zephyr community, but for the ecosystem and for the industry itself. What are those?

Kate Stewart: I think one of the game changers early on for us is when we actually became, we had enough of a security posture and the processes in place, we could become a CVE numbering authority back in 2017. And so, you know, we worked and so we’ve been able to handle and manage our own vulnerabilities ever since. And I think that was a significant milestone for us because it said, yeah, we’re serious about it. And then we’ve, you know, another one was when someone did an audit on us and found a bunch of things on the security side when we thought we were doing okay, and oh, okay, fine, we’ll go fix them. So we fixed it and we, you know, moved it forward. And so adopting the security best practices, you know, getting the Bluetooth, working with the, like the Bluetooth Sig and getting another thing that served in the early years for us, the fact that we Nordic was spearheading a lot of this at the time, the fact that we were working on Bluetooth and the participating and using Zephyr to prove out the spec before it became public. So as soon as the spec was public, we had a reference Zephyr app, Zephyr Port, so people could use it. That’s why we got a lot of audio stuff. Another thing was the integration of some of the AI and basically TensorFlow Lite and being able to show that you can work with Zephyr on the very smart, small, resource constrained edges that helped to shift things and then figuring out a path for doing our safety. Finally, like I say, doing open source and safety is an interesting challenge, to put it mildly. And that we got concept approval. We don’t have our certification, but we have our concept. So if we can do what we say we want to do, the authority says they’ll be able to give us our certification. So we got our concept approval at the end of 94. And so these are some of the things I think are milestones. But I was doing the security best practices all the way along meant that, you know, we were pretty much ready for the cra. The only thing we’ve got left to do from a steward perspective is to figure out who we’re going to work with in Europe as our csearch. Over there we’ll probably line up behind Linux. But we do need to make a decision, we do need to reach out and get that established and that’s something we’re planning for working on this summer, next month and the month after to the deal. So I’m hoping that I can go tick, tick, tick, tick, tick on my little checklist for this stuff by the summer. And so when we head towards the fall, we’re pretty much ready to go and we can work from there. So that’s kind of. Those are some of the make cells. But the fact that we’ve. Oh, I’m silly. The other thing that was a milestone for us was five years ago we were generating spdxs bombs. So we’ve had automatic SBOM generation for five years. And so you can be very precise about what is actually in your images, which for embedded is important. But to do it all you have to do is a couple of command line options. So it’s pretty straightforward.

Swapnil Bhartiya: And that’s your background is spdx and either way. And we have talked about SBOM before SBOM became a thing, if you remember that. Yeah, ye. So I mean you folks have been kind of ahead of the curve. Yeah. So now I also. You talked about some of these. I’m also kind of curious about the milestone in terms of devices support. You used to talk about the storage device at one point, but what devices when they were added or they started using Zephyr and you’re like this is also kind of game changer in the terms of ecosystem growth from the hardware perspective. What are those?

Kate Stewart: The tracking devices. So the things like the pet trackers or like the solar power trackers that go on the side of a container and the fact that they’re running Zephyr in there and they’ve got enough battery life that they can work off of solar for months on end. One of the ones that is I learned about a couple years Emba Int is one of our members that they joined about two years ago, Emma Ema int.

Swapnil Bhartiya: Okay.

Kate Stewart: And what they do is they’ve got effectively they’ve got a device for helping manage cattle herds on and animals on ranges and they’ve got a collar with some solar on it and they Zephyr sense and doing some sensing about, you know, motion tracking and you know, so they can sort of make sure the areas don’t get over grazed and the health of the animal and things like that. And they use AI to basically process things on the cow. And then once again, once a day or so they send the signal up to a satellite with no other infrastructure. The fact that you can go from a solar device, you can go from cow to satellite completely blows my mind. You know, just that that type of communication is possible with such a low power device. There’s not a battery, it’s all being. And so you can completely see the application, the stuff, the technology orig originate in Australia where you understand that. And then so it’s being spreading around. But some of these very low power areas and devices that are very low power have been making a bit of a game changer, especially when people become aware of them. One of our members, Demont or Oticon does hearing aids with Zephyr. And so. And then I guess about two. Yeah, about a year ago. Zephyr started out as a satellite rtos. Okay before it was, it was a small, it was part of a company that was doing the work. It was fully safety certified at the time. Wind river acquired it and then put it on the shelf. And then when intel had acquired Wind river and they needed to look for a good starting point, they looked at this little satellite operating system that’s sitting there and they, and then the team in intel started, you know, working with the Wind river team to pull it apart and reformate it into something that they can use. And that was the start of Zephyr. So it started off as something satellite and then last year it’s back up in, it’s back up in space now too. So in the Thera satellite they’ve been visible that they’ve, they’re using Zephyr up there. And so, you know, finding, you know, these sorts of things are kind of cool. And so on our website you’ll see the Atheros one and you can read up about it a little bit more if you’re curious. But these are the sort of interesting changes because from simple biomonitoring types of devices like rings, smart rings, things like that, wearable technologies like there’s an open source watch, but things to help people’s lives be better, things to track your pets, things to there’s low power, they’ve got to last for a long time to sensors on the top of garbage cans to make sure there’s a can full, does it need to be empty? Because it’s running AI to figure out is everything full or not. And so it’s very Pervasive and very everywhere, like in your laptops, in the Chromebooks and so forth. In fact, from the survey, some of the respondents in the survey are already shipping over tens of millions of units with Zephyr in them. Okay. Which I confess, after 10 years, it really quite blows my mind how much. You know, I think it was like 13% of our respondents are shipping over 10 million units. Something like that. So the world’s changing with this stuff. And it’s just hidden underneath the covers.

Swapnil Bhartiya: Yep. When you talk about space, I am just. This may be a very, very stupid and dumb question to be unrelated, is that. I mean, we hear a lot about Voyager, which is just reaching the edge of the solar and. And of course, it is running out of fuel, it’s running nuclear, but they had to shut down some sensors. These kind of applications, space exploration, where you have low power, but you want to run forever. If Zephyr was back then, in 1970s, you know, when some of the space exploration was happening, now, we are not doing that much. What is the. What is the what? What role can Zephyr play in these kind of history? I’m just kind of curious to hear, because these are the things where you just send the device, you don’t talk to it ever. Yeah, of course, you send signals, you update back, you keep it updated. But what I’m trying to say, these are not very powerful devices. You know, they’re very, very. So what kind of applications do you see of Zephyr in those use cases? Because we don’t hear about.

Kate Stewart: So where everything is seriously restrained and the power budget is very intense. Zephyr is showing it. It’s a good fit these days. But for, like, you know, for the power budgets, you sort of design to

Swapnil Bhartiya: them, well, you have power bed, but you also want all those capabilities. You want those sensors.

Kate Stewart: You know, you want the sensors to be running and everything else. And they take power when they run. Yeah. And so you have to figure out how often do you run them, how often do you cycle things up. The thing about Zephyr is the cycling is pretty. You know, the cost is small. That’s why it’s running. When, you know, when the machine is, quote, unquote, powered off, that’s usually when Zephyr’s running. And so that’s what sort of why we’re happening. What’s happening with us up there is figuring out how do we use Zephyr, how does it work? And as you design a system, this is all system design, you know, what sort of redundancy you want to build into your calculations. Realistically, the fact that we’ve had Voyager running all these years is, you know, like, when did it launch? Gosh. Do you remember?

Swapnil Bhartiya: I think it was launched. I think year date I 77. I think I was something like that. So it’s as old as I am.

Kate Stewart: Okay, that’s why you came to pay attention. Very good. But, yeah, one of the things that was coming out of the survey is the bulk of people doing products with Zephyr expected to be lasting for more than five years. Okay. Like I say, the. The tale of it all is very much into five to 10 years is the biggest point. But we have, you know, people using Zephyr for 30 years. And, you know, at home, I have a car that’s 30 years old. Yeah. And so, in fact, it’s more than that now. And it had a very simple microprocessor in it at the time. And I compare it to what I’m seeing in current cars, modern cars. Like, in fact, I couldn’t find a replacement AC unit for it. Yeah, yeah, yeah, yeah, yeah. And in Texas, you need a replacement AC unit. So I broke down and I finally bought another car as a second car last year. So I would have a car with AC for driving around in the summer because I couldn’t replace my Volvo AC unit or my mechanic couldn’t find a replacement part. But Honda, for instance, is one of the members of Zephyr. Oh, okay. So I’m kind of looking forward to seeing what shows up in Zephyr in cars in the future, you know, on the sensors, actuators, and the high assurance. And so after we get 61508 safety certification for industrial, we’ll probably go up to 26, 26 too. So these are the plans we have in the project.

Swapnil Bhartiya: When we look at the ecosystem, I’m not talking about the software, but also the hardware players that play. How has Zephyr changed the way companies, some of them were new to open source, or so you folks actually became a catalyst for them to adopt. Once they start using Zephyr, they did not look at Zephyr because it’s open source. They look at Zephyr because it was, you know, time to market and resource efficiency was there. But when they came to the fold, then they also learned about how open source works. And then suddenly they realized, hey, you know what? I don’t have to do a lot of work. You know, the whole community is doing. That also changed the way they used to work with other companies. Now the companies they will never interact with, they have kind of become partners. So can you also talk about how Zephyr has also changed the way companies used to work with embedded systems or rtos and now the way they of

Kate Stewart: the silos are coming down and what we’re seeing is a lot of those companies, I roughly estimate this is mine. And some of the. For every 1 upstream contributor, it’s probably 10 people using it. That’s my rule of thumb. And a lot of those 10 other, you know, nine that are using it, nine, 10 that are using it, they’re not used to contributing upstream. But then they start getting really frustrated when the things start shifting out from underneath them and they have to play catch up. And that tends to provide the catalyst for them to start looking at participating in upstream. And so Zephyr has gotten a lot of people starting to participate in upstream that weren’t there before and also quite frankly join the project, which helps support the project, quite frankly, because a lot of this stuff takes money to keep running. And so if they’re making money with Zephyr, if they can help support the infrastructure, we certainly appreciate it. If they contribute code, sure, that’s going to help them and help us in the long run, but it’s figuring out how we get the virtuous cycle going so that they can build off of what they’ve got. Up till now, these are the things that we need to sort of. We’ve brought some people, there’s been a lot of people that have just done bare metal only just very simple loops, very small and bringing it up from that side. And so those people had never participated. A lot of those have participated in open source and now they’re getting there. And so now they’re becoming part of the community. And it’s good for these organizations. It’s also good for the developers in the sense that, you know, we’ve had people in our community at this point in time who stayed in the community and have gone through two other jobs since I knew them first. And they’ve all stayed in Zephyr each time. And so that was a trait of the Linux community, had. And the fact that we’re seeing that same behavior in Zephyr has made me happy because that means people can build careers on it as well, which is exciting.

Swapnil Bhartiya: I also want to ask you one thing, is that is there anything that you would like is on your wish list that the whole ecosystem that is around Zephyr, that either companies do a little bit more in the way of contribution? Second is that there are a lot of other organizations, there are other ecosystem that you feel they are. Zephyr is sweet spot for them, but they are not still tapping on that. So this is twofold. Question one is what do you expect from the members of the people who are already interacting with Zephyr? And second is the community you’re seeing, they can benefit from Zephyr.

Kate Stewart: So what I welcome our community to be doing is the project, a subset of the members is working very closely on getting the safety certification and understanding the code to the level that we can create. Requirements and links to the code and links to the test so we have the traceability for safety. Getting more of our members engaged in doing that and others in the community interested in doing this will accelerate us getting the kernel up there. But this pattern is a pattern of contribution of analysis. And so my wish list is that beyond what we do for our initial certification scope, others use, as they use our work there and they do this driver over here that wasn’t in the scope or this stack over here that wasn’t in the scope, they do all the analysis that they contributed upstream. So they contribute the analysis upstream. The maintainers sign off that, yes, what you’ve articulated as a requirement truly is a requirement, and then everyone else can benefit from it. And that way we build our safety scope out the same way we build our code out. So that’s my wish list. That’s my big wish list. Now, your second part of your question

Swapnil Bhartiya: was that there are a lot of industries or ecosystem which you feel are Jeffrey’s sweetest spot for them. They should be tapping into it and they have not yet one of those.

Kate Stewart: So the ones that have not really tapped in, they’ve tapped into it a bit. And I think there’s a lot more to come is the home consumer. So I know some of the home consumers. I think you’re going to be seeing a lot of appliances start to emerge in the next few years. I’ve been having discussions with people and we have, we have some.

Swapnil Bhartiya: When you say home appliance, I’m just interrupting you for a second. Like I cook a lot, so I have a lot in early time. It used to be just electric motor. I pushed a button and it starts to make my dough and a dumb. But these days, almost every device has some microprocessor. Yeah, exactly. So I feel that there’s already. They’re already a customer, you know, potential. So when you say appliance, I say almost every appliance when you look at home. So I just want to set the stage for that.

Kate Stewart: And then they. You have a Bluetooth signal working to tell you what my dishwasher, I can

Swapnil Bhartiya: control it from here. And I like why, but it, it, it makes sense, you know, that I, you know, the cycle. I have Bosch. So when you’re talking about appliances, just for the sake of, you know, audience who may not know what appliances already have them or what appliances are using. No, they don’t. They’re not using Zephyr. They should be using it because this is the.

Kate Stewart: There’s some that are working towards using Zephyr and so. But more should be, I think, the heart of it. It’s basically got the connectivity built in. It’s got a wide range of processors available to it. So some supply chain resilience. And so people can be focusing on the applications, the interfaces, things that make their brand, you know, efficient for people and make people happy using their brand. And the lower levels are just, you know, built up from there. And so that just makes it so much easier if, you know, something happens with, oh, some weird tariffs come into play or something like that, and they can’t get the parts they want. The fact that they can keep production going with a different processor with a fairly lightweight movement, because Zephyr’s interfaces are all standardized, it allows them some flexibility and supply chain resilience. So the places where there’s, you know, aspects of that, people want to have some resilience in their supply chain. I think Zephyr is a good fit right now because of the number of boards we have, especially when the power is a consideration. You know, devices that have to work unconnected to the walls and the wires, but also with the ones connected too, as well, because you don’t want your power bill being high either. Right. And so, you know, I think printers, I think, and, you know, various laptops, you’re seeing, you say we’re seeing it all that way.

Swapnil Bhartiya: When you say printers, I mostly think about 3D printers because I do a lot of 3D printing there.

Kate Stewart: I wouldn’t be surprised. But I don’t know for certain.

Swapnil Bhartiya: Yeah. Because I think the biggest players are Prusa. There I have Prusa. And then, you know, of course, Bambu Lab is becoming a part. What are they using? The appliances? Are they using the proprietary one or.

Kate Stewart: There’s some industries I’m not, I have not been into.

Swapnil Bhartiya: But this is the market that should be tapping into these types of markets.

Kate Stewart: And I think, quite frankly, the camera market, I think, I think there’s some good, I Think there’s some good potential in some of the.

Swapnil Bhartiya: All these audio equipment.

Kate Stewart: Yeah, the audio equipment’s mostly all gone. A lot of it’s gone Zephyr already.

Swapnil Bhartiya: Okay. Wow.

Kate Stewart: Yeah. Because of the Bluetooth audio and so these pieces are coming in. But yeah, there’s. I think there’s a lot of it.

Swapnil Bhartiya: Any device which is running any kind of rtos or, you know, any connectivity, Zephyr is the right spot. One thing I would like to know is that if you look at Zephyr and if you look at your success, and I’ll bring back the pun, despite not being as big as Linux kernel, both in size. Because when I talk to Greg, you’re like, no, you can use Linux kernel there as well. You know, we have a smart. That’s funny. What the question I’m asking is that if you look at the success of Zephyr, despite catering to a niche market, you know, what lessons can be learned from it? The way you folks have looked at the security, you have maintained the size, resource efficiency, at the same time, the community. What lesson? Other open source community. Because sometimes those community, despite having one of the best products, they don’t succeed.

Kate Stewart: So my talk tomorrow is effort by developers for developers, and listening to developers and making sure that developers are being heard as a community grows larger is one of the things that wasn’t obvious that we’re doing, but we have been doing it and it has served us well. So, like, there’s an annual survey that’s about to open for the developers and happy to give you the link to share when you posting this video, if you could. It’ll be open till the end of June. And we want, you know, anyone who’s using Zelfur as a developer to weigh in. What are we doing well? What are we not doing well? But we’ve been doing this annually since 2018, roughly. And by getting that annual feedback across our developer range, we get a feel for the things we might be overlooking from a TSC perspective. And we get the input and, and we get the justification to spend money on improving things and communicating things in a better way. So these are the things that have been significant to the project that really are not, you know, these are. Listening to the developers is the thing that you should be doing as a project and being a way of making sure that your developers feel that they are empowered. We have in rtsc, we have community members who’ve been elected by their peers in the community because of their contributions as well as, you know, developers in rtsc that have come from our member companies. So we’ve got that mix. And keeping that healthy mix is important too because. And figuring out what you want to prioritize for spending the money on, you know, like our CI infrastructure costs a lot, our testing of the Bluetooth stack costs a lot for the project. And these are all things we need for quality and to keep the reputation up. And then right now the big challenge is going to be obviously working with how is AI going to interact with Zephyr?

Swapnil Bhartiya: That was going to my final question as well. Initially you were talking about somebody found a bug in the very you you thought everything was secure. And so with AI, two things are happening, of course, especially for open source, suddenly you can get big tsunami of contribution, you know, because with AI generated code. But AI can also help in finding that. But also talk about a lot of AI. You already talked about your trash cans and everything that are using AI and they’re using it for, you know, to track. But what, what role do you see of Zephyr in the edge embedded AI world and what role do you see of AI in the Zephyr board? So it’s two fold question.

Kate Stewart: So AI into Zephyr. I see AI as an assist to the developers. A lot of the agents are able to help us improve the code quality and find bugs before they go into the code base. And so I think you’re going to be seeing a lot of the maintainers starting to employ agents as part of their review process. We’re seeing that happening in the kernel and we’re seeing it starting to happen in Zephyr. We’re seeing people look at refactoring code and working with it that way to make it more efficient over time. And so leveraging these agents and tools, I think you’ll see us work to improve the quality of the code base. And I’m excited about that because we can also be leveraging these types of tools to help us understand the requirements, traceability. So that’s my personal excitement point. So using these tools as assistance to the developers is where I see the positives. The negatives are people who want to contribute, run an AI tool, don’t know what they’re doing and waste maintainers times because they really don’t understand what’s going on. And it’s an on ramp. And I think we need to figure out how to manage it, manage that. But the key part is that making sure that the ones who really understand the architecture and the system engineering aspects have the tools to be efficient and then figuring out how we can, you know, make sure that we don’t burn them out with a lot of slop coming in. So that’s sort of the AI interaction in terms of what Zephyr and AI can do, is they can help recognize key patterns and become more efficient. So that in sort of mixed critical, like if you have a system that has like Zephyr and Linux on different processors or something like that, you keep the power consumption down when you’re running Zephyr and you’re running an AI to recognize something is happening. And then you might power up your FPGA with your Linux kernel to do some proper full AI type of thing. That’s actually the pattern we’ve got in that satellite I was mentioning. Okay. And I think we’re going to see a lot more of that of using Zephyr with a simple AI or simplistic AIs to recognize some trigger condition for more resource intensive computation.

Swapnil Bhartiya: Thank you so much once again for of course, sharing the whole evolution history, but also where it’s heading and which industries should actually be taking advantage of Zephyr because it will save them so much time, so much resources, and they will actually make them this tariff and geopolitical crisis resilient as well, because that will free them to use whatever technology. And also you’re using open source, you don’t have to worry about. And then you can also become a partner in the ecosystem and contribute because whatever you contribute will help you back as well.

Kate Stewart: That’s right.

Swapnil Bhartiya: And thank you so much. And I look forward to the next 20 years of Zephyr. Thank you.

Kate Stewart: Thank you very much. Thank you for being here with the journey with us. I’ve appreciated talking to you all through these years. My pleasure.

Swapnil Bhartiya: Thank you.

Kate Stewart: Thank you.