Guest: Rob Hirschfeld (LinkedIn)
Company: RackN (Twitter)
Show: Let’s Talk

Along with CentOS 7 reaching its end of life in June 2024, Red Hat recently announced that CentOS Stream will be the sole repository for public Red Hat Enterprise Linux (RHEL)-related source code releases. This means only paying customers will be able to obtain the source code to RHEL.

In this episode of TFiR: Let’s Talk, RackN Co-Founder and CEO Rob Hirschfeld shares his insights on what this means for developers, RackN customers, and the industry, in general.

What happened:

• The Red Hat Enterprise Linux (RHEL) is the dominant enterprise Linux distribution. Red Hat has been allowing in the community a free, open, and bug-for-bug copy of the enterprise Linux. A lot of companies depend on this free open version to test, modify, and run their infrastructure and there is a huge ecosystem around it.
• Red Hat announced that it will no longer keep making the old versions available. It is only going to allow people to have the tip version, i.e., the very latest version of that code. People will no longer have access to prior versions of CentOS or access to the free open Red Hat.
• The industry responded vigorously. People created CentOS copies that reflected other versions. Rocky, Alma, and Oracle started maintaining versions that had back revs of that distro. If you wanted an older version, you could go to one of their distros and get that license or get that code.
• Even though Red Hat only builds the very latest, tip or stream version, you could still go and see the code for the older version. Red Hat stopped making all of those changes to anything but that very tip or stream version, which meant that in all the other distros, you would not get bug-for-bug changes to previous releases.

How this affects developers:

• If you are relying on Red Hat 8.1 or even older versions and there’s a bug found in that Red Hat fixes, other distributions won’t know how that bug was fixed and could cause behavioral changes. There will be drift between the Red Hat Enterprise Linux Standard and all of the other copies, which will magnify over time. You can no longer count on the compatibility between the Red Hat Enterprise Linux version and anything else.
• If there is drift in the operating systems, then when you build your RPM as a target for Red Hat Enterprise Linux 7.5, that RPM might not work in any other distribution. Or if you use Rocky 7.5 as your source and you test against Rocky 7.5, and then you show up at your Red Hat Enterprise Linux customers, you might find that that RPM doesn’t work because they’re not binary compatible anymore, that they have behaviors that you didn’t expect.
• If you are going to build any packaging for Linux, you will have to account for the variances across the different Linux distributions in a much more granular way.
• You will have to ask people which version of RPM packaging they are using because you have to package their Rocky and AlmaLinux differently than their Amazon Linux. That drift can be very expensive for the software ecosystem to manage.

How this affects RackN:

• RackN had to step back and think through how to support those types of Linux flavors and variances. It’s going to cause a lot of teams to tie in deeper to their vendors. It is going to cause RackN to build more walled gardens because the portability between vendors disappeared a bit.
• Last year, when CentOS changed the CentOS streams, RackN migrated away from CentOS as the basis for its discovery image, which is now based on AlmaLinux. It produces at least 4 distinctive versions of its discovery image. It already has to support flavors of Linux, each of which has its own boot environment and its own deployment process.
• From RackN’s perspective, it’s a little bit more work for them to support, but it’s highly valuable work to provide abstraction layers for our customers, so that they don’t have to reinvent the wheel every time. The more variation in the market, the more you need support in addressing that, and the harder it is for customers to lock in and have one choice.

How this affects RackN customers:

• Machine learning has a lot of Ubuntu or Debian-based work. Customers are struggling to get their new machine learning pieces into the corporate standard, because the Red Hat family, by design, does not move as quickly. It will push people into the Debian releases and into the Debian process.
• Most companies could run other Linuxes, but don’t. There was not a lot of incentive for an enterprise that has a relationship with Red Hat to maintain multiple Linux relationships. This might be opening up the door for that to happen.
• People will move to the multi-vendor version and variety is potentially good for the market.
• From a customer perspective, the ability to have just one thing keeps getting harder, but the benefit is that as you get better at supporting multiple things, your supply chain neutrality, your ability to absorb shocks and supply chain changes goes up.
• Customers will feel the hardship, but adapting to that hardship puts them in a much better commercial position.
• At the very core of RackN’s mission and philosophy is the belief that the market is not served by having megalithic vendors. Customers who have distributed supply chains have choices in how they operate are better positioned to own their own infrastructure, have control over their destiny and have leverage in those relationships.

How this affects the Linux ecosystem:

• It opens up the possibility for one of the other Red Hat derivatives to step forward and become dominant.
• It will push more people to give up on packaging and accepting containers because they don’t want to have that battle.
• It might push the industry even more aggressively towards agnostic distribution, ways to get your code into the market that don’t rely on a distribution.
• The way the community came vigorously together to rally around what they want to happen will result in a healthier market and a healthier ecosystem for everybody.
• Opportunities abound for what that new distribution should look like.

How open source can help:

• Hirschfeld hopes to see collaboration, instead of having Rocky Linux or something else become a replacement for Red Hat. This happens in open source — it creates collaboration, a place where there is a mutual benefit, and an ecosystem.
• Enterprises don’t have the expertise, bandwidth, and time to collaborate. But if there is a core of tens of people (not millions or thousands) at the core of governance, then there can be a pyramid effect of downstream people who will benefit from that and pay into the ecosystem.
• Companies can then shop for the commercial distribution and the support they want.
• Companies can put their licensing and dollars behind some of the distribution companies that then would still collaborate around how the distribution works across multiple vendors.
• Collaborating around an enterprise distribution of Linux would mean commercial support (i.e., what it actually looks like and why it’s such a challenge to maintain a distribution in production) would be more open.  (Hat tip to Red Hat for having done that work for so long because it is hard work, and it is expensive work.)

This summary was written by Camille Gregory.