Is It Still Difficult to Release a Linux Distribution in 2024?

This doesn't quite count as a blog post; it's more of a record of recent discussions and exchanges with others.

Recently, out of boredom, I was chatting with group members, and we discussed the idea of building an operating system together.

It sounds difficult. However, after actually trying it, we found it wasn't that hard.

For example, if you have read the ArchLinux documentation and experienced a manual installation of Arch, you'll realize that you are essentially building a system from scratch on the fly:

https://wiki.archlinux.org/title/installation_guide

Of course, it doesn't have a graphical interface. If you need a graphical interface, you can:

pacman -S gnome-shell

One more reboot, and both gdm and gnome show up:

Looks like this kernel is quite new.

A group member joked about me:

Tools like neofetch are purely a matter of “user loyalty.” For such Legendary software, we all know it won’t be Deprecated anytime soon, and we all know there are better alternatives, but legacy tools often retain a loyal influence within the community.

It’s like how Windows 1.0 introduced the Control Panel. It was restructured in Windows Vista, but after Windows 8, it essentially stopped receiving new features. Subsequent versions buried it deep, seemingly afraid users would still use the Control Panel. In fact, clicking on settings inside it often redirects you to the modern Settings app.

But people just insist on using it. There’s no help for it—too many manuals, too much documentation, and too much experience from network engineers is already based on the Control Panel. Even for a long time into the future, the Control Panel likely won’t be removed, even if no new features are added. I myself prefer using the Control Panel to change IP and firewall settings. My assessment of it is simply Legendary. No matter how good the new alternatives are, we don’t want to lose the old classics.

Back on topic, a group member said that Arch is considered relatively simple. If you were using Gentoo, wouldn’t you be compiling everything on the spot?

Yes, users of Gentoo and NixOS are probably all Linux distribution publishers themselves.

I Googled and found https://github.com/clayrisser/linux-factory. What a divine bulk Linux distribution production machine. The accompanying image is quite amusing:

Clone it, tweak the configuration file .yaml inside, then run make—and just like that, you get an ISO. Add whatever you want. The Linux distribution that gets built is Debian-based.

Tweak these few properties, and the built ISO changes accordingly.

How do I put this? The ecosystem for building custom Linux distributions has become incredibly mature. Even a complete novice can spend three minutes searching online and easily create a domestic operating system. In fact, countless Linux users are essentially producers of distributions. When you think about it, isn't this almost barrier-free?

But at the same time, I'm genuinely curious about one thing: every Arch Linux user I know has gone through the story of a rolling update breaking their system. After all, Arch uses rolling updates; there's no such thing as fixed, critical components spanning major versions.

So how do Arch users actually live? Imagine your girlfriend comes over and says, "Let's watch a movie." You grab your laptop, sit down, and are about to start playing the movie—only to hit a Kernel Panic. Are you really going to debug the kernel in front of your girlfriend?

Some might say: "Just use snapshots, right?"

The problem is that when you need to roll back to a snapshot, you're already in a state of extreme distress. If it's acceptable from a user's perspective, it's unacceptable if you're the one building the operating system.

If you're building your own operating system, it might be used by your relatives and friends. You can't rely on users having the skills to ensure the high availability (HA) of your messy components.

So if you're an architect using a bunch of technologies, you can afford to let some of those technologies fail. You design your own HA, your own distributed systems, your own backups, your own snapshots, your own AB partitioning. You have countless ways to handle it. But if you're building infrastructure, can you really rely on others to take backups? Rely on others to design HA? Rely on others to manage distributed systems? Rely on others to roll back snapshots? That's simply unacceptable.

When your relatives and friends break their systems with rolling updates, they'll just come and scold you. It's like this: if you're building a car, and a feature only works if the seatbelt is functioning properly, the accident has already happened. You should be trying to prevent the accident in the first place.

Why, after so many years, with RAID technology and backup technology being so advanced, do hard drive manufacturers still promise a certain lifespan before failure? It's because, even though you know you can fix it, there will always be users who buy your hard drive, forget to back up their data, and when it fails, they'll come looking for you.

Back to the main point:

If you don't plan to use rolling updates, the only way is to lock your dependencies to a major version that is free of bugs and has a long maintenance cycle, and then backfill some security updates. Then, every two years, when the components have already changed significantly, you test the stability and compatibility of the LTS versions of each component together, lock them to that version again, backfill security updates, and repeat this cycle.

If I wanted to have a life, I would definitely do this. If I wanted to have a life, I would definitely do this, rather than dealing with rolling updates where configuration file syntax changes. I would prefer to stay stable for two years, and then spend an afternoon every two years researching what changed in each package over that period.

Using it this way on your own is fine, but it is much harder if you are releasing a distribution. In fact, the distribution itself has very low difficulty; the real pain is how you manage the versions of the packages you support.

A new version upgrade of any package in any given context can potentially cause a breaking change. You need to lock these breaking changes into a controllable scope. Of course, if you don't mind the hassle, you can manually develop migration scripts. But as everyone knows, there are more packages in the world than stars in the sky. If you go all-in on the latest versions, your users will definitely break every day. You also have to precisely identify, among the various branch management strategies used by different packages, the specific commits that contain no breaking changes and are fixing vulnerabilities, backfill them, and release security updates...

Doing all this just to ensure that when users run an upgrade on your operating system, it completes without breaking—this is definitely not a job for one person. It still requires a company, hundreds of engineers, monitoring packages, and holding weekly meetings to discuss upgrade strategies for base packages.

https://repology.org/repositories/graphs

As shown in the image above, Ubuntu's package list is already quite large (although they are mostly older versions):

Readers might be confused. So many packages in Ubuntu? What about Nix? What about Arch?

This is because the image I showed earlier is actually just a small portion. The complete graph looks like this:

In 2024, the threshold for releasing a self-developed domestic operating system is: too low, yet also too high.

It is too low because: Linux is already very mature. You can grab the kernel, set up a package manager, install sh, and you have a command line. Install gdm, install gnome, install mutter, and the graphical interface is already there. Install a few applications, and it’s ready to use. Hand-coding an operating system in a day is not a dream.

It is too high because: if you want to ensure that your operating system runs stably, securely, and quickly over the long term, you must maintain stable versions and provide continuous security updates. This requires a deep understanding of what users might do on your OS, how much software they might install, what branches these software packages have, how their quality is managed, how to avoid breaking changes while fixing vulnerabilities... This is not something a team of a dozen people can accomplish in an afternoon.

A community member asked: What if all software used a Monolithic architecture?

Some dependencies are not so easily managed. For example: it is well known that older versions of GNOME network settings only support NetworkManager. So if the network manager you install is not NetworkManager, GNOME’s graphical settings become completely useless. Here, NetworkManager becomes a dependency of GNOME.

But newer versions of GNOME have dropped NetworkManager and switched to networkd. If a user updates GNOME, will they lose internet access?

At this point, your only option is to write a migration script to help users translate NetworkManager’s configuration files into networkd’s. However, these two configuration files have different syntax and support different features. Not to mention whether you have the manpower to develop a migration script for each such piece of software, your translation may not even be equivalent.

And if you release your own Linux distribution, breaking changes are a daily occurrence. Maybe a browser’s engine is self-developed today, and switches to Chromium tomorrow. Maybe it’s written in C++ today, and switches to Python tomorrow...

Or maybe one day, a user of your operating system gets a sudden urge to switch from GNOME to KDE. You can’t just stop them, right? So you have to test all these compatibility issues thoroughly. Even Ubuntu, a loyal GNOME user, officially tests KDE support.

I used to wonder, too: a company like RedHat hires so many people, but isn't it an open-source-based company? Isn't it just a company that provides technical support to enterprises and answers questions every day? Later, I learned that this is exactly what they do. They meticulously organize and isolate standards, APIs, and abstractions for various packages, then drive the implementation of these standards within those packages.

I realize this might sound a bit abstract, and you may still not fully grasp what RedHat does. Let me give a concrete example:

RedHat championed a concept: OCI, the Open Container Initiative. They successfully stripped Docker of its dominance.

Originally, Docker was doing just fine. Then suddenly, the message was: "Docker shouldn't be locked in; you should define a set of standards for containers, and everyone else can implement them their own way." Docker agreed, even though they held a monopoly position, and participated in OCI. In the end, even Kubernetes started supporting OCI, and Docker became just one of many OCI providers, completely losing its monopoly. Kubernetes explicitly declared it would no longer support Docker (referring to the legacy Docker).

Of course, your Docker can still publish OCI-compliant images; OCI didn't say you couldn't play. But the result was this: cloud providers like Microsoft, Google, and Amazon no longer needed Docker. They developed their own OCI implementations, heavily customized Kubernetes, and ultimately, the cloud providers became the biggest winners.

With RedHat's move, I can see clearly that Docker was destined to be stripped of its power.

Of course, this isn't necessarily a bad thing for RedHat. After all, they contributed effort and helped advance abstract standards. This is just one of the many positive things RedHat has done. They have contributed to a large number of standards: Ceph, OpenStack, Kubernetes, GNOME... they are involved in all of them.

Of course, this is also a good thing, at least better than making money through closed-source systems. Don't ever expect these big companies to suddenly develop a conscience; their goal has always been profit.