Introduction
In this Blog, we will provide an overview of the Linux kernel development process, and we will learn how small incremental changes (patches) sent to a mailing list get included in the upstream kernel.
Learning Objectives?
By the end of this Blog, you should be able to:
Understand how the Linux kernel is developed.
Discuss the Linux kernel release cycle and merge window.
Discuss active kernel releases. Explain the concept of kernel trees.
About the Linux Kernel?
The Linux kernel is the result of collaborative development efforts from developers across the globe. Small incremental changes, also known as patches, add new features, make enhancements, and fix bugs.
It is a 24-hour, seven days a-week, and 365 days of continuous development process that results in a new release once every 2+ months, and several stable and extended stable releases once a week. New development and current release integration cycles run in parallel.
A new release comes out once every 10 to 11 weeks. Releases are time-based rather than feature-based. That means releases are not held up for features. There is no set date for these releases.
What Does the Release Cycle Look Like?
While kernel development is a continuous process, at certain points during the development, when a set of features and bug fixes are ready, a new version of the kernel is released. These new versions are called kernel releases. Linus Torvalds releases a new kernel and opens a 2-week merge window. During this merge window, he pulls code for the next release from subsystem maintainers. Subsystem maintainers send signed git pull requests to Linus either during the merge window or before. All major new development is added to the kernel during the merge window.10,000+ change sets (patches) get pulled into Linus's tree during these 2 weeks, at the end of which he releases the first release candidate, known as rc1.
At this point, the release cycle moves into a bug fixes-only mode, with a series of the release candidate (RC) releases from Linus. One week after rc1 is released, rc2 comes out; rc3 comes out a week after, and so on, until all major bug fixes and regressions (if any) are resolved.
The new cycle begins with a 3-week quiet period, which starts a week before the release and continues through the 2-week merge window. Maintainers and key contributors are busy getting their trees ready to send pull requests to Linus. Please note that the quiet period isn't formalized, and each sub-system might handle it differently. This period isn't well advertised, and new developers might see a slow response from the community.
Active Kernel Releases?
Release Candidate (RC):
Release Candidate or RC releases are mainline kernel pre-releases that are used for testing new features in the mainline (we will talk about the mainline tree shortly). These releases must be compiled from sources. Kernel developers test these releases for bugs and regressions.
Close Stable:
Stable releases are bugfix-only releases. After Linus releases a mainline kernel, it moves into stable mode. Any bug fixes for a stable kernel are backported from the mainline kernel and applied to stable git by a designated stable kernel release maintainer. Stable kernel updates are released on average, once a week, or on an as-needed basis.
Close Long-term:
Long-term releases are stable releases selected for long-term maintenance to provide critical bug fixes for older kernel trees.
Stable releases are normally only maintained for a few mainline release cycles unless they are marked as long-term releases (LTR). A long-term release, as the name suggests, is maintained for a longer period to allow multiple vendors to collaborate on a specific kernel release that they plan on maintaining for an extended period.
You can refer to The Linux Kernel Archives for more details about current stable and long-term releases, and their release cadence.
Kernel Trees-What are They?
You have probably heard about kernel trees, and we have already mentioned the term above. The kernel code is organized in several main and subsystem git repositories called trees.
Kernel Trees
The Mainline kernel tree This tree is maintained by Linus Torvalds. This is where Linus releases mainline kernels and RC releases.
The Stable tree
This tree is maintained by Greg Kroah-Hartman. This tree consists of stable-release branches. Stable releases are based on this tree.
The Linux-next tree
This is the integration tree maintained by Stephen Rothwell. Code from a large number of subsystem trees gets pulled into this tree periodically and then released for integration testing. The process of pulling changes from various trees catches merge conflicts (if any) between trees
One of my first actions as a maintainer was to request that my tree be added to Linux-next. After I commit patches to my tree, I wait 3 to 7 days before sending a pull request to Linus, giving enough time to find problems and regressions, if any.
Patches applied to a tree that will be added to Linux-next is only for the next merge window, including during the merge window.
Patches for the next release may be added to Linux next after the merge window has closed, and the rc1 candidate has been released by Linus.
Subsystem Maintainers
Each major subsystem has its own tree and designated maintainer(s). Subsystems may have multiple maintainers. You can find a list of subsystems and their maintainers in the MAINTAINERS file in the kernel source.
In addition, almost every kernel subsystem has a mailing list. Please refer to the Linux kernel mailing lists and the list archives on lore.kernel.org for a list of mailing lists and their archives. Archives are a great reference for patch discussions and the history of changes.
The development process itself happens entirely over emails. Contributors send patches to mailing lists through email, and contributions are then discussed through emails. Most maintainers send an email response to contributors saying "Applied to tree-name" or "Thanks Applied" to contributors when they accept and commit patches. Some maintainers have a bot that will send patch-applied notifications with the patch included in them. Please refer to the kernel.org git repositories for a list of kernel trees. Not all trees are on kernel.org. You can find subsystem git information and mailing lists for each of the subsystems in the MAINTAINERS file.
You can refer to A guide to the Kernel Development Process for additional information.
