From c388177596e5c576db6435adc7b872cd1bdf3ab6 Mon Sep 17 00:00:00 2001 From: Avi Deitcher Date: Wed, 31 Jan 2024 10:50:49 +0200 Subject: [PATCH] updated kernel documents Signed-off-by: Avi Deitcher --- docs/kernels.md | 214 ++++++++++++++++++++++++++++++++++++++---------- 1 file changed, 173 insertions(+), 41 deletions(-) diff --git a/docs/kernels.md b/docs/kernels.md index 54b543826..691955070 100644 --- a/docs/kernels.md +++ b/docs/kernels.md @@ -10,17 +10,51 @@ The LinuxKit kernels are based on the latest stable releases and are updated frequently to include bug and security fixes. For some kernels we do carry additional patches, which are mostly back-ported fixes from newer kernels. The full kernel source with patches can be -found on [github](https://github.com/linuxkit/linux). Each kernel -image is tagged with the full kernel version (e.g., -`linuxkit/kernel:4.9.33`) and with the full kernel version plus the -hash of the files it was created from (git tree hash of the `./kernel` -directory). For selected kernels (mostly the LTS kernels and latest -stable kernels) we also compile/push kernels with additional debugging -enabled. The hub images for these kernels have the `-dbg` suffix in -the tag. For some kernels, we also provide matching packages -containing the `perf` utility for debugging and performance tracing. -The perf package is called `kernel-perf` and is tagged the same way as -the kernel packages. +found on [github](https://github.com/linuxkit/linux). + +## Kernel Image Naming and Tags + +We publish the following kernel images: + +* primary kernel +* debug kernel +* tools for the specific kernel build +* builder image for the specific kernel build, useful for compiling compatible kernel modules + +### Primary Kernel Images + +Each kernel image is tagged with: + +* the full kernel version, e.g. `linuxkit/kernel:6.6.13`. This is a multi-arch index, and should be used whenever possible. +* the full kernel version plus hash of the files it was created from (git tree hash of the `./kernel` directory), e.g. `6.6.13-c0d96951e9892a7447a8e7965d2d6bd7e621c3fd`. This is a multi-arch index. +* the full kernel version plus architecture, e.g. `linuxkit/kernel:6.6.13-amd64` or `linuxkit/kernel:6.6.13-arm64`. Each of these is architecture specific. +* the full kernel version plus hash of the files it was created from (git tree hash of the `./kernel` directory) plus architecture, e.g. `6.6.13-c0d96951e9892a7447a8e7965d2d6bd7e621c3fd-arm64`. + +### Debug Kernel Images + +With each kernel image, we also publish kernels with additional debugging enabled. +These have the same image name and the same tags as the primary kernel, with the `-dbg` +suffix added immediately after the version. E.g. + +* `linuxkit/kernel:6.6.13-dbg` +* `linuxkit/kernel:6.6.13-dbg-c0d96951e9892a7447a8e7965d2d6bd7e621c3fd` +* `linuxkit/kernel:6.6.13-dbg-amd64` +* `linuxkit/kernel:6.6.13-dbg-c0d96951e9892a7447a8e7965d2d6bd7e621c3fd-amd64` + +### Tools + +With each kernel image, we also publish images with various tools. As of this writing, +those tools are `perf` and `bcc`. + +The tools images are named `linuxkit/kernel-`, followed by the same tags as the +primary kernel. For example: + +* `linuxkit/kernel-perf:6.6.13` +* `linuxkit/kernel-perf:6.6.13-c0d96951e9892a7447a8e7965d2d6bd7e621c3fd` +* `linuxkit/kernel-perf:6.6.13-amd64` +* `linuxkit/kernel-perf:6.6.13-c0d96951e9892a7447a8e7965d2d6bd7e621c3fd-amd64` + +## Additional Contributions In addition to the official images, there are also some [scripts](../contrib/foreign-kernels) which repackage kernels packages @@ -32,7 +66,6 @@ use cases for the promising IoT scenarios. All -rt patches are grabbed from https://www.kernel.org/pub/linux/kernel/projects/rt/. But so far we just enable it over 4.14.x. - ## Loading kernel modules Most kernel modules are autoloaded with `mdev` but if you need to `modprobe` a module manually you can use the `modprobe` package in the `onboot` section like this: @@ -121,40 +154,22 @@ FROM linuxkit/kernel:5.10.104 AS ksrc FROM linuxkit/alpine:2be490394653b7967c250e86fd42cef88de428ba AS build ``` -## Modifying the kernel config +## Building and Modifying -Each series of kernels has a config file dedicated to it -in [../kernel/](../kernel), -e.g. -[config-5.10.x-x86_64](../kernel/config-5.10.x-x86_64), -which is applied during the kernel build process. +This section describes how to build kernels, and how to modify existing ones. -If you need to modify the kernel config, `make kconfig` in -the [kernel](../kernel) directory will create a local -`linuxkit/kconfig` Docker image, which contains the patched sources -for all support kernels and architectures in -`/linux-4..`. The kernel source also has the kernel config -copied to the default kernel config. +Throughout the document, the terms used are: -Running the image like: +* kernel version: actual semver version of a kernel, e.g. `6.6.13` or `5.15.27` +* kernel series: major.minor version of a kernel, e.g. `6.6.x` or `5.15.x` -```sh -docker run --rm -ti -v $(pwd):/src linuxkit/kconfig -``` +Each series of kernels has a config file dedicated to it in [../kernel/](../kernel), +e.g. [config-5.10.x-x86_64](../kernel/config-5.10.x-x86_64), +one per target architecture. Note that the architecture used as the `uname -m` one +and not the alpine or golang one. Thus `x86_64` rather than `amd64`, and `aarch64` rather +than `arm64`. -will give you a interactive shell where you can modify the kernel -configuration you want, either by editing the config file, or via -`make menuconfig` etc. Once you are done, save the file as `.config` -and copy it back to the source tree, -e.g. `/src/kernel-config-4.9.x-x86_64`. - -You can also configure other architectures other than the native -one. For example to configure the arm64 kernel on x86_64, use: - -``` -make ARCH=arm64 defconfig -make ARCH=arm64 oldconfig # or menuconfig -``` +The series+arch config file is applied during the kernel build process. **Note**: We try to keep the differences between kernel versions and architectures to a minimum, so if you make changes to one @@ -167,6 +182,123 @@ configuration also try to apply it to the others. The script [kconfig-split.py]( creates a file with the common and the x86_64 and arm64 specific config options for the 4.9.x kernel series. +**Note**: The CI pipeline does *not* push out kernel images. +Anyone modifying a kernel should: + +1. Follow the steps below for the desired changes and commit them. +1. Run appropriate `make build` or variants to ensure that it works. +1. Open a PR with the changes. This may fail, as the CI pipeline may not have access to the modified kernels. +1. A maintainer should run `make push` to push out the images. +1. Run (or rerun) the tests. + +### Modifying the kernel config + +The process of modifying the kernel configuration is as follows: + +1. Create a `linuxkit/kconfig` container image: `make kconfig`. This is not pushed out. +1. Run a container based on `linuxkit/kconfig`. +1. In the container, modify the config to suit your needs using normal kernel tools like `make defconfig` or `make menuconfig`. +1. Save the config from the image. + +The `linuxkit/kconfig` image contains the patched sources +for all support kernels and architectures in `/linux-..`. +The kernel source also has the kernel config copied to the default kernel config location, +so that `make menuconfig` and `make defconfig` work correctly. + +Run the container as follows: + +```sh +docker run --rm -ti -v $(pwd):/src linuxkit/kconfig +``` + +This will give you a interactive shell where you can modify the kernel +configuration you want, while mounting the directory, so that you can save the +modified config. + +To create or modify the config, you must cd to the correct directory, +e.g. + +```sh +cd /linux-6.6.13 +# or +cd /linux-5.15.27 +``` + +Now you can build the config. + +When `make defconfig` or `make menuconfig` is done, +the modified config file will be in `.config`; save the file back to `/src`, +e.g. + +```sh +cp .config /src/kernel-config-6.6.x-x86_64 +``` + +You can also configure other architectures other than the native +one. For example to configure the arm64 kernel on x86_64, use: + +```sh +make ARCH=arm64 defconfig +make ARCH=arm64 oldconfig # or menuconfig +``` + +Note that the generated file **must** be final. When you actually build the kernel, +it will check that running `make defconfig` will have no changes. If there are changes, +the build will fail. + +The easiest way to check it is to rerun `make defconfig` inside the kconfig container. + +1. Finish your creation of the config file, as above. +1. Copy the `.config` file to the target location, as above. +1. Copy the `.config` file to the source location for defconfig, e.g. `cp .config arch/x86/configs/x86_64_config` or `cp. config /linux/arch/arm64/configs/defconfig` +1. Run `make defconfig` again, and check that there are no changes, e.g. `diff .config arch/x86/configs/x86_64_config` or `diff .config /linux/arch/arm64/configs/defconfig` + +If there are no differences, then you can commit the new config file. + +Finally, test that you can build the kernel with that config as `make build-`, e.g. `make build-5.15.148`. + +## Adding a new kernel version + +If you want to add a new kernel version within an existing series, e.g. `5.15.27` already exists +and you want to add (or replace it with) `5.15.148`, apply the following process. + +1. Modify the list of kernels inside the `Makefile` to include the new version, and, optionally, remove the old one. +1. Create a new `linuxkit/kconfig` container image: `make kconfig`. This is not pushed out. +1. Run a container based on `linuxkit/kconfig`. +```sh +docker run --rm -ti -v $(pwd):/src linuxkit/kconfig +``` +1. In the container, change directory to the kernel source directory for the new version, e.g. `cd /linux-5.15.148`. +1. Run `make defconfig` to create the default config file. +1. If the config file has changed, copy it out of the container and check it in, e.g. `cp .config /src/kernel-config-5.15.x-x86_64`. +1. Repeat for other architectures. +1. Commit the changed config files. +1. Test that you can build the kernel with that config as `make build-`, e.g. `make build-5.15.148`. + +## Adding a new kernel series + +To add a new kernel series, you need to create a new config file. Since the last major series +likely is the best basis for the new one, subject to additional modifications, you can use +the previous one as a starting point. + +1. Modify the list of kernels inside the `Makefile` to include the new version. You do not need to specify the series anywhere, as the `Makefile` calculates it. E.g. adding `7.0.5` will cause it to calculate the series as `7.0.x` automatically. +1. Create a new `linuxkit/kconfig` container image: `make kconfig`. This is not pushed out. +1. Run a container based on `linuxkit/kconfig`. +```sh +docker run --rm -ti -v $(pwd):/src linuxkit/kconfig +``` +1. In the container, change directory to the kernel source directory for the new version, e.g. `cd /linux-7.0.5`. +1. Copy the existing config file for the previous series, e.g. `cp /src/kernel-config-6.6.x-x86_64 .config`. +1. Run `make oldconfig` to create the config file for the new series from the old one. Answer any questions. +1. Save the newly generated config file `.config` to the source directory, e.g. `cp .config /src/kernel-config-7.0.x-x86_64`. +1. Repeat for other architectures. +1. Commit the new config files. +1. Test that you can build the kernel with that config as `make build-`, e.g. `make build-7.0.5`. + +In addition, there are tests that are applied to a specific kernel version, notably the tests in +[020_kernel](../test/cases/020_kernel/). You will need to add a new test case for the new series, +copying an existing one and modifying it as needed. + ## Building and using custom kernels To build and test locally modified kernels, e.g., to try a different