docs: Update kernel docs

- Update section on how to change the kernel config
- Reword kernel module section. It was messy
- General tidying up:
  - empty lines around quoted areas
  - two empty line above and one empty line below section headings
  - Limit line length.

Signed-off-by: Rolf Neugebauer <rolf.neugebauer@docker.com>

docs/kernels.md

@@ -40,21 +40,22 @@ In summary, LinuxKit offers a choice of the following kernels:
 
 ## Compiling external kernel modules
 
-This section describes how to build external (out-of-tree) kernel modules. It is assumed you have
+This section describes how to build external (out-of-tree) kernel
-the source available to those modules, and require the correct kernel version headers and compile tools.
+modules. It is assumed you have the source available to those modules,
+and require the correct kernel version headers and compile tools.
 
 The LinuxKit kernel packages include `kernel-dev.tar` which contains
 the headers and other files required to compile kernel modules against
 the specific version of the kernel. Currently, the headers are not
 included in the initial RAM disk, but it is possible to compile custom
-modules offline and include then include the modules in the initial
+modules offline and then include the modules in the initial RAM disk.
-RAM disk.
+
+There is a [example](../test/cases/020_kernel/010_kmod_4.9.x), but
+basically one can use a multi-stage build to compile the kernel
+modules:
 
-There is a [example](../test/cases/020_kernel/010_kmod_4.9.x), but basically one can use a
-multi-stage build to compile the kernel modules:
 ```
 FROM linuxkit/kernel:4.9.33 AS ksrc
-# Extract headers and compile module
 FROM linuxkit/kernel-compile:1b396c221af673757703258159ddc8539843b02b@sha256:6b32d205bfc6407568324337b707d195d027328dbfec554428ea93e7b0a8299b AS build
 COPY --from=ksrc /kernel-dev.tar /
 RUN tar xf kernel-dev.tar
@@ -66,113 +67,89 @@ To use the kernel module, we recommend adding a final stage to the
 Dockerfile above, which copies the kernel module from the `build`
 stage and performs a `insmod` as the entry point. You can add this
 package to the `onboot` section in your YAML
-file. [kmod.yml](../test/cases/020_kernel/010_kmod_4.9.x/kmod.yml) contains an example for the
+file. [kmod.yml](../test/cases/020_kernel/010_kmod_4.9.x/kmod.yml)
-configuration.
+contains an example for the configuration.
 
-## Compiling internal kernel modules
-If you want to compile in-tree kernel modules, i.e. those whose source is already in the
-kernel tree but have not been included in `linuxkit/kernel`, you have two options:
 
-1. Follow the external kernel modules process from above
+## Modifying the kernel config
-2. Modify the kernel config in [../kernel/](../kernel/) and rebuild the kernel.
 
-In general, if it is an in-tree module, we prefer to include it in the standard linuxkit kernel
+Each series of kernels has a config file dedicated to it
-distribution, i.e. option 2 above. Once you have it working, please open a Pull Request to include it.
+in [../kernel/](../kernel),
+e.g.
+[kernel.config-4.9.x-x86_64](../kernel/kernel_config-4.9.x-x86_64),
+which is applied during the kernel build process.
 
-### External Process
+If you need to modify the kernel config, `make kconfig` in
-The `kernel-dev.tar` included with each kernel does *not* include the kernel sources, *only* the headers.
+the [kernel](../kernel) directory will create a local
-To build those modules, you will need to download the kernel source separately and recompile. The
+`linuxkit/kconfig` Docker image, which contains the patched sources
-in-container process that downloads the source is available in the [Dockerfile](../kernel/Dockerfile).
+for all support kernels and architectures in
+`/linux-4.<minor>.<rev>`. The kernel source also has the kernel config
+copied to the default kernel config.
 
-### Modify Config
+Running the image like:
-Building an in-tree module is very similar to building a new modified kernel (see below):
 
-1. Modify the appropriate `kernel.config-*` file(s)
+```sh
-2. Compile
+docker run --rm -ti -v $(pwd):/src linuxkit/kconfig
+```
+
+will give you a interactive shell where you can modify the kernel
+configuration you want, either by editing the config file, or via
+`make menuconfig` etc. Once you are done, save the file as `.config`
+and copy it back to the source tree,
+e.g. `/src/kernel-config-4.9.x-x86_64`.
+
+You can also configure other architectures other than the native
+one. For example to configure the arm64 kernel on x86_64, use:
+
+```
+make ARCH=arm64 defconfig
+make ARCH=arm64 oldconfig # or menuconfig
+```
+
+**Note**: We try to keep the differences between kernel versions and
+architectures to a minimum, so if you make changes to one
+configuration also try to apply it to the others. The script [kconfig-split.py](../scripts/kconfig-split.py) can be used to compare kernel config files. For example:
+
+```sh
+../scripts/kconfig-split.py kernel_config-4.9.x-aarch64 kernel_config-4.9.x-x86_64
+```
+
+creates a file with the common and the x86_64 and arm64 specific
+config options for the 4.9.x kernel series.
 
 ## Building and using custom kernels
 
 To build and test locally modified kernels, e.g., to try a different
-kernel config or new patches, the existing kernel build system in the
+kernel config or new patches, the existing kernel build system in
-[`../kernel`](../kernel/) can be re-used. For example, assuming the
+the [`kernel`](../kernel/) directory can be re-used. For example,
-current 4.9 kernel is 4.9.33, you can build a local kernel with:
+assuming the current 4.9 kernel is 4.9.33, you can build a local
+kernel with:
 
-```
+```sh
 make build_4.9.x
 ```
 
 This will create a local kernel image called
-`linuxkit/kernel:4.9.33-<hash>-dirty` assuming you haven't committed you local changes. You can then use this in your YAML file as:
+`linuxkit/kernel:4.9.33-<hash>-dirty` assuming you haven't committed
+you local changes. You can then use this in your YAML file as:
+
 ```
 kernel:
   image: linuxkit/kernel:4.9.33-<hash>-dirty
 ```
 
-If you have committed your local changes, the `-dirty` will not be appended. Then you can also override the Hub organisation to use the image elsewhere with:
+If you have committed your local changes, the `-dirty` will not be
-```
+appended. Then you can also override the Hub organisation to use the
+image elsewhere with:
+
+```sh
 make ORG=<your hub org>
 ```
+
 The image will be uploaded to Hub and can be use in a YAML file as
 `<your hub org>/kernel:4.9.33` or as `<your hub
 org>/kernel:4.9.33-<hash>`.
 
-### Modifying the Config
-Each series of kernels has a config file dedicated to it in [../kernel/](../kernel), e.g.
-[kernel.config-4.9.x](../kernel/kernel_config-4.9.x). To build a particular series of kernel:
-
-1. Create a separate `git` branch (not required but *strongly* recommended)
-2. Modify the appropriate `kernel.config`, e.g. `kernel.config-4.9.x`
-3. Run `make build_<series>` with appropriate arguments per this section, e.g. `make build_4.9.x ORG=foo HASH=bar`
-4. Create a `.yml`, build and test
-
-You can modify the config in one of two ways:
-
-* Manually, editing the config file
-* Using a standard config generator, like `menuconfig`
-
-Generally, you will manually edit a file if you are a Linux kernel expert and _fully_ understand all of the dependencies, or if the change is minor and you are _highly confident_ there are no dependencies.
-
-If you wish to use `menuconfig`, which figures out dependencies for you, you will need an environment in which to run it. Fortunately, the linuxkit project's kernel compile process already sets one up for you.
-To get an appropriate environment:
-
-1. `cd kernel/`
-2. Run a build for your desired kernel series, e.g. `make build_4.9.x ORG=foo HASH=bar`
-3. When you see the output from `make defconfig && make oldconfig` complete, hit `Ctrl-C` to stop the build
-4. Note the hash from the intermediate container. That intermediate container has all of the tools and source in it, and can be used to build.
-5. Get a shell in that intermediate container, mounting the current directory in: `docker run -it --rm -v ${PWD}:/src <hash> sh`
-
-This will give you a read-to-run kernel build environment, with all of the config files in `/src/`.
-
-For the output of step 4, e.g.:
-
-```
-Step X/29 : COMMAND
- ---> b2a4a976d661
-```
-
-Once you have your shell, and you want to run the config, you can do the following. We assume you have launched your config container using the steps above, i.e. `docker run -it --rm -v ${PWD}:/src <hash> sh`. The kernel source is in `/linux/`, while the `kernel/` directory from linuxkit is in `/src/`:
-
-Unless you are building the config from scratch, you probably want to make small modifications to the existing config.
-
-The appropriate config at `/src/kernel.config-<series>` was already copied over to `/linux/.config` by the build.
-
-1. `cd /linux`
-2. `make menuconfig`
-3. Load in the existing config: On the bottom menu, use the left-right arrow keys to `Load`
-4. Load it from `.config`
-5. `Exit` from the `Load` pop-up and make the desired changes
-6. Save the modified config: On the bottom menu, use the left-right arrow keys to `Save`
-7. Save it to `.config`
-8. Exit the menu by selecting `Exit` from the bottom meny as many times as necessary
-9. Copy the saved config to the mount location: `cp /linux/.config /src/some-saved-name.config` (replace with an appropriate name)
-10. Exit out of the container
-11. Check the differences generated by menuconfig with `diff kernel.config-<series> some-saved-name.config`.
-    * If the changes are as you expected, proceed to the next step
-    * If the changes are different, either return to the container and menuconfig, or edit manually
-12. Copy the new config file to the build location: `cp some-saved-name.config kernel.config-<series>`
-13. Run your build: `make build_<series>`
-
-
 
 ## Working with Linux kernel patches for LinuxKit
 
@@ -205,20 +182,25 @@ future.
 
 ### Preparation
 
-Patches are applied to point releases of the linux stable tree. You need an up-to-date copy of that tree:
+Patches are applied to point releases of the linux stable tree. You
+need an up-to-date copy of that tree:
+
 ```sh
 git clone git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git
 ```
+
 Add it as a remote to a clone of the [LinuxKit clone](https://github.com/linuxkit/linux).
 
 We use the following variables:
 - `KITSRC`: Base directory of LinuxKit repository
 - `LINUXSRC`: Base directory of Linux stable kernel repository
 e.g.:
+
 ```sh
 KITSRC=~/src/linuxkit/linuxkit
 LINUXSRC=~/src/linuxkit/linux
 ```
+
 to refer to the location of the LinuxKit and Linux kernel trees.
 
 
@@ -228,6 +210,7 @@ There are different ways to do this, but we recommend applying the
 patches to the current version and then rebase to the new version. We
 define the following variables to refer to the current base tag and
 the new tag you want to rebase the patches to:
+
 ```sh
 CURTAG=v4.9.14
 NEWTAG=v4.9.15
@@ -235,6 +218,7 @@ NEWTAG=v4.9.15
 
 If you don't already have a branch, it's best to import the current
 patch set and then rebase:
+
 ```sh
 cd $LINUXSRC
 git checkout -b ${NEWTAG}-linuxkit ${CURTAG}
@@ -249,12 +233,14 @@ conflicts resolve them, then `git add <files>` and `git rebase
 If you already have linux tree with a `${CURTAG}-linuxkit` branch, you
 can rebase by creating a new branch from the current branch and then
 rebase:
+
 ```sh
 cd $LINUXSRC
 git checkout ${CURTAG}-linuxkit
 git branch ${NEWTAG}-linuxkit ${CURTAG}-linuxkit
 git rebase --onto ${NEWTAG} ${NEWTAG} ${NEWTAG}-linuxkit
 ```
+
 Again, resolve any conflicts as described above.
 
 
@@ -271,6 +257,7 @@ If the patch is not cherry-picked try to include as much information
 in the commit message as possible as to where the patch originated
 from. The canonical form would be to add a `Origin:` line after the
 DCO lines, e.g.:
+
 ```
 Origin: https://patchwork.ozlabs.org/patch/622404/
 ```
@@ -279,6 +266,7 @@ Origin: https://patchwork.ozlabs.org/patch/622404/
 
 To export patches to LinuxKit, you should use `git format-patch` from
 the Linux tree, e.g., something along these lines:
+
 ```sh
 cd $LINUXSRC
 rm $KITSRC/kernel/patches-4.9.x/*
@@ -296,8 +284,11 @@ The simplest way to use the `perf` utility is to add the package to
 the `init` section in the YAML file. This adds the binary to the root
 filesystem.
 
-To use the binary, you can either bind mount it into the `getty` or `ssh` service container or you can access the root filesystem from the `getty` container via `nsenter`:
+To use the binary, you can either bind mount it into the `getty` or
-```
+`ssh` service container or you can access the root filesystem from the
+`getty` container via `nsenter`:
+
+```sh
 nsenter -m/proc/1/ns/mnt ash
 ```