Merge pull request #1744 from rneugeba/kernels-up

Add CentOS/Fedora kernels. Fix/Improve kernel images
2025-07-19 01:06:27 +00:00 · 2017-05-01 10:01:57 +01:00 · 2017-05-01 10:01:57 +01:00 · 0a2fa3eec6
commit 0a2fa3eec6
parent 7ec84e1ed2 721ffeeccd
8 changed files with 214 additions and 28 deletions
--- a/docs/kernels.md
+++ b/docs/kernels.md
@ -1,19 +1,36 @@
-# LinuxKit kernels
+# Linux kernels

-Currently, LinuxKit supports a number of kernels. These kernels are
-typically based on the latest stable releases and are updated
-frequently to include bug and security fixes.  For some kernels we do
-carry some additional patches, which are mostly back-ported fixes from
-newer kernels. The full kernel source with patches is on
-[github](https://github.com/linuxkit/linux).
+LinuxKit kernel images are distributed as hub images which contain the
+kernel, kernel modules, kernel config file, and optionally, kernel
+headers to compile kernel modules against. The repository containing
+the official LinuxKit kernels is at
+[linuxkit/kernels](https://hub.docker.com/r/linuxkit/kernel/).

-The kernel images are stored on Hub under
-[linuxkit/kernel](https://hub.docker.com/r/linuxkit/kernel/). Each
-kernel image is tagged with the full kernel version plus the hash of
-the files it was created from (git tree hash of the `./kernel`
+The LinuxKit kernels are based on the latest stable releases and are
+updated frequently to include bug and security fixes.  For some
+kernels we do carry additional patches, which are mostly back-ported
+fixes from newer kernels. The full kernel source with patches can be
+found on [github](https://github.com/linuxkit/linux). Each kernel
+image is tagged with the full kernel version plus the hash of the
+files it was created from (git tree hash of the `./kernel`
 directory). For convenience, the latest kernel of each stable series
-is also available under the a shorthand tag,
-e.g. `linuxkit/kernel:4.9.x` for the latest `4.9` kernel.
+is also available under a shorthand tag, e.g. `linuxkit/kernel:4.9.x`
+for the latest `4.9` kernel.
+
+In addition to the official kernel images, LinuxKit offers the ability
+to build bootable Linux images with kernels from various
+distributions. We mostly offer this mostly for testing
+purposes. "Foreign" kernel images are created by re-packing the native
+kernel packages into hub images. The hub images are typically tagged
+with the kernel version.
+
+In summary, LinuxKit offers a choice of the following kernels:
+- [linuxkit/kernel](https://hub.docker.com/r/linuxkit/kernel/): Official LinuxKit kernels.
+- [linuxkit/kernel-mainline](https://hub.docker.com/r/linuxkit/kernel-mainline/): Mainline [kernel.org](http://kernel.org) kernels from the [Ubuntu Mainline PPA](http://kernel.ubuntu.com/~kernel-ppa/mainline/).
+- [linuxkit/kernel-ubuntu](https://hub.docker.com/r/linuxkit/kernel-ubuntu/): Selected Ubuntu kernels.
+- [linuxkit/kernel-debian](https://hub.docker.com/r/linuxkit/kernel-debian/): Selected Debian kernels.
+- [linuxkit/kernel-centos](https://hub.docker.com/r/linuxkit/kernel-centos/): Selected CentOS kernels.
+- [linuxkit/kernel-fedora](https://hub.docker.com/r/linuxkit/kernel-fedora/): Selected Fedora kernels.


 ## Working with Linux kernel patches for LinuxKit
@ -66,13 +83,17 @@ to refer to the location of the LinuxKit and Linux kernel trees.

 ### Updating the patches to a new kernel version

-There are different ways to do this, but we recommend applying the patches to the current version and then rebase to the new version. We define the following variables to refer to the current base tag and the new tag you want to rebase the patches to:
+There are different ways to do this, but we recommend applying the
+patches to the current version and then rebase to the new version. We
+define the following variables to refer to the current base tag and
+the new tag you want to rebase the patches to:
 ```sh
 CURTAG=v4.9.14
 NEWTAG=v4.9.15
 ```

-If you don't already have a branch, it's best to import the current patch set and then rebase:
+If you don't already have a branch, it's best to import the current
+patch set and then rebase:
 ```sh
 cd $LINUXSRC
 git checkout -b ${NEWTAG}-linuxkit ${CURTAG}
@ -80,9 +101,13 @@ git am ${KITSRC}/kernel/patches/*.patch
 git rebase ${NEWTAG}-linuxkit ${NEWTAG}
 ```

-The `git am` should not have any conflicts and if the rebase has conflicts resolve them, then `git add <files>` and `git rebase --continue`.
+The `git am` should not have any conflicts and if the rebase has
+conflicts resolve them, then `git add <files>` and `git rebase
+--continue`.

-If you already have linux tree with a `${CURTAG}-linuxkit` branch, you can rebase by creating a new branch from the current branch and then rebase:
+If you already have linux tree with a `${CURTAG}-linuxkit` branch, you
+can rebase by creating a new branch from the current branch and then
+rebase:
 ```sh
 cd $LINUXSRC
 git checkout ${CURTAG}-linuxkit
@ -94,7 +119,12 @@ Again, resolve any conflicts as described above.

 ### Adding/Removing patches

-If you want to add or remove patches make sure you have an up-to-date branch with the currently applied patches (see above). Then either any normal means (`git cherry-pick -x`, `git am`, or `git commit`, etc) to add new patches. For cherry-picked patches also please add a `Origin:` line after the DCO lines with a reference the git tree the patch was cherry-picked from.
+If you want to add or remove patches make sure you have an up-to-date
+branch with the currently applied patches (see above). Then either any
+normal means (`git cherry-pick -x`, `git am`, or `git commit`, etc) to
+add new patches. For cherry-picked patches also please add a `Origin:`
+line after the DCO lines with a reference the git tree the patch was
+cherry-picked from.

 If the patch is not cherry-picked try to include as much information
 in the commit message as possible as to where the patch originated
@ -106,7 +136,8 @@ Origin: https://patchwork.ozlabs.org/patch/622404/

 ### Export patches to LinuxKit

-To export patches to LinuxKit, you should use `git format-patch` from the Linux tree, e.g., something along these lines:
+To export patches to LinuxKit, you should use `git format-patch` from
+the Linux tree, e.g., something along these lines:
 ```sh
 cd $LINUXSRC
 rm $KITSRC/kernel/patches-4.9.x/*
--- a/scripts/kernels/Dockerfile.deb
+++ b/scripts/kernels/Dockerfile.deb
@ -11,11 +11,14 @@ RUN for url in ${DEB_URLS}; do \
        dpkg-deb -x dl.deb . ;\
    done

+RUN for d in lib/modules/*; do depmod -b . $(basename $d); done
+
 RUN mkdir /out
 RUN cp -a boot/vmlinuz-* /out/kernel
 RUN cp -a boot/config-* /out/kernel_config
+RUN cp -a boot/System.map-* /out/System.map
 RUN tar cf /out/kernel.tar lib
-RUN tar cf /out/kernel-dev.tar usr
+RUN tar cf /out/kernel-dev.tar usr || true

 FROM linuxkit/toybox-media:d7e82a7d19ccc84c9071fa7a88ecaa58ae958f7c@sha256:4c7d25f2be2429cd08417c36e04161cb924e46f3e419ee33a0aa9ff3a0942e02
 WORKDIR /
--- a/scripts/kernels/Dockerfile.rpm
+++ b/scripts/kernels/Dockerfile.rpm
@ -0,0 +1,28 @@
+FROM alpine:3.5 AS extract
+
+ARG RPM_URLS
+
+RUN apk add --no-cache curl rpm tar && true
+WORKDIR /rpm
+RUN mkdir extract
+RUN for url in ${RPM_URLS}; do \
+        echo "Extracting: $url"; \
+        curl -fsSL -o dl.rpm $url && \
+        rpm2cpio dl.rpm | cpio -idm ;\
+    done
+
+RUN for d in lib/modules/*; do depmod -b . $(basename $d); done
+
+RUN mkdir /out
+# With some fedora rpms, the kernel and system map are in modules directory
+RUN cp -a boot/vmlinuz-* /out/kernel || mv lib/modules/*/vmlinuz /out/kernel
+RUN cp -a boot/config-* /out/kernel_config || mv lib/modules/*/config /out/kernel_config
+RUN cp -a boot/System.map-* /out/System.map || mv lib/modules/*/System.map /out/System.map
+RUN tar cf /out/kernel.tar lib
+RUN tar cf /out/kernel-dev.tar usr || true
+
+FROM linuxkit/toybox-media:d7e82a7d19ccc84c9071fa7a88ecaa58ae958f7c@sha256:4c7d25f2be2429cd08417c36e04161cb924e46f3e419ee33a0aa9ff3a0942e02
+WORKDIR /
+ENTRYPOINT []
+CMD []
+COPY --from=extract /out/* /
--- a/scripts/kernels/centos.sh
+++ b/scripts/kernels/centos.sh
@ -0,0 +1,47 @@
+#! /bin/sh
+
+REPO="linuxkit/kernel-centos"
+BASE_URL=http://mirror.centos.org/centos/
+
+TAGS=$(curl --silent -f -lSL https://registry.hub.docker.com/v1/repositories/${REPO}/tags)
+
+LINKS=$(curl -s ${BASE_URL}/ | sed -n 's/.*href="\([^"]*\).*/\1/p')
+# Just get names for Centos 7
+RELEASES=$(echo $LINKS | grep -o "7\.[^ ]*")
+RELEASES="7/ $RELEASES"
+
+# Add updates
+URLS=""
+for RELEASE in $RELEASES; do
+    URLS="$URLS ${BASE_URL}/${RELEASE}/os/x86_64/Packages/"
+done
+URLS="$URLS ${BASE_URL}/7/updates/x86_64/Packages/"
+
+for URL in $URLS; do
+    PACKAGES=$(curl -s ${URL}/ | sed -n 's/.*href="\([^"]*\).*/\1/p')
+
+    KERNEL_RPMS=$(echo $PACKAGES | \
+        grep -o "kernel-[0-9]\+\.[0-9]\+\.[0-9]\+-[0-9]\+\.[^ ]\+\.rpm")
+    for KERNEL_RPM in $KERNEL_RPMS; do
+        RPM_URLS="${URL}/${KERNEL_RPM}"
+
+        VERSION=$(echo $KERNEL_RPM | \
+                      grep -o "[0-9]\+\.[0-9]\+\.[0-9]\+-[0-9\.]\+\.el[0-9]\+")
+
+        if echo $TAGS | grep -q "\"${VERSION}\""; then
+            echo "${REPO}:${VERSION} exists"
+            continue
+        fi
+
+        # Don't pull in the headers. This is mostly for testing
+        # HEADERS_RPM="kernel-headers-${VERSION}.x86_64.rpm"
+        # RPM_URLS="${RPM_URLS} ${URL}/${HEADERS_RPM}"
+
+        docker build -t ${REPO}:${VERSION} -f Dockerfile.rpm --no-cache \
+               --build-arg RPM_URLS="${RPM_URLS}" . &&
+            DOCKER_CONTENT_TRUST=1 docker push ${REPO}:${VERSION}
+
+        docker rmi ${REPO}:${VERSION}
+        docker system prune -f
+    done
+done
--- a/scripts/kernels/debian.sh
+++ b/scripts/kernels/debian.sh
@ -3,6 +3,8 @@
 REPO="linuxkit/kernel-debian"
 BASE_URL=http://mirrors.kernel.org/debian/pool/main/l/linux/

+TAGS=$(curl --silent -f -lSL https://registry.hub.docker.com/v1/repositories/${REPO}/tags)
+
 ARCH=amd64
 LINKS=$(curl -s ${BASE_URL}/ | sed -n 's/.*href="\([^"]*\).*/\1/p')
 # Just get names for 4.x kernels
@ -13,8 +15,10 @@ for KERN_DEB in $KERNELS; do
    VERSION=$(echo $KERN_DEB | \
        grep -o "[0-9]\+\.[0-9]\+\.[0-9]\+-[0-9]\+" | head -1)

-    echo "$VERSION -> $KERN_DEB"
-    DOCKER_CONTENT_TRUST=1 docker pull ${REPO}:${VERSION} && continue
+    if echo $TAGS | grep -q "\"${VERSION}\""; then
+        echo "${REPO}:${VERSION} exists"
+        continue
+    fi

    URLS="${BASE_URL}/${KERN_DEB}"

@ -22,4 +26,7 @@ for KERN_DEB in $KERNELS; do
    docker build -t ${REPO}:${VERSION} -f Dockerfile.deb --no-cache \
           --build-arg DEB_URLS="${URLS}" . &&
        DOCKER_CONTENT_TRUST=1 docker push ${REPO}:${VERSION}
+
+    docker rmi ${REPO}:${VERSION}
+    docker system prune -f
 done
--- a/scripts/kernels/fedora.sh
+++ b/scripts/kernels/fedora.sh
@ -0,0 +1,55 @@
+#! /bin/sh
+
+REPO="linuxkit/kernel-fedora"
+BASE_URL=http://mirrors.kernel.org/fedora/
+
+TAGS=$(curl --silent -f -lSL https://registry.hub.docker.com/v1/repositories/${REPO}/tags)
+
+LINKS=$(curl -s ${BASE_URL}/releases/ | sed -n 's/.*href="\([^"]*\).*/\1/p')
+# Just get releases 20+
+RELEASES=$(echo $LINKS | grep -o "2[0-9]")
+
+ARCH=x86_64
+URLS=""
+for RELEASE in $RELEASES; do
+    URLS="$URLS ${BASE_URL}/releases/${RELEASE}/Everything/${ARCH}/os/Packages/k/"
+    URLS="$URLS ${BASE_URL}/updates/${RELEASE}/${ARCH}/k/"
+done
+
+for URL in $URLS; do
+    PACKAGES=$(curl -s ${URL}/ | sed -n 's/.*href="\([^"]*\).*/\1/p')
+
+    KERNEL_RPMS=$(echo $PACKAGES | \
+        grep -o "kernel-[0-9]\+\.[0-9]\+\.[0-9]\+-[0-9]\+\.[^ ]\+\.rpm")
+    for KERNEL_RPM in $KERNEL_RPMS; do
+        RPM_URLS="${URL}/${KERNEL_RPM}"
+
+        VERSION=$(echo $KERNEL_RPM | \
+                      grep -o "[0-9]\+\.[0-9]\+\.[0-9]\+-[0-9\.]\+\.fc[0-9]\+")
+
+        if echo $TAGS | grep -q "\"${VERSION}\""; then
+            echo "${REPO}:${VERSION} exists"
+            continue
+        fi
+
+        CORE_RPM="kernel-core-${VERSION}.${ARCH}.rpm"
+        RPM_URLS="${RPM_URLS} ${URL}/${CORE_RPM}"
+
+        MOD_RPM="kernel-modules-${VERSION}.${ARCH}.rpm"
+        RPM_URLS="${RPM_URLS} ${URL}/${MOD_RPM}"
+
+        MOD_EXTRA_RPM="kernel-modules-extra-${VERSION}.${ARCH}.rpm"
+        RPM_URLS="${RPM_URLS} ${URL}/${MOD_EXTRA_RPM}"
+
+        # Don't pull in the headers. This is mostly for testing
+        # HEADERS_RPM="kernel-headers-${VERSION}.x86_64.rpm"
+        # RPM_URLS="${RPM_URLS} ${URL}/${HEADERS_RPM}"
+
+        docker build -t ${REPO}:${VERSION} -f Dockerfile.rpm --no-cache \
+               --build-arg RPM_URLS="${RPM_URLS}" . &&
+            DOCKER_CONTENT_TRUST=1 docker push ${REPO}:${VERSION}
+
+        docker rmi ${REPO}:${VERSION}
+        docker system prune -f
+    done
+done
--- a/scripts/kernels/mainline.sh
+++ b/scripts/kernels/mainline.sh
@ -3,6 +3,8 @@
 REPO="linuxkit/kernel-mainline"
 BASE_URL=http://kernel.ubuntu.com/~kernel-ppa/mainline

+TAGS=$(curl --silent -f -lSL https://registry.hub.docker.com/v1/repositories/${REPO}/tags)
+
 build_image() {
    VERSION=$1
    KDIR=$2
@ -43,7 +45,13 @@ for KDIR in $KDIRS; do
    # Strip the Ubuntu release name for the tag and also the 'v' like with
    # the other kernel packages
    VERSION=$(echo $KDIR | grep -o "[0-9]\+\.[0-9]\+\.[0-9]\+")
-    DOCKER_CONTENT_TRUST=1 docker pull ${REPO}:${VERSION} && continue
+    if echo $TAGS | grep -q "\"${VERSION}\""; then
+        echo "${REPO}:${VERSION} exists"
+        continue
+    fi
    build_image ${VERSION} ${KDIR} && \
        DOCKER_CONTENT_TRUST=1 docker push ${REPO}:${VERSION}
+
+    docker rmi ${REPO}:${VERSION}
+    docker system prune -f
 done
--- a/scripts/kernels/ubuntu.sh
+++ b/scripts/kernels/ubuntu.sh
@ -3,6 +3,8 @@
 REPO="linuxkit/kernel-ubuntu"
 BASE_URL=http://mirrors.kernel.org/ubuntu/pool/main/l/linux/

+TAGS=$(curl --silent -f -lSL https://registry.hub.docker.com/v1/repositories/${REPO}/tags)
+
 ARCH=amd64
 LINKS=$(curl -s ${BASE_URL}/ | sed -n 's/.*href="\([^"]*\).*/\1/p')
 # Just get names for 4.x kernels
@ -13,22 +15,27 @@ for KERN_DEB in $KERNELS; do
    VERSION=$(echo $KERN_DEB | \
        grep -o "[0-9]\+\.[0-9]\+\.[0-9]\+-[0-9]\+" | head -1)

-    echo "$VERSION -> $KERN_DEB"
-    DOCKER_CONTENT_TRUST=1 docker pull ${REPO}:${VERSION} && continue
+    if echo $TAGS | grep -q "\"${VERSION}\""; then
+        echo "${REPO}:${VERSION} exists"
+        continue
+    fi

    EXTRA_DEB=$(echo $LINKS | \
        grep -o "linux-image-extra-${VERSION}-generic_[^ ]\+${ARCH}\.deb")

+    URLS="${BASE_URL}/${KERN_DEB} ${BASE_URL}/${EXTRA_DEB}"
+
    # Don't pull in the headers. This is mostly for testing
    # HDR_DEB=$(echo $LINKS | \
    #     grep -o "linux-headers-${VERSION}_[^ ]\+_all\.deb")
    # HDR_ARCH_DEB=$(echo $LINKS | \
    #     grep -o "linux-headers-${VERSION}-generic_[^ ]\+_${ARCH}\.deb")
+    # URLS="${URLS} ${BASE_URL}/${HDR_DEB} ${BASE_URL}/${HDR_ARCH_DEB}"

-    URLS="${BASE_URL}/${KERN_DEB} ${BASE_URL}/${EXTRA_DEB}"
-
-    # Doesn't exist build and push
    docker build -t ${REPO}:${VERSION} -f Dockerfile.deb --no-cache \
           --build-arg DEB_URLS="${URLS}" . &&
        DOCKER_CONTENT_TRUST=1 docker push ${REPO}:${VERSION}
+
+    docker rmi ${REPO}:${VERSION}
+    docker system prune -f
 done