diff --git a/docs/kernels.md b/docs/kernels.md index 09136e9ef..241ba5a07 100644 --- a/docs/kernels.md +++ b/docs/kernels.md @@ -1,19 +1,36 @@ -# LinuxKit kernels +# Linux kernels -Currently, LinuxKit supports a number of kernels. These kernels are -typically based on the latest stable releases and are updated -frequently to include bug and security fixes. For some kernels we do -carry some additional patches, which are mostly back-ported fixes from -newer kernels. The full kernel source with patches is on -[github](https://github.com/linuxkit/linux). +LinuxKit kernel images are distributed as hub images which contain the +kernel, kernel modules, kernel config file, and optionally, kernel +headers to compile kernel modules against. The repository containing +the official LinuxKit kernels is at +[linuxkit/kernels](https://hub.docker.com/r/linuxkit/kernel/). -The kernel images are stored on Hub under -[linuxkit/kernel](https://hub.docker.com/r/linuxkit/kernel/). Each -kernel image is tagged with the full kernel version plus the hash of -the files it was created from (git tree hash of the `./kernel` +The LinuxKit kernels are based on the latest stable releases and are +updated frequently to include bug and security fixes. For some +kernels we do carry additional patches, which are mostly back-ported +fixes from newer kernels. The full kernel source with patches can be +found on [github](https://github.com/linuxkit/linux). Each kernel +image is tagged with the full kernel version plus the hash of the +files it was created from (git tree hash of the `./kernel` directory). For convenience, the latest kernel of each stable series -is also available under the a shorthand tag, -e.g. `linuxkit/kernel:4.9.x` for the latest `4.9` kernel. +is also available under a shorthand tag, e.g. `linuxkit/kernel:4.9.x` +for the latest `4.9` kernel. + +In addition to the official kernel images, LinuxKit offers the ability +to build bootable Linux images with kernels from various +distributions. We mostly offer this mostly for testing +purposes. "Foreign" kernel images are created by re-packing the native +kernel packages into hub images. The hub images are typically tagged +with the kernel version. + +In summary, LinuxKit offers a choice of the following kernels: +- [linuxkit/kernel](https://hub.docker.com/r/linuxkit/kernel/): Official LinuxKit kernels. +- [linuxkit/kernel-mainline](https://hub.docker.com/r/linuxkit/kernel-mainline/): Mainline [kernel.org](http://kernel.org) kernels from the [Ubuntu Mainline PPA](http://kernel.ubuntu.com/~kernel-ppa/mainline/). +- [linuxkit/kernel-ubuntu](https://hub.docker.com/r/linuxkit/kernel-ubuntu/): Selected Ubuntu kernels. +- [linuxkit/kernel-debian](https://hub.docker.com/r/linuxkit/kernel-debian/): Selected Debian kernels. +- [linuxkit/kernel-centos](https://hub.docker.com/r/linuxkit/kernel-centos/): Selected CentOS kernels. +- [linuxkit/kernel-fedora](https://hub.docker.com/r/linuxkit/kernel-fedora/): Selected Fedora kernels. ## Working with Linux kernel patches for LinuxKit @@ -66,13 +83,17 @@ to refer to the location of the LinuxKit and Linux kernel trees. ### Updating the patches to a new kernel version -There are different ways to do this, but we recommend applying the patches to the current version and then rebase to the new version. We define the following variables to refer to the current base tag and the new tag you want to rebase the patches to: +There are different ways to do this, but we recommend applying the +patches to the current version and then rebase to the new version. We +define the following variables to refer to the current base tag and +the new tag you want to rebase the patches to: ```sh CURTAG=v4.9.14 NEWTAG=v4.9.15 ``` -If you don't already have a branch, it's best to import the current patch set and then rebase: +If you don't already have a branch, it's best to import the current +patch set and then rebase: ```sh cd $LINUXSRC git checkout -b ${NEWTAG}-linuxkit ${CURTAG} @@ -80,9 +101,13 @@ git am ${KITSRC}/kernel/patches/*.patch git rebase ${NEWTAG}-linuxkit ${NEWTAG} ``` -The `git am` should not have any conflicts and if the rebase has conflicts resolve them, then `git add ` and `git rebase --continue`. +The `git am` should not have any conflicts and if the rebase has +conflicts resolve them, then `git add ` and `git rebase +--continue`. -If you already have linux tree with a `${CURTAG}-linuxkit` branch, you can rebase by creating a new branch from the current branch and then rebase: +If you already have linux tree with a `${CURTAG}-linuxkit` branch, you +can rebase by creating a new branch from the current branch and then +rebase: ```sh cd $LINUXSRC git checkout ${CURTAG}-linuxkit @@ -94,7 +119,12 @@ Again, resolve any conflicts as described above. ### Adding/Removing patches -If you want to add or remove patches make sure you have an up-to-date branch with the currently applied patches (see above). Then either any normal means (`git cherry-pick -x`, `git am`, or `git commit`, etc) to add new patches. For cherry-picked patches also please add a `Origin:` line after the DCO lines with a reference the git tree the patch was cherry-picked from. +If you want to add or remove patches make sure you have an up-to-date +branch with the currently applied patches (see above). Then either any +normal means (`git cherry-pick -x`, `git am`, or `git commit`, etc) to +add new patches. For cherry-picked patches also please add a `Origin:` +line after the DCO lines with a reference the git tree the patch was +cherry-picked from. If the patch is not cherry-picked try to include as much information in the commit message as possible as to where the patch originated @@ -106,7 +136,8 @@ Origin: https://patchwork.ozlabs.org/patch/622404/ ### Export patches to LinuxKit -To export patches to LinuxKit, you should use `git format-patch` from the Linux tree, e.g., something along these lines: +To export patches to LinuxKit, you should use `git format-patch` from +the Linux tree, e.g., something along these lines: ```sh cd $LINUXSRC rm $KITSRC/kernel/patches-4.9.x/* diff --git a/scripts/kernels/Dockerfile.deb b/scripts/kernels/Dockerfile.deb index e5ba75a67..ae03f8c44 100644 --- a/scripts/kernels/Dockerfile.deb +++ b/scripts/kernels/Dockerfile.deb @@ -11,11 +11,14 @@ RUN for url in ${DEB_URLS}; do \ dpkg-deb -x dl.deb . ;\ done +RUN for d in lib/modules/*; do depmod -b . $(basename $d); done + RUN mkdir /out RUN cp -a boot/vmlinuz-* /out/kernel RUN cp -a boot/config-* /out/kernel_config +RUN cp -a boot/System.map-* /out/System.map RUN tar cf /out/kernel.tar lib -RUN tar cf /out/kernel-dev.tar usr +RUN tar cf /out/kernel-dev.tar usr || true FROM linuxkit/toybox-media:d7e82a7d19ccc84c9071fa7a88ecaa58ae958f7c@sha256:4c7d25f2be2429cd08417c36e04161cb924e46f3e419ee33a0aa9ff3a0942e02 WORKDIR / diff --git a/scripts/kernels/Dockerfile.rpm b/scripts/kernels/Dockerfile.rpm new file mode 100644 index 000000000..343aad7f1 --- /dev/null +++ b/scripts/kernels/Dockerfile.rpm @@ -0,0 +1,28 @@ +FROM alpine:3.5 AS extract + +ARG RPM_URLS + +RUN apk add --no-cache curl rpm tar && true +WORKDIR /rpm +RUN mkdir extract +RUN for url in ${RPM_URLS}; do \ + echo "Extracting: $url"; \ + curl -fsSL -o dl.rpm $url && \ + rpm2cpio dl.rpm | cpio -idm ;\ + done + +RUN for d in lib/modules/*; do depmod -b . $(basename $d); done + +RUN mkdir /out +# With some fedora rpms, the kernel and system map are in modules directory +RUN cp -a boot/vmlinuz-* /out/kernel || mv lib/modules/*/vmlinuz /out/kernel +RUN cp -a boot/config-* /out/kernel_config || mv lib/modules/*/config /out/kernel_config +RUN cp -a boot/System.map-* /out/System.map || mv lib/modules/*/System.map /out/System.map +RUN tar cf /out/kernel.tar lib +RUN tar cf /out/kernel-dev.tar usr || true + +FROM linuxkit/toybox-media:d7e82a7d19ccc84c9071fa7a88ecaa58ae958f7c@sha256:4c7d25f2be2429cd08417c36e04161cb924e46f3e419ee33a0aa9ff3a0942e02 +WORKDIR / +ENTRYPOINT [] +CMD [] +COPY --from=extract /out/* / diff --git a/scripts/kernels/centos.sh b/scripts/kernels/centos.sh new file mode 100755 index 000000000..8be7d63f2 --- /dev/null +++ b/scripts/kernels/centos.sh @@ -0,0 +1,47 @@ +#! /bin/sh + +REPO="linuxkit/kernel-centos" +BASE_URL=http://mirror.centos.org/centos/ + +TAGS=$(curl --silent -f -lSL https://registry.hub.docker.com/v1/repositories/${REPO}/tags) + +LINKS=$(curl -s ${BASE_URL}/ | sed -n 's/.*href="\([^"]*\).*/\1/p') +# Just get names for Centos 7 +RELEASES=$(echo $LINKS | grep -o "7\.[^ ]*") +RELEASES="7/ $RELEASES" + +# Add updates +URLS="" +for RELEASE in $RELEASES; do + URLS="$URLS ${BASE_URL}/${RELEASE}/os/x86_64/Packages/" +done +URLS="$URLS ${BASE_URL}/7/updates/x86_64/Packages/" + +for URL in $URLS; do + PACKAGES=$(curl -s ${URL}/ | sed -n 's/.*href="\([^"]*\).*/\1/p') + + KERNEL_RPMS=$(echo $PACKAGES | \ + grep -o "kernel-[0-9]\+\.[0-9]\+\.[0-9]\+-[0-9]\+\.[^ ]\+\.rpm") + for KERNEL_RPM in $KERNEL_RPMS; do + RPM_URLS="${URL}/${KERNEL_RPM}" + + VERSION=$(echo $KERNEL_RPM | \ + grep -o "[0-9]\+\.[0-9]\+\.[0-9]\+-[0-9\.]\+\.el[0-9]\+") + + if echo $TAGS | grep -q "\"${VERSION}\""; then + echo "${REPO}:${VERSION} exists" + continue + fi + + # Don't pull in the headers. This is mostly for testing + # HEADERS_RPM="kernel-headers-${VERSION}.x86_64.rpm" + # RPM_URLS="${RPM_URLS} ${URL}/${HEADERS_RPM}" + + docker build -t ${REPO}:${VERSION} -f Dockerfile.rpm --no-cache \ + --build-arg RPM_URLS="${RPM_URLS}" . && + DOCKER_CONTENT_TRUST=1 docker push ${REPO}:${VERSION} + + docker rmi ${REPO}:${VERSION} + docker system prune -f + done +done diff --git a/scripts/kernels/debian.sh b/scripts/kernels/debian.sh index ca0a5adc5..c5661750f 100755 --- a/scripts/kernels/debian.sh +++ b/scripts/kernels/debian.sh @@ -3,6 +3,8 @@ REPO="linuxkit/kernel-debian" BASE_URL=http://mirrors.kernel.org/debian/pool/main/l/linux/ +TAGS=$(curl --silent -f -lSL https://registry.hub.docker.com/v1/repositories/${REPO}/tags) + ARCH=amd64 LINKS=$(curl -s ${BASE_URL}/ | sed -n 's/.*href="\([^"]*\).*/\1/p') # Just get names for 4.x kernels @@ -13,8 +15,10 @@ for KERN_DEB in $KERNELS; do VERSION=$(echo $KERN_DEB | \ grep -o "[0-9]\+\.[0-9]\+\.[0-9]\+-[0-9]\+" | head -1) - echo "$VERSION -> $KERN_DEB" - DOCKER_CONTENT_TRUST=1 docker pull ${REPO}:${VERSION} && continue + if echo $TAGS | grep -q "\"${VERSION}\""; then + echo "${REPO}:${VERSION} exists" + continue + fi URLS="${BASE_URL}/${KERN_DEB}" @@ -22,4 +26,7 @@ for KERN_DEB in $KERNELS; do docker build -t ${REPO}:${VERSION} -f Dockerfile.deb --no-cache \ --build-arg DEB_URLS="${URLS}" . && DOCKER_CONTENT_TRUST=1 docker push ${REPO}:${VERSION} + + docker rmi ${REPO}:${VERSION} + docker system prune -f done diff --git a/scripts/kernels/fedora.sh b/scripts/kernels/fedora.sh new file mode 100755 index 000000000..74fe3e5e9 --- /dev/null +++ b/scripts/kernels/fedora.sh @@ -0,0 +1,55 @@ +#! /bin/sh + +REPO="linuxkit/kernel-fedora" +BASE_URL=http://mirrors.kernel.org/fedora/ + +TAGS=$(curl --silent -f -lSL https://registry.hub.docker.com/v1/repositories/${REPO}/tags) + +LINKS=$(curl -s ${BASE_URL}/releases/ | sed -n 's/.*href="\([^"]*\).*/\1/p') +# Just get releases 20+ +RELEASES=$(echo $LINKS | grep -o "2[0-9]") + +ARCH=x86_64 +URLS="" +for RELEASE in $RELEASES; do + URLS="$URLS ${BASE_URL}/releases/${RELEASE}/Everything/${ARCH}/os/Packages/k/" + URLS="$URLS ${BASE_URL}/updates/${RELEASE}/${ARCH}/k/" +done + +for URL in $URLS; do + PACKAGES=$(curl -s ${URL}/ | sed -n 's/.*href="\([^"]*\).*/\1/p') + + KERNEL_RPMS=$(echo $PACKAGES | \ + grep -o "kernel-[0-9]\+\.[0-9]\+\.[0-9]\+-[0-9]\+\.[^ ]\+\.rpm") + for KERNEL_RPM in $KERNEL_RPMS; do + RPM_URLS="${URL}/${KERNEL_RPM}" + + VERSION=$(echo $KERNEL_RPM | \ + grep -o "[0-9]\+\.[0-9]\+\.[0-9]\+-[0-9\.]\+\.fc[0-9]\+") + + if echo $TAGS | grep -q "\"${VERSION}\""; then + echo "${REPO}:${VERSION} exists" + continue + fi + + CORE_RPM="kernel-core-${VERSION}.${ARCH}.rpm" + RPM_URLS="${RPM_URLS} ${URL}/${CORE_RPM}" + + MOD_RPM="kernel-modules-${VERSION}.${ARCH}.rpm" + RPM_URLS="${RPM_URLS} ${URL}/${MOD_RPM}" + + MOD_EXTRA_RPM="kernel-modules-extra-${VERSION}.${ARCH}.rpm" + RPM_URLS="${RPM_URLS} ${URL}/${MOD_EXTRA_RPM}" + + # Don't pull in the headers. This is mostly for testing + # HEADERS_RPM="kernel-headers-${VERSION}.x86_64.rpm" + # RPM_URLS="${RPM_URLS} ${URL}/${HEADERS_RPM}" + + docker build -t ${REPO}:${VERSION} -f Dockerfile.rpm --no-cache \ + --build-arg RPM_URLS="${RPM_URLS}" . && + DOCKER_CONTENT_TRUST=1 docker push ${REPO}:${VERSION} + + docker rmi ${REPO}:${VERSION} + docker system prune -f + done +done diff --git a/scripts/kernels/mainline.sh b/scripts/kernels/mainline.sh index d93d9d55a..5694eb6c7 100755 --- a/scripts/kernels/mainline.sh +++ b/scripts/kernels/mainline.sh @@ -3,6 +3,8 @@ REPO="linuxkit/kernel-mainline" BASE_URL=http://kernel.ubuntu.com/~kernel-ppa/mainline +TAGS=$(curl --silent -f -lSL https://registry.hub.docker.com/v1/repositories/${REPO}/tags) + build_image() { VERSION=$1 KDIR=$2 @@ -43,7 +45,13 @@ for KDIR in $KDIRS; do # Strip the Ubuntu release name for the tag and also the 'v' like with # the other kernel packages VERSION=$(echo $KDIR | grep -o "[0-9]\+\.[0-9]\+\.[0-9]\+") - DOCKER_CONTENT_TRUST=1 docker pull ${REPO}:${VERSION} && continue + if echo $TAGS | grep -q "\"${VERSION}\""; then + echo "${REPO}:${VERSION} exists" + continue + fi build_image ${VERSION} ${KDIR} && \ DOCKER_CONTENT_TRUST=1 docker push ${REPO}:${VERSION} + + docker rmi ${REPO}:${VERSION} + docker system prune -f done diff --git a/scripts/kernels/ubuntu.sh b/scripts/kernels/ubuntu.sh index ea285cd47..b5aa11c7f 100755 --- a/scripts/kernels/ubuntu.sh +++ b/scripts/kernels/ubuntu.sh @@ -3,6 +3,8 @@ REPO="linuxkit/kernel-ubuntu" BASE_URL=http://mirrors.kernel.org/ubuntu/pool/main/l/linux/ +TAGS=$(curl --silent -f -lSL https://registry.hub.docker.com/v1/repositories/${REPO}/tags) + ARCH=amd64 LINKS=$(curl -s ${BASE_URL}/ | sed -n 's/.*href="\([^"]*\).*/\1/p') # Just get names for 4.x kernels @@ -13,22 +15,27 @@ for KERN_DEB in $KERNELS; do VERSION=$(echo $KERN_DEB | \ grep -o "[0-9]\+\.[0-9]\+\.[0-9]\+-[0-9]\+" | head -1) - echo "$VERSION -> $KERN_DEB" - DOCKER_CONTENT_TRUST=1 docker pull ${REPO}:${VERSION} && continue + if echo $TAGS | grep -q "\"${VERSION}\""; then + echo "${REPO}:${VERSION} exists" + continue + fi EXTRA_DEB=$(echo $LINKS | \ grep -o "linux-image-extra-${VERSION}-generic_[^ ]\+${ARCH}\.deb") + URLS="${BASE_URL}/${KERN_DEB} ${BASE_URL}/${EXTRA_DEB}" + # Don't pull in the headers. This is mostly for testing # HDR_DEB=$(echo $LINKS | \ # grep -o "linux-headers-${VERSION}_[^ ]\+_all\.deb") # HDR_ARCH_DEB=$(echo $LINKS | \ # grep -o "linux-headers-${VERSION}-generic_[^ ]\+_${ARCH}\.deb") + # URLS="${URLS} ${BASE_URL}/${HDR_DEB} ${BASE_URL}/${HDR_ARCH_DEB}" - URLS="${BASE_URL}/${KERN_DEB} ${BASE_URL}/${EXTRA_DEB}" - - # Doesn't exist build and push docker build -t ${REPO}:${VERSION} -f Dockerfile.deb --no-cache \ --build-arg DEB_URLS="${URLS}" . && DOCKER_CONTENT_TRUST=1 docker push ${REPO}:${VERSION} + + docker rmi ${REPO}:${VERSION} + docker system prune -f done