https://www.shellcheck.net/wiki/SC2153
this is not a mispelling of KUBE_OUTPUT_SUBPATH, the unprefixed version is an optional override which sets _KUBE_OUTPUT_SUBPATH, which is handled in hack/lib/init.sh where golang.sh is sourced
Since a few releases, Go supports `go build -race` and then produces
binaries which do data race detection when invoked. Some changes are needed to
enable using this in a kind cluster:
- `-race` must be passed when building dynamically linked binaries.
Only those support -race because CGO is required.
To avoid adding yet another env variables, the existing KUBE_RACE
gets used to convey the intent.
- KUBE_RACE must be passed into the dockerized build.
- Logging the base image of a release image makes it easier
to figure out whether the binary has a chance to run.
The base image is important because dynamically linked binaries need a base
image with libc. By default, control plane components are linked statically,
so users need to explicitly override the defaults:
KUBE_RACE=-race KUBE_CGO_OVERRIDES="kube-apiserver kube-controller-manager kube-scheduler" KUBE_GORUNNER_IMAGE=gcr.io/k8s-staging-test-infra/kubekins-e2e:v20250815-171060767f-master kind build node-image ...
KUBE_GORUNNER_IMAGE changes the base image for kube-apiserver,
kube-controller-manager and kube-scheduler. The kubekins image was picked for
this example because a Prow job definition already uses it. Reusing
it in a job avoids the need to maintain another image definition.
Running conformance tests against such a cluster with alpha+beta features
enabled revealed one new data race:
$ kubectl logs -n kube-system kube-controller-manager-kind-control-plane
...
WARNING: DATA RACE
Write at 0x00c00019a730 by goroutine 216:
k8s.io/client-go/tools/leaderelection.(*LeaderElector).setObservedRecord()
k8s.io/client-go/tools/leaderelection/leaderelection.go:529 +0x179
k8s.io/client-go/tools/leaderelection.(*LeaderElector).tryCoordinatedRenew()
k8s.io/client-go/tools/leaderelection/leaderelection.go:367 +0x5ca
...
Run "gimme master" and use the newly built binaries. Adjust
GOROOT_BOOTSTRAP so we can use the golang that is already
on disk to build the new commit from golang.
To enable this, edit `.go-version` and set the version to
```
devel
```
and then run `make quick-release` (for example)
Signed-off-by: Davanum Srinivas <davanum@gmail.com>
Adds the KUBE_BUILD_WINDOWS option to make release-images and quick-release-images,
which will allow it to build the a Windows kube-proxy image as well. That image can
then be used with Windows Host Process Containers to start the kube-proxy
service on Windows nodes.
`make` is able to build project binaries, but fails with
error `hack/lib/golang.sh: line 455: go: command not found`
trying to place them if go binary is not in the PATH.
This happens because kube::golang::place_bins uses different
environment than kube::golang::build_binaries.
Setting up an one environment for both `kube::golang::place_bins`
and `kube::golang::build_binaries` should solve this issue and allow
default make target to fully work without go binary in the PATH.
Prior to this it would download Go, setup the environment, then (maybe)
download Go again.
I renamed verify_go_version to be internal and fixed all callers.
This helps 2 things:
1) output is cleaner
2) whether a build is static or non-static depends on string matching,
which is easier after this
If you specify a `...` pattern, this will preserve it, which breaks the
static/non-static matching but really, can we just agree that `make` is
for binaries? It works for other things but so does `go build`.
Example:
```
$ make WHAT="cmd/kubectl/ ./staging/src/k8s.io/api/... ./test/e2e/e2e.test ./test/e2e/e2e.test ./cmd/kubectl/ "
go version go1.21.4 linux/amd64
+++ [1208 22:24:43] Building go targets for linux/amd64
k8s.io/api/... (non-static)
k8s.io/kubernetes/cmd/kubectl (static)
k8s.io/kubernetes/test/e2e/e2e.test (test)
```
Result: `make test` works:
```
$ #### BUILD ####
$ make WHAT="./cmd/kubectl/ ./staging/src/k8s.io/api"
+++ [1211 11:20:46] Building go targets for linux/amd64
./cmd/kubectl/ (non-static)
./staging/src/k8s.io/api (non-static)
$ #### TEST ####
$ make test WHAT=./pkg/proxy/iptables/
+++ [1211 11:22:31] Set GOMAXPROCS automatically to 6
+++ [1211 11:22:31] Running tests without code coverage and with -race
ok k8s.io/kubernetes/pkg/proxy/iptables 9.517s
$ make test WHAT=pkg/proxy/iptables/
+++ [1211 11:22:53] Set GOMAXPROCS automatically to 6
+++ [1211 11:22:53] Running tests without code coverage and with -race
ok k8s.io/kubernetes/pkg/proxy/iptables (cached)
$ make test WHAT=k8s.io/kubernetes/pkg/proxy/iptables/
+++ [1211 11:23:09] Set GOMAXPROCS automatically to 6
+++ [1211 11:23:09] Running tests without code coverage and with -race
ok k8s.io/kubernetes/pkg/proxy/iptables (cached)
$ make test WHAT=./staging/src/k8s.io/api
+++ [1211 11:23:24] Set GOMAXPROCS automatically to 6
+++ [1211 11:23:24] Running tests without code coverage and with -race
ok k8s.io/api 21.981s
$ make test WHAT=staging/src/k8s.io/api
+++ [1211 11:23:54] Set GOMAXPROCS automatically to 6
+++ [1211 11:23:54] Running tests without code coverage and with -race
ok k8s.io/api (cached)
$ make test WHAT=k8s.io/api
+++ [1211 11:24:06] Set GOMAXPROCS automatically to 6
+++ [1211 11:24:06] Running tests without code coverage and with -race
ok k8s.io/api (cached)
```
and use it from test
Result: `make` works, but `make test` still broken
```
$ make kubectl
+++ [1211 11:15:46] Building go targets for linux/amd64
./cmd/kubectl (static)
$ make WHAT=./cmd/kubectl/
+++ [1211 11:15:53] Building go targets for linux/amd64
./cmd/kubectl/ (non-static)
$ make WHAT=cmd/kubectl/
+++ [1211 11:16:02] Building go targets for linux/amd64
./cmd/kubectl/ (non-static)
$ make WHAT=k8s.io/kubernetes/cmd/kubectl
+++ [1211 11:16:16] Building go targets for linux/amd64
k8s.io/kubernetes/cmd/kubectl (static)
$ make WHAT=./staging/src/k8s.io/api
+++ [1211 11:16:31] Building go targets for linux/amd64
./staging/src/k8s.io/api (non-static)
$ make WHAT=staging/src/k8s.io/api
+++ [1211 11:16:43] Building go targets for linux/amd64
./staging/src/k8s.io/api (non-static)
$ make WHAT=k8s.io/api
+++ [1211 11:16:50] Building go targets for linux/amd64
k8s.io/api (non-static)
```
```
$ make test WHAT=./cmd/kubectl
+++ [1211 11:17:23] Set GOMAXPROCS automatically to 6
+++ [1211 11:17:23] Running tests without code coverage and with -race
cmd/kubectl/kubectl.go:25:2: cannot find package "k8s.io/client-go/plugin/pkg/client/auth" in any of:
/home/thockin/src/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/client-go/plugin/pkg/client/auth (vendor tree)
/home/thockin/sdk/gotip/src/k8s.io/client-go/plugin/pkg/client/auth (from $GOROOT)
/home/thockin/src/kubernetes/_output/local/go/src/k8s.io/client-go/plugin/pkg/client/auth (from $GOPATH)
cmd/kubectl/kubectl.go:20:2: cannot find package "k8s.io/component-base/cli" in any of:
/home/thockin/src/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/component-base/cli (vendor tree)
/home/thockin/sdk/gotip/src/k8s.io/component-base/cli (from $GOROOT)
/home/thockin/src/kubernetes/_output/local/go/src/k8s.io/component-base/cli (from $GOPATH)
cmd/kubectl/kubectl.go:21:2: cannot find package "k8s.io/kubectl/pkg/cmd" in any of:
/home/thockin/src/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/kubectl/pkg/cmd (vendor tree)
/home/thockin/sdk/gotip/src/k8s.io/kubectl/pkg/cmd (from $GOROOT)
/home/thockin/src/kubernetes/_output/local/go/src/k8s.io/kubectl/pkg/cmd (from $GOPATH)
cmd/kubectl/kubectl.go:22:2: cannot find package "k8s.io/kubectl/pkg/cmd/util" in any of:
/home/thockin/src/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/kubectl/pkg/cmd/util (vendor tree)
/home/thockin/sdk/gotip/src/k8s.io/kubectl/pkg/cmd/util (from $GOROOT)
/home/thockin/src/kubernetes/_output/local/go/src/k8s.io/kubectl/pkg/cmd/util (from $GOPATH)
make: *** [Makefile:191: test] Error 1
```
This makes "new" and "old" setup_env functions. In subsequent commits,
all callers of the "old" form will be fixed, and the "new" will be
renamed back.
The old and new functions diff:
```diff
--- /tmp/a 2023-12-14 09:02:57.804092696 -0800
+++ /tmp/b 2023-12-14 09:03:09.679999585 -0800
@@ -1,4 +1,4 @@
-kube::golang::old::setup_env() {
+kube::golang::new::setup_env() {
kube::golang::verify_go_version
# Set up GOPATH. We have tools which depend on being in a GOPATH (see
@@ -7,9 +7,9 @@
# Even in module mode, we need to set GOPATH for `go build` and `go install`
# to work. We build various tools (usually via `go install`) from a lot of
# scripts.
- # * We can't set GOBIN because that does not work on cross-compiles.
- # * We could use `go build -o <something>`, but it's subtle when it comes
- # to cross-compiles and whether the <something> is a file or a directory,
+ # * We can't just set GOBIN because that does not work on cross-compiles.
+ # * We could always use `go build -o <something>`, but it's subtle wrt
+ # cross-compiles and whether the <something> is a file or a directory,
# and EVERY caller has to get it *just* right.
# * We could leave GOPATH alone and let `go install` write binaries
# wherever the user's GOPATH says (or doesn't say).
@@ -20,16 +20,6 @@
#
# Eventually, when we no longer rely on run-in-gopath.sh we may be able to
# simplify this some.
- local go_pkg_dir="${KUBE_GOPATH}/src/${KUBE_GO_PACKAGE}"
- local go_pkg_basedir
- go_pkg_basedir=$(dirname "${go_pkg_dir}")
-
- mkdir -p "${go_pkg_basedir}"
-
- # TODO: This symlink should be relative.
- if [[ ! -e "${go_pkg_dir}" || "$(readlink "${go_pkg_dir}")" != "${KUBE_ROOT}" ]]; then
- ln -snf "${KUBE_ROOT}" "${go_pkg_dir}"
- fi
export GOPATH="${KUBE_GOPATH}"
# If these are not set, set them now. This ensures that any subsequent
@@ -40,24 +30,10 @@
# Make sure our own Go binaries are in PATH.
export PATH="${KUBE_GOPATH}/bin:${PATH}"
- # Change directories so that we are within the GOPATH. Some tools get really
- # upset if this is not true. We use a whole fake GOPATH here to collect the
- # resultant binaries.
- local subdir
- subdir=$(kube::realpath . | sed "s|${KUBE_ROOT}||")
- cd "${KUBE_GOPATH}/src/${KUBE_GO_PACKAGE}/${subdir}" || return 1
-
- # Set GOROOT so binaries that parse code can work properly.
- GOROOT=$(go env GOROOT)
- export GOROOT
-
# Unset GOBIN in case it already exists in the current session.
# Cross-compiles will not work with it set.
unset GOBIN
- # This seems to matter to some tools
- export GO15VENDOREXPERIMENT=1
-
- # Disable workspaces
- export GOWORK=off
+ # Explicitly turn on modules.
+ export GO111MODULE=on
}
```
Result: `make` works for k/k:
```
$ make kubectl
+++ [1211 11:07:31] Building go targets for linux/amd64
k8s.io/kubernetes/cmd/kubectl (static)
$ make WHAT=./cmd/kubectl/
+++ [1211 11:08:19] Building go targets for linux/amd64
k8s.io/kubernetes/./cmd/kubectl/ (non-static)
$ make WHAT=k8s.io/kubernetes/cmd/kubectl
+++ [1211 11:08:52] Building go targets for linux/amd64
k8s.io/kubernetes/cmd/kubectl (static)
```
Result: `make` works for staging by package:
```
$ make WHAT=k8s.io/api
+++ [1211 11:11:37] Building go targets for linux/amd64
k8s.io/api (non-static)
```
Result: `make` fails for staging by path:
```
$ make WHAT=./staging/src/k8s.io/api
+++ [1211 11:12:44] Building go targets for linux/amd64
k8s.io/kubernetes/./staging/src/k8s.io/api (non-static)
cannot find module providing package k8s.io/kubernetes/staging/src/k8s.io/api: import lookup disabled by -mod=vendor
(Go version in go.work is at least 1.14 and vendor directory exists.)
!!! [1211 11:12:44] Call tree:
!!! [1211 11:12:44] 1: /home/thockin/src/kubernetes/hack/lib/golang.sh:850 kube::golang::build_some_binaries(...)
!!! [1211 11:12:44] 2: /home/thockin/src/kubernetes/hack/lib/golang.sh:1012 kube::golang::build_binaries_for_platform(...)
!!! [1211 11:12:44] 3: hack/make-rules/build.sh:27 kube::golang::build_binaries(...)
!!! [1211 11:12:44] Call tree:
!!! [1211 11:12:44] 1: hack/make-rules/build.sh:27 kube::golang::build_binaries(...)
!!! [1211 11:12:44] Call tree:
!!! [1211 11:12:44] 1: hack/make-rules/build.sh:27 kube::golang::build_binaries(...)
make: *** [Makefile:96: all] Error 1
```
Result: `make test` fails:
```
$ make test WHAT=./cmd/kubectl
+++ [1211 11:13:38] Set GOMAXPROCS automatically to 6
+++ [1211 11:13:38] Running tests without code coverage and with -race
cmd/kubectl/kubectl.go:25:2: cannot find package "k8s.io/client-go/plugin/pkg/client/auth" in any of:
/home/thockin/src/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/client-go/plugin/pkg/client/auth (vendor tree)
/home/thockin/sdk/gotip/src/k8s.io/client-go/plugin/pkg/client/auth (from $GOROOT)
/home/thockin/src/kubernetes/_output/local/go/src/k8s.io/client-go/plugin/pkg/client/auth (from $GOPATH)
cmd/kubectl/kubectl.go:20:2: cannot find package "k8s.io/component-base/cli" in any of:
/home/thockin/src/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/component-base/cli (vendor tree)
/home/thockin/sdk/gotip/src/k8s.io/component-base/cli (from $GOROOT)
/home/thockin/src/kubernetes/_output/local/go/src/k8s.io/component-base/cli (from $GOPATH)
cmd/kubectl/kubectl.go:21:2: cannot find package "k8s.io/kubectl/pkg/cmd" in any of:
/home/thockin/src/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/kubectl/pkg/cmd (vendor tree)
/home/thockin/sdk/gotip/src/k8s.io/kubectl/pkg/cmd (from $GOROOT)
/home/thockin/src/kubernetes/_output/local/go/src/k8s.io/kubectl/pkg/cmd (from $GOPATH)
cmd/kubectl/kubectl.go:22:2: cannot find package "k8s.io/kubectl/pkg/cmd/util" in any of:
/home/thockin/src/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/kubectl/pkg/cmd/util (vendor tree)
/home/thockin/sdk/gotip/src/k8s.io/kubectl/pkg/cmd/util (from $GOROOT)
/home/thockin/src/kubernetes/_output/local/go/src/k8s.io/kubectl/pkg/cmd/util (from $GOPATH)
make: *** [Makefile:191: test] Error 1
```
for now:
- shim FORCE_HOST_GO to GOTOOLCHAIN=local
- treat GOTOOLCHAIN set and !=auto like FORCE_HOST_GO
- otherwise set GOTOOLCHAIN=go${GO_VERSION} and fallback to gimme if necessary
TODO: set toolchain statements in go.mod files and keep them in sync
