For unknown reasons, hack/make-rules/test-e2e-node.sh adds -timeout instead of
--timeout. Therefore the fallback code in test/e2e_node/remote/remote.go didn't
find it and added its own --timeout=60m after it. This effectively limits E2E
node test runs to 60 minutes, regardless of what is specified in the job:
W0206 09:53:51.425532 7151 remote.go:158] ginkgo flags are missing explicit --timeout (ginkgo defaults to 60 minutes)
I0206 09:53:51.425565 7151 remote.go:165] updated ginkgo flags: -timeout=24h --label-filter="Feature: containsAny DynamicResourceAllocation && Feature: isSubsetOf { Beta, DynamicResourceAllocation } && !Flaky && !Slow" --no-color -v --timeout=60m
...
I0206 09:53:57.767096 7151 ssh.go:146] Running the command ssh, with args: ... timeout -k 30s 3600.000000s ./ginkgo -timeout=24h --label-filter="Feature: containsAny DynamicResourceAllocation && Feature: isSubsetOf { Beta, DynamicResourceAllocation } && !Flaky && !Slow" --no-color -v --timeout=60m ...
Note that the timeout for the test was 60m in this case (hence the "timeout -k
30s 3600.000000s") but it could also be something larger.
Around the 1.31 release, we discovered that a change introduced in 1.27 allowead
clients to open WATCH requests directly to etcd. This had detrimental consequences,
enabling abusive clients to bypass caching and overwhelm etcd.
Unlike the API server, etcd lacks protection against such behavior.
To mitigate this, we redirected all WATCH requests to be served from the cache.
The WatchFromStorageWithoutResourceVersion feature gate was retained as an escape hatch.
However, since we have no plans to allow direct WATCH requests to etcd again,
this flag is now obsolete.
Direct WATCH requests to etcd offer no advantage, as they don't provide stronger
consistency guarantees. WATCH operations are inherently inconsistent; unlike LIST
operations, they do not confirm the resource version with a quorum. While Kubernetes
uses the WithRequireLeader option on WATCH requests to prevent maintaining connections
to isolated etcd members, the API server provides the same level of guarantee through
its health checks, which fail if it cannot connect to etcd member. Therefore,
the WatchFromStorageWithoutResourceVersion feature gate can be deprecated and removed.
They were enabled yesterday and executed seven times, with results that (so
far) seem to be fairly stable with just one run that was slower across the
board.
The links in the YAML can be used to navigate to each test case quickly. The
thresholds were chose with a 20% security margin below what seems to be a
common result.
Watch requests to etcd are mapped to a single stream that has a limited throughput.
By opening a lot of concurrent watch requests to single resource, users
could starve other watches from getting any events.
Separating the RPC was meant to protect the watch opened by cache.
However, as we are no longer planning to allow users to open watch directly to etcd,
the flag is not needed.
