This raises the number of allowed taints per device to 16 by lowering
the number of allowed devices to 64 per ResourceSlice if (and only if!)
taints are used.
"effect: None" and DeviceTaintRule status with conditions get added
to support giving feedback to admins.
Instead of merely adding the new effect value, this also changes validation of
the enum so that unknown values are valid if they were already stored. This
will simplify adding new effects in the future because validation won't fail
for them after a downgrade. Consumers must treat them like this new None
effect, i.e. ignore them.
The v1alpha3 version is still needed for DeviceTaintRule, but the rest of the
types and most structs became obsolete in v1.32 when we introduced v1beta1 and
bumped the storage version to v1beta1.
Removing them now simplifies adding new features because new fields don't need
to be added to these obsolete types. This could have been done already in 1.33,
but wasn't to minimize disrupting on-going work.
Previously, etcd wrote to stderr in JSON format:
{"level":"warn","ts":"2025-04-11T03:32:06.676527Z","caller":"embed/config.go:689","msg":"Running http and grpc server on single port. This is not recommended for production."}
{"level":"warn","ts":"2025-04-11T03:32:06.676707Z","caller":"embed/config.go:689","msg":"Running http and grpc server on single port. This is not recommended for production."}
{"level":"warn","ts":"2025-04-11T03:32:06.677056Z","caller":"etcdmain/etcd.go:146","msg":"failed to start etcd","error":"listen tcp 127.0.0.1:37803: bind: address already in use"}
{"level":"fatal","ts":"2025-04-11T03:32:06.677104Z","caller":"etcdmain/etcd.go:204","msg":"discovery failed","error":"listen tcp 127.0.0.1:37803: bind: address already in use","stacktrace":"go.etcd.io/etcd/server/v3/etcdmain.startEtcdOrProxyV2\n\tgo.etcd.io/etcd/server/v3/etcdmain/etcd.go:204\ngo.etcd.io/etcd/server/v3/etcdmain.Main\n\tgo.etcd.io/etcd/server/v3/etcdmain/main.go:40\nmain.main\n\tgo.etcd.io/etcd/server/v3/main.go:31\nruntime.main\n\truntime/proc.go:272"}
This has several drawbacks:
- Not very readable.
- When used in tests which start etcd themselves (for example, scheduler_perf),
the output is not associated with the current test.
- Contains warnings that are confusing for developers who don't know that they
are harmless.
Intercepting output, parsing it and reformating makes the output nicer. Instead
of a mixture of JSON messages (see above) and normal test output, we now get
the etcd output embedded inside the test output. We can also filter out some
known harmless messages. Cleaning up more output or avoiding it in the first
place might be a good next step.
With `go test -v ./test/integration/scheduler_perf/dra`:
=== RUN TestSchedulerPerf
=== RUN TestSchedulerPerf/SchedulingWithResourceClaimTemplate
=== RUN TestSchedulerPerf/SchedulingWithResourceClaimTemplate/fast
I0411 13:21:03.353458 65212 feature_gate.go:385] feature gates: {map[SchedulerQueueingHints:false]}
...
I0411 13:21:10.552975 65212 cidrallocator.go:210] stopping ServiceCIDR Allocator Controller
I0411 13:21:10.567327 65212 etcd.go:210] "etcd output" logger="TestSchedulerPerf/SchedulingWithResourceClaimTemplate/fast" error="accept tcp 127.0.0.1:42245: use of closed network connection" level="warn" ts="2025-04-11T13:21:10.567045+0200" caller="embed/serve.go:160" msg="stopping insecure grpc server due to error"
I0411 13:21:10.567398 65212 etcd.go:210] "etcd output" logger="TestSchedulerPerf/SchedulingWithResourceClaimTemplate/fast" ts="2025-04-11T13:21:10.567198+0200" caller="embed/serve.go:162" msg="stopped insecure grpc server due to error" error="accept tcp 127.0.0.1:42245: use of closed network connection" level="warn"
I0411 13:21:15.567917 65212 etcd.go:227] "etcd didn't exit in 5 seconds, killing it" logger="TestSchedulerPerf/SchedulingWithResourceClaimTemplate/fast"
I0411 13:21:15.567964 65212 etcd.go:234] "etcd exited" logger="TestSchedulerPerf/SchedulingWithResourceClaimTemplate/fast" err="signal: terminated"
With per-test output `go test -v ./test/integration/scheduler_perf/dra -args -use-testing-log`:
=== RUN TestSchedulerPerf
=== RUN TestSchedulerPerf/SchedulingWithResourceClaimTemplate
=== RUN TestSchedulerPerf/SchedulingWithResourceClaimTemplate/fast
I0411 13:19:03.540497 28645 feature_gate.go:385] feature gates: {map[DynamicResourceAllocation:true]}
...
I0411 13:19:10.519994 28645 cidrallocator.go:210] stopping ServiceCIDR Allocator Controller
etcd.go:210: I0411 13:19:10.533131] etcd output msg="stopping insecure grpc server due to error" error="accept tcp 127.0.0.1:46637: use of closed network connection" level="warn" ts="2025-04-11T13:19:10.532900+0200" caller="embed/serve.go:160"
etcd.go:210: I0411 13:19:10.533274] etcd output msg="stopped insecure grpc server due to error" error="accept tcp 127.0.0.1:46637: use of closed network connection" level="warn" ts="2025-04-11T13:19:10.533022+0200" caller="embed/serve.go:162"
etcd.go:227: I0411 13:19:15.533715] etcd didn't exit in 5 seconds, killing it
etcd.go:234: I0411 13:19:15.533803] etcd exited err="signal: terminated"
This adds the "DeviceTaint" top-level type to v1alpha3 and related fields to
ResourceSlice and ResourceClaim. It's complete enough bring up an API server
and generate files.
Integration testing has to this point relied on patching serving codecs for built-in APIs. The
test-only patching is removed and replaced by feature gated checks at runtime.
For alpha, there is one apiserver feature gate and two client-go feature gates controlling
CBOR. They were initially wired to separate test-only feature gate instances in order to prevent
them from being configurable at runtime via command-line flags or environment variables (for
client-go feature gates outside of Kubernetes components). All of the integration tests required by
the KEP as alpha criteria have been implemented. This adds the feature gates to the usual feature
gate instances and removes the temporary code to support separate test-only feature gate instances.
This removes the DRAControlPlaneController feature gate, the fields controlled
by it (claim.spec.controller, claim.status.deallocationRequested,
claim.status.allocation.controller, class.spec.suitableNodes), the
PodSchedulingContext type, and all code related to the feature.
The feature gets removed because there is no path towards beta and GA and DRA
with "structured parameters" should be able to replace it.
This is a complete revamp of the original API. Some of the key
differences:
- refocused on structured parameters and allocating devices
- support for constraints across devices
- support for allocating "all" or a fixed amount
of similar devices in a single request
- no class for ResourceClaims, instead individual
device requests are associated with a mandatory
DeviceClass
For the sake of simplicity, optional basic types (ints, strings) where the null
value is the default are represented as values in the API types. This makes Go
code simpler because it doesn't have to check for nil (consumers) and values
can be set directly (producers). The effect is that in protobuf, these fields
always get encoded because `opt` only has an effect for pointers.
The roundtrip test data for v1.29.0 and v1.30.0 changes because of the new
"request" field. This is considered acceptable because the entire `claims`
field in the pod spec is still alpha.
The implementation is complete enough to bring up the apiserver.
Adapting other components follows.
This is in preparation for revamping the resource.k8s.io completely. Because
there will be no support for transitioning from v1alpha2 to v1alpha3, the
roundtrip test data for that API in 1.29 and 1.30 gets removed.
Repeating the version in the import name of the API packages is not really
required. It was done for a while to support simpler grepping for usage of
alpha APIs, but there are better ways for that now. So during this transition,
"resourceapi" gets used instead of "resourcev1alpha3" and the version gets
dropped from informer and lister imports. The advantage is that the next bump
to v1beta1 will affect fewer source code lines.
Only source code where the version really matters (like API registration)
retains the versioned import.
The fact that the .status.loadBalancer field can be set while .spec.type
is not "LoadBalancer" is a flub. Any spec update will already clear
.status.ingress, so it's hard to really rely on this. After this
change, updates which try to set this combination will fail validation.
Existing cases of this will not be broken. Any spec/metadata update
will clear it (no error) and this is the only stanza of status.
New gate "AllowServiceLBStatusOnNonLB" is off by default, but can be
enabled if this change actually breaks someone, which seems exceeedingly
unlikely.
this check needs to go after any mutations. After the mutating admission chain, rest.BeforeUpdate (which is responsible for reverting updates to immutable timestamp fields, among other things.) is called in the store.Update function. Without moving this check, it will be possible for an object to be written to etcd with only a change to its managed fields timestamp.
The name "PodScheduling" was unusual because in contrast to most other names,
it was impossible to put an article in front of it. Now PodSchedulingContext is
used instead.
