Commit Graph

62 Commits

Author SHA1 Message Date
Paco Xu
7d8437933e retry on conflict for negative TerminationGracePeriodSeconds update 2023-03-01 12:55:58 +08:00
Kubernetes Prow Robot
aa98f6f4da
Merge pull request #115606 from wzshiming/fix/termination_grace_period_seconds
`pod.spec.terminationGracePeriodSeconds` is a negative then convert to 1
2023-02-23 07:35:35 -08:00
Shiming Zhang
b70d1377bb Add test for Pod TerminationGracePeriodSeconds is negative 2023-02-22 13:36:16 +08:00
Michal Wozniak
fd28f69ca4 Add e2e_node test for oom killed container reason 2023-02-20 08:15:45 +01:00
Patrick Ohly
2f6c4f5eab e2e: use Ginkgo context
All code must use the context from Ginkgo when doing API calls or polling for a
change, otherwise the code would not return immediately when the test gets
aborted.
2022-12-16 20:14:04 +01:00
Patrick Ohly
d4729008ef e2e: simplify test cleanup
ginkgo.DeferCleanup has multiple advantages:
- The cleanup operation can get registered if and only if needed.
- No need to return a cleanup function that the caller must invoke.
- Automatically determines whether a context is needed, which will
  simplify the introduction of context parameters.
- Ginkgo's timeline shows when it executes the cleanup operation.
2022-12-13 08:09:01 +01:00
Patrick Ohly
df5d84ae81 e2e: accept context from Ginkgo
Every ginkgo callback should return immediately when a timeout occurs or the
test run manually gets aborted with CTRL-C. To do that, they must take a ctx
parameter and pass it through to all code which might block.

This is a first automated step towards that: the additional parameter got added
with

    sed -i 's/\(framework.ConformanceIt\|ginkgo.It\)\(.*\)func() {$/\1\2func(ctx context.Context) {/' \
        $(git grep -l -e framework.ConformanceIt -e ginkgo.It )
    $GOPATH/bin/goimports -w $(git status | grep modified: | sed -e 's/.* //')

log_test.go was left unchanged.
2022-12-10 19:50:18 +01:00
Patrick Ohly
dfdf88d4fa e2e: adapt to moved code
This is the result of automatically editing source files like this:

    go install golang.org/x/tools/cmd/goimports@latest
    find ./test/e2e* -name "*.go" | xargs env PATH=$GOPATH/bin:$PATH ./e2e-framework-sed.sh

with e2e-framework-sed.sh containing this:

sed -i \
    -e "s/\(f\|fr\|\w\w*\.[fF]\w*\)\.ExecCommandInContainer(/e2epod.ExecCommandInContainer(\1, /" \
    -e "s/\(f\|fr\|\w\w*\.[fF]\w*\)\.ExecCommandInContainerWithFullOutput(/e2epod.ExecCommandInContainerWithFullOutput(\1, /" \
    -e "s/\(f\|fr\|\w\w*\.[fF]\w*\)\.ExecShellInContainer(/e2epod.ExecShellInContainer(\1, /" \
    -e "s/\(f\|fr\|\w\w*\.[fF]\w*\)\.ExecShellInPod(/e2epod.ExecShellInPod(\1, /" \
    -e "s/\(f\|fr\|\w\w*\.[fF]\w*\)\.ExecShellInPodWithFullOutput(/e2epod.ExecShellInPodWithFullOutput(\1, /" \
    -e "s/\(f\|fr\|\w\w*\.[fF]\w*\)\.ExecWithOptions(/e2epod.ExecWithOptions(\1, /" \
    -e "s/\(f\|fr\|\w\w*\.[fF]\w*\)\.MatchContainerOutput(/e2eoutput.MatchContainerOutput(\1, /" \
    -e "s/\(f\|fr\|\w\w*\.[fF]\w*\)\.PodClient(/e2epod.NewPodClient(\1, /" \
    -e "s/\(f\|fr\|\w\w*\.[fF]\w*\)\.PodClientNS(/e2epod.PodClientNS(\1, /" \
    -e "s/\(f\|fr\|\w\w*\.[fF]\w*\)\.TestContainerOutput(/e2eoutput.TestContainerOutput(\1, /" \
    -e "s/\(f\|fr\|\w\w*\.[fF]\w*\)\.TestContainerOutputRegexp(/e2eoutput.TestContainerOutputRegexp(\1, /" \
    -e "s/framework.AddOrUpdateLabelOnNode\b/e2enode.AddOrUpdateLabelOnNode/" \
    -e "s/framework.AllNodes\b/e2edebug.AllNodes/" \
    -e "s/framework.AllNodesReady\b/e2enode.AllNodesReady/" \
    -e "s/framework.ContainerResourceGatherer\b/e2edebug.ContainerResourceGatherer/" \
    -e "s/framework.ContainerResourceUsage\b/e2edebug.ContainerResourceUsage/" \
    -e "s/framework.CreateEmptyFileOnPod\b/e2eoutput.CreateEmptyFileOnPod/" \
    -e "s/framework.DefaultPodDeletionTimeout\b/e2epod.DefaultPodDeletionTimeout/" \
    -e "s/framework.DumpAllNamespaceInfo\b/e2edebug.DumpAllNamespaceInfo/" \
    -e "s/framework.DumpDebugInfo\b/e2eoutput.DumpDebugInfo/" \
    -e "s/framework.DumpNodeDebugInfo\b/e2edebug.DumpNodeDebugInfo/" \
    -e "s/framework.EtcdUpgrade\b/e2eproviders.EtcdUpgrade/" \
    -e "s/framework.EventsLister\b/e2edebug.EventsLister/" \
    -e "s/framework.ExecOptions\b/e2epod.ExecOptions/" \
    -e "s/framework.ExpectNodeHasLabel\b/e2enode.ExpectNodeHasLabel/" \
    -e "s/framework.ExpectNodeHasTaint\b/e2enode.ExpectNodeHasTaint/" \
    -e "s/framework.GCEUpgradeScript\b/e2eproviders.GCEUpgradeScript/" \
    -e "s/framework.ImagePrePullList\b/e2epod.ImagePrePullList/" \
    -e "s/framework.KubectlBuilder\b/e2ekubectl.KubectlBuilder/" \
    -e "s/framework.LocationParamGKE\b/e2eproviders.LocationParamGKE/" \
    -e "s/framework.LogSizeDataTimeseries\b/e2edebug.LogSizeDataTimeseries/" \
    -e "s/framework.LogSizeGatherer\b/e2edebug.LogSizeGatherer/" \
    -e "s/framework.LogsSizeData\b/e2edebug.LogsSizeData/" \
    -e "s/framework.LogsSizeDataSummary\b/e2edebug.LogsSizeDataSummary/" \
    -e "s/framework.LogsSizeVerifier\b/e2edebug.LogsSizeVerifier/" \
    -e "s/framework.LookForStringInLog\b/e2eoutput.LookForStringInLog/" \
    -e "s/framework.LookForStringInPodExec\b/e2eoutput.LookForStringInPodExec/" \
    -e "s/framework.LookForStringInPodExecToContainer\b/e2eoutput.LookForStringInPodExecToContainer/" \
    -e "s/framework.MasterAndDNSNodes\b/e2edebug.MasterAndDNSNodes/" \
    -e "s/framework.MasterNodes\b/e2edebug.MasterNodes/" \
    -e "s/framework.MasterUpgradeGKE\b/e2eproviders.MasterUpgradeGKE/" \
    -e "s/framework.NewKubectlCommand\b/e2ekubectl.NewKubectlCommand/" \
    -e "s/framework.NewLogsVerifier\b/e2edebug.NewLogsVerifier/" \
    -e "s/framework.NewNodeKiller\b/e2enode.NewNodeKiller/" \
    -e "s/framework.NewResourceUsageGatherer\b/e2edebug.NewResourceUsageGatherer/" \
    -e "s/framework.NodeHasTaint\b/e2enode.NodeHasTaint/" \
    -e "s/framework.NodeKiller\b/e2enode.NodeKiller/" \
    -e "s/framework.NodesSet\b/e2edebug.NodesSet/" \
    -e "s/framework.PodClient\b/e2epod.PodClient/" \
    -e "s/framework.RemoveLabelOffNode\b/e2enode.RemoveLabelOffNode/" \
    -e "s/framework.ResourceConstraint\b/e2edebug.ResourceConstraint/" \
    -e "s/framework.ResourceGathererOptions\b/e2edebug.ResourceGathererOptions/" \
    -e "s/framework.ResourceUsagePerContainer\b/e2edebug.ResourceUsagePerContainer/" \
    -e "s/framework.ResourceUsageSummary\b/e2edebug.ResourceUsageSummary/" \
    -e "s/framework.RunHostCmd\b/e2eoutput.RunHostCmd/" \
    -e "s/framework.RunHostCmdOrDie\b/e2eoutput.RunHostCmdOrDie/" \
    -e "s/framework.RunHostCmdWithFullOutput\b/e2eoutput.RunHostCmdWithFullOutput/" \
    -e "s/framework.RunHostCmdWithRetries\b/e2eoutput.RunHostCmdWithRetries/" \
    -e "s/framework.RunKubectl\b/e2ekubectl.RunKubectl/" \
    -e "s/framework.RunKubectlInput\b/e2ekubectl.RunKubectlInput/" \
    -e "s/framework.RunKubectlOrDie\b/e2ekubectl.RunKubectlOrDie/" \
    -e "s/framework.RunKubectlOrDieInput\b/e2ekubectl.RunKubectlOrDieInput/" \
    -e "s/framework.RunKubectlWithFullOutput\b/e2ekubectl.RunKubectlWithFullOutput/" \
    -e "s/framework.RunKubemciCmd\b/e2ekubectl.RunKubemciCmd/" \
    -e "s/framework.RunKubemciWithKubeconfig\b/e2ekubectl.RunKubemciWithKubeconfig/" \
    -e "s/framework.SingleContainerSummary\b/e2edebug.SingleContainerSummary/" \
    -e "s/framework.SingleLogSummary\b/e2edebug.SingleLogSummary/" \
    -e "s/framework.TimestampedSize\b/e2edebug.TimestampedSize/" \
    -e "s/framework.WaitForAllNodesSchedulable\b/e2enode.WaitForAllNodesSchedulable/" \
    -e "s/framework.WaitForSSHTunnels\b/e2enode.WaitForSSHTunnels/" \
    -e "s/framework.WorkItem\b/e2edebug.WorkItem/" \
    "$@"

for i in "$@"; do
    # Import all sub packages and let goimports figure out which of those
    # are redundant (= already imported) or not needed.
    sed -i -e '/"k8s.io.kubernetes.test.e2e.framework"/a e2edebug "k8s.io/kubernetes/test/e2e/framework/debug"' "$i"
    sed -i -e '/"k8s.io.kubernetes.test.e2e.framework"/a e2ekubectl "k8s.io/kubernetes/test/e2e/framework/kubectl"' "$i"
    sed -i -e '/"k8s.io.kubernetes.test.e2e.framework"/a e2enode "k8s.io/kubernetes/test/e2e/framework/node"' "$i"
    sed -i -e '/"k8s.io.kubernetes.test.e2e.framework"/a e2eoutput "k8s.io/kubernetes/test/e2e/framework/pod/output"' "$i"
    sed -i -e '/"k8s.io.kubernetes.test.e2e.framework"/a e2epod "k8s.io/kubernetes/test/e2e/framework/pod"' "$i"
    sed -i -e '/"k8s.io.kubernetes.test.e2e.framework"/a e2eproviders "k8s.io/kubernetes/test/e2e/framework/providers"' "$i"
    goimports -w "$i"
done
2022-10-06 08:19:47 +02:00
Lars Ekman
e93f1161ea Replace fallocate with dd in e2e tests 2022-07-23 17:22:49 +02:00
Dave Chen
857458cfa5 update ginkgo from v1 to v2 and gomega to 1.19.0
- update all the import statements
- run hack/pin-dependency.sh to change pinned dependency versions
- run hack/update-vendor.sh to update go.mod files and the vendor directory
- update the method signatures for custom reporters

Signed-off-by: Dave Chen <dave.chen@arm.com>
2022-07-08 10:44:46 +08:00
Antonio Ojea
ffdbce6291 e2e test for evicted pods 2022-05-27 06:43:03 +02:00
Sergiusz Urbaniak
373c08e0c7
test/e2e/framework: configure pod security admission level for e2e tests 2022-03-28 15:42:10 +02:00
Clayton Coleman
ca98714ec0
test: Add E2E for init container pod deletion
Exploring termination revealed we have race conditions in certain
parts of pod initialization and termination. To better catch these
issues refactor the existing test so it can be reused, and then test
a number of alternate scenarios.
2022-03-16 13:15:03 -04:00
Elana Hashman
f9a827b895
Fix bugs in e2e pod test
- Update test to account for late synced statuses
- Terminated containers are not running
- Add missing format values to test logline
2021-10-05 11:51:12 -07:00
Davanum Srinivas
07332ad398
fix ineffassign and varcheck
Signed-off-by: Davanum Srinivas <davanum@gmail.com>
2021-07-14 08:41:22 -04:00
Clayton Coleman
3eadd1a9ea
Keep pod worker running until pod is truly complete
A number of race conditions exist when pods are terminated early in
their lifecycle because components in the kubelet need to know "no
running containers" or "containers can't be started from now on" but
were relying on outdated state.

Only the pod worker knows whether containers are being started for
a given pod, which is required to know when a pod is "terminated"
(no running containers, none coming). Move that responsibility and
podKiller function into the pod workers, and have everything that
was killing the pod go into the UpdatePod loop. Split syncPod into
three phases - setup, terminate containers, and cleanup pod - and
have transitions between those methods be visible to other
components. After this change, to kill a pod you tell the pod worker
to UpdatePod({UpdateType: SyncPodKill, Pod: pod}).

Several places in the kubelet were incorrect about whether they
were handling terminating (should stop running, might have
containers) or terminated (no running containers) pods. The pod worker
exposes methods that allow other loops to know when to set up or tear
down resources based on the state of the pod - these methods remove
the possibility of race conditions by ensuring a single component is
responsible for knowing each pod's allowed state and other components
simply delegate to checking whether they are in the window by UID.

Removing containers now no longer blocks final pod deletion in the
API server and are handled as background cleanup. Node shutdown
no longer marks pods as failed as they can be restarted in the
next step.

See https://docs.google.com/document/d/1Pic5TPntdJnYfIpBeZndDelM-AbS4FN9H2GTLFhoJ04/edit# for details
2021-07-06 15:55:22 -04:00
Clayton Coleman
2d361d4306
e2e: Flag questionable durations in container status on termination
The e2e that create/deletes pods rapidly and verifies their status
was reporting a very long timing:

timings total=12.211347385s t=491ms run=2s execute=450402h8m25s

in a few scenarios. Add error checks that clarify when this happens
and why. Report p50/75/90/99 latencies on teardown as observed from
the test for baseline for future changes.
2021-05-24 11:48:42 -04:00
Jordan Liggitt
8bbb6704c1 Update flaky volume toleration regex to include projected volumes 2021-03-06 11:39:36 -05:00
wojtekt
8df355b80a Replace some nested KubeDescribe calls with ginkgo.Describe 2021-02-25 14:44:39 +01:00
Kubernetes Prow Robot
5e44d8ebaf
Merge pull request #95015 from Jefftree/bdd-remove-dependency
Remove dependency on behaviors for conformance walk.go
2020-10-05 13:04:19 -07:00
Seth Jennings
b7c74338e8 test: e2e: fix race in pods test 2020-10-02 14:32:31 -05:00
Jefftree
fe972b8f2f Remove dependency on behaviors for conformance generation 2020-09-23 13:28:06 -07:00
Claudiu Belu
89eae73323 tests: Refactors agnhost image pod usage in tests
A previous commit created a few agnhost related functions that creates agnhost
pods / containers for general purposes.

Refactors tests to use those functions.
2020-09-22 13:09:49 -07:00
Kubernetes Prow Robot
48d5d204c3
Merge pull request #92614 from tnqn/onfailure-recreate
Don't create a new sandbox for pod with RestartPolicyOnFailure if all containers succeeded
2020-09-03 14:57:40 -07:00
Jefftree
ace97738e2 Update formatting of conformance comment 2020-07-29 20:50:44 -07:00
Quan Tian
b2b082f54f Don't create a new sandbox for pod with RestartPolicyOnFailure if all containers succeeded
The kubelet would attempt to create a new sandbox for a pod whose
RestartPolicy is OnFailure even after all container succeeded. It caused
unnecessary CRI and CNI calls, confusing logs and conflicts between the
routine that creates the new sandbox and the routine that kills the Pod.

This patch checks the containers to start and stops creating sandbox if
no container is supposed to start.
2020-07-07 22:49:48 +08:00
Ryan Phillips
d9d1fc6077 e2e pods: add missing pod and node name in the formatted log line 2020-06-11 14:50:27 -05:00
Kubernetes Prow Robot
f834c92ce1
Merge pull request #91632 from liggitt/ginkgo-recover-e2e
Defer ginkgo recovers
2020-06-01 16:24:14 -07:00
Jordan Liggitt
c9638d54d0 Defer ginkgo recovers 2020-06-01 11:02:41 -04:00
Jordan Liggitt
ee193bcd40 Add debugging and timeouts to pending pod e2e 2020-05-29 11:54:16 -04:00
Jefftree
ceb67482c4 link 2 pod conformance tests with behaviors 2020-05-18 23:35:25 -07:00
Derek Carr
02e4ce6835 deflake grace period for pod deletion in node e2e 2020-03-23 15:23:54 -04:00
Kenichi Omichi
2c8955fd4a Use e2epod.WaitForPodNameRunningInNamespace directly
WaitForPod*() are just wrapper functions for e2epod package, and they
made an invalid dependency to sub e2e framework from the core framework.
So this replaces WaitForPodRunning() with the e2epod function.
2020-03-17 00:13:14 +00:00
Clayton Coleman
d80981b657
test: Detect flakes caused by container teardown races on CRI
CRI returns a different reason than the dockershim, and we must also
catch the symptoms of https://github.com/kubernetes/kubernetes/issues/88766
in that environment.
2020-03-08 16:21:57 -04:00
Kubernetes Prow Robot
5708511499
Merge pull request #88708 from mikedanese/deleteopts
Migrate clientset metav1.DeleteOpts to pass-by-value
2020-03-05 23:09:23 -08:00
Mike Danese
c58e69ec79 automated refactor 2020-03-05 14:59:46 -08:00
Clayton Coleman
8c35e32a40
test: Properly detect container runtime flake in e2e test
The condition was not part of the message and so would not
match:

OCI runtime create failed: container_linux.go:349: starting container process caused "process_linux.go:449: container init caused \"rootfs_linux.go:58: mounting \\\"/var/lib/kubelet/pods/128aea1f-bde3-43d5-8b5f-dd86b9a5ef33/volumes/kubernetes.io~secret/default-token-v55hm\\\" to rootfs \\\"/var/lib/docker/overlay2/813487ba91d534ded546ae34f2a05e7d94c26bd015d356f9b2641522d8f0d6da/merged\\\" at \\\"/var/run/secrets/kubernetes.io/serviceaccount\\\" caused \\\"stat /var/lib/kubelet/pods/128aea1f-bde3-43d5-8b5f-dd86b9a5ef33/volumes/kubernetes.io~secret/default-token-v55hm: no such file or directory\\\"\"": unknown

Updated the check and regex.
2020-03-05 14:20:59 -05:00
Clayton Coleman
6d98b0a0f4
Test that an always-fail container can't report the pod Succeeded
The kubelet can race when a pod is deleted and report that a container succeeded
when it instead failed, and thus the pod is reported as succeeded. Create an e2e
test that demonstrates this failure.
2020-03-04 13:34:20 -05:00
Mike Danese
3aa59f7f30 generated: run refactor 2020-02-07 18:16:47 -08:00
Sakura
b9a586e382
Fix non-ascii characters in test/e2e/node/pods.go
Signed-off-by: Sakura <longfei.shang@daocloud.io>
2020-02-05 15:14:21 +08:00
Mike Danese
d55d6175f8 refactor 2020-01-29 08:50:45 -08:00
Aaron Crickenberger
4d731d92bf Demote Delete Grace Period test to [Flaky]
This is currently the top flake against PRs, so I'm tagging it
as [Flaky]. Flaky tests can't be conformance tests, so I'm
removing it from [Conformance] as well until this is resolved.
2020-01-08 10:12:47 -08:00
Jordan Liggitt
176c95ff3e Add debugging for delete grace period e2e flake 2019-12-16 14:15:19 -05:00
Kubernetes Prow Robot
9ae79ef8e3
Merge pull request #85849 from liggitt/deflake-delete-grace-period
Deflake delete grace period e2e
2019-12-03 13:18:34 -08:00
Jordan Liggitt
28327efe6b Deflake delete grace period e2e 2019-12-03 09:46:45 -05:00
Adelina Tuvenie
9a1f631c5c Replace nginx image with agnhost in sig-node pods tests 2019-11-05 13:18:15 +02:00
carlory
d1290ffdef clean up test code 2019-09-05 23:44:19 +08:00
Kenichi Omichi
0d2e5866d4 Use framework functions under test/e2e/node/
There are some functions of e2e test framework and it is useful to
read the test code by using these functions.
This replaces gomega calls with these functions under test/e2e/node/
2019-08-21 02:01:49 +00:00
WanLinghao
e43572f02b Move pod related functions file in test/e2e/framework/util.go to its sub direcotry 2019-07-31 13:53:41 +08:00
draveness
0ec95afb76 feat: use framework.ExpectEqual instead of should 2019-07-07 21:07:48 +08:00