Commit Graph

262 Commits

Author SHA1 Message Date
Sergey Kanzhelev
1e6281e4a2 first iteration to add standalone mode 2023-03-14 20:46:41 +00:00
Kubernetes Prow Robot
04e7021d06
Merge pull request #114625 from Divya063/feature-private-image-registry
[E2E] Add support for pulling images from private registry
2023-02-28 06:27:17 -08:00
Divya Rani
a8b1e57246 add support for pulling images from private registry 2023-02-28 00:40:51 -08:00
Patrick Ohly
3e2b26ce52 e2e: revise complete report creation
The previous approach was based on the observation that some Prow jobs use the
--report-dir parameter instead of the E2E_REPORT_DIR env variable. Parsing the
command line was necessary to use the --json-report and --junit-report
parameters.

But that is complex and can be avoided by triggering the creation of complete
reports in the E2E test suite. The paths are hard-coded and relative to the
report directory to keep the code simple.

There was a report that k8s-triage started processing more data after
6db4b741dd was merged. It's unclear whether
that was because of the new <report-dir>/ginkgo_report.xml file. To avoid
this potential problem, the reports are now in a "ginkgo" sub-directory.

While at it, error checking gets enhanced:
- Create directories at the start of
  the suite and bail out early if that fails.
- *All* e2e suites using the framework do this, not just test/e2e.
- Added missing error checking of truncated JUnit report writing.
2023-02-10 10:20:20 +01:00
Patrick Ohly
0f9a8d38be e2e framework: configure poll interval+duration for gomega
Primarily this protects against accidentally polling with the default interval
of 10ms. Setting these defaults may also make some tests simpler because they
don't need to override the defaults.
2023-01-09 17:29:35 +01:00
Patrick Ohly
db394db398 e2e: move several timeouts from TestContext into TimeoutContext
This consolidates timeout handling. In the future, configuration of all
timeouts via a configuration file might get added. For now, the same three
legacy command line flags for the timeouts that get moved continue to be
supported.
2023-01-09 17:29:35 +01:00
Patrick Ohly
0c2c027ebc e2e: clean up listing
It doesn't make sense for the E2E framework to have command line options that
don't do anything because then all test suites built with the framework inherit
those options.

For -list-images and -list-conformance-tests the solution is to move the
implementation into the framework (-list-images) respectively move the flag
into test/e2e (-list-conformance-tests).

The placement was decided based on the observation that image patching is
common functionality while conformance testing is specific to one test suite.
2022-12-23 10:36:06 +01:00
Kubernetes Prow Robot
020fb2eb77
Merge pull request #113538 from pohly/e2e-ginkgo-timeline
e2e: ginkgo timeline
2022-12-21 13:53:28 -08:00
Patrick Ohly
f6cdd37046 e2e framework: move JUnit write function
This makes it possible to unit test it.
2022-12-21 07:55:07 +01:00
Patrick Ohly
a53753575a e2e framework: stop overriding ginkgo verbosity/progress
The default settings from Ginkgo are useful enough for Kubernetes since
timeline support was added (https://github.com/onsi/ginkgo/issues/1069).
2022-12-21 07:55:07 +01:00
Patrick Ohly
f4a837affb e2e framework: avoid storing redundant failure message string
All information that we want will be written into the failure XML element's
data. We don't need the message tag and don't want it because our
tools (kettle, testgrid, spyglass) would then just concatenate the two strings.

This gets implemented for us by Ginkgo. However, truncating the failure message
is not supported there at the moment. It's unclear how important that is,
therefore this (recently added feature) gets removed.
2022-12-21 07:55:07 +01:00
Patrick Ohly
6db4b741dd e2e: dump full report information
The Ginkgo CLI output no longer has the full output. We need the reports
generated by Ginkgo for that.
2022-12-21 07:55:07 +01:00
Patrick Ohly
2f6c4f5eab e2e: use Ginkgo context
All code must use the context from Ginkgo when doing API calls or polling for a
change, otherwise the code would not return immediately when the test gets
aborted.
2022-12-16 20:14:04 +01:00
Patrick Ohly
023baa5e45 e2e framework: truncate too long failure messages when writing JUnit
Our tooling cannot handle very long failure messages well:
- when unfolding a test in the spyglass UI, it fills the entire screen
- failure correlation for http://go.k8s.io/triage has resource constraints

We cannot enforce that all tests only produce short failure messages and even
if we could, depending on the test failure, including more information may be
useful to understand it.

To achieve both goals (summary for correlation and overview, all details
available when digging deeper), too longer failure messages now get truncated,
with the full message guaranteed to be captured in the test output.

"Too long" is arbitrarily chosen to be similar to the gomega.MaxLength because
that has been a limit for failure message size in the past.
2022-10-28 15:43:48 +02:00
Patrick Ohly
b3f4cd66cd e2e framework: add -gomega-max-length parameter
When gomega.format exceeds the default size of 4000, it truncates and prints:

  Gomega truncated this representation as it exceeds 'format.MaxLength'.
  Consider having the object provide a custom 'GomegaStringer' representation
  or adjust the parameters in Gomega's 'format' package.

  Learn more here: https://onsi.github.io/gomega/#adjusting-output

These instructions don't help the user of the e2e.test binary unless we provide
a command line flag.
2022-10-28 15:43:48 +02:00
Patrick Ohly
8af3258526 e2e framework: move NodeKiller into framework/node
This removes some more dependencies from the framework.

Removing the configuration needs to be handled as part of refactoring the
TestContext.
2022-10-06 08:16:47 +02:00
Kubernetes Prow Robot
d2e3c70f1d
Merge pull request #111961 from pohly/e2e-klog-output
e2e: klog output
2022-09-17 14:38:28 -07:00
Patrick Ohly
d33e66b81b e2e: reduce built time for framework
Pulling the CreateKubeConfig function from the expensive to build
test/utils/apiserver package had a considerable impact on the overall build
time because that package depends on a lot of other packages.

Because only that one function is needed by the framework, that extra build
time can be avoided by moving it into its own package.
2022-09-10 17:56:23 +02:00
Patrick Ohly
7eb472246f e2e framework: use new apiserver helper package 2022-09-09 13:51:07 +02:00
Davanum Srinivas
9bbf01bae9
Remove references to openstack and cinder
Signed-off-by: Davanum Srinivas <davanum@gmail.com>
2022-08-22 16:43:14 -04:00
Patrick Ohly
96e9a93a6b e2e: disable double printing of errors
We don't want klog to print to anything other than GinkgoWriter, but it still
used os.Stderr in addition to GinkgoWriter when printing log entries with
severity >= error. Changing "stderrthreshold" fixes that.

The unit test for framework output handling didn't test klog behavior. Now it
does:
- os.Stderr is redirected, should be empty
- a new test invokes klog
2022-08-22 17:04:26 +02:00
Dave Chen
c299a12cf2 e2e: trim junit report for Spyglass, avoid "open stdout"
Including the full information for successful tests makes the resulting XML
file too large for the 200GB limit in Spyglass when running large jobs (like
scale testing).

The original solution from https://github.com/kubernetes/kubernetes/pull/111627
broke JUnit reporting in other test suites, in particular
test/e2e_node. Keeping the code inside the framework ensures that all test
suites continue to have the JUnit reporting.

AfterReadingAllFlags is a good place to set this up because all test suites
using the test context are expected to call it before running tests and after
parsing flags.

Removing the ReportEntries added by ginkgo.By from all test reports usually
avoids the `system-err` part in the JUnit file, which in Spyglass avoids
the extra "open stdout" button.

Co-authored-by: Patrick Ohly <patrick.ohly@intel.com>
Co-authored-by: Dave Chen <dave.chen@arm.com>
2022-08-03 17:56:36 +02:00
Danielle
5b312246c8
Revert "e2e: Trim junit reporter to adapt with testgrid" 2022-08-03 15:45:09 +02:00
Kubernetes Prow Robot
cf777ab782
Merge pull request #111627 from chendave/trim_report
e2e: Trim junit reporter to adapt with testgrid
2022-08-03 04:51:52 -07:00
Dave Chen
5fcb8e5153 e2e: trim junit report for Spyglass
Including the full information for successful tests makes the resulting XML
file too large for the 200GB limit in Spyglass when running large jobs (like
scale testing).

Co-authored-by: Patrick Ohly <patrick.ohly@intel.com>
Co-authored-by: Dave Chen <dave.chen@arm.com>
2022-08-03 00:32:24 +08:00
Matthew Cary
738b458f8e enable gce pd driver via a flag rather than an env var
Change-Id: Ib7c99083ad334e5f6dd152e376a303de794e0bf1
2022-07-28 17:23:28 -07:00
Davanum Srinivas
a9593d634c
Generate and format files
- Run hack/update-codegen.sh
- Run hack/update-generated-device-plugin.sh
- Run hack/update-generated-protobuf.sh
- Run hack/update-generated-runtime.sh
- Run hack/update-generated-swagger-docs.sh
- Run hack/update-openapi-spec.sh
- Run hack/update-gofmt.sh

Signed-off-by: Davanum Srinivas <davanum@gmail.com>
2022-07-26 13:14:05 -04:00
Dave Chen
3833695e75 Redirect klog out to GinkgoWriter
Co-authored-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Dave Chen <dave.chen@arm.com>
2022-07-08 10:46:11 +08:00
Dave Chen
82ac6be0e9 Custom reporter of Junit report is no longer needed
Ginkgo is now writing the JUnit file itself. The -report-dir parameter is used
as fallback for enabling JUnit output in case that users haven't migrated to
the new -junit-report parameter.

Co-authored-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Dave Chen <dave.chen@arm.com>
2022-07-08 10:46:11 +08:00
Dave Chen
5ac8105b86 Set Ginkgo config by the method of GinkgoConfiguration()
Signed-off-by: Dave Chen <dave.chen@arm.com>
2022-07-08 10:46:11 +08:00
Dave Chen
857458cfa5 update ginkgo from v1 to v2 and gomega to 1.19.0
- update all the import statements
- run hack/pin-dependency.sh to change pinned dependency versions
- run hack/update-vendor.sh to update go.mod files and the vendor directory
- update the method signatures for custom reporters

Signed-off-by: Dave Chen <dave.chen@arm.com>
2022-07-08 10:44:46 +08:00
Humble Chirammal
3988ac1498 correct the e2e framework flag description typos
Signed-off-by: Humble Chirammal <hchiramm@redhat.com>
2022-06-06 19:23:28 +05:30
Patrick Ohly
2664740043 e2e: move feature gate support from test/e2e to test/e2e_node
The test/e2e suite has never supported feature gates:
- it cannot discover at runtime how the cluster is configured
- its --feature-gates parameter had no effect

Despite that, tests were written that used
e2eskipper.SkipUnlessFeatureGateEnabled even though that function then only
checked the default feature gate state.  To catch such mistakes, e2e tests
suites now must explicitly enable feature gate checking via
e2eskipper.InitFeatureGates. They also must register their own command line
flag. When that is not done, then using SkipUnlessFeatureGateEnabled or
SkipIfFeatureGateEnabled leads to a test failure.

test/e2e_node does both and therefore continues to work as before.
2022-04-25 15:41:41 +02:00
Patrick Ohly
ff5558edc3 test: support storage tests with non-standard kubelet root directory
Some storage tests deploy DaemonSets which hard-code /var/lib/kubelet as root
directory for kubelet registration and pod directory. There was already a
parameter which allowed specifying the root directory, just with a very
confusing name ("--volume-dir") and matching field name. A --kubelet-root-dir
parameters gets added because this may make it easier to find the parameter,
with the old name preserved as an alias for the same field for backwards
compatibility.
2022-03-03 12:34:18 +01:00
Ciprian Hacman
3834761187 Remove unused --dockershim-checkpoint-dir e2e.test flag
Signed-off-by: Ciprian Hacman <ciprian@hakman.dev>
2022-02-16 15:34:28 +02:00
Kubernetes Prow Robot
1fa888529e
Merge pull request #107533 from neolit123/1.24-update-master-label-taint
kubeadm: apply "master" label/taint migration for 1.24
2022-02-15 21:44:36 -08:00
Ciprian Hacman
7d5afb322d Remove unused --container-runtime e2e.test flag
Signed-off-by: Ciprian Hacman <ciprian@hakman.dev>
2022-02-14 08:49:56 +02:00
ahrtr
fe95aa614c io/ioutil has already been deprecated in golang 1.16, so replace all ioutil with io and os 2022-02-03 05:32:12 +08:00
Lubomir I. Ivanov
8641897057 e2e/framework/test_context: add "control-plane" to non-blocking-taints
Having only the "master" taint in the list of non-blocking taints
blocks kubeadm / kind clusters from migrating to applying
both the "control-plane" and "master" taints in 1.24.
Add "control-plane" to the list of taints.

Leave TODO to cleanup the "master" taint in 1.25+.
It has to be removed either way as part of the inclusive
language cleanup efforts.
2022-01-21 14:12:49 +02:00
Ciprian Hacman
d01e9cedb1 Clean up logic for deprecated flag --container-runtime in scripts
Signed-off-by: Ciprian Hacman <ciprian@hakman.dev>
2022-01-20 11:19:08 +02:00
Varsha Teratipally
94ff5ea1f1 change default container-runtime-endpoint to containerd.sock 2022-01-18 22:00:31 +00:00
Ciprian Hacman
a0abe5aa33 Clean up dockershim in tests
Signed-off-by: Ciprian Hacman <ciprian@hakman.dev>
2021-12-22 13:05:34 +02:00
Claudiu Belu
35e23afa50 tests: Prepull images commonly used test images
Some tests have a short timeout for starting the pods (1 minute), but if
those tests happen to be the first ones to run, and the images have to be
pulled, then the test could timeout, especially with larger images. This
commit will allow us to prepull commonly used E2E test images, so this issue
can be avoided.
2021-11-05 14:22:50 -07:00
Francesco Romani
54c7d8fbb1 e2e: TM: add option to fail instead of skip
The Topology Manager e2e tests wants to run on real multi-NUMA system
and want to consume real devices supported by device plugins; SRIOV
devices happen to be the most commonly available of such devices.

CI machines aren't multi NUMA nor expose SRIOV devices, so the biggest portion
of the tests will just skip, and we need to keep it like this until we
figure out how to enable these features.

However, some organizations can and want to run the testsuite on bare metal;
in this case, the current test will skip (not fail) with misconfigured
boxes, and this reports a misleading result. It will be much better to
fail if the test preconditions aren't met.

To satisfy both needs, we add an option, controlled by an environment
variable, to fail (not skip) if the machine on which the test run
doesn't meet the expectations (multi-NUMA, 4+ cores per NUMA cell,
expose SRIOV VFs).
We keep the old behaviour as default to keep being CI friendly.

Signed-off-by: Francesco Romani <fromani@redhat.com>
2021-09-13 13:23:36 +02:00
Sascha Grunert
484b027536
Remove unused --allow-gathering-profiles e2e.test flag
The profile gatherer has been removed in
https://github.com/kubernetes/kubernetes/pull/85304, so those options
are unused since then and can therefore be removed.

Signed-off-by: Sascha Grunert <sgrunert@redhat.com>
2021-08-31 11:33:33 +02:00
Niu Lechuan
12dcd2f84d Remove usage of github.com/pkg/errors
Signed-off-by: Niu Lechuan <lechuan.niu@daocloud.io>
2021-06-23 17:46:23 +08:00
Kubernetes Prow Robot
af60bebde3
Merge pull request #97028 from knabben/e2e-restart-kubelet
Adding restart kubelet flag on e2e test
2021-06-22 21:00:09 -07:00
Grant Griffiths
564e531aa7 Add Snapshot Controller e2e metric tests
Signed-off-by: Grant Griffiths <ggriffiths@purestorage.com>
2021-05-20 23:29:23 -07:00
Amim Knabben
0341e4c2f3 Enabling runtime config on E2E node tests 2021-03-08 15:45:06 -05:00
Kubernetes Prow Robot
26744ac58d
Merge pull request #98781 from smarterclayton/skip_all_nodes
Bug 1920221: Allow test invokers to skip test waits before and after
2021-02-05 06:32:52 -08:00