Commit Graph

534 Commits

Author SHA1 Message Date
Chelsea Mafrica
84ccdd8ef2 vendor: update OpenTelemetry to v0.20.0
Update OpenTelemetry from v0.15.0 to v0.20.0.

    Git log

    02d8bdd5 Release v0.20.0 (1837)
    aa66fe75 OS and Process resource detectors (1788)
    7374d679 Fix Links documents (1835)
    856f5b84 Add feature request issue template (1831)
    0fdc3d78 Remove bundler from Jaeger exporter (1830)
    738ef11e Fix flaky global ErrorHandler delegation test (1829)
    e43d9c00  Update Default Value for Jaeger Exporter Endpoint  (1824)
    0032bd64 Fix default merging of resource attributes from environment variable (1785)
    96c5e4ba Add SpanProcessor example for Span annotation on start (1733)
    543c8144 Remove the WithSDKOptions from the Jaeger exporter (1825)
    66389ad6 Update function docs in sdk.go (1826)
    70bc9eb3 Adds support for timeout on the otlp/gRPC exporter (1821)
    081cc61d Update Jaeger exporter convenience functions (1822)
    1b9f16d3 Remove the WithDisabled option from Jaeger exporter (1806)
    6867faa0 Bump actions/cache from v2.1.4 to v2.1.5 (1818)
    a2bf04dc Build context pipeline in Jaeger upload process (1809)
    2de86f23 Remove locking from Jaeger exporter shutdown/export (1807)
    4f9fec29 Add ExportSpans benchmark to Jaeger exporter (1805)
    d9566abe Fix OTLP testing flake: signal connection from mock collector (1816)
    a2cecb6e add support for env var configuration to otlp/gRPC (1811)
    d616df61 Fix flaky OTLP exporter reconnect test (1814)
    b09df84a Changes stdout to expose the `*sdktrace.TracerProvider` (1800)
    04890608 Remove options field from Jaeger exporter (1808)
    6db20e00 Remove the abandoned Process struct in Jaeger exporter (1804)
    086abf34 docs: use test example to document prometheus.InstallNewPipeline (1796)
    d0cea04b Bump google.golang.org/api from 0.43.0 to 0.44.0 in /exporters/trace/jaeger (1792)
    99c477fe Fixed typo for default service name in Jaeger Exporter (1797)
    95fd8f50 Bump google.golang.org/grpc from 1.36.1 to 1.37.0 in /exporters/otlp (1791)
    9b251644 Zipkin Exporter: Use default resouce's serviceName as default serivce name (1777) (1786)
    4d141e47 Add k8s.node.name and k8s.node.uid to semconv (1789)
    5c99a34c Fix golint issue caused by incorrect comment (1795)
    c5d006c0 Update Jaeger environment variables (1752)
    58432808 add NewExportPipeline and InstallNewPipeline for otlp (1373)
    7d8e6bd7 Zipkin Exporter: Adjust span transformation to comply with the spec (1688)
    2817c091 Merge sdk/export/trace into sdk/trace (1778)
    c61e654c Refactor prometheus exporter tests to match file headers as well (1470)
    23422c56 Remove process config for Jaeger exporter (1776)
    0d49b592 Add test to check bsp ignores `OnEnd` and `ForceFlush` post Shutdown` (1772)
    e9aaa04b Record links/events attribute drops independently (1771)
    5bbfc22c Make ExportSpans for Jaeger Exporter honor deadline (1773)
    0786fe32 Add Bug report issue templates (1775)
    3c7facee Add `ExportTimeout` option to batch span processor (1755)
    c6b92d5b Make TraceFlags spec-compliant (1770)
    ee687ca5 Bump github.com/itchyny/gojq from 0.12.2 to 0.12.3 in /internal/tools (1774)
    52a24774 add support for configuring tls certs via env var to otlp/HTTP (1769)
    35cfbc7e Update precedence of event name in Jaeger exporter (1768)
    33699d24 Adds semantic conventions for exceptions (1492)
    928e3c38 Modify ForceFlush to abort after timeout/cancellation (1757)
    3947cab4 Fix testCollectorEndpoint typo and add tag assertions in jaeger_test (1753)
    ecc635dc add website docs (1747)
    07a8d195 Fix Jaeger span status reporting and unify tag keys (1761)
    4fa35c90 add partial support for env var config to otlp/HTTP (1758)
    bf180d0f improve OTLP/gRPC connection errors (1737)
    d575865b Fix span IsRecording when not sampling (1750)
    20c93b01 Update SamplingParameters (1749)
    97501a3f Update SpanSnapshot to use parent SpanContext (1748)
    604b05cb Store current Span instead of local and remote SpanContext in context.Context (1731)
    c61f4b6d Set @lizthegrey to emeritus status (1745)
    b1342fec Bump github.com/golangci/golangci-lint in /internal/tools (1743)
    54e1bd19 Bump google.golang.org/api from 0.41.0 to 0.43.0 in /exporters/trace/jaeger (1741)
    4d25b6a2 Bump github.com/prometheus/client_golang from 1.9.0 to 1.10.0 in /exporters/metric/prometheus (1740)
    0a47b66f Bump google.golang.org/grpc from 1.36.0 to 1.36.1 in /exporters/otlp (1739)
    26f006b8 Reinstate @paivagustavo as an Approver (1734)
    382c7ced Remove hasRemoteParent field from SDK span (1728)
    862a5a68 Remove setting error status while recording error with Span from oteltest package (1729)
    6defcfdf Remove links on NewRoot spans (1726)
    a9b2f851 upgrade thrift to v0.14.1 in jaeger exporter (1712)
    5a6a854d Bump google.golang.org/protobuf from 1.25.0 to 1.26.0 in /exporters/otlp (1724)
    23486213 Migrate to using go.opentelemetry.io/proto/otlp (1713)
    5d559b40 Remove makeSamplingDecision func (1711)
    e24702da Update the TraceContext.Extract docs (1720)
    9d4eb1f6 Update dates in CHANGELOG.md for 2021 releases (1723)
    2b4fa968 Release v0.19.0 (1710)
    4beb7041 sdk/trace: removing ApplyConfig and Config (1693)
    1d42be16 Rename WithDefaultSampler TracerProvider option to WithSampler and update docs (1702)
    860d5d86 Add flag to determine whether SpanContext is remote (1701)
    0fe65e6b Comply with OpenTelemetry attributes specification (1703)
    88884351 Bump google.golang.org/api from 0.40.0 to 0.41.0 in /exporters/trace/jaeger (1700)
    345f264a breaking(zipkin): removes servicName from zipkin exporter. (1697)
    62cbf0f2 Populate Jaeger's Span.Process from Resource (1673)
    28eaaa9a Add a test to prove the Tracer is safe for concurrent calls (1665)
    8b1be11a Rename resource pkg label vars and methods (1692)
    a1539d44 OpenCensus metric exporter bridge (1444)
    77aa218d Fix issue #1490, apply same logic as in the SDK (1687)
    9d3416cc Fix synchronization issues in global trace delegate implementation (1686)
    58f69f09 Span status from HTTP code: Do not set status message if it can be inferred (1681)
    9c305bde Flush metric events prior to shutdown in OTLP example (1678)
    66b1135a Fix CHANGELOG (1680)
    90bd4ab5 Update employer information for maintainers (1683)
    36841913 Remove WithRecord() option from trace.SpanOption when starting a span (1660)
    65c7de20 Remove trace prefix from NoOp src files. (1679)
    e88a091a Make SpanContext Immutable (1573)
    d75e2680 Avoid overriding configuration of tracer provider (1633)
    2b4d5ac3 Bump github.com/golangci/golangci-lint in /internal/tools (1671)
    150b868d Bump github.com/google/go-cmp from 0.5.4 to 0.5.5 (1667)
    76aa924e Fix the examples target info messaging (1676)
    a3aa9fda Bump github.com/itchyny/gojq from 0.12.1 to 0.12.2 in /internal/tools (1672)
    a5edd79e Removed setting error status while recording err as span event (1663)
    e9814758 chore(zipkin): improves zipkin example to not to depend on timeouts. (1566)
    3dc91f2d Add ForceFlush method to TracerProvider (1608)
    bd0bba43 exporter: swap pusher for exporter (1656)
    56904859 Update the SimpleSpanProcessor (1612)
    a7f7abac  SpanStatus description set only when status code is set to Error (1662)
    05252f40 Jaeger Exporter: Fix minor mapping discrepancies (1626)
    238e7c61 Add non-empty string check for attribute keys (1659)
    e9b9aca8 Add tests for propagation of Sampler Tracestate changes (1655)
    875a2583 Add docs on when reviews should be cleared (1556)
    7153ef2d Add HTTP/JSON to the otlp exporter (1586)
    62e2a0f7 Unexport the simple and batch SpanProcessors (1638)
    992837f1 Add TracerProvider tests to oteltest harness (1607)
    bb4c297e Pre release v0.18.0 (1635)
    712c3dcc Fix makefile ci target and coverage test packages (1634)
    841d2a58 Rename local var new to not collide with builtin (1610)
    13938ab5 Update SpanProcessor docs (1611)
    e25503a0 Add compatibility tests to CI (1567)
    1519d959 Use reasonable interval in sdktrace.WithBatchTimeout (1621)
    7d4496e0 Pass metric labels when transforming to gaugeArray (1570)
    6d4a5e0d Bump google.golang.org/grpc from 1.35.0 to 1.36.0 in /exporters/otlp (1619)
    a93393a0 Bump google.golang.org/grpc in /example/prom-collector (1620)
    e499ca86 Fix validation for tracestate with vendor and add tests (1581)
    43886e52 Make timestamps sequential in lastvalue agg check (1579)
    37688ef6 revent end-users from implementing some interfaces (1575)
    85e696d2 Updating documentation with an working example for creating NewExporter (1513)
    562eb28b Unify the Added sections of the unreleased changes (1580)
    c4cf1aff Fix Windows build of Jaeger tests (1577)
    4a163bea Fix stdout TestStdoutTimestamp failure with sleep (1572)
    bd4701eb Stagger timestamps in exact aggregator tests (1569)
    b94cd4b2 add code attributes to semconv package (1558)
    78c06cef Update docs from gitter to slack for communication (1554)
    1307c911 Remove vendor exclude from license-check (1552)
    5d2636e5 Bump github.com/golangci/golangci-lint in /internal/tools (1565)
    d7aff473 Vendor Thrift dependency (1551)
    298c5a14 Update span limits to conform with OpenTelemetry specification (1535)
    ecf65d79 Rename otel/label -> otel/attribute (1541)
    1b5b6621 Remove resampling on span.SetName (1545)
    8da52996 fix: grpc reconnection  (1521)
    3bce9c97 Add Keys() method to propagation.TextMapCarrier (1544)
    0b1a1c72 Make oteltest.SpanRecorder into a concrete type (1542)
    7d0e3e52 SDK span no modification after ended (1543)
    7de3b58c Remove extra labels types (1314)
    73194e44 Bump google.golang.org/api from 0.39.0 to 0.40.0 in /exporters/trace/jaeger (1536)
    8fae0a64 Create resource.Default() with required attributes/default values (1507)
    76f93422 Release v0.17.0 (1534)
    9b242bc4 Organize API into Go modules based on stability and dependencies (1528)
    e50a1c8c Bump actions/cache from v2 to v2.1.4 (1518)
    a6aa7f00 Bump google.golang.org/api from 0.38.0 to 0.39.0 in /exporters/trace/jaeger (1517)
    38efc875 Code Improvement - Error strings should not be capitalized (1488)
    6b340501 Update default branch name (1505)
    b39fd052 nit: Fix comment to be up-to-date (1510)
    186c2953 Fix golint error of package comment form (1487)
    9308d662 Bump google.golang.org/api from 0.37.0 to 0.38.0 in /exporters/trace/jaeger (1506)
    1952d7b6 Reverse order of attribute precedence when merging two Resources (1501)
    ad7b4715 Remove build flags for runtime/trace support (1498)
    4bf4b690 Remove inaccurate and unnecessary import comment (1481)
    7e19eb6a Bump google.golang.org/api from 0.36.0 to 0.37.0 in /exporters/trace/jaeger (1504)
    c6a4406a Bump github.com/golangci/golangci-lint in /internal/tools (1503)
    9524ac09 Update workflows to include main branch as trigger (1497)
    c066f15e Bump github.com/gogo/protobuf from 1.3.1 to 1.3.2 in /internal/tools (1478)
    894e0240 Bump github.com/golangci/golangci-lint in /internal/tools (1477)
    71ffba39 Bump google.golang.org/grpc from 1.34.0 to 1.35.0 in /exporters/otlp (1471)
    515809a8 Bump github.com/itchyny/gojq from 0.12.0 to 0.12.1 in /internal/tools (1472)
    3e96ad1e gitignore: remove unused example path (1474)
    c5622777 Histogram aggregator functional options (1434)
    0df8cd62 Rename Makefile.proto to avoid interpretation as proto file (1468)
    979ff51f Bump github.com/stretchr/testify from 1.6.1 to 1.7.0 (1453)
    1df8b3b8 Bump github.com/gogo/protobuf from 1.3.1 to 1.3.2 in /exporters/otlp (1456)
    4c30a90a Bump github.com/stretchr/testify from 1.6.1 to 1.7.0 in /sdk (1455)
    5a9f8f6e Bump github.com/stretchr/testify from 1.6.1 to 1.7.0 in /exporters/stdout (1454)
    7786f34c Bump github.com/stretchr/testify from 1.6.1 to 1.7.0 in /exporters/trace/zipkin (1457)
    4352a7a6 Bump github.com/stretchr/testify from 1.6.1 to 1.7.0 in /exporters/otlp (1460)
    6990b3b3 Bump github.com/stretchr/testify from 1.6.1 to 1.7.0 in /exporters/metric/prometheus (1461)
    7af40d22 Bump github.com/stretchr/testify from 1.6.1 to 1.7.0 in /exporters/trace/jaeger (1463)
    f16f1892 Bump google.golang.org/grpc in /example/otel-collector (1465)
    fe363be3 Move Span Event to API (1452)
    43922240 Bump google.golang.org/grpc in /example/prom-collector (1466)
    0aadfb27 Prepare release v0.16.0 (1464)
    207587b6 Metric histogram aggregator: Swap in SynchronizedMove to avoid allocations (1435)
    c29c6fd1 Shutdown underlying span exporter while shutting down BatchSpanProcessor (1443)
    dfece3d2 Combine the Push and Pull metric controllers (1378)
    74deeddd Handle tracestate in TraceContext propagator  (1447)
    49f699d6 Remove Quantile aggregation, DDSketch aggregator; add Exact timestamps (1412)
    9c949411 Rename internal/testing to internal/internaltest (1449)
    8d809814 Move gRPC driver to a subpackage and add an HTTP driver (1420)
    9332af1b Bump github.com/golangci/golangci-lint in /internal/tools (1445)
    5ed96e92 Update exporters/otlp Readme.md (1441)
    bc9cb5e3 Switch CircleCI badge to GitHub Actions (1440)
    716ad082 Remove CircleCI config (1439)
    0682db1e Adding Security Workflows to GitHub Actions (2/2): gosec workflow (1429)
    11f732b8 Adding Security Workflows to GitHub Actions (1/2): codeql workflow (1428)
    40f1c003 Add Tracestate into the SamplingResult struct (1432)
    db06c8d1 Flush metric events before shutdown in collector example (1438)
    f6f458e1 Fix golint issue caused by typo in trace.go (1436)
    fe9d1f7e Use uint64 Count consistently in metric aggregation (1430)
    3a337d0b Bump github.com/golangci/golangci-lint in /internal/tools (1433)
    1e4c8321 cleanup: drop the removed examples in gitignore (1427)
    5c9221cf Unify endpoint API that related to OTel exporter (1401)
    045c3ffe Build scripts: Replace mapfile with read loop for old bash versions (1425)
    2def8c3d Add Versioning Documentation (1388)
    6bcd1085 Bump github.com/itchyny/gojq from 0.11.2 to 0.12.0 in /internal/tools (1424)
    38e76efe Add a split protocol driver for otlp exporter (1418)
    439cd313 Add TraceState to SpanContext in API (1340)
    35215264 Split connection management away from exporter (1369)
    add9d933 Bump github.com/prometheus/client_golang from 1.8.0 to 1.9.0 in /exporters/metric/prometheus (1414)
    93d426a1 Add @dashpole as a project Approver (1410)
    6fe20ef3 Fix small typo (1409)
    b22d0d70 Mention the getting started guide (1406)
    3fb80fb2 Fix duplicate checkout action in GitHub workflow (1407)
    2051927b Correct CI workflow syntax (1403)
    f11a86f7 Fix typo in comment (1402)
    bdf87a78 Migrate CircleCI ci.yml workflow to GitHub Actions (1382)
    4e59dd1f Bump google.golang.org/grpc from 1.32.0 to 1.34.0 in /example/otel-collector (1400)
    83513f70 Bump google.golang.org/api from 0.32.0 to 0.36.0 in /exporters/trace/jaeger (1398)
    a354fc41 Bump github.com/prometheus/client_golang from 1.7.1 to 1.8.0 in /exporters/metric/prometheus (1397)
    3528e42c Bump google.golang.org/grpc from 1.32.0 to 1.34.0 in /exporters/otlp (1396)
    af114baf Call otel.Handle with non-nil errors (1384)
    c3c4273e Add RO/RW span interfaces (1360)

Fixes #2591

Signed-off-by: Chelsea Mafrica <chelsea.e.mafrica@intel.com>
2021-11-04 12:30:45 -07:00
Chelsea Mafrica
b5cfb73466
Merge pull request #2931 from YchauWang/wyc-runtime-shim2
runtime# make sure the "Shutdown" trace span have a correct end
2021-11-04 11:33:22 -07:00
Chelsea Mafrica
09d5d8836b runtime: tracing: Change method for adding tags
In later versions of OpenTelemetry label.Any() is deprecated. Create
addTag() to handle type assertions of values. Change AddTag() to
variadic function that accepts multiple keys and values.

Fixes #2547

Signed-off-by: Chelsea Mafrica <chelsea.e.mafrica@intel.com>
2021-11-04 10:19:05 -07:00
wangyongchao.bj
9d3ec58370 runtime: make sure the "Shutdown" trace span have a correct end
We only added span.End() in the main process of the shim2 Shutdown method.
The "Shutdown" span would keep alive, when the containers number is not 0.
This PR make sure the "Shutdown" trace span have a correct end.

Fixes: #2930

Signed-off-by: wangyongchao.bj <wangyongchao.bj@inspur.com>
2021-11-02 14:24:31 +08:00
GabyCT
c8553ea427
Merge pull request #2046 from littlejawa/issue_2042
test: Fix random failure for TestIoCopy
2021-10-29 17:29:31 -05:00
Bin Liu
eb248b0c66
Merge pull request #2750 from liubin/fix/2749-remove-fixme
runtime: set tags for trace span
2021-10-29 11:42:49 +08:00
James O. D. Hunt
ec3aa1694b
Merge pull request #2844 from jongwu/unit_test
enable unit test on arm
2021-10-25 10:58:21 +01:00
David Gibson
57ab408576 runtime: Introduce "vfio_mode" config variable and annotation
In order to support DPDK workloads, we need to change the way VFIO devices
will be handled in Kata containers.  However, the current method, although
it is not remotely OCI compliant has real uses.  Therefore, introduce a new
runtime configuration field "vfio_mode" to control how VFIO devices will be
presented to the container.

We also add a new sandbox annotation -
io.katacontainers.config.runtime.vfio_mode - to override this on a
per-sandbox basis.

For now, the only allowed value is "guest-kernel" which refers to the
current behaviour where VFIO devices added to the container will be bound
to whatever driver in the VM kernel claims them.

Signed-off-by: David Gibson <david@gibson.dropbear.id.au>
2021-10-25 12:29:29 +11:00
Jianyong Wu
c59c36732b runtime: current vcpu number should be limited
The physical current vcpu number should not be used directly as the
largest vcpu number is limited to defaultMaxQemuVCPUs.
Here, a new helper is introduced in pkg/katautils/config.go to get
current vcpu number.

Fixes: #2809
Signed-off-by: Jianyong Wu <jianyong.wu@arm.com>
2021-10-23 15:07:25 +08:00
Jianyong Wu
fa922517d9 runtime: kernel version with '+' as suffix panic in parse
The current kernel version parse lib can't process suffix '+', as the
modified kernel version will add '+' as suffix, thus panic will occur.

For example, if the current kernel version is "5.14.0-rc4+", test
TestHostNetworkingRequested will panic:
--- FAIL: TestHostNetworkingRequested (0.00s)
panic: &{DistroName:ubuntu DistroVersion:18.04
KernelVersion:5.11.0-rc3+ Issue: Passed:[] Failed:[] Debug:true
ActualEUID:0}: failed to check test constraints: error: Build meta data
is empty

Here, remove the suffix '+' in kernel version fix helper.

Fixes: #2809
Signed-off-by: Jianyong Wu <jianyong.wu@arm.com>
2021-10-23 15:07:25 +08:00
James O. D. Hunt
09741272bc
Merge pull request #2783 from likebreath/1001/clh_enable_seccomp
virtcontainers: clh: Enable the `seccomp` feature
2021-10-21 09:21:33 +01:00
Bin Liu
72d1a04cf1
Merge pull request #2761 from liubin/fix/2752-optimize-test-code
runtime: optimize test code
2021-10-19 12:21:04 +08:00
bin
273a1a9ac6 runtime: optimize test code
This PR includes these optimize changes:

- Remove the dependency on the container engine.
  The old code uses runc to generate config.json and
  Docker to export rootfs, that will be heavy and need
  additional dependency.
  Using a fixed config for busybox image can avoid
  the heavy processing above.

- Moved duplicate code to pkg/katatestutils package

Fixes: #2752

Signed-off-by: bin <bin@hyper.sh>
2021-10-19 09:54:49 +08:00
bin
76f16fd1a7 runtime: use containerd package instead of cri-containerd
cri-containerd project has been merged into containerd repo, and
we should not reference it any more in code and docs.

This commit will use containerd package instead of cri-containerd
package.

Fixes: #2791

Signed-off-by: bin <bin@hyper.sh>
2021-10-19 09:40:20 +08:00
James O. D. Hunt
41c49a7bf5
Merge pull request #2771 from fengwang666/debug-pid
runtime: update sandbox root dir cleanup behavior in rootless hypervisor
2021-10-18 17:47:47 +01:00
Julien Ropé
17a8c5c685 runtime: Fix random failure for TestIoCopy
When running the TestIoCopy test, on some occasions, the test
runs too quick, and closes the stdin pipe before the ioCopy()
routine start to read from it. This causes a SIGSEGV error.

To fix this issue, I am adding additional read/write tests before
closing the pipes. As the read operation waits for the writer to
be done, this actually synchronizes the threads and make sure
the final tests (with closed pipes) works as expected.

Fixes: #2042

Signed-off-by: Julien Ropé <jrope@redhat.com>
2021-10-18 15:25:57 +02:00
Chelsea Mafrica
6ffe9e5afe
Merge pull request #2816 from cmaf/add-var-name-kata
runtime: change name in config settings back to "kata"
2021-10-15 14:09:41 -07:00
James O. D. Hunt
321be0f794 tracing: Remove trace mode and trace type
Remove the `trace_mode` and `trace_type` agent tracing options as
decided in the Architecture Committee meeting.

See:

- https://github.com/kata-containers/kata-containers/pull/2062

Fixes: #2352.

Signed-off-by: James O. D. Hunt <james.o.hunt@intel.com>
2021-10-15 10:09:38 +01:00
Chelsea Mafrica
3f95469a78 runtime: logging: Add variable for syslog tag
The variable for 'name' in config-settings.go.in was previously
hardcoded as "kata". In e7c42fb it was changed to the runtime name,
which is "kata-runtime". Add a variable to specify a syslog identifier
for consistency for tests and documentation that use it.

Fixes #2806

Signed-off-by: Chelsea Mafrica <chelsea.e.mafrica@intel.com>
2021-10-11 02:12:13 -07:00
Feng Wang
adc9e0baaf runtime: fix two bugs in rootless hypervisor
Update the sandbox dir clean up logic to be more appropriate
Add different seeds for randInt() method

Fixes #2770

Signed-off-by: Feng Wang <feng.wang@databricks.com>
2021-10-08 15:52:42 -07:00
Bo Chen
51cbe14584 runtime: Add option "disable_seccomp" to config hypervisor.clh
This patch adds an option "disable_seccomp" to the config
hypervisor.clh, from which users can disable the `seccomp`
feature from Cloud Hypervisor when needed (for debugging purposes).

Fixes: #2782

Signed-off-by: Bo Chen <chen.bo@intel.com>
2021-10-08 15:10:30 -07:00
bin
5c77cc2c49 runtime: don't start shim management server in tests
Shim management server is running in a go routine, in test mode
this will cause the directory where the listen socket
file(/run/vc/sbs/777-77-77777777/shim-monitor.sock) in leak
after the tests finished.

Fixes: #2805

Signed-off-by: bin <bin@hyper.sh>
2021-10-08 18:41:53 +08:00
Fupan Li
988eb95621
Merge pull request #2760 from liubin/fix/2759-optimize-code-for-managing-temp-users
runtime: optimize code for managing temp users for rootless mode
2021-10-08 13:49:14 +08:00
bin
bf8f582c1d runtime: optimize code for managing temp users for rootless mode
This commit does two chagnes:

- move code for managing temp users to rootless.go.
- use common function in qemu.go when shutdown the VM.

Fixes: #2759

Signed-off-by: bin <bin@hyper.sh>
2021-10-08 11:04:21 +08:00
Chelsea Mafrica
6e3fcce2a2
Merge pull request #2748 from liubin/fix/2747-add-test
runtime: Optimize func noNeedForOutput and add test cases
2021-10-06 11:24:57 -07:00
bin
46720c61c1 runtime: set tags for trace span
Set tags for trace span in hook.go and remove FIXME.

Fixes: #2749

Signed-off-by: bin <bin@hyper.sh>
2021-09-28 18:05:03 +08:00
bin
18bff58487 runtime: Optimize func noNeedForOutput and add test cases
Optimize func noNeedForOutput and add test cases for this func.

Fixes: #2747

Signed-off-by: bin <bin@hyper.sh>
2021-09-28 16:58:44 +08:00
Francesco Giudici
2304a59601 runtime: set the sandbox storage path static
Since we now have "unix://" kind of socket returned by the
SocketAddress() function, there is no more need to build the sandbox
storage path dynamically to keep OS compatibility.

Fixes: #2738
Suggested-by: Christophe de Dinechin <dinechin@redhat.com>
Signed-off-by: Francesco Giudici <fgiudici@redhat.com>
2021-09-27 15:57:34 +02:00
Francesco Giudici
315295e0ef runtime: rename GetSanboxesStoragePath() --> GetSandboxesStoragePath()
Add the missing 'd'.

Fixes: #2738
Suggested-by: Jakob Naucke <jakob.naucke@ibm.com>
Signed-off-by: Francesco Giudici <fgiudici@redhat.com>
2021-09-27 15:56:14 +02:00
Peng Tao
05995632c3
Merge pull request #2566 from fgiudici/kata-monitor_improvements
Kata monitor: cache improvements
2021-09-27 12:29:13 +08:00
Julio Montes
1766c93b08
Merge pull request #2662 from cmaf/tracing-stop-rootctx
runtime: tracing: Use root context to stop tracing
2021-09-23 11:50:35 -05:00
Francesco Giudici
bfb556d56a kata-monitor: refresh kata sandbox list on fs events
This commit stops the container engine polling in favor of
the kata sandbox storage path monitoring.
The pod cache list is now refreshed based on fs events and synced with
the container engine only when needed.

Signed-off-by: Francesco Giudici <fgiudici@redhat.com>
2021-09-23 14:32:09 +02:00
Francesco Giudici
0e854f3b80 kata-monitor: improve detection of kata workloads
When the container engine is different than containerd or CRI-O we
lack proper detection of kata workloads and consider all the pods as
kata ones.
Instead of querying the container engine for the lower level runtime
used in each pod, check if a directory matching the pod exists in
the virtualcontainers sandboxes storage path.
This provides a container engine independent way to check for kata pods.

Signed-off-by: Francesco Giudici <fgiudici@redhat.com>
2021-09-23 14:32:09 +02:00
Francesco Giudici
afad910d0e kata-monitor: add getSandboxFS()
Retrieve the absolute sandbox storage path. We will soon need this to
monitor the creation/deletion of new kata sandboxes.

Signed-off-by: Francesco Giudici <fgiudici@redhat.com>
2021-09-20 10:37:55 +02:00
Francesco Giudici
e38686f74d runtime: add GetSandboxesStoragePath()
The storage path we use to collect the sandbox files is defined in the
virtcontainers/persist/fs package.
We create the runtime socket in that storage path, by hardcoding the
full path in the SocketAddress() function in the runtime package.
This commit splits the hardcoded path by the socket address path so that
the runtime package will be able to provide the storage path to all the
components that may need it.

Signed-off-by: Francesco Giudici <fgiudici@redhat.com>
2021-09-20 10:37:55 +02:00
Francesco Giudici
245a12bbb7 kata-monitor: improve sandbox caching
In order to retrieve the list of sandboxes, we poll the container engine
every 15 seconds via the CRI. Once we have the list we have to inspect
each pod to find out the kata ones.
This commit extend the sandbox cache to keep track of all the pods,
marking the kata ones, so that during the next polling only the new
sandboxes should be inspected to figure out which ones are using the
kata runtime.

Fixes: #2563
Signed-off-by: Francesco Giudici <fgiudici@redhat.com>
2021-09-20 10:37:55 +02:00
Francesco Giudici
fc067d61d4 kata-monitor: warn when unable to retrive the lower level runtime
this is an unexpected event (likely a change in how containerd/cri-o
record the lower level runtime in the pod) and should be more visible:
raise the log level to "warning".

Signed-off-by: Francesco Giudici <fgiudici@redhat.com>
2021-09-20 10:37:54 +02:00
Francesco Giudici
53ec4df953 kata-monitor: minor fixes
fix comment and use literals

Signed-off-by: Francesco Giudici <fgiudici@redhat.com>
2021-09-20 10:37:54 +02:00
Chelsea Mafrica
39cd05e0bb runtime: tracing: Use root context to stop tracing
Call StopTracing with s.rootCtx, which is the root context for tracing,
instead of s.ctx, which is parent to a subset of trace spans.

Fixes #2661

Signed-off-by: Chelsea Mafrica <chelsea.e.mafrica@intel.com>
2021-09-17 11:39:13 -07:00
Feng Wang
1cfe59304d runtime: Run QEMU using a non-root user/group
A random generated user/group is used to start QEMU VMM process.
The /dev/kvm group owner is also added to the QEMU process to grant it access.

Fixes #2444

Signed-off-by: Feng Wang <feng.wang@databricks.com>
2021-09-17 11:28:44 -07:00
Peng Tao
e7c42fbc76 runtime: unify generated config
We don't need to maintain two generated config.go and even have
duplicates between them.

Signed-off-by: Peng Tao <bergwolf@hyper.sh>
2021-09-16 17:19:18 +08:00
Peng Tao
4f7cc18622 runtime: refactor commandline code directory
Move all command line code to `cmd` and move containerd-shim-v2 to pkg.

Fixes: #2627
Signed-off-by: Peng Tao <bergwolf@hyper.sh>
2021-09-16 17:19:18 +08:00
bin
2abc450a4d test: enable running tests under root user
Add tests that run under root user to test special cases.

Fixes: #2446

Signed-off-by: bin <bin@hyper.sh>
2021-09-09 14:21:34 +08:00
Peng Tao
c0daa4ebff
Merge pull request #2513 from cmaf/tracing-tracingtags-consistency
tracing: Change runtime tracing tags to vars
2021-08-31 10:25:10 +08:00
Peng Tao
a9de761d71 runtime: drop qemu-lite support
As the project is not maintained and we have not been testing against it
for a long time.

Fixes: #2529
Signed-off-by: Peng Tao <bergwolf@hyper.sh>
2021-08-30 16:58:12 +08:00
Peng Tao
8ae3edbc18 runtime: fix default hypervisor path
Should not be qemu-lite.

Signed-off-by: Peng Tao <bergwolf@hyper.sh>
2021-08-30 16:09:02 +08:00
Chelsea Mafrica
87de26bda3 tracing: Modify Trace() to accept multiple tag maps
The general Trace() function accepts one map as a set of tags. Modify it
to accept multiple sets of tags so that additional ones can be added at
Trace() and not as a subsequent call.

Additionally, we should not iterate over the maps unless tracing tracing
is enabled.

Fixes #2512

Signed-off-by: Chelsea Mafrica <chelsea.e.mafrica@intel.com>
2021-08-26 15:55:32 -07:00
wangyongchao.bj
2304f935b4 docs: update the GoDoc url from kata 1.x to 2.x
the katatestutils GoDoc url stilled using the kata 1.x branch url. This PR fixed the
url from kata-containers/runtime/pkg/katatestutils to
kata-containers/kata-containers/src/runtime/pkg/katatestutils

Fixes: #2500

Signed-off-by: wangyongchao.bj <wangyongchao.bj@inspur.com>
2021-08-25 11:21:36 +08:00
Fabiano Fidêncio
348795e282
Merge pull request #2233 from fgiudici/kata-monitor_liubin_cri
use CRI in kata-monitor
2021-08-20 13:58:12 +02:00
Chelsea Mafrica
9586d48254 tracing: Return context in runHooks() span creation
The call to Trace() in runHooks() should return a context so that
subsequent calls to runHook() produce properly ordered trace spans.

Fixes #2423

Signed-off-by: Chelsea Mafrica <chelsea.e.mafrica@intel.com>
2021-08-12 10:09:56 -07:00
Francesco Giudici
2d8386ea52 kata-monitor: add few unit tests
Add cri.go unit tests

Signed-off-by: Francesco Giudici <fgiudici@redhat.com>
2021-08-05 11:41:54 +02:00
Francesco Giudici
8714a35063 kata-monitor: make code to identify kata pods simpler
just search for the "kata" substring in the runtime value and log at
info level when the runtime name/type is not found.

Signed-off-by: Francesco Giudici <fgiudici@redhat.com>
2021-08-05 11:41:54 +02:00
Francesco Giudici
68a6f011b5 kata-monitor: drop the runtime info from the sandbox cache
We keep the container engine info in the sandbox cache map, as the value
associated to the pod id (the key). Since we used that in
getMonitorAddress() only (which is gone) we can avoid storing that
information. Let's drop it.
Keep the map structure and the [put,delete]IfExists functions as we may
want to move to an event based cache update process sooner or later, and
we will need those.

Signed-off-by: Francesco Giudici <fgiudici@redhat.com>
2021-08-05 11:41:54 +02:00
Francesco Giudici
97dcc5f78a kata-monitor: drop getMonitorAddress()
since the shim socket path is statically defined in the containerd-shimv2
code, we don't need to retrieve the socket name from the filesystem:
construct the socket name using the containerd-shimv2 code.

Signed-off-by: Francesco Giudici <fgiudici@redhat.com>
2021-08-05 11:41:54 +02:00
Francesco Giudici
c2f03e8993 kata-monitor: talk to the container engine via the CRI
kata-monitor uses containerd client to retrieve information from the
container engine. This makes kata-monitor work with the containerd
container engine only.
Bin Liu (bin <bin@hyper.sh>) worked on a kata-monitor version able
to talk to any container engine leveraging the standard CRI[1].
Here, the original work of Bin Lui has been adapted on the current
kata-monitor to make it container engine independent.

[1] https://github.com/liubin/kata-containers/tree/fix/1030-use-cri-in-kata-monitor

Fixes: #1030
Signed-off-by: Francesco Giudici <fgiudici@redhat.com>
2021-08-05 11:41:54 +02:00
James O. D. Hunt
4f0726bc49 docs: Remove table of contents
Removed all TOCs now that GitHub auto-generates them.

Also updated the documentation requirements doc removing the requirement
to add a TOC.

Fixes: #2022.

Signed-off-by: James O. D. Hunt <james.o.hunt@intel.com>
2021-07-30 10:58:22 +01:00
Julio Montes
31de8eb75b runtime: pkg: fix govet fieldalignment
Fix structures alignment

Signed-off-by: Julio Montes <julio.montes@intel.com>
2021-07-20 10:30:30 -05:00
Hui Zhu
cb6b7667cd runtime: Add option "enable_guest_swap" to config hypervisor.qemu
This commit add option "enable_guest_swap" to config hypervisor.qemu.
It will enable swap in the guest. Default false.
When enable_guest_swap is enabled, insert a raw file to the guest as the
swap device if the swappiness of a container (set by annotation
"io.katacontainers.container.resource.swappiness") is bigger than 0.
The size of the swap device should be
swap_in_bytes (set by annotation
"io.katacontainers.container.resource.swap_in_bytes") - memory_limit_in_bytes.
If swap_in_bytes is not set, the size should be memory_limit_in_bytes.
If swap_in_bytes and memory_limit_in_bytes is not set, the size should be
default_memory.

Fixes: #2201

Signed-off-by: Hui Zhu <teawater@antfin.com>
2021-07-19 23:22:06 +08:00
fupan.lfp
8e0daf6780 shimv2: fix the issue of kata-runtime exec failed
Commit 32c9ae1388 upgrade the
containerd vendor, which used the socket path to replace
the abstract socket address for socket listen and dial, and
there's an bug in containerd's abstract socket dialing.

Thus we should replace our monitor and exec socket server
with the socket path to fix this issue.

Fixes: #2238

Signed-off-by: fupan.lfp <fupan.lfp@antgroup.com>
2021-07-16 11:41:09 +08:00
Benjamin Porter
b10e3e22b5
tracing: Consolidate tracing into a new katatrace package
Removes custom trace functions defined across the repo and creates
a single trace function in a new katatrace package. Also moves
span tag management into this package and provides a function to
dynamically add a tag at runtime, such as a container id, etc.

Fixes #1162

Signed-off-by: Benjamin Porter <bporter816@gmail.com>
2021-07-11 14:19:51 -05:00
fupan.lfp
32c9ae1388 shimv2: update containerd vendor
Since the latest containerd's shimv2 had changed the socket
from abstract unix socket to path unix socket, thus we'd
better to update the vendor to match with the latest containerd.

containerd from v1.3.9, v1.4.3 and v1.5.0 used the path unix socket
instead of abstract socket, thus kata wouldn's support the
containerd's version older than them.

Fixes: #1441

short logs:

15d9703d6 Remove ARM64 releases from release notes
    5d2e8e86d Revert "Release artifacts for Linux ARM64"
    7942ae68b Revert "Specify seccomp target arch for CC"
    3187b6dc8 tests: Adds consumed memory stats test
    969ec8949 Specify seccomp target arch for CC
    c19b7b64d RELEASES.md: recommend alternatives for deprecated
features
    8a62aa1c3 Deprecate built-in aufs snapshotter
    4e7915f80 CI: allow Go 1.13 for Docker/Moby compatibility
    8e589e873 Vagrantfile: update to Fedora 34
    5847340a7 tests: Refactors container image usage
    9f43eade6 Prepare v1.5.0-rc.3 release notes
    4c7b960cb prow needs some additional setup for docker buildx
    2e4c1d4b7 Use the multi-arch version of the test images
    4e00c4b65 integration tests needs lsof
    177273680 Add script to build test images
    1b5d59dfe Add multi-arch support for test images
    78e529727 add integration tests
    2b0e6cdd4 Separate jobs for build and test for openlab/arm64
    cdd075853 Release artifacts for Linux ARM64
    efcb18742 Add unit tests for PID NamespaceMode_TARGET validation
    b48f27df6 Support PID NamespaceMode_TARGET
    909660ea9 process: use the unbuffered channel as the done signal
    0f332dadd Update cgroups for regenerated protos
    391b123a5 adds quiet option for ref
    ab1654d0e Fix PushHandler cannot push image that contains duplicated
blobs
    00f8d32ef add not found debug out for check cmd; update usage
    55734b1c5 Prepare 1.5.0-rc.2 release notes
    3ef337ae3 Update containerd vendors to tags
    fbe1e140f Update Go to 1.16.3
    c1d1edbad gha: use sudo -E in some places to prevent dropping
env-vars
    7966a6652 Cleanup code
    5d79d3adb go.mod: update kubernetes to v1.20.6
    1c03c377e go.mod: github.com/containerd/fifo v1.0.0
    12a2a2108 go.mod: github.com/google/uuid v1.2.0
    3292ea586 pkg/seccomp: use sync.Once to speed up IsEnabled
    00b5c99b1 pkg/seccomp: simplify IsEnabled, update doc
    6dd29c25f go.mod: github.com/containerd/aufs
    330a2a809 go.mod: github.com/containerd/zfs
    34780d67a runtime/shim: check the namespace flag first
    c3dde8c4b freebsd: add zfs to the default plugins
    b431fe4fc freebsd: don't run shim delete in deleted dir
    1f4192daf freebsd: exclude v1 runtimes
    cb1580937 metadata: improve deleting a non-empty namespace's error
message
    5bf84034d Remove junit test result processor
    b83d04f91 Add variable names to runtime's interface definitions
    993b86399 Add shim start opts
    9e576b889 Optimize backoff
    5c02688b5 converter: use OpenWriter helper function
    fcf3b275f Add lock for ListPids
    fdb76f55d Fix backword-compatibility issue of non-versioned config
file
    d21fe4625 adds log for each failed host and status not found on host
    8a4cbabc6 Reimport windows layers when comitting snapshots
    2de38a926 fix(windows): create debug npipe failure
    41fc516a2 docs/rootless.md: recommend "easy way" over "hard way"
    864a3322b go.mod: github.com/containerd/go-cni v1.0.2
    ee34caccb go.mod: github.com/Microsoft/go-winio v0.4.17
    d478676d3 go.mod: github.com/containerd/imgcrypt v1.1.1
    1dd45d51c go.mod: github.com/containerd/typeurl v1.0.2
    abd4be07a fix the 404 url
    978ebbef6 Prepare 1.5.0-rc.1 release
    ce116d4c5 go.mod: github.com/containerd/imgcrypt
v1.1.1-0.20210412181126-0bed51b9522c
    0550c3233 containerd-stress: add snapshotter option for stress test
to use
    8a04bd052 address recent runtimes config confusion
    c4778fe1b go.mod: github.com/containernetworking/plugins v0.9.1
    5ce35ac39 devmapper: log pool status when mkfs fails
    75097b8ca hcsshim seems to have been updated
    9ad087947 Switch all our tests to version 2
    e96d2a5d9 Revert "remove two very old no longer used runtime
options"
    14f357b90 CI: update crun to 0.19
    294331060 go.mod: github.com/containerd/console v1.0.2
    bb6c0c2de Add more bolt utils
    0ad8c0a16 Decouple shim start from task creation
    c7504987e Implement windowsDiff.Compare via hcsshim/pkg/ociwclayer
    a64a76846 Replace inline applyWindowsLayer using hcsshim
    149fa366f Don't tease the logger with a %-less format string
    b399e2ef6 Don't lose Compare failure if aborting diff upload fails
    36bf3f0e8 go.mod: github.com/Microsoft/hcsshim v0.8.16
    8e1a8ecd8 Prepare v1.5.0-rc.0
    45df696bf Fix return event publishing error
    4bc8f692f optimize cri redirect logs
    9bc8d63c9 cri/server: use containerd/oci instead of
libcontainer/devices
    dd16b006e merge in the move to the new options type
    9144ce967 shows our runc.v2 default options in the containerd
default config
    3d20fa930 fix TestSetOOMScoreBoundaries
    4d4117415 Change CRI config runtime options type
    21ebeef74 integration: use busybox:1.32.0 since latest is
unavailable
    f9bcf4a8a add section link
    d4be6aa8f rm mirror defaults; doc registry deprecations
    7bb73da6b runtime/v2/shim: remove unused SetScore() and remove
sys.OOMScoreMaxKillable
    91e7d21ee sys: add AdjustOOMScore() utility
    44240116a sys: add boundary checks to SetOOMScore()
    ace1912bb sys: use assert for error checks in OOM tests
    6e7271522 sys: add missing pre-condition checks in tests
    badd60d3f sys: un-export runningPrivileged(), remove
runningUnprivileged()
    21a175860 go.mod github.com/klauspost/compress v1.11.13
    58c5fd09e re-enable cri test
    da998c81e move to gcr.io/k8s-staging-cri-tools test images
    8ba8533bd pkg/cri/opts.WithoutRunMount -> oci.WithoutRunMount
    92ea98eda cri-cni-release: add imgcrypt binaries (v1.1.0)
    4c1fa5719 remotes/docker: Only return "already exists" on push when
the upload was successful
    0186a329e remove two very old no longer used runtime options
    58a07754a Temporarily disable cri-tools critest
    7ae0a60fb Add OCI ref.name to unique key in remotes handler
    5ada2f74a Keep host order as defined in TOML file
    d9ff8ebef support multi-arch images for windows via ctr
    af1e2af72 ci: upload junit formatted test results
    6866b36ab Add workaround to keep docker hosts structs private
    c54d92c79 image: use generic decompressor for calculating DiffID
    1faca349e integration/client: rename package to "client"
    6fc9e4500 synchronize replace rules in integration/client go.mod
with main go.mod
    9e19a2984 Fix hosts test on Windows
    3f406d4af Cleanup vendor
    d56b49c13 Rewrite Docker hosts parser
    e1f51ba73 Use os.File#Seek() to get the size of a block device
    ddd4298a1 Migrate current TOML code to github.com/pelletier/go-toml
    499c2f7d4 Vendor github.com/pelletier/go-toml
    61c749036 integration/util: remove dependency on k8s.io/klog/v2
    d9765f7bf Extend default timeout for nested VM integration run
    5e94745f2 ctr: add --user for task exec
    f8c2f0475 remotes/ctr: allow to limit max concurrent uploads like
downloads
    4674ad7be Ignore some tests on darwin
    55450e773 Run unit tests on CI for MacOS
    311e326a1 Add CI job to cross compile all the things
    10a498c7c Update go-winio to fix compile error on armv7
    1a9c6f557 Revendor zfs to to fix integer overflow
    1fd3d12f9 `go mod tidy` the client integration test module
    da7d96ba3 Clean up WCOW layers after tests in the correct order
    9ad87b9ba adds critools-version
    72b7f4bab task: allow checkpoint on pause state
    e4b9b1038 Make CRI registry docs more clear
    ec4d7736d Increase timeout for linux integration tests
    eb7c7c71e Fix oom tests on non Linux
    708299ca4 Move RunningInUserNS() to its own package
    0886ceaea Fix reference ordering in CRI image store
    bf9db47e8 add caller info to the testHook
    305b42583 use happy-eyeballs for port-forwarding
    22ef69d77 Support HTTP debug in ctr
    01765d097 night ci fix: add packages for ubuntu 20.04
    8cdc1f13b go.mod: github.com/containerd/zfs
v0.0.0-20210322090317-0e92c2247fb7
    30e1e66e5 runtime/v2: Fix defer cleanup
    33776ada0 Use specific image for user namespaces tests
    7704fe72d Specifically mention "mkfs.ext4" on the error from the
command
    1410220d8 Fix error log when copy file
    fe787efa2 Fix error log when kill shim
    8d8c15ca5 contentproxy: ensure grpc stream is closed on commit
    6e343f25e Switch test image to a non rate-limited manifest list
    9fdc96c09 runtime/v2: add comment for checkCopyShimLogError
    24602e7a9 change default runtime for containerd-stress app
    8731888ec Re-enable CRIU tests by not using overlayfs snapshotter
    b520428b5 Fix CRIU
    4e76bcf06 gofmt -s -w all the things
    569023fd5 go.mod: github.com/containerd/nri
v0.0.0-20210316161719-dbaa18c31c14
    0e1f59e89 go.mod: github.com/containerd/zfs
v0.0.0-20210315114300-dde8f0fda960
    ffff68866 upgrade pause image to 3.5 for non-root
    88d3881e1 go.mod: github.com/containerd/fifo
v0.0.0-20210316144830-115abcc95a1d
    a22c43fa4 go.mod: github.com/containerd/aufs
v0.0.0-20210316121734-20793ff83c97
    f6f861736 go.mod: github.com/containerd/btrfs
v0.0.0-20210316141732-918d888fb676
    460b35236 go.mod: kubernetes v1.20.4
    5e484c961 runtime/v2/runc: fix the defer cleanup of the NewContainer
    e6086d9c0 Prepare release notes for v1.5.0-beta.4
    34b7a5f09 Update mailmap
    ba8f9845e move overlay-checks to an overlayutils package
    7776e5ef2 Support adding devices by dir
    d895118c7 runtime/v2/runc: fix leaking socket path
    a76cefd12 plugin status should be skip, not error
    766e7953a Change dgst to digest in debug
    4e8b2f309 rootfs: fix the error handling of the createInitLayer
    d3ad7f390 cmd/ctr: use e.g. in the command usage
    231bbdc37 cmd/ctr: fix export command
    ecb881e5e add imgcrypt stream processors to the default config
    ac2726e12 cmd/containerd: deduplicate config*.go
    9a7ca39cb defaults: add DefaultConfigDir
    8f863afd3 Use net.IP.IsLoopback() to match loopback addresses
    eabd9b98b runtime: ignore file-already-closed error if dead shim

Signed-off-by: fupan.lfp <fupan.lfp@antgroup.com>
2021-06-30 22:53:24 +08:00
Chelsea Mafrica
ac0bd57748
Merge pull request #2003 from cmaf/fix-span-runHooks
tracing: Make runHooks() span creation return context
2021-06-24 07:50:42 -07:00
Marcel Apfelbaum
789a59549e virtcontainers: Remove the pc machine
Keeping around two different x86 machines has no added value
and require more tests and maintenance. Prefer the q35 machine
since it has more features and drop the pc machine.

Fixes #1953
Depends-on: github.com/kata-containers/tests#3586
Signed-off-by: Marcel Apfelbaum <marcel@redhat.com>
2021-06-22 11:54:07 +03:00
Julio Montes
7834f4127f virtcontainers: change memory_offset to uint64
`memory_offset` is used to increase the maximum amount of memory
supported in a VM, this offset is equal to the NVDIMM/PMEM device that
is hot added, in real use case workloads such devices are bigger than
4G, which is the current limit (uint32).

fixes #2006

Signed-off-by: Julio Montes <julio.montes@intel.com>
2021-06-16 07:16:49 -05:00
Chelsea Mafrica
cabddcc735 tracing: Make runHooks() span creation return context
The call to Trace() in runHooks() does not return a context; fix this so
that the subsequent calls to runHook() produces a properly ordered trace
span.

Fixes #2001

Signed-off-by: Chelsea Mafrica <chelsea.e.mafrica@intel.com>
2021-06-10 23:50:51 -07:00
Chelsea Mafrica
8ca0207281 tracing: Add sandbox and container ID to trace spans
Add sandbox, container, and hypervisor IDs to trace spans. Note that
some spans in sandbox.go are created with a trace() call from api.go.
These spans have additional attributes set after span creation to
overwrite the api attributes.

Fixes #1878

Signed-off-by: Chelsea Mafrica <chelsea.e.mafrica@intel.com>
2021-06-02 21:53:54 -07:00
Chelsea Mafrica
05a46fede0 tracing: Make runtime span attributes more consistent
Span attributes (tags) are not consistent in runtime tracing, so
designate and use core attributes such source, package, subsystem, and
type as span metadata for more understandable output.

Use WithAttributes() during span creation to reduce calls to
SetAttributes().

Modify Trace() in katautils to accept slice of attributes so multiple
functions using different attributes can use it.

Fixes #1852

Signed-off-by: Chelsea Mafrica <chelsea.e.mafrica@intel.com>
2021-05-27 10:07:11 -07:00
Fabiano Fidêncio
ac61e60492
Merge pull request #1790 from snir911/configure_timeout
runtime: make dialing timeout configurable
2021-05-11 16:52:05 +02:00
Bin Liu
bffb099d99
Merge pull request #1816 from egernst/get-sandbox-metrics-cli
Get sandbox metrics cli
2021-05-11 13:10:30 +08:00
Samuel Ortiz
2c4e4ca1ac
Merge pull request #1590 from devimc/2021-02-02/ConfidentialComputing
Support TDx
2021-05-10 22:19:40 +02:00
Eric Ernst
3787306107 kata-monitor: export get stats for sandbox
Gathering stats for a given sandbox is pretty useful; let's export a
function from katamonitor pkg to do this.

Signed-off-by: Eric Ernst <eric_ernst@apple.com>
2021-05-10 08:53:56 -07:00
Snir Sheriber
01b56d6cbf runtime: make dialing timeout configurable
allow to set dialing timeout in configuration.toml
default is 30s

Fixes: #1789
Signed-off-by: Snir Sheriber <ssheribe@redhat.com>
2021-05-10 16:39:37 +03:00
Eric Ernst
3caed6f88d runtime: shim: dedup client, socket addr code
(1) Add an accessor function, SocketAddress, to the shim-v2 code for
determining the shim's abstract domain socket address, given the sandbox
ID.

(2) In kata monitor, create a function, BuildShimClient, for obtaining the appropriate
http.Client for communicating with the shim's monitoring endpoint.

(3) Update the kata CLI and kata-monitor code to make use of these.

(4) Migrate some kata monitor methods to be functions, in order to ease
future reuse.

(5) drop unused namespace from functions where it is no longer needed.

Signed-off-by: Eric Ernst <eric_ernst@apple.com>
2021-05-07 15:20:37 -07:00
Julio Montes
0affe8860d virtcontainers: define confidential guest framework
Define the structure and functions needed to support confidential
guests, this commit doesn't add support for any specific technology,
support for TDX, SEV, PEF and others will be added in following
commits.

Signed-off-by: Julio Montes <julio.montes@intel.com>
2021-05-06 10:09:05 -05:00
Chelsea Mafrica
8587e3a00b
Merge pull request #1732 from liubin/fix/1731-delete-builtin-parameter
runtime: delete not used function parameter builtIn
2021-04-23 18:30:55 -07:00
bin
677f0d9904 runtime: delete not used function parameter builtIn
Parametr builtIn is not used in function updateRuntimeConfigAgent,
delete it from updateRuntimeConfigAgent and LoadConfiguration
function signature.

Fixes: #1731

Signed-off-by: bin <bin@hyper.sh>
2021-04-23 17:42:42 +08:00
Christophe de Dinechin
dcb9f40394 config: Protect annotation for entropy_source
It would be undesirable to be given an annotation like "/dev/null".
Filter out bad annotation values.

Fixes: #1043

Suggested-by: James O. D. Hunt <james.o.hunt@intel.com>
Signed-off-by: Christophe de Dinechin <dinechin@redhat.com>
2021-04-22 15:26:40 +02:00
Bo Chen
1511d966aa
Merge pull request #1616 from egernst/dechat-deruntime
Dechat deruntime
2021-04-01 11:02:27 -07:00
Eric Ernst
50fff97753 trace: move trace span chatter to trace rather than info
No human should ever read that ouptut. Let's at least move it to trace for now.

Fixes: #1615

Signed-off-by: Eric Ernst <eric.g.ernst@gmail.com>
2021-04-01 09:02:56 -07:00
bin
48e5e4f2f3 test: install mock hook binary before test
`make test` depends mock hook in virtcontainers directory,
before test, install it first.

And also run test as normal user and root in GitHub actions.

Fixes: #1554

Signed-off-by: bin <bin@hyper.sh>
2021-03-29 22:40:45 +08:00
Chelsea Mafrica
6fcfea8dcf runtime: Fix static check errors
Fix comment formatting and unused variable to make static checks pass.

Fixes #1550

Signed-off-by: Chelsea Mafrica <chelsea.e.mafrica@intel.com>
2021-03-25 12:59:01 -07:00
Bin Liu
5b5b5cc611
Merge pull request #1539 from bergwolf/ut
fix runtime UTs and enable static check
2021-03-25 16:29:45 +08:00
James O. D. Hunt
2fc7f75724
Merge pull request #1521 from jodh-intel/verify-cid
Verify container ID
2021-03-24 13:27:58 +00:00
Peng Tao
74192d179d runtime: fix static check errors
It turns out we have managed to break the static checker in many
difference places with the absence of static checker in github action.
Let's fix them while enabling static checker in github actions...

Signed-off-by: Peng Tao <bergwolf@hyper.sh>
2021-03-24 20:10:19 +08:00
Peng Tao
190f813427 runtime/katautils: PFlash should be initialized
newQemuHypervisorConfig() sets it to an empty slice. We have to set the
same in the test config otherwise it is nil and reflect DeepEqual would
fail.

Signed-off-by: Peng Tao <bergwolf@hyper.sh>
2021-03-23 17:49:47 +08:00
James O. D. Hunt
b265870997 runtime: Validate CID
Validate the container ID as we cannot rely on the container manager
doing this.

Fixes: #1520.

Signed-off-by: James O. D. Hunt <james.o.hunt@intel.com>
2021-03-18 11:19:32 +00:00
Chelsea Mafrica
6b0dc60dda runtime: Fix ordering of trace spans
A significant number of trace calls did not use a parent context that
would create proper span ordering in trace output. Add local context to
functions for use in trace calls to facilitate proper span ordering.
Additionally, change whether trace function returns context in some
functions in virtcontainers and use existing context rather than
background context in bindMount() so that span exists as a child of a
parent span.

Fixes #1355

Signed-off-by: Chelsea Mafrica <chelsea.e.mafrica@intel.com>
2021-03-16 17:39:28 -07:00
Eric Ernst
48ed8f3c4a runtime: add support for readonly sandbox bindmounts
If specified, sandbox_bind_mounts identifies host paths to be
mounted (ro) into the sandboxes shared path. This is only valid
if filesystem sharing is utilized.

The provided path(s) will be bindmounted (ro) into the shared fs directory on
the host, and thus mapped into the guest. If defaults are utilized,
these mounts should be available in the guest at
`/var/run/kata-containers/shared/containers/sandbox-mounts`

These will not be exposed to the container workloads, and are only
added for potential guest-services to consume (example: expose certs
into the guest that are available on the host).

Fixes: #1464

Signed-off-by: Eric Ernst <eric.g.ernst@gmail.com>
2021-03-04 10:04:25 -08:00
Eric Ernst
4f67bf919d
Merge pull request #1344 from liubin/fix/1329-improvements-for-console-access
runtime: connect guest debug console bypass kata-monitor
2021-02-18 17:59:32 -08:00
Eric Ernst
2e2749ad3f runtime: clh-config: add runtime hooks to the clh toml
Today hooks are only described in the QEMU toml. This shouldn't be VMM
specific -- let's make sure these are advertised for Cloud Hypervisor as
well.

Fixes: #1401

Signed-off-by: Eric Ernst <eric.g.ernst@gmail.com>
2021-02-14 20:26:02 -08:00
bin
44cde6e464 runtime: connect guest debug console bypass kata-monitor
Parse agent socket address by conversation to improve usability of
using guest debug console.

Fixes: #1329

Signed-off-by: bin <bin@hyper.sh>
2021-02-09 19:36:48 +08:00
Fupan Li
5d1432210c
Merge pull request #1352 from liubin/fix/migrate-opentracing-to-opentelemetry
runtime: migrate from opentracing to opentelemetry
2021-02-09 10:18:10 +08:00
bin
17df9b119d runtime: migrate from opentracing to opentelemetry
This commit includes two changes:
- migrate from opentracing to opentelemetry
- add jaeger configuration items

Fixes: #1351

Signed-off-by: bin <bin@hyper.sh>
2021-02-03 17:30:49 +08:00
Jianyong Wu
b7a1f752c0 arm64: enable acpi for qemu/virt.
acpi is enabled for kata 1.x, port and rebase code for 2.x
including:
runtime: enable pflash;
agent: add acpi support for pci bus path;
packaging: enable CONFIG_RTC_DRV_EFI;

Fixes: #1317
Signed-off-by: Jianyong Wu <jianyong.wu@arm.com>
2021-01-29 22:12:43 +08:00
bin liu
40418f6d88 runtime: add geust memory dump
When guest panic, dump guest kernel memory to host filesystem.
And also includes:
- hypervisor config
- hypervisor version
- and state of sandbox

Fixes: #1012

Signed-off-by: bin liu <bin@hyper.sh>
2020-11-05 16:04:21 +08:00
bin liu
5b065eb599 runtime: change govmm package
Change govmm package name from github.com/intel/govmm
to github.com/kata-containers/govmm

Fixes: #859

Signed-off-by: bin liu <bin@hyper.sh>
2020-10-22 21:27:49 +08:00
Peng Tao
0d5d69e8cd
Merge pull request #902 from c3d/bug-v2/launchpad-1878234-access
Validate runtime annotations
2020-10-16 15:47:45 +08:00
bin liu
1b7ed32836 kata-monitor: use regexp to check if runtime is kata containers
To support a few common configurations for Kata, including:

- `io.containerd.kata.v2`
- `io.containerd.kata-qemu.v2`
- `io.containerd.kata-clh.v2`

`kata-monintor` changes to use regexp instead of direct string comparison.

Fixes: #957

Signed-off-by: bin liu <bin@hyper.sh>
2020-10-15 18:42:44 +08:00
Christophe de Dinechin
be6ee2550d makefile: Improve names of config entries for annotation checks
The entries used to be things like PATH_LIST, which are too generic.
Replace them with more precise name with a distinguishing keyword,
namely VALID. For example valid_hypervisor_paths.

Fixes: #901

Suggested-by: James O.D. Hunt <james.o.hunt@intel.com>
Signed-off-by: Christophe de Dinechin <dinechin@redhat.com>
2020-10-14 16:10:12 +02:00
Christophe de Dinechin
7c6aede5d4 config: Whitelist hypervisor annotations by name
Add a field "enable_annotations" to the runtime configuration that can
be used to whitelist annotations using a list of regular expressions,
which are used to match any part of the base annotation name, i.e. the
part after "io.katacontainers.config.hypervisor."

For example, the following configuraiton will match "virtio_fs_daemon",
"initrd" and "jailer_path", but not "path" nor "firmware":

  enable_annotations = [ "virtio.*", "initrd", "_path" ]

The default is an empty list of enabled annotations, which disables
annotations entirely.

If an anontation is rejected, the message is something like:

  annotation io.katacontainers.config.hypervisor.virtio_fs_daemon is not enabled

Fixes: #901

Suggested-by: Peng Tao <tao.peng@linux.alibaba.com>
Signed-off-by: Christophe de Dinechin <dinechin@redhat.com>
2020-10-14 16:10:12 +02:00
Christophe de Dinechin
4e89b885d2 config: Protect file_mem_backend against annotation attacks
This one could theoretically be used to overwrite data on the host.
It seems somewhat less risky than the earlier ones for a number
of reasons, but worth protecting a little anyway.

Fixes: #901

Signed-off-by: Christophe de Dinechin <dinechin@redhat.com>
2020-10-14 16:10:12 +02:00
Christophe de Dinechin
aae9656d8b config: Protect vhost_user_store_path against annotation attacks
This path could be used to overwrite data on the host.

Fixes: #901

Signed-off-by: Christophe de Dinechin <dinechin@redhat.com>
2020-10-14 16:10:12 +02:00
Christophe de Dinechin
b21a829c61 config: Protect ctlpath from annotation attack
This also adds annotation for ctlpath which were not present
before. It's better to implement the code consistenly right now to make
sure that we don't end up with a leaky implementation tacked on later.

Fixes: #901

Signed-off-by: Christophe de Dinechin <dinechin@redhat.com>
2020-10-14 16:10:12 +02:00
Christophe de Dinechin
27b6620b23 config: Protect jailer_path annotation
The jailer_path annotation can be used to execute arbitrary code on
the host. Add a jailer_path_list configuration entry providing a list
of regular expressions that can be used to filter annotations that
represent valid file names.

Fixes: #901

Signed-off-by: Christophe de Dinechin <dinechin@redhat.com>
2020-10-14 16:10:12 +02:00
Christophe de Dinechin
bf13ff0a3a config: Protect virtio_fs_daemon annotation
Sending the virtio_fs_daemon annotation can be used to execute
arbitrary code on the host. In order to prevent this, restrict the
values of the annotation to a list provided by the configuration
file.

Fixes: #901

Signed-off-by: Christophe de Dinechin <dinechin@redhat.com>
2020-10-14 16:10:12 +02:00
Christophe de Dinechin
8c75de1966 config: Add 'List' alternates for hypervisor configuration paths
Paths mentioned in the hypervisor configuration can be overriden
using annotations, which is potentially dangerous. For each path,
add a 'List' variant that specifies the list of acceptable values
from annotations.

Bug: https://bugs.launchpad.net/katacontainers.io/+bug/1878234

Fixes: #901

Signed-off-by: Christophe de Dinechin <dinechin@redhat.com>
2020-10-14 16:10:12 +02:00
Julio Montes
6df165c19d runtime: add support for SGX
Support the `sgx.intel.com/epc` annotation that is defined by the intel
k8s plugin. This annotation enables SGX. Hardware-based isolation and
memory encryption.

For example, use `sgx.intel.com/epc = "64Mi"` to create a container
with 1 EPC section with pre-allocated memory.

At the time of writing this patch, SGX patches have not landed on the
linux kernel project.
The following github kernel fork contains all the SGX patches for the
host and guest: https://github.com/intel/kvm-sgx

fixes #483

Signed-off-by: Julio Montes <julio.montes@intel.com>
2020-10-01 08:24:29 -05:00
bin liu
484a595f1a runtime: add enable_debug_console configuration item for agent
Set enable_debug_console=true in Kata's congiguration file,
runtime will pass `agent.debug_console`
and `agent.debug_console_vport=1026` to agent.

Fixes: #245

Signed-off-by: bin liu <bin@hyper.sh>
2020-09-27 15:04:59 +08:00
bin liu
febdf8f68c runtime: add debug console service
Add `kata-runtime exec` to enter guest OS
through shell started by agent

Fixes: #245

Signed-off-by: bin liu <bin@hyper.sh>
2020-09-27 10:57:17 +08:00
Qi Feng Huo
f5598a1bc2 Subject: [PATCH] qemu: add annotations for iommu_platform
for s390x virtio devices

Add iommu_platform annotations for qemu for ccw,
other supported devices can also make use of that.

  Fixes #603

Signed-off-by: Qi Feng Huo <huoqif@cn.ibm.com>
2020-08-28 11:25:14 +08:00
Penny Zheng
1099a28830 kata 2.0: delete use_vsock option and proxy abstraction
With kata containers moving to 2.0, (hybrid-)vsock will be the only
way to directly communicate between host and agent.
And kata-proxy as additional component to handle the multiplexing on
serial port is also no longer needed.
Cleaning up related unit tests, and also add another mock socket type
`MockHybridVSock` to deal with ttrpc-based hybrid-vsock mock server.

Fixes: #389

Signed-off-by: Penny Zheng penny.zheng@arm.com
2020-07-16 04:20:02 +00:00
Peng Tao
2bff7a16f5
Merge pull request #363 from liubin/feature/delete-sub-commands-332
runtime: delete unused sub-commands.
2020-07-13 11:06:27 +08:00
bin liu
bbf8517050 runtime: add pprof interface for shim
Add new http interfaces to support pprof:

- /sandboxes
- /debug/vars
- /debug/pprof/
- /debug/pprof/cmdline
- /debug/pprof/profile
- /debug/pprof/symbol
- /debug/pprof/trace

Fixes: #397

Signed-off-by: bin liu <bin@hyper.sh>
2020-07-10 13:05:25 +08:00
bin liu
3cf8b470cd runtime: delete Stateful from SandboxConfig
Since all containers are started from shim v2, `Stateful` is not needed.

Fixes: #332

Signed-off-by: bin liu <bin@hyper.sh>
2020-07-08 21:59:44 +08:00
bin liu
069505e2d5 runtime: delete unused sub-commands.
This PR delete codes not used anymore.

Fixes: #332

Signed-off-by: bin liu <bin@hyper.sh>
2020-07-08 21:59:03 +08:00
Hui Zhu
e3a3818f7a
Merge pull request #388 from lifupan/2.0-dev
config: there is no need to check vhost-vosck for FC
2020-07-08 20:58:14 +08:00
bin liu
bd8f03a5ef runtime: remove agent abstraction
This PR will delete agent abstraction and use Kata agent as the only one agent.

Fixes: #377

Signed-off-by: bin liu <bin@hyper.sh>
2020-07-08 10:07:40 +08:00
Fupan Li
92697a197f
Merge pull request #297 from liubin/2-dev/metrics
Kata 2.0: add metrics for agent/guest/shim
2020-07-08 09:44:20 +08:00
Fupan Li
57dfda9b3d
Merge pull request #365 from liubin/feature/delete-shim-364
runtime: clean up shim abstraction
2020-07-07 09:54:27 +08:00
fupan.lfp
83b2366560 config: there is no need to check vhost-vosck for FC
Since the FC used the hybrid vsock, there's no need
to check whether the vhost vsock suported by host.

Fixes: #387

Signed-off-by: fupan.lfp <fupan.lfp@antfin.com>
2020-07-06 16:02:37 +08:00
bin liu
1b75daa00f runtime: add new command to collect metrics from Kata containers
Add a new command to collect metrics and return metrics to Prometheus.

Signed-off-by: bin liu <bin@hyper.sh>
2020-07-02 17:54:54 +08:00
bin liu
186fed2a11 runtime: add implementation of GetMetrics
add implementation of `GetMetrics` for runtime.

Signed-off-by: bin liu <bin@hyper.sh>
2020-07-02 10:58:14 +08:00
Jose Carlos Venegas Munoz
94fdec4e4b clh: Allow add virtiofs args and cache options from config
[ port from runtime commit e5a3211c74e20e9878fd0f5d1c80a3c4354eabd1 ]

Today some options for virtiofsd could improve compatibility
for example xattrs for dnf or cache=auto for file consistency
for changes in the host. Allow users can enabled as requiered.

Signed-off-by: Jose Carlos Venegas Munoz <jose.carlos.venegas.munoz@intel.com>
Signed-off-by: Peng Tao <bergwolf@hyper.sh>
2020-06-30 04:04:39 -07:00
bin liu
3ece4130c9 runtime: clean up shim abstraction
This PR will delete shim abstraction from sandbox.

Fixes: #364

Signed-off-by: bin liu <bin@hyper.sh>
2020-06-30 15:52:20 +08:00
Jia He
fa9d619e8a qemu: add cpu_features option
[ port from runtime commit 0100af18a2afdd6dfcc95129ec6237ba4915b3e5 ]

To control whether guest can enable/disable some CPU features. E.g. pmu=off,
vmx=off. As discussed in the thread [1], the best approach is to let users
specify them. How about adding a new option in the configuration file.

Currently this patch only supports this option in qemu,no other vmm.

[1] https://github.com/kata-containers/runtime/pull/2559#issuecomment-603998256

Signed-off-by: Jia He <justin.he@arm.com>
Signed-off-by: Peng Tao <bergwolf@hyper.sh>
2020-06-29 20:16:11 -07:00
Liam Merwick
d3b3e8bee6 virtcontainers: x86: Support microvm machine type
[ port from runtime commit 6aff077901021d9a0075c446dfe281b2487e1487 ]

With the addition of support to govmm for multiple transports (intel/govmm#111)
and microvm (intel/govmm#121) we can now enable support for the 'microvm'
machine type in kata-runtime.

Signed-off-by: Liam Merwick <liam.merwick@oracle.com>
Signed-off-by: Peng Tao <bergwolf@hyper.sh>
2020-06-29 01:06:30 -07:00
Peng Tao
3bbb97add3
Merge pull request #312 from Pennyzct/network_throttle_on_qemu
rate-limiter: network I/O throttling on VM level
2020-06-25 04:59:44 +08:00
Penny Zheng
527c3f4634 test: Add unit test TestNewFirecrackerHypervisor
We have defined specific config file configuration-fc.toml for firecracker,
including specific features and requirements, but the related unit test
TestNewFirecrackerHypervisor is missing.

Fixes: #250

Signed-off-by: Penny Zheng <penny.zheng@arm.com>
2020-06-24 06:14:42 +00:00
Penny Zheng
c2645f5d5a rate-limiter: add rate limiter configuration/annotation on VM level
Add configuration/annotation about network I/O throttling on VM level.
rx_rate_limiter_max_rate is dedicated to control network inbound
bandwidth per pod.
tx_rate_limiter_max_rate is dedicated to control network outbound
bandwidth per pod.

Fixes: #250

Signed-off-by: Penny Zheng <penny.zheng@arm.com>
2020-06-24 06:14:04 +00:00
Christophe de Dinechin
042426d73a katatestutils: Use the configured virtiofs daemon path
The current path is hardcoded as follows:
  virtio_fs_daemon = "/path/to/virtiofsd"

Switch to using the value of config.VirtioFSDaemon instead.

Fixes: #2686

Signed-off-by: Christophe de Dinechin <dinechin@redhat.com>
Signed-off-by: Fabiano Fidêncio <fidencio@redhat.com>
2020-06-23 22:10:44 +02:00
Adrian Moreno
b97287090b qemu: enable iommu on q35
Add a configuration option and a Pod Annotation

If activated:
- Add kernel parameters to load iommu
- Add irqchip=split in the kvm options
- Add a vIOMMU to the VM

Fixes #2694
Signed-off-by: Adrian Moreno <amorenoz@redhat.com>
Signed-off-by: Fabiano Fidêncio <fidencio@redhat.com>
2020-06-22 16:37:20 +02:00
Peng Tao
4fdb4cffd5 runtime: update vendor/modules.txt
go 1.14 requires it to match go.mod and exposes a backtrace with
slightly difference.

Fixes: #227
Signed-off-by: Peng Tao <bergwolf@hyper.sh>
2020-05-25 23:30:42 -07:00
Peng Tao
6de95bf36c gomod: update runtime import path
To use the kata-containers repo path.

Most of the change is generated by script:
find . -type f -name "*.go" |xargs sed -i -e \
's|github.com/kata-containers/runtime|github.com/kata-containers/kata-containers/src/runtime|g'

Fixes: #201
Signed-off-by: Peng Tao <bergwolf@hyper.sh>
2020-04-29 18:39:03 -07:00
Peng Tao
a02a8bda66 runtime: move all code to src/runtime
To prepare for merging into kata-containers repository.

Signed-off-by: Peng Tao <bergwolf@hyper.sh>
2020-04-27 19:39:25 -07:00