Adds logic which adds virtual fields resources. This allows these fields
to be sorted/filtered on when the SQL cache is enabled. Id and
metadata.state.name were added as the first two fields.
Prior schema calculations started with openapiv2 models which included a
model for APIGroups. However, new schema calculations use
ServerGroupsAndResources first, which omitted these values. This
re-adds this type using a static schema.
This uses SQLite-backed informers provided by Lasso with https://github.com/rancher/lasso/pull/65 to implement Steve API (/v1/) functionality.
This new functionality is available behind a feature flag to be specified at Steve startup
See https://confluence.suse.com/pages/viewpage.action?pageId=1359086083
Co-authored-by: Ricardo Weir <ricardo.weir@suse.com>
Co-authored-by: Michael Bolot <michael.bolot@suse.com>
Co-authored-by: Silvio Moioli <silvio@moioli.net>
Signed-off-by: Silvio Moioli <silvio@moioli.net>
Fixes two bugs with the schema definitions:
- Adds resources that are a part of the baseSchema (but aren't k8s resources).
- Adds logic to handle resources which aren't in a prefered version, but
are still in some version.
Some tests which relied on timeouts were a bit flaky in CI. This PR
refactors a few of them to work on a more reliable method of receiving
from a channel and raises the timeout of another test.
In the original implementation of the definition handler, the resource
list was checked for preferred version, and this overrode the preferred
version of the overall group. However, this logic was inaccurate and
did not use the group as the source of truth on the preferred version
like it should have.
Updates the schema definitions refresh method to use a new debounce
method. Adds a handler which refreshes the definitions every 2
seconds after adding a CRD and every 10 minutes by default.
Changes steve to only return top level schema definitions rather
than a full defined schema for every field. Also adds basic docs
and fixes a bug with schema generation on versioned crds
ByNames could previously return a nil value and a nil error. This caused
issues when other parts of the application
(pkg/stores/partition/parallel.go) tried to use the result. Now this
will return an empty list on the error condition, instead of nil
Remove unnecessary items from the `access` slice in one of the unit
tests. Each item in `access` corresponds to one request in `apiOps`, so
they should have the same number of elements.
Add a nested store to the proxy store to strip non-Kubernetes fields
from the object being updated.
The steve formatter and proxy store adds fields to objects when it
outputs them to the client, for usability by the UI. It adds the
object's fields[1], relationships to other objects[2], a summary of the
object's state[3], and additional information in the conditions[4].
These fields are not native to Kubernetes, so when a client submits the
object back as an update, Kubernetes reports a warning that they are
unrecognized. This change ensures the extra fields are removed before
submitting the update.
[1] bf2e9655f5/pkg/stores/proxy/proxy_store.go (L189)
[2] bf2e9655f5/pkg/resources/common/formatter.go (L106)
[3] bf2e9655f5/pkg/resources/common/formatter.go (L100)
[4] bf2e9655f5/pkg/resources/common/formatter.go (L108)
Add a new query parameter to filter resources by their namespace or
their namespace's project. This parameter is separate from the existing
`filter` parameter.
Filter by a comma-separated list of projects and/or namespaces with:
?projectsornamespaces=p1,n1,n2
The result can be negated with the ! operator:
?projectsornamespaces!=p1,n1,n2
This change adds support for excluding results using the != operator.
Example to exclude all results with name "example":
?filter=metadata.name!=example
Include all results from namespace "example" but exclude those with name
"example":
?filter=metadata.namespace=example&metadata.name!=example
Exclude results with name "foo" OR exclude results with name "bar"
(effectively includes results of both types):
?filter=metadata.name!=foo,metadata.name!=bar
Currently, multiple filters can be appended on the query string, and
each subsequent filter is ANDed with the set. Items that pass through
the filter set must match every filter in the set.
This change adds support for OR filters. A single filter key can specify
multiple filters, separated by ','. An item that passes this filter can
match any filter in the set.
For example, this filter matches items that have either "name" or
"namespace" that match "example":
?filter=metadata.name=example,metadata.namespace=example
This filter matches items that have "name" that matches either "foo" or
"bar":
?filter=metadata.name=foo,metadata.name=bar
Specifying more than one filter key in the query still ANDs each inner
filter set together. This set of filters can match either a name of
"foo" or "bar", but must in all cases match namespace "abc":
?filter=metadata.name=foo,metadata.name=bar&filter=metadata.namespace=abc
Prior, table client configs were mistakenly swapped out for regular
client configs. This caused requests using these methods to no
longer return table column data under metadata.fields. Consequently,
table data was blank in the dashboard UI. Now, the client config
swap has been reverted and table fields should be populated in the
dashboard UI.
Without this change, the cache is checked if a revision is specified,
but the revision is ignored if the value is not found in the cache. This
is a problem if by chance sequential paginated requests land on separate
pods, because a different version number may be fetched on the sequent
request and cause the caches to be inconsistent with one another. This
change ensures that `revision` is passed as `resourceVersion` to the
Kubernetes request, since `revision` has no meaning to Kubernetes.
The default chunk size in the partition was set to 100000. It could be
overridden as a larger or smaller number, but not disabled altogether.
This change adds the ability for users to explicitly opt out of the
limit by specifying a negative number or zero. The default behavior is
the same.
Extend the sorting functionality in the partition store to support
primary and secondary sorting criteria. Sorting parameters are specified
by a single 'sort' query and comma separated. Example:
Sort by name and creation time:
GET /v1/secrets?sort=metadata.name,metadata.creationTimestamp
Reverse sort by name, normal sort by creation time:
GET /v1/secrets?sort=-metadata.name,metadata.creationTimestamp
Normal sort by name, reverse sort by creation time:
GET /v1/secrets?sort=metadata.name,-metadata.creationTimestamp
Cache filtered, sorted results for fast subsequent page retrieval.
Requests for cached queries need to include the list revision number
along with other queries. If no specific revision is requested, a new
fetch is done in order to get the latest revision. The revision is
included in the list response.
Example first request:
GET /v1/secrets?pagesize=10
Example subsequent page request:
GET /v1/secrets?pagesize=10&page=1&revision=107740
Extend the partition store to parse page and pagesize parameters and
return a subset of list results. The total number of pages is included
in the list response.
Request an initial page:
GET /v1/secrets?pagesize=10
or
GET /v1/secrets?pagesize=10&page=1
Request subsequent pages, or arbitrary pages:
GET /v1/secrets?pagesize=10&page=37
If a page number is out of bounds, an empty list is returned.
Extend the partition store to parse the "sort" query parameter as a
sorting condition. Dot notation is used to denote the object field.
Preceding the key with "-" denotes descending (reverse) order.
Example sorting by name:
GET /v1/secrets?sort=metadata.name
Reverse sorting by name:
GET /v1/secrets?sort=-metadata.name
All values are converted to strings and sorted lexicographically.
Extend the partition store to parse query params as list filters. Filter
keys use dot notation to denote the subfield of an object to filter on.
Example:
GET /v1/secrets?filter=metadata.name=foo
Filters are ANDed together, so an object must match all filters to be
included in the list. Example:
GET /v1/secrets?filter=metadata.name=foo&filter=metadata.namespace=bar
Arrays are searched for matching items. If any item in the array
matches, the item is included in the list. Example:
GET /v1/pods?filter=spec.containers.image=alpine
Filtering and sorting needs to operate on unstructured data. It also
needs to operate after the parallel partitioner, higher in the store
stack. This means that the proxy Store needs to return raw, unstructured
data up to the partitioner. This change moves all conversions from
unstructured Kubernetes types to apiserver objects up from the proxy
store into the partitioner.
The use of AddNamespaceConstraint was removed in e35b8304 of
rancher/rancher, so there is no possibility of there being a namespace
constraint in the request context. Remove the unused function and the
unused codepath from the rbac store.
The header defined in RFC 9110[1] is "Accept", not "Accepts". This
change corrects the route filter. Since this API is not documented and
the header was plainly incorrect, no attempt is made at backwards
compatibility.
[1] https://www.rfc-editor.org/rfc/rfc9110.html#name-accept
Three main changes:
- When a count changes, only send that count. This makes the
count socket significantly more usable
- The count buffer will now send a count right away. This allows
a message to launch wihtout an inital buffered wait period
- The count buffer now sends messages every 5 seconds instead of 1
By default, a watch times out after 30 minutes. For debugging purposes,
it's convenient if this can be decreased. Add an environment variable
CATTLE_WATCH_TIMEOUT_SECONDS to enable setting the timeout in seconds.
Add debug logs and send websocket messages when the watch is closed
unexpectedly.
In addition to being helpful for debugging, the dashboard specifically
looks for a `resource.error` event containing the string "too old" in
order to trigger the watch to be resynced with a refreshed revision
number. Without this error returned, the dashboard will only see
`resource.stop` events and never change its behavior, continuing to try
to restart the watch with an incorrect resource version.
This change enables steve to work with three new query parameters:
"include": only include the named fields from the kubernetes object.
Subfields are denoted with ".". Subfields within arrays are ignored.
Multiple fields can be included by repeating the parameter. Example:
GET /v1/configmaps?include=kind&include=apiVersion =>
{
"type": "collection",
"links": {
"self": "http://server/v1/configmaps"
},
"createTypes": {
"configmap": "http://server/v1/configmaps"
},
"actions": {},
"resourceType": "configmap",
"revision": "327238",
"data": [
{
"id": "c-m-w466b2vg/kube-root-ca.crt",
"type": "configmap",
"links": {
"remove": "http://server/v1/configmaps/c-m-w466b2vg/kube-root-ca.crt",
"self": "http://server/v1/configmaps/c-m-w466b2vg/kube-root-ca.crt",
"update": "http://server/v1/configmaps/c-m-w466b2vg/kube-root-ca.crt",
"view": "http://server/api/v1/namespaces/c-m-w466b2vg/configmaps/kube-root-ca.crt"
},
"apiVersion": "v1",
"kind": "ConfigMap"
},
}
...
}
"exclude": exclude the named fields from the kubernetes object.
Subfields are denoted with ".". Subfields within arrays are ignored.
Multiple fields can be excluded by repeating the parameter. Example:
GET /v1/configmaps?exclude=data&exclude=metadata.managedFields =>
{
"type": "collection",
"links": {
"self": "http://server/v1/configmaps"
},
"createTypes": {
"configmap": "http://server/v1/configmaps"
},
"actions": {},
"resourceType": "configmap",
"revision": "328086",
"data": [
{
"id": "c-m-w466b2vg/kube-root-ca.crt",
"type": "configmap",
"links": {
"remove": "http://server/v1/configmaps/c-m-w466b2vg/kube-root-ca.crt",
"self": "http://server/v1/configmaps/c-m-w466b2vg/kube-root-ca.crt",
"update": "http://server/v1/configmaps/c-m-w466b2vg/kube-root-ca.crt",
"view": "http://server/api/v1/namespaces/c-m-w466b2vg/configmaps/kube-root-ca.crt"
},
"apiVersion": "v1",
"kind": "ConfigMap",
"metadata": {
"creationTimestamp": "2022-04-11T22:05:27Z",
"fields": [
"kube-root-ca.crt",
1,
"25h"
],
"name": "kube-root-ca.crt",
"namespace": "c-m-w466b2vg",
"relationships": null,
"resourceVersion": "36948",
"state": {
"error": false,
"message": "Resource is always ready",
"name": "active",
"transitioning": false
},
"uid": "1c497934-52cb-42ab-a613-dedfd5fb207b"
}
},
...
}
"excludeValues": replace the values of an object with empty strings, leaving
the keys in place. Useful for showing a summary of an object with large
values, such as the data in a ConfigMap. Only works on fields that are
object. Multiple fields can have values excluded by repeating the
parameter. Example:
GET /v1/configmaps?excludeValues=data =>
{
"type": "collection",
...
"data": [
{
...
"data": {
"ca.crt": ""
},
...
},
...
]
}
