This introduces the a `WithTransaction` function, which is then used for all transactional work in Steve.
Because `WithTransaction` takes care of all `Begin`s, `Commit`s and `Rollback`s, it eliminates the problem where forgotten open transactions can block all other operations (with long stalling and `SQLITE_BUSY` errors).
This also:
- merges together the disparate `DBClient` interfaces in one only `db.Client` interface with one unexported non-test implementation. I found this much easier to follow
- refactors the transaction package in order to make it as minimal as possible, and as close to the wrapped `sql.Tx` and `sql.Stmt` functions as possible, in order to reduce cognitive load when working with this part of the codebase
- simplifies tests accordingly
- adds a couple of known files to `.gitignore`
Credits to @tomleb for suggesting the approach: https://github.com/rancher/lasso/pull/121#pullrequestreview-2515872507
* Add more fields to index when sql-caching is on.
* Restore the gvkKey helper, add event fields.
The UI team wasn't sure whether the event fields should go in the empty-string group or in 'events.k8s.io',
so let's go with both until/unless specified otherwise.
* More fixes to the fields to index:
- Remove the erroneously added management.cattle.io.nodes fields
- Use the builtin Event class, not events.k8s.io (by looking at the dashboard client code)
* Start on the virtual-field work.
* Map `Event.type` to `Event._type` for indexing.
* Add a unit test for field replacement for Event.type
* Add label processing.
* Don't test for transformation of event objects in the common module.
* Parse metadata.label queries differently.
* Improve a variable name that turned out to not be temporary.
* No need to specifically cache certain labels, as all are now cached.
* Add a test to verify simple label (m.labels.foo=blah) queries work.
* 'addLabelFields' never returns an error.
* Delete superseded function.
* Was calling 'addLabelFields' one time too many.
* Start using k8s ParseToRequirements
* Pull in the k8s parser.
* Successfully test for quotation marks.
* Add quoted strings to the lexer.
* Move to a forked k8s label lexer to include non-label tests.
* Improve and test the way quoted strings in the query are detected.
* Reinstate the original Apache license in the derived code.
Following clause 4.3 of the Apache license:
"You must cause any modified files to carry prominent notices stating that You changed the files..."
* Ignore case for operators.
* Test IN multiple-target-values
* Test the not-in operator.
* Ignore case for operators.
SQL is case-insensitive on field names and values, so this just adds consistency.
* Added tests for parsing EXISTS and NOT-EXISTS queries.
* Parse less-than and greater-than ops
* Lasso's `CacheFor` now takes a `watchable` argument.
* Support 'gt' and 'lt' as synonyms for '<' and '>'.
I see both types of operators being bandied about -- it's easy to support the aliases.
* typo fix
* Have the filter parser allow exist tests only on labels.
Also reduce the case where there's no namespace function.
* Specify hard-wired fields to index alphabetically.
* Remove unused variable.
* Parser: 'metadata.labels[FIELD]' is valid
* Pull in new gvk fields from main (and keep in alpha order).
* Fixed a couple of drops done during the last rebase.
* Add a reminder to keep the entries in alpha order.
* Test TransformLabels
* Remove TransformLabels
* Remove unused/unneeded code.
* Describe diffs between our label-selector parser and upstream's.
* Use the merged lasso 46333 work.
* Drop unused field.
* Tighten up the code.
* Specify which commit the label selector parser is based on.
* Allow both single-quoted and double-quoted value matching, doc difference.
* More review-driven changes:
- Stricter processing of m.l.name keys:
Require ending close-bracket for a start-bracket
- Comment fix
- Moving sql processing from lasso to steve: some changes missed in rebase
* Drop support for double-quotes for string values.
For now on only single-quotes (or none where possible) are allowed.
* Renaming and dropping an init block.
* Quoted strings are dropped from the filter queries
In particular, label values have a specific syntax: they must
start and end with a letter, and their innards may contain
only alnums '.', '-' and '_'. So there's no need for quoting.
And that means now that `=` and `==` do exact matches,
and the `~` operator does a partial match.
`!=` and `!~` negate -- note that `!~` is a stricter operation than `!=`,
in that given a set of possible string values, `!=` will accept more
of them than `!~`. Maybe I shouldn't have gone here, but these operators
reminded me of learning about `nicht durfen` and `nicht sollen` in German,
or something like that.
* Move a constant definition to the module level.
* Remove commented-out code.
* Remove unused func and adjacent redundant comment.
* Replace primary/secondary sort fields with an array of sort directives.
* Allow more than 2 sort-params in a search query.
* Add a virtual 'status.ready' field to clusters.
* Rename status.ready -> status.connected
* Set virtual field 'spec.internal' <- spec.displayName == 'local'
* Need to declare all virtual fields to index.
* Ready clusters have condition[type==Ready && status=True]
* Update the README to reflect generalized sorting.
* Bump lasso to get revised sort directives.
* Review-driven changes, mostly comments and drop unneeded code.
* Add unit tests to verify sort-order stringification.
* Ignore empty-string sort components.
* Fix a rebase mishap.
* Drop unneeded commented-out code.
* Clusters have a 'spec.internal' field, no need to synthesize one.
* Added a note on square-brackets for label references.
This should be added to the README for filter queries in the PR for 46333.
* Bump to latest sqlcache-free lasso
* Checkpoint
* Add support for custom columns
* Remove old Store and Delegate abstraction
* Fix nits and rewording
* Remove unused mock file
* Update documentation for extension api server
* Remove the need for scheme for ConvertListOptions
* Rename store to utils
* fixup! Remove the need for scheme for ConvertListOptions
* Move watch helper to tests
* Add convertError at a few places
* Ignore misspell on creater
* Fix comments and remove unused params
* Add convertError to missing error returns
* Fix watcher implementation
* Document request.UserFrom and request.NamespaceFrom
The original PR, steve/pull/158 drifted too far from changes
in main, so it's easier to create a new PR:
* Bring in and update DefaultSchemaTemplatesForStore
* Move from based on `master` to `main`
* Update k8s version
* bump lasso to include pull #111
Signed-off-by: Silvio Moioli <silvio@moioli.net>
* Make IsListWatchable public to be reused in other packages
Signed-off-by: Silvio Moioli <silvio@moioli.net>
* Let lasso know whether a type is watchable upon requesting a cache
Signed-off-by: Silvio Moioli <silvio@moioli.net>
* Adapt existing tests
Signed-off-by: Silvio Moioli <silvio@moioli.net>
* Add a test to check watchability is detected correctly
Signed-off-by: Silvio Moioli <silvio@moioli.net>
---------
Signed-off-by: Silvio Moioli <silvio@moioli.net>
* adding NonResourceURLs support to access_store
* added tests to AccessSet NonResourceURLs handling
* change on test script suggested by @tomleb + go mod tidy
* added nonresource to ext api authorization
* added NonResourceURLs implementation in Authorizes + test
* removed non-resource-url tests from the main test
* added new tests for non-resource-urls
* removed unused test data
* changed nonResourceKey to point to struct{}
* addressed comments from @tomleb
* addressed more comments
* fixing typo
* check for empty accessSet
* Refactor ID based partitioning, add unit tests
This resolves an issue where the requested namespace filter was not
always honored.
* Correct naming issues to appease the linter
* changing secret and configmap formatters to return decoded helm data if includeHelmData query parameter is present
* adding tests for gzip magic header
* refactor(accesscontrol): make addAccess directly accept PolicyRules
* refactor(accesscontrol): add new types for encapsulating all needed data
* refactor(accesscontrol): make getRules return resource version
* refactor(accesscontrol): add new getRoleRefs to policyRuleIndex
* refactor(accesscontrol): make accessStore use the new types and method
* cleanup(accesscontrol): remove unused code
* cleanup(accesscontrol): adapt tests
* cleanup(accesscontrol): add some comments and remove unused function
* refactor(accesscontrol): rework indexer to make it more readable and testable
* Fix typo
* test: consistent use of t.Error
* test: refactor policyRulesMock to just use a map
* misc: rename toUserInfo function
* refactor: consistent sort by UID
* Add more fields to index when sql-caching is on.
Misc changes:
- Use the builtin Event class, not events.k8s.io (by looking at the dashboard client code)
- Specify full path to the management.cattle.io fields.
- Map `Event.type` to `Event._type` for indexing.
Use a compound transform-func to first check for a "signal",
and then to run all the relevant transformers until either
one fails or the list is exhausted.
- Includes moving the fakeSummaryCache type into a common area.
Use a simpler way of running transforms.
* Inline the function to get the gvk key.
* Create a '--sql-cache' flag to turn on caching for the steve CLI.
* Improve error-handling in object transformer.
* Drop the 'GetTransform' function.
* Inline the code that transforms a payload into a k8s-unstructured object.
Previously, the formatter for state/relationships was disabled when the
sql cache was enabled, since a transform function was adding those
values before they were added to the cache. However, the get/watch calls
currently don't use the cache, causing the state/relationships to be
missing.
This implements the Imperative API that is served at /ext with Steve. The imperative API is compatible with Kubernetes' API server and will be used as an extension API server.
* refactor(accesscontrol): use interface for AccessStore cache
* refactor(accesscontrol): early return when cache is disabled
* test(accesscontrol): add failing unit test
* test(accesscontrol): skip failing test
* added namespace check to proxy_store create
* added namespaced resources checks and create tests
* Update pkg/stores/proxy/proxy_store.go
* changed error message and added missing name only test
* updated sql/proxy_store
* changed return to use apierror.NewAPIError
---------
Co-authored-by: Felipe C. Gehrke <felipe@localhost.localdomain>
Co-authored-by: Tom Lebreux <tom.lebreux@suse.com>
Adds logic which adds virtual fields resources. This allows these fields
to be sorted/filtered on when the SQL cache is enabled. Id and
metadata.state.name were added as the first two fields.
* README: make common header for list parameters
* README: document differences when SQLite caching is enabled
* README: clarify support of sorting with the SQL cache
* README: mention additional printer columns
Signed-off-by: Silvio Moioli <silvio@moioli.net>
