1
0
mirror of https://github.com/rancher/steve.git synced 2025-09-18 00:08:17 +00:00
Commit Graph

387 Commits

Author SHA1 Message Date
Eric Promislow
3119b3d8fa Ensure database is reset after an api-service change. (#785) 2025-08-18 15:07:49 -07:00
Eric Promislow
4cc7e7b947 Fix new projectsornamespaces negative tests. (#769)
Remember to select objects that don't have the project label assigned to them
when doing a negative test.
2025-08-08 17:23:56 -07:00
Felipe Gehrke
3cac88141b #50968 - Single SQL Transaction for projectsornamespaces filter (#758)
* working for positive case

* changed to custom code

* removed comment

* added tests

* fixing ci error

* one more fix

* Update proxy_store tests for projectornamespace parsing.

Since we no longer need to execute a db query, and just return a filter,
we can drop a lot of code.

* added distinct back to conform with other queries

* one more fix

---------

Co-authored-by: Eric Promislow <epromislow@suse.com>
2025-08-08 16:07:54 -03:00
Eric Promislow
dbd2818d22 VAI: Replace namespace+name default sort with id (#724)
* Sort by ID rather than metadata.namespace,metadata.name

- In tests add an id field to the list of fields to cache/create and to the objects.

* Add benchmark tests to compare sorting by ns/name vs id.
2025-08-07 15:42:33 -07:00
Eric Promislow
97b595d551 Implement sorting by status.lastScheduleTime and status.lastSuccessfulTime. (#766)
Note that it will be possible to filter against these fields, but that doesn't sound too useful,
as you'd have to compare against a date string.
2025-08-07 13:28:30 -07:00
Chad Roberts
c81c0d21a4 Dropping debug-level default to 1 to reduce noise from reflector. (#765) 2025-08-07 14:39:13 -04:00
Chad Roberts
ac12e33508 Update to use klog v2. Also turned down the default debug-level to 2 from 7 to avoid the lower level http logs that show up with klog v2 (#764) 2025-08-07 10:40:36 -07:00
Sakala Venkata Krishna Rohit
6946ffe8aa Add an attribute observedgeneration (#714)
* Add an attribute observedgeneration

* add unit tests
2025-08-07 10:30:06 -07:00
Tom Lebreux
a020084518 Fix slow labels queries (#762)
* Add PRAGMA temp_store=2 to write temporary tables in memory

* Only use DISTINCT when filtering labels

* Log long queries to debug

* Add unit test
2025-08-06 20:07:09 -04:00
Eric Promislow
5f08feeb5f Reset db more selectively: approach #2 (#745)
* On Reset, remove all the sqlite files, not just the main one.

* Avoid resetting the database and cache when not needed.

Avoid resetting when a change doesn't bring in new schema fields.

Always reset when a CRD is deleted.

* Force a database reset when a new CRD is added.

* Improve code readability. No functional change.
2025-07-31 10:43:55 -07:00
Chad Roberts
e92b9cf7ea Revert "Update to use klog v2 (#744)" (#757)
This reverts commit dd64c2d200.
2025-07-30 13:19:25 -04:00
Chad Roberts
dd64c2d200 Update to use klog v2 (#744) 2025-07-29 15:42:25 -04:00
Sakala Venkata Krishna Rohit
1c7948d4c9 Revert "Revert "Make view link related to GET RBAC permission. (#699)" (#703)" (#704)
This reverts commit 8cd7bd0a8f.
2025-07-18 15:36:53 -07:00
Felipe Gehrke
b556256ed3 #50914 - Changed ParseQuery projectornamespace Handling (#718)
* changed ParseQuery projectornamespace handling

* addressed comments from eric/chad and tom

* fixing lint error

* added resourceVersion to list

* updated mocks

* updated tests

* Update tests after rebase

---------

Co-authored-by: Tom Lebreux <tom.lebreux@suse.com>
2025-07-17 13:51:13 -04:00
Tom Lebreux
0c2c554c8c Respect shouldEncrypt configuration when storing events (#725) 2025-07-15 09:57:28 -04:00
Eric Promislow
904e52dc11 Index fields used for testing/QA to stay compatible with v2.11 (#719) 2025-07-11 10:32:32 -07:00
Tom Lebreux
faa5ad63e9 Fix CRD Created At field (#723)
* Add Tokens and Kubeconfig fields to date mapping

* Fix missing cols in transform func for watch

* Mark CRDs as.. CRD

* Don't transform CRD's date field
2025-07-11 13:28:52 -04:00
Tom Lebreux
127d37391d Better gc (#717)
* More error wrapping for SQL cache

* Use context.Context instead of stopCh

* Move CacheFor to Info log level

* Implement time-based GC

* Set default GC param when running standalone steve
2025-07-11 12:48:19 -04:00
Eric Promislow
3692666375 Index the secrets annotation "management.cattle.io/project-scoped-secret-copy" (#706)
Tests will be in the rancher integration tests.
2025-07-09 13:13:53 -07:00
Eric Promislow
76477e98df Hard-wire external associations: 6 and 7/7: unit tests for updating A=>B links when instances of A and B change (#702)
* Implement hard-wired external associations:

* The table is in sqlproxy.proxy_store
  - externalGVKDependencies - a map of GVKs to dependencies.
    When the key GVK is updated, it triggers the updates in the database for the dependent GVKs,
    replacing fields as specified in the table.

* This is done in an afterUpsert handler, but it's done after the transaction for the core
  GVK update is finished, because most likely the dependent GVK updates will depend on the
  final database values for the GVK being updated, and if we do it as part of the transaction
  the new values won't be committed to the database.

* Wrote unit tests for external associations.

* When an object is modified/created, check for external deps that need updating.

* Stop emitting errors when joining tables if one of the tables doesn't exist.

* Finish rebasing manually.

* Fix external association unit-tests after rebasing.
2025-07-09 13:11:42 -07:00
Eric Promislow
a3cd651c75 Revert part of steve PR 633 (#708)
Specifically, recomment a test that was uncommented in this PR,
and revert the underlying code that made this test pass.  That
code change causes user-based integration tests to fail, so something
is not correct in that code.

This was actually unrelated to the reason for PR 633, to fix
a problem when sorting on a label that dropped objects that
didn't have that label.  That part remains untouched.
2025-07-09 11:12:47 -07:00
Eric Promislow
2a86733c64 Use SQL WITH statements to sort unbound labels. (#663)
* Use SQL `WITH` statements to sort unbound labels.

These are labels whose names are never positively referenced in a filter,
so they don't need to exist on the row that we still want to display.

Here I create a virtual table of all the rows, substituting a null
value for each label that isn't associated on the row, and then sort on that.

* Just always select-distinct for now.

* Add more tests for filtering/sorting

- Assign more values to the cattle and horses labels
- Move the sortfield value to a number in the 100s -- keep in mind
  these values are sorted by ascii value of underlying chars, not numerically
- Rename the var names to better reflect the values they contain

* Remove mentions of the prepared SQL statement in the test descriptions.

We care either about the items we get back from the query, or in some
cases the SQL that gets generated by the AST interpreter.

* Simplify the use of WITH stmts in SQL (thx Tom)

* Fix the comment about an unexpected situation.

* Fix post-rebase tests.
2025-07-08 10:07:05 -07:00
Sakala Venkata Krishna Rohit
8cd7bd0a8f Revert "Make view link related to GET RBAC permission. (#699)" (#703)
This reverts commit 542c471584.
2025-07-04 14:27:43 -07:00
Sakala Venkata Krishna Rohit
542c471584 Make view link related to GET RBAC permission. (#699)
* Add view link based on GET RBAC permission

* Add tests for view link
2025-07-04 14:06:24 -07:00
Eric Promislow
3be82a28d1 Hard-wire external associations: 5/7: update A=>B links when instances of A change (#646)
* Continue rebasing.

* Wrote unit tests for external associations.

* Fix the generated SQL.

Some syntactic sugar (capitalizing the keywords), but use the 'ON' syntax on JOINs.

* We want "management.cattle.io.projects:spec.displayName" not "...spec.clusterName"

* Implement hard-wired external associations:

* The table is in sqlproxy.proxy_store
  - externalGVKDependencies - a map of GVKs to dependencies.
    When the key GVK is updated, it triggers the updates in the database for the dependent GVKs,
    replacing fields as specified in the table.

* This is done in an afterUpsert handler, but it's done after the transaction for the core
  GVK update is finished, because most likely the dependent GVK updates will depend on the
  final database values for the GVK being updated, and if we do it as part of the transaction
  the new values won't be committed to the database.

* When an object is modified/created, check for external deps that need updating.

* Stop emitting errors when joining tables if one of the tables doesn't exist.

* Update unit test syntax for SQL queries.

* And an override check

This ensures we don't overwrite good data when
pulling data from one table to another.

* Drop labels, and use mgmt.cattle.io/spec.displayName

There's no need to hardwire labels in proxy_store:typeSpecificIndexedFields
because all labels are indexed in the shadow labels table.

* Keep clusterName, add displayName for mgmt.cattle.io

* Fix rebase/merge breakage.

* Finish the merge: add the 'selfUpdateInfo' param where it didn't get inserted during merge.

* Patch up rebase failures.

* Now gomock generates named args. I give up.
2025-07-03 14:35:09 -07:00
Alejandro Ruiz
61173104af Fix parsing of metadata.fields when age is already a duration (#700)
* Fix parsing of metadata.fields when age is already a duration

* Fix frozen "now"

* Use ParseTimestampOrHumanReadableDuration instead
2025-07-01 13:53:51 -03:00
Eric Promislow
c782fea615 Update the READMEs to cover SQL-based caching. (#693) 2025-06-26 09:29:51 -07:00
Peter Matseykanets
952305e096 vai: Rework events garbage collection delete query (#682) 2025-06-26 09:22:40 -07:00
Alejandro Ruiz
6495e7879c Make waiting for ExtensionAPIServer configurable (#694) 2025-06-26 17:04:49 +02:00
Eric Promislow
496a6f8968 Hard-wire external associations: 3 sections in, this one is 4/7 (#645)
* Continue rebasing.

* Wrote unit tests for external associations.

* Fix the generated SQL.

Some syntactic sugar (capitalizing the keywords), but use the 'ON' syntax on JOINs.

* whitespace fix post rebase

* We want "management.cattle.io.projects:spec.displayName" not "...spec.clusterName"

* Fix the database calls: drop the key

* Fix breakage during automatic rebase merging gone wrong.

* ws fix - NFC

* Post rebase-merge fixes

* Fix rebase-driven merge.

* Fix rebase breakage.

* go-uber v0.5.2 prefers real arg names to '<argN>'
2025-06-25 16:10:48 -07:00
Swastik Gour
e17ca28461 Bumped dependencies to be compatible with k8s-v1.33 (#681)
Signed-off-by: swastik959 <Sswastik959@gmail.com>
2025-06-20 17:53:42 +05:30
Eric Promislow
c32995dba4 Log and ignore transform errors. (#683) 2025-06-18 11:02:46 -07:00
Felipe Gehrke
4212386e13 changing ParseHumanDuration to support timestamp values too (#684) 2025-06-18 10:57:49 -07:00
Felipe Gehrke
b3539616e0 #48673 - Added Timestamp Cache Handling to metadata.fields (#648)
* added timestamp convertion to metadata.fields

* fixed duration parsing

* fixed tests

* removed tags file

* added comments

* added better error handling

* changed ParseHumanDuration to use Fscanf

* added builtins handling

* adding mock updates

* fixing tests

* another try

* added timestamp convertion to metadata.fields

* addressing comments from @ericpromislow

* converting error to warning

* added template options
2025-06-16 15:33:28 -07:00
Eric Promislow
2e8a0f2851 Support indexing on array-like fields (#673)
* Run tests using sqlite DB in a temp directory.

I was running into write-file errors which happens when two sqlite processes try to update the DB at the same time.

* Implement and test the extractBarredValue custom SQL function.

* Explain the DB path constants better.
2025-06-16 15:06:07 -07:00
Eric Promislow
2cd7997e6b Fix non-vai sorting involving string arrays, add tests. (#618)
- Bump wrangler to get fix for 'GetValueFromAny'.
  This adds string arrays to the types it supports.

- Use a newer go library sort function to maintain a stable sort

- Use the map-sort-map pattern to avoid repeatedly calculating the
  same underlying value for `object[arg1][arg2][...]`
2025-06-13 10:18:48 -07:00
Chad Roberts
5de54d6a4a Handle int64 field type when adding index fields (#671)
* Handle int64 field type when adding index fields

* add support for float64
2025-06-13 13:04:55 -04:00
Chad Roberts
58f15a4349 check err after client.List call when creating informer (#672) 2025-06-12 17:03:52 -04:00
Josh Meranda
c67ddf2de4 [main] block UI until we receive requests from kube-apiserver (#668)
* block UI until we receive requests from kube-apiserver

* satisfy CI

* undo test code

---------

Co-authored-by: joshmeranda <joshua.meranda@gmail.com>
2025-06-11 14:58:36 -04:00
Tom Lebreux
1157865ea3 Fix panic due to concurrent map writes (#669)
The same object is given to many HTTP request and they proceed to modify
that object.
2025-06-11 11:57:48 -04:00
Tom Lebreux
cf97607be5 Return resourceversion too old error to UI instead of logging (#667)
* Return watch error instead of logging it

The UI needs to know about watch error like `resourceversion too old` so
we need to return it.

* Sort by resourceVersion as number

The UI makes some assumption on resourceVersion. It assumes they are a
number and they are ordered by the number value. We'll want to fix this
at some point most likely but for now let's give something in a way that
UI wants.

* Remove -d suffix

After much testing, a delete of an object seems to have its own
resourceVersion so we don't need the -d suffix, we can simply use the
new resourceVersion.
2025-06-10 17:29:42 -04:00
Eric Promislow
7db113a1fd Add vai access-control for ext Tokens and Kubeconfigs (#651)
* Add vai access-control for tokens.

* Check for both Token and Kubeconfig resources

* Add a unit test for verifying the generated filters for restricted resources.

* Remove a TODO comment as Tom points out we no longer need it.

* Return error if we can't get userinfo from apiOp.Request.Context

* Stop using camelCase for the user ID label.

* Add a test for the admin user.

* And fold the two user-access tests into a single parameterized test.

* Address reviewer comments.

* post-rebase merge fixes

* WIP - add a comment about determining admin users.

---------

Co-authored-by: Peter Matseykanets <peter.matseykanets@suse.com>
2025-06-10 14:28:49 -07:00
Tom Lebreux
f258ebcf31 Add garbage collection for events table (#664)
* Add ListOptionIndexerOptions to create ListOptionIndexer

* Remove unused function

* Implement configurable garbage collection on ListOptionIndexer

* Propagate Steve options from Server
2025-06-10 12:02:55 -04:00
Tom Lebreux
b4db257cdb Handle transaction failure due to canceled context.Context (#662)
* Re-order SQL event hooks so events are last

* Add QueryRowContext for single line queries

* Add test case for unknown resource version

* Properly check rows and close it

* More accurate error message when context.Context is canceled

* Re-order test check
2025-06-09 15:39:09 -04:00
Tom Lebreux
b695567794 Keep track of events for SQL cache watches (#661)
* Keep track of past events

* Clarify RV acronym

* Fix comment typo

* Rename listEvents variables

* Improve filter readability

* Move defer
2025-06-05 18:40:19 -04:00
Tom Lebreux
b0aa90cd22 Add watch filtering support for vai (#654)
* Add support for watch filtering

* Remove debug Println

* Validate watch selector
2025-06-03 18:10:36 -04:00
Tom Lebreux
e3f207ddc2 Add basic watch functionality for SQL cache (#653)
* Remove unused method

* Add basic watch functionality

* Remove TestWatchNamesErrReceive test
2025-06-03 18:07:18 -04:00
Tom Lebreux
2672969496 Add object to RegisterAfterDelete and introduce RegisterAfterDeleteAll (#649)
* Add object to AfterDelete callbacks

* Add RegisterAfterDeleteAll
2025-06-03 17:32:43 -04:00
Tom Lebreux
a8f3ce48d6 Convert tests to real SQL store (#652) 2025-06-02 16:01:45 -06:00
Alejandro Ruiz
06fe9c3ef4 Calculate AccessSets once per request instead of per resource (#647) 2025-05-30 16:32:56 +02:00