Commit Graph

28 Commits

Author SHA1 Message Date
RoyUP9
2df9fb49db
Changed entry protocol field from struct to unique string (#1167) 2022-06-26 17:08:09 +03:00
M. Mert Yıldıran
eef0ee8023
Fetch N number of records in M milliseconds timeout before streaming the records (#1056)
* Fetch N number of records in M milliseconds timeout before streaming the records

* Implement the functionality inside socket data streamer

* Reverse the `fetchData` slice

* #run_acceptance_tests

* Trying to fix the tests.
#run_acceptance_tests

* javascript compilation error.

* #run_acceptance_tests

* Name the method better

* Upgrade Basenine version to `v0.8.0`

* Fix some issues related to `Fetch`

* Upgrade the Basenine version in `Dockerfile` as well

* Remove underscore from the parameter name

* Parameterize fetch timeout ms

Co-authored-by: gadotroee <55343099+gadotroee@users.noreply.github.com>
Co-authored-by: Roee Gadot <roee.gadot@up9.com>
2022-05-09 19:11:27 +03:00
M. Mert Yıldıran
e1ad302c29
Make logger a separate module such that don't depend on shared module as a whole for logging (#1047)
* Make `logger` a separate module such that don't depend on `shared` module as a whole for logging

* Update `Dockerfile`
2022-04-27 22:26:27 +03:00
Igor Gov
2de57a6630
Fix: websocket timeouts when many traffic etries sent (#1038) 2022-04-24 11:23:59 +03:00
RamiBerm
76a6a77a14
Refactor ws (#961)
* Separate socket and basenine logic

* WIP

* Update socket_server_handlers.go

* Update socket_data_streamer.go and socket_server_handlers.go

* Update socket_server_handlers.go

* Merge branch 'develop' into refactor_ws
# Please enter a commit message to explain why this merge is necessary,
# especially if it merges an updated upstream into a topic branch.
#
# Lines starting with '#' will be ignored, and an empty message aborts
# the commit.

* empty commit for actions

* empty commit for actions

* commit for actions

* Revert "commit for actions"

This reverts commit 8ba2ecf7d3.

Co-authored-by: RoyUP9 <87927115+RoyUP9@users.noreply.github.com>
2022-04-04 17:33:53 +03:00
M. Mert Yıldıran
774f07fccd
Add /db/flush and /db/reset API endpoints (#896)
* Add `/db/flush` and `/db/reset` API endpoints

* Handle the unmarshalling errors better in the WebSocket

* Handle Basenine connection error better in the WebSocket

* Upgrade to Basenine `v0.6.5`

* Fix the duplicated `StartTime` state

Co-authored-by: gadotroee <55343099+gadotroee@users.noreply.github.com>
2022-03-21 15:54:36 +02:00
M. Mert Yıldıran
dd430c31d5
Always derive the summary and method fields from the entry in the database on read (#877)
* Always derive the summary and method fields from the entry in the database on read

* Update the expected JSONs in the protocol unit tests

* Add test cases for `Summarize` method

* Remove unused `GetEntry` method, `DataUnmarshaler` struct and `UnmarshalData` method

* Temporarily enable the acceptance tests

* Temporarily disable Slack notification on failure

* Update the Cypress tests

* Fix an issue in Redis

* Fix a typo and the Cypress tests

* Revert "Temporarily disable Slack notification on failure"

This reverts commit cad1901ea4.

* Revert "Temporarily enable the acceptance tests"

This reverts commit bad7706c9b.
2022-03-06 15:41:36 +03:00
M. Mert Yıldıran
73f3e448cf
Fix the issue in the WebSocket that's introduced by #819 (#839) 2022-02-22 15:27:50 +03:00
M. Mert Yıldıran
27fa0afb72
TRA-4331 Implement full data streaming over WebSocket (#819)
* Implement full data streaming over WebSocket

* Fix the linting error

* Make the empty being the criteria

* Use a label to break the nested loop
2022-02-17 17:01:44 +03:00
RamiBerm
0a4674ea7c
TRA-4263 extensibility refactor (#770)
* Refactor routers

* refactor extension loading

* refactor server creation code

* Update main.go
2022-02-08 14:59:56 +02:00
Igor Gov
fd64c1bb14
Refactor mizuserver module rename (#742)
Co-authored-by: Igor Gov <igor.govorov1@gmail.com>
2022-02-01 17:19:24 +02:00
Igor Gov
602225bb36
Adding go lint to more modules (#738) 2022-02-01 12:08:55 +02:00
Igor Gov
0f6c56986f
Technical depth: Adding Go linter to CI (#734) 2022-02-01 08:47:26 +02:00
RamiBerm
83c9194703
TRA-4202 role management (#688)
* WIP

* wip

* Update keto.yml, socket_routes.go, and 12 more files...

* fixes and docs

* Update api.js

* Update auth.go and api.js

* Update user_role_provider.go

* Update config_routes.go and api.js

* Update consts.go
2022-01-25 14:25:24 +02:00
RamiBerm
50c89e6245
Revert "TRA-4157 fix ws auth (#669)" (#671)
This reverts commit 676e50b0b1.
2022-01-20 16:29:23 +02:00
RamiBerm
676e50b0b1
TRA-4157 fix ws auth (#669)
* Update socket_routes.go, user_controller.go, and 2 more files...

* Update user_controller.go

* Switch to http-only cookies for more security
2022-01-20 14:10:25 +02:00
M. Mert Yıldıran
80e881fee2
Upgrade Basenine to 0.3.0, do a refactor to enable redact helper and update the cheatsheet (#614)
* Upgrade Basenine version from `0.2.26` to `0.3.0`

* Remove `Summarize` method from `Dissector` interface and refactor the data structures in `tap/api/api.go`

* Rename `MizuEntry` to `Entry` and `BaseEntryDetails` to `BaseEntry`

* Populate `ContractStatus` field as well

* Update the cheatsheet

* Upgrade the Basenine version in the helm chart as well

* Remove a forgoten `console.log` call
2022-01-11 12:51:30 +03:00
RamiBerm
833d08bb40
TRA-4075 integrate kratos user management (#583)
* WIP

* WIP

* WIP

* WIP

* Update App.tsx and Header.tsx

* Update createResources.go, provider.go, and 2 more files...

* WIP

* fix eof newlines

* Fix ts imports, add readiness probe to kratos to prevent mizu being used while kratos isnt ready

* cleaned code

* fix install create namespace

* Update package-lock.json

* Update provider.go

* Update provider.go

* Update provider.go

* Update install_controller.go

* Update kratos.yml

* Update start.sh

* Update provider.go

* Update provider.go

* Update main.go, socket_routes.go, and 8 more files...

* Update App.tsx

* Update installRunner.go

* Update App.tsx
2022-01-05 17:50:27 +02:00
RoyUP9
02b2cbaa03
Added update config route for install mode (#581) 2022-01-04 17:27:14 +02:00
M. Mert Yıldıran
1ef3778051
Add type switch for Base field of MizuEntry (#520) 2021-12-08 16:53:57 +03:00
M. Mert Yıldıran
9f1e311689
TRA-4017 Bring back getOldEntries method using fetch API and always start streaming from now (#518)
* Bring back `getOldEntries` method using fetch API

* Determine no more data on top based on `leftOff` value

* Remove `entriesBuffer` state

* Always open WebSocket with some `leftOff` value

* Rename `leftOff` state to `leftOffBottom`

* Don't set the `focusedEntryId` through WebSocket if the WebSocket is closed

* Call `setQueriedCurrent` with addition

* Close WebSocket upon reaching to top

* Open WebSocket upon snapping to bottom

* Close the WebSocket on snap broken event instead

* Set queried current value to zero upon filter submit

* Upgrade `react-scrollable-feed-virtualized` version and use `scrollToIndex` function

* Change the footer text format

* Improve no more data top logic

* Fix `closeWebSocket()` call logic in `onSnapBrokenEvent` and handle `data.meta` being `null` in `getOldEntries`

* Fix the issues around fetching old records

* Clean up `EntriesList.module.sass`

* Decrement initial `leftOffTop` value by `2`

* Fix the order of `incomingEntries` in `getOldEntries`

* Request `leftOffTop - 1` from `fetchEntries`

* Limit the front-end total entries fetched through WebSocket count to `10000`

* Lose the UI performance gain that's provided by #452

* Revert "Fix the selected entry behavior by propagating the `focusedEntryId` through WebSocket (before #452) TRA-3983 (#513)"

This reverts commit 873f252544.

* Fix the issues caused by 09371f141f

* Upgrade Basenine version from `0.2.13` to `0.2.14`

* Upgrade Basenine version from `0.2.14` to `0.2.15`

* Fix the condition of "Fetch old records" button visibility

* Upgrade Basenine version from `0.2.15` to `0.2.16` and fix the UI code related to fetching old records

* Make `newEntries` constant
2021-12-08 15:19:35 +03:00
M. Mert Yıldıran
873f252544
Fix the selected entry behavior by propagating the focusedEntryId through WebSocket (before #452) TRA-3983 (#513)
* Revert the select entry behavior into its original state RACING! (before #452) [TRA-3983 alternative 3]

* Remove the remaining `forceSelect`(s)

* Add a missing `focusedEntryId` prop

* Fix the race condition

* Propagate the `focusedEntryId` through WebSocket to prevent racing
2021-11-30 15:27:10 +03:00
RoyUP9
763f72a640
remove newline in logs, fixed logs time format (#469) 2021-11-16 12:07:48 +02:00
M. Mert Yıldıran
ed8d36cdad
Send the message into this WebSocket connection instead of all (#449) 2021-11-11 11:10:42 +03:00
M. Mert Yıldıran
d2fe3f6620
Migrate from SQLite to Basenine and introduce a new filtering syntax (#279)
* Fix the OOMKilled error by calling `debug.FreeOSMemory` periodically

* Remove `MAX_NUMBER_OF_GOROUTINES` environment variable

* Change the line

* Increase the default value of `TCP_STREAM_CHANNEL_TIMEOUT_MS` to `10000`

* Write the client and integrate to the new real-time database

* Refactor the WebSocket implementaiton for `/ws`

* Adapt the UI to the new filtering system

* Fix the rest of the issues in the UI

* Increase the buffer of the scanner

* Implement accessing single records

* Increase the buffer of another scanner

* Populate `Request` and `Response` fields of `MizuEntry`

* Add syntax highlighting for the query

* Add database to `Dockerfile`

* Fix some issues

* Update the `realtime_dbms` Git module commit hash

* Upgrade Gin version and print the query string

* Revert "Upgrade Gin version and print the query string"

This reverts commit aa09f904ee.

* Use WebSocket's itself to query instead of the query string

* Fix some errors related to conversion to HAR

* Fix the issues caused by the latest merge

* Fix the build error

* Fix PR validation GitHub workflow

* Replace the git submodule with latest Basenine version `0.1.0`

Remove `realtime_client.go` and use the official client library `github.com/up9inc/basenine/client/go` instead.

* Move Basenine host and port constants to `shared` module

* Reliably execute and wait for Basenine to become available

* Upgrade Basenine version

* Properly close WebSocket and data channel

* Fix the issues caused by the recent merge commit

* Clean up the TypeScript code

* Update `.gitignore`

* Limit the database size

* Add `Macros` method signature to `Dissector` interface and set the macros provided by the protocol extensions

* Run `go mod tidy` on `agent`

* Upgrade `github.com/up9inc/basenine/client/go` version

* Implement a mechanism to update the query using click events in the UI and use it for protocol macros

* Update the query on click to timestamps

* Fix some issues in the WebSocket and channel handling

* Update the query on clicks to status code

* Update the query on clicks to method, path and service

* Update the query on clicks to is outgoing, source and destination ports

* Add an API endpoint to validate the query against syntax errors

* Move the query background color state into `TrafficPage`

* Fix the logic in `setQuery`

* Display a toast message in case of a syntax error in the query

* Remove a call to `fmt.Printf`

* Upgrade Basenine version to `0.1.3`

* Fix an issue related to getting `MAX_ENTRIES_DB_BYTES` environment variable

* Have the `path` key in request details, in HTTP

* Rearrange the HTTP headers for the querying

* Do the same thing for `cookies` and `queryString`

* Update the query on click to table elements

Add the selectors for `TABLE` type representations in HTTP extension.

* Update the query on click to `bodySize` and `elapsedTime` in `EntryTitle`

* Add the selectors for `TABLE` type representations in AMQP extension

* Add the selectors for `TABLE` type representations in Kafka extension

* Add the selectors for `TABLE` type representations in Redis extension

* Define a struct in `tap/api.go` for the section representation data

* Add the selectors for `BODY` type representations

* Add `request.path` to the HTTP request details

* Change the summary string's field name from `path` to `summary`

* Introduce `queryable` CSS class for queryable UI elements and underline them on hover

* Instead of `N requests` at the bottom, make it `Displaying N results (queried X/Y)` and live update the values

Upgrade Basenine version to `0.2.0`.

* Verify the sha256sum of Basenine executable inside `Dockerfile`

* Pass the start time to web UI through WebSocket and always show the `EntriesList` footer

* Pipe the `stderr` of Basenine as well

* Fix the layout issues related to `CodeEditor` in the UI

* Use the correct `shasum` command in `Dockerfile`

* Upgrade Basenine version to `0.2.1`

* Limit the height of `CodeEditor` container

* Remove `Paused` enum `ConnectionStatus` in UI

* Fix the issue caused by the recent merge

* Add the filtering guide (cheatsheet)

* Update open cheatsheet button's title

* Update cheatsheet content

* Remove the old SQLite code, adapt the `--analyze` related code to Basenine

* Change the method signature of `NewEntry`

* Change the method signature of `Represent`

* Introduce `HTTPPair` field in `MizuEntry` specific to HTTP

* Remove `Entry`, `EntryId` and `EstimatedSizeBytes` fields from `MizuEntry`

Also remove the `getEstimatedEntrySizeBytes` method.

* Remove `gorm.io/gorm` dependency

* Remove unused `sensitiveDataFiltering` folder

* Increase the left margin of open cheatsheet button

* Add `overflow: auto` to the cheatsheet `Modal`

* Fix `GetEntry` method

* Fix the macro for gRPC

* Fix an interface conversion in case of AMQP

* Fix two more interface conversion errors in AMQP

* Make the `syncEntriesImpl` method blocking

* Fix a grammar mistake in the cheatsheet

* Adapt to the changes in the recent merge commit

* Improve the cheatsheet text

* Always display the timestamp in `en-US`

* Upgrade Basenine version to `0.2.2`

* Fix the order of closing Basenine connections and channels

* Don't close the Basenine channels at all

* Upgrade Basenine version to `0.2.3`

* Set the initial filter to `rlimit(100)`

* Make Basenine persistent

* Upgrade Basenine version to `0.2.4`

* Update `debug.Dockerfile`

* Fix a failing test

* Upgrade Basenine version to `0.2.5`

* Revert "Do not show play icon when disconnected (#428)"

This reverts commit 8af2e562f8.

* Upgrade Basenine version to `0.2.6`

* Make all non-informative things informative

* Make `100` a constant

* Use `===` in JavaScript no matter what

* Remove a forgotten `console.log`

* Add a comment and update the `query` in `syncEntriesImpl`

* Don't call `panic` in `GetEntry`

* Replace `panic` calls in `startBasenineServer` with `logger.Log.Panicf`

* Remove unnecessary `\n` characters in the logs
2021-11-09 19:54:48 +03:00
M. Mert Yıldıran
147e812edb
Replace all rlog occurrences with the shared logger (#350)
* Replace all `rlog` occurrences with the shared logger

* Use the same log format in `InitLoggerStderrOnly` as well

* Convert one more `log.Fatal` to `logger.Log.Errorf` as well in the `cli`

* Replace `log.` occurrences with `logger.Log.` in `agent`

* Fix `cannot use err (type error)`

* Change the logging level to `DEBUG`

* Replace an `Errorf` with `Fatal`

* Add informative message
2021-10-17 12:15:30 +03:00
Igor Gov
fa632b49a7
Introducing mizu logs dump & Log prints alignment in API server using rlog (#165) 2021-08-05 11:01:08 +03:00
Igor Gov
d18f1f8316
Tapped pods report via endpoint instead of web socket (#164) 2021-08-04 10:41:33 +03:00