Commit Graph

29 Commits

Author SHA1 Message Date
M. Mert Yıldıran
d7fcf273c0
TRA-4494 Remove all non-functional OutboundLink code that was providing /status/recentTLSLinks endpoint (#1008)
* Remove non-critical TLS detected log that causes `slice bounds out of range` error

* Remove all non-functional `OutboundLink` code that was providing `/status/recentTLSLinks` endpoint

* Fix more unused code
2022-04-17 19:01:21 +03:00
M. Mert Yıldıran
774f07fccd
Add /db/flush and /db/reset API endpoints (#896)
* Add `/db/flush` and `/db/reset` API endpoints

* Handle the unmarshalling errors better in the WebSocket

* Handle Basenine connection error better in the WebSocket

* Upgrade to Basenine `v0.6.5`

* Fix the duplicated `StartTime` state

Co-authored-by: gadotroee <55343099+gadotroee@users.noreply.github.com>
2022-03-21 15:54:36 +02:00
RoyUP9
edbe4ab00b
Removed all code related to install (#892) 2022-03-15 16:41:42 +02:00
RamiBerm
145004fe43
TRA-4263 revert extensible routing (#774)
* Update main.go, extensions.go, and 2 more files...

* Update config_routes.go, entries_routes.go, and 7 more files...
2022-02-08 17:48:30 +02:00
RamiBerm
0a4674ea7c
TRA-4263 extensibility refactor (#770)
* Refactor routers

* refactor extension loading

* refactor server creation code

* Update main.go
2022-02-08 14:59:56 +02:00
Igor Gov
fd64c1bb14
Refactor mizuserver module rename (#742)
Co-authored-by: Igor Gov <igor.govorov1@gmail.com>
2022-02-01 17:19:24 +02:00
gadotroee
4be7164f20
Fix routing in frontend not working (#723) 2022-01-31 10:49:44 +02:00
RamiBerm
83c9194703
TRA-4202 role management (#688)
* WIP

* wip

* Update keto.yml, socket_routes.go, and 12 more files...

* fixes and docs

* Update api.js

* Update auth.go and api.js

* Update user_role_provider.go

* Update config_routes.go and api.js

* Update consts.go
2022-01-25 14:25:24 +02:00
Gustavo Massaneiro
d5fd2ff1da
Service Map (#623)
* debug builds and gcflags

* update dockerfile for debug

* service map routes and controller

* service map graph structure

* service map interface and new methods

* adding service map edges from mizu entries

* new service map count methods

* implementing the status endpoint

* ServiceMapResponse and ServiceMapEdge models

* service map get endpoint logic

* reset logic and endpoint

* fixed service map get status

* improvements to graph node structure

* front-end implementation and service map buttons

* new render endpoint to render the graph in real time

* spinner sass

* new ServiceMapModal component

* testing react-force-graph-2d lib

* Improvements to service map graph structure, added node id and updated edge source/destination type

* Revert "testing react-force-graph-2d lib"

This reverts commit 1153938386.

* testing react-graph-vis lib

* updated to work with react-graph-vis lib

* removed render endpoint

* go mod tidy

* serviceMap config flag

* using the serviceMap config flag

* passing mizu config to service map as a dependency

* service map tests

* Removed print functions

* finished service map tests

* new service property

* service map controller tests

* moved service map reset button to service map modal
reset closes the modal

* service map modal refresh button and logic

* reset button resets data and refresh

* service map modal close button

* node size/edge size based on the count value
edge color based on protocol

* nodes and edges shadow

* enabled physics to avoid node overlap, changed kafka protocol color to dark green

* showing edges count values and fixed bidirectional edges overlap

* go mod tidy

* removed console.log

* Using the destination node protocol instead of the source node protocol

* Revert "debug builds and gcflags"
Addressed by #624 and #626

This reverts commit 17ecaece3e.

* Revert "update dockerfile for debug"
Addressed by #635

This reverts commit 5dfc15b140.

* using the entire tap Protocol struct instead of only the protocol name

* using the backend protocol background color for node colors

* fixed test, the node list order can change

* re-factoring to get 100% coverage

* using protocol colors just for edges

* re-factored service map to use TCP Entry data. Node key is the entry ip-address instead of the name

* fallback to ip-address when entry name is unresolved

* re-factored front-end

* adjustment to main div style

* added support for multiple protocols for the same edge

* using the item protocol instead of the extension variable

* fixed controller tests

* displaying service name and ip-address on graph nodes

* fixed service map test, we cannot guarantee the slice order

* auth middleware

* created a new pkg for the service map

* re-factoring

* re-factored front-end

* reverting the import order as previous

* Aligning with other UI feature flags handling

* we don't need to get the status anymore, we have window["isServiceMapEnabled"]

* small adjustments

* renamed from .tsx to .ts

* button styles and minor improvements

* moved service map modal from trafficPage to app component

Co-authored-by: Igor Gov <igor.govorov1@gmail.com>
2022-01-19 15:27:12 -03:00
Igor Gov
8963630e9e
Experimental feature: OAS Generator (#632) 2022-01-13 09:34:55 +02:00
RoyUP9
26a9c31d1e
Extracted agent status to consistent volume (#628) 2022-01-12 16:03:50 +02:00
RoyUP9
b88bdb90f6
Fixed standalone config, small refactor (#589) 2022-01-06 12:04:58 +02:00
RamiBerm
833d08bb40
TRA-4075 integrate kratos user management (#583)
* WIP

* WIP

* WIP

* WIP

* Update App.tsx and Header.tsx

* Update createResources.go, provider.go, and 2 more files...

* WIP

* fix eof newlines

* Fix ts imports, add readiness probe to kratos to prevent mizu being used while kratos isnt ready

* cleaned code

* fix install create namespace

* Update package-lock.json

* Update provider.go

* Update provider.go

* Update provider.go

* Update install_controller.go

* Update kratos.yml

* Update start.sh

* Update provider.go

* Update provider.go

* Update main.go, socket_routes.go, and 8 more files...

* Update App.tsx

* Update installRunner.go

* Update App.tsx
2022-01-05 17:50:27 +02:00
RoyUP9
bf68689212
renamed standalone route to config route (#586) 2022-01-04 18:46:03 +02:00
RoyUP9
02b2cbaa03
Added update config route for install mode (#581) 2022-01-04 17:27:14 +02:00
Igor Gov
90c210452d
API server stores tappers status (#531) 2021-12-15 14:52:49 +02:00
M. Mert Yıldıran
c47959dbd8
Bring back GetEntries HTTP endpoint (#515)
* Bring back `GetEntries` HTTP endpoint

* Upgrade Basenine version from `0.2.12` to `0.2.13`

* Accept negative `leftOff` value

* Remove `max`es from the validations

* Make `timeoutMs` optional

* Update the route comment

* Add `EntriesResponse` struct
2021-12-01 11:55:13 +03:00
M. Mert Yıldıran
d2fe3f6620
Migrate from SQLite to Basenine and introduce a new filtering syntax (#279)
* Fix the OOMKilled error by calling `debug.FreeOSMemory` periodically

* Remove `MAX_NUMBER_OF_GOROUTINES` environment variable

* Change the line

* Increase the default value of `TCP_STREAM_CHANNEL_TIMEOUT_MS` to `10000`

* Write the client and integrate to the new real-time database

* Refactor the WebSocket implementaiton for `/ws`

* Adapt the UI to the new filtering system

* Fix the rest of the issues in the UI

* Increase the buffer of the scanner

* Implement accessing single records

* Increase the buffer of another scanner

* Populate `Request` and `Response` fields of `MizuEntry`

* Add syntax highlighting for the query

* Add database to `Dockerfile`

* Fix some issues

* Update the `realtime_dbms` Git module commit hash

* Upgrade Gin version and print the query string

* Revert "Upgrade Gin version and print the query string"

This reverts commit aa09f904ee.

* Use WebSocket's itself to query instead of the query string

* Fix some errors related to conversion to HAR

* Fix the issues caused by the latest merge

* Fix the build error

* Fix PR validation GitHub workflow

* Replace the git submodule with latest Basenine version `0.1.0`

Remove `realtime_client.go` and use the official client library `github.com/up9inc/basenine/client/go` instead.

* Move Basenine host and port constants to `shared` module

* Reliably execute and wait for Basenine to become available

* Upgrade Basenine version

* Properly close WebSocket and data channel

* Fix the issues caused by the recent merge commit

* Clean up the TypeScript code

* Update `.gitignore`

* Limit the database size

* Add `Macros` method signature to `Dissector` interface and set the macros provided by the protocol extensions

* Run `go mod tidy` on `agent`

* Upgrade `github.com/up9inc/basenine/client/go` version

* Implement a mechanism to update the query using click events in the UI and use it for protocol macros

* Update the query on click to timestamps

* Fix some issues in the WebSocket and channel handling

* Update the query on clicks to status code

* Update the query on clicks to method, path and service

* Update the query on clicks to is outgoing, source and destination ports

* Add an API endpoint to validate the query against syntax errors

* Move the query background color state into `TrafficPage`

* Fix the logic in `setQuery`

* Display a toast message in case of a syntax error in the query

* Remove a call to `fmt.Printf`

* Upgrade Basenine version to `0.1.3`

* Fix an issue related to getting `MAX_ENTRIES_DB_BYTES` environment variable

* Have the `path` key in request details, in HTTP

* Rearrange the HTTP headers for the querying

* Do the same thing for `cookies` and `queryString`

* Update the query on click to table elements

Add the selectors for `TABLE` type representations in HTTP extension.

* Update the query on click to `bodySize` and `elapsedTime` in `EntryTitle`

* Add the selectors for `TABLE` type representations in AMQP extension

* Add the selectors for `TABLE` type representations in Kafka extension

* Add the selectors for `TABLE` type representations in Redis extension

* Define a struct in `tap/api.go` for the section representation data

* Add the selectors for `BODY` type representations

* Add `request.path` to the HTTP request details

* Change the summary string's field name from `path` to `summary`

* Introduce `queryable` CSS class for queryable UI elements and underline them on hover

* Instead of `N requests` at the bottom, make it `Displaying N results (queried X/Y)` and live update the values

Upgrade Basenine version to `0.2.0`.

* Verify the sha256sum of Basenine executable inside `Dockerfile`

* Pass the start time to web UI through WebSocket and always show the `EntriesList` footer

* Pipe the `stderr` of Basenine as well

* Fix the layout issues related to `CodeEditor` in the UI

* Use the correct `shasum` command in `Dockerfile`

* Upgrade Basenine version to `0.2.1`

* Limit the height of `CodeEditor` container

* Remove `Paused` enum `ConnectionStatus` in UI

* Fix the issue caused by the recent merge

* Add the filtering guide (cheatsheet)

* Update open cheatsheet button's title

* Update cheatsheet content

* Remove the old SQLite code, adapt the `--analyze` related code to Basenine

* Change the method signature of `NewEntry`

* Change the method signature of `Represent`

* Introduce `HTTPPair` field in `MizuEntry` specific to HTTP

* Remove `Entry`, `EntryId` and `EstimatedSizeBytes` fields from `MizuEntry`

Also remove the `getEstimatedEntrySizeBytes` method.

* Remove `gorm.io/gorm` dependency

* Remove unused `sensitiveDataFiltering` folder

* Increase the left margin of open cheatsheet button

* Add `overflow: auto` to the cheatsheet `Modal`

* Fix `GetEntry` method

* Fix the macro for gRPC

* Fix an interface conversion in case of AMQP

* Fix two more interface conversion errors in AMQP

* Make the `syncEntriesImpl` method blocking

* Fix a grammar mistake in the cheatsheet

* Adapt to the changes in the recent merge commit

* Improve the cheatsheet text

* Always display the timestamp in `en-US`

* Upgrade Basenine version to `0.2.2`

* Fix the order of closing Basenine connections and channels

* Don't close the Basenine channels at all

* Upgrade Basenine version to `0.2.3`

* Set the initial filter to `rlimit(100)`

* Make Basenine persistent

* Upgrade Basenine version to `0.2.4`

* Update `debug.Dockerfile`

* Fix a failing test

* Upgrade Basenine version to `0.2.5`

* Revert "Do not show play icon when disconnected (#428)"

This reverts commit 8af2e562f8.

* Upgrade Basenine version to `0.2.6`

* Make all non-informative things informative

* Make `100` a constant

* Use `===` in JavaScript no matter what

* Remove a forgotten `console.log`

* Add a comment and update the `query` in `syncEntriesImpl`

* Don't call `panic` in `GetEntry`

* Replace `panic` calls in `startBasenineServer` with `logger.Log.Panicf`

* Remove unnecessary `\n` characters in the logs
2021-11-09 19:54:48 +03:00
RamiBerm
a3ec5d147e
TRA-3842 daemon mode (#427)
* Update config.go, tapConfig.go, and models.go

* WIP

* Update go.sum

* Update tapRunner.go

* Update tap.go

* WIP

* WIP

* Update Dockerfile, main.go, and 2 more files...

* WIP

* Update utils.go, tapClusterResourceManagement.go, and utils.go

* Merge branch 'develop'

* Update metadata_controller.go, utils.go, and 2 more files...

* Update main.go, utils.go, and tapRunner.go

* Update tapRunner.go

* Update config.go, config.go, and models.go

* Update main.go, main.go, and stats_provider_test.go

* Update provider.go

* bug fixes

* Update main.go, metadata_controller.go, and 13 more files...

* Update metadata_controller.go, status_controller.go, and 4 more files...

* Update main.go, config.go, and 3 more files...

* Update tapRunner.go

* Update config.go, stats_provider_test.go, and consts.go
2021-11-04 11:46:45 +02:00
RoyUP9
26788bb3a6
organize routes (#348) 2021-10-13 17:31:15 +03:00
RoyUP9
b40104b74c
changed sync entries to start on startup (#344) 2021-10-13 11:48:42 +03:00
RoyUP9
837e35255b
auth status route to api server (#342) 2021-10-12 11:03:58 +03:00
RoyUP9
8b8c4609ce
renamed upload entries to sync entries (#330) 2021-10-07 18:33:14 +03:00
gadotroee
073b0b72d3
Remove fetch command (and direction) (#264) 2021-09-06 10:16:04 +03:00
RoyUP9
afd5757315
added tapper count route and wait time for tappers in test (#226) 2021-08-22 11:38:19 +03:00
Igor Gov
d18f1f8316
Tapped pods report via endpoint instead of web socket (#164) 2021-08-04 10:41:33 +03:00
RamiBerm
ceb8d714e3
TRA-3420 inform user of tls traffic (#149)
* Update passive_tapper.go and tls_utils.go

* Update go.mod, go.sum, and 18 more files...

* go fmt

* Update http_reader.go, passive_tapper.go, and 3 more files...

* Update status_controller.go and status_provider.go

Co-authored-by: RamiBerm <rami.berman@up9.com>
2021-08-01 14:57:43 +03:00
RamiBerm
6dd2bf705b
TRA-3437 switch fiber and ikisocket with gin-gonic and gorilla websocket (#136)
* WIP

* WIP

* WIP

* Update socket_server_handlers.go and socket_routes.go

* Fix stuck sockets

* Update go.mod, go.sum, and 5 more files...

* Update socket_routes.go

* Update Dockerfile, go.sum, and fiber_middleware.go

* fix analyze

Co-authored-by: RamiBerm <rami.berman@up9.com>
2021-07-25 13:08:29 +03:00
nimrod-up9
803681a239
Renamed collector, aggregator to api server, api folder to agent (#133)
* Renamed aggregator -> apiServer.

* Format errors with container names.

* Renamed collector -> apiServer.

* Rephrased help messages.

* Moved api -> agent.

* Continue renameing api -> agent in Makefile and Dockerfiles.
2021-07-22 17:17:17 +03:00