Commit Graph

209 Commits

Author SHA1 Message Date
Igor Gov
371e513249
Remove config dependency from basenine init (#846)
* Remove config dependency from basenine init
2022-02-23 10:06:19 +02:00
Igor Gov
97cce32e3f
Fix: service map component aware of agent config (#845)
* Fix: Service map component aware of mizu config
2022-02-23 09:35:05 +02:00
Igor Gov
d2e91b4ffa
Fix: tapper tries to load agent config map (#844) 2022-02-23 09:20:19 +02:00
M. Mert Yıldıran
73f3e448cf
Fix the issue in the WebSocket that's introduced by #819 (#839) 2022-02-22 15:27:50 +03:00
Igor Gov
58a17897cf
Agent logs: output to stderr errors and above (#838)
* Agent logs: output to stderr errors and above
2022-02-22 14:02:51 +02:00
M. Mert Yıldıran
cf3106f636
Silence the logging that comes from Basenine and protocol dissectors (#835) 2022-02-21 18:08:16 +03:00
Andrey Pokhilko
a553a1b683
OAS: use resolved service names (#827)
* OAS service names to be resolved

* fix test
2022-02-21 17:20:59 +03:00
M. Mert Yıldıran
27fa0afb72
TRA-4331 Implement full data streaming over WebSocket (#819)
* Implement full data streaming over WebSocket

* Fix the linting error

* Make the empty being the criteria

* Use a label to break the nested loop
2022-02-17 17:01:44 +03:00
David Levanon
87ef469e25
Add tls tapper (#683)
* initial tls tapper commit

* add tls flag to mizu cli

* support ssl_read_ex/ssl_write_ex

* use hostproc to find libssl

* auto discover tls processes

* support libssl1.0

* recompile ebpf with old clang/llvm

* Update tap/passive_tapper.go

Co-authored-by: M. Mert Yıldıran <mehmet@up9.com>

* Update tap/tlstapper/tls_poller.go

Co-authored-by: M. Mert Yıldıran <mehmet@up9.com>

* Update tap/tlstapper/tls_poller.go

Co-authored-by: M. Mert Yıldıran <mehmet@up9.com>

* Update tap/tlstapper/tls_poller.go

Co-authored-by: M. Mert Yıldıran <mehmet@up9.com>

* Update tap/tlstapper/tls_poller.go

Co-authored-by: M. Mert Yıldıran <mehmet@up9.com>

* Update tap/tlstapper/tls_poller.go

Co-authored-by: M. Mert Yıldıran <mehmet@up9.com>

* Update tap/tlstapper/tls_poller.go

Co-authored-by: M. Mert Yıldıran <mehmet@up9.com>

* Update tap/tlstapper/tls_poller.go

Co-authored-by: M. Mert Yıldıran <mehmet@up9.com>

* upgrade ebpf go lib

* handling big tls messages

* fixing max buffer size in ebpf

* remove unused import

* fix linter issues

* minor pr fixes

* compile with old clang

* fix cgroup file format

* pr fixes + cgroup extract enhance

* fix linter

* adding indirect ebpf dep to agent go.mod

* adding ebpf docker builder

* minor pr fixes

* add req resp matcher to dissect

* rename ssl hooks to ssl hooks structs

* move to alpine, use local copy of mizu instead of git, add readme

* use global req resp mather for tls

Co-authored-by: M. Mert Yıldıran <mehmet@up9.com>
Co-authored-by: gadotroee <55343099+gadotroee@users.noreply.github.com>
2022-02-16 15:34:51 +02:00
Gustavo Massaneiro
bb3ae1ef70
Service Map node key as entry.Name instead of entry.IP (#818) 2022-02-16 09:01:59 +02:00
RamiBerm
4197ec198c
Tag entries destination namespace (#803)
* Update main.go, socket_server_handlers.go, and 7 more files...

* Update cors.go

* omit empty namespace in json so tests pass

* fix indent
2022-02-15 10:51:38 +02:00
Igor Gov
88e3fba1ef
Fix: tapper status race condition (#788) 2022-02-10 12:13:40 +02:00
Igor Gov
9318388101
Update Go dependencies to latest and greatest (#772) 2022-02-09 11:25:41 +02:00
RamiBerm
145004fe43
TRA-4263 revert extensible routing (#774)
* Update main.go, extensions.go, and 2 more files...

* Update config_routes.go, entries_routes.go, and 7 more files...
2022-02-08 17:48:30 +02:00
RamiBerm
0a4674ea7c
TRA-4263 extensibility refactor (#770)
* Refactor routers

* refactor extension loading

* refactor server creation code

* Update main.go
2022-02-08 14:59:56 +02:00
Igor Gov
98ca1b2840
Moving to go 1.17 (#766)
* Updating to go1.17
2022-02-08 11:58:08 +02:00
Andrey Pokhilko
0f77cf8ef9
OAS: Some minor fixes (#762)
* Commit some fixes

* Accept expected

Co-authored-by: Igor Gov <iggvrv@gmail.com>
2022-02-07 10:12:05 +03:00
Igor Gov
bd253e1ff7
Fix cli build param ver (#758)
Co-authored-by: Igor Gov <igor.govorov1@gmail.com>
2022-02-06 20:35:41 +02:00
Andrey Pokhilko
9cddb0c7e8
OAS: Maintain required flag for request body (#752)
* sample data

* Maintain required flag

* accept some changes

* Make required field stable

* Fix an unneeded warning
2022-02-03 11:41:07 +02:00
Andrey Pokhilko
5934be4da6
OAS: handle urlencoded and multipart form data correctly (#730)
* Start with tests

* Expected file

* Almost works

* Make test stab;e

* 5 examples, not 6

* Add param type

* multipart example

* parsing multipart

* Commit

* Commit

* multipart

* Write encoding into schema

* Stable test

* Add reset

* Refactoring

* Refactoring

* Maintain the required flag

* lint
2022-02-03 10:46:31 +02:00
Andrey Pokhilko
a2118b869e
OAS: Improve URL clustering (#711)
* Evidence of problems

* Improving it

* Progressing

* Progress

* Fixes

* progress

* log is not right

* Fixing thresholds

* Almost there

* Imptovements

* Move some false negatives into TODO

* cosmetics

* linter (I disagree)

* linter (I disagree)

* fix test
2022-02-02 12:33:24 +02:00
Andrey Pokhilko
d9ed9098eb
OAS: Count entries per endpoint (#715)
* 0.9 (#37)

* Update .gitignore

* WIP

* WIP

* Update README.md, root.go, and 4 more files...

* Update README.md

* Update README.md

* Update root.go

* Update provider.go

* Update provider.go

* Update root.go, go.mod, and go.sum

* Update mizu.go

* Update go.sum and provider.go

* Update portForward.go, watch.go, and mizu.go

* Update README.md

* Update watch.go

* Update mizu.go

* Update mizu.go

* no message

* no message

* remove unused things and use external for object id (instead of copy)

* no message

* Update mizu.go

* Update go.mod, go.sum, and 2 more files...

* no message

* Update README.md, go.mod, and resolver.go

* Update README.md

* Update go.mod

* Update loader.go

* some refactor

* Update loader.go

* no message

* status to statusCode

* return data directly

* Traffic viewer

* cleaning

* css

* no message

* Clean warnings

* Makefile - first draft

* Update Makefile

* Update Makefile

* Update Makefile, README.md, and 4 more files...

* Add api build and clean to makefile (files restructure) (#9)

* no message
* add clean api command

* no message

* stating with web socket

* Add tap as a separate executable (#10)

* Added tap.

* Ignore build directories.

* Added tapper build to Makefile.

* Improvements  (#12)

* no message

* no message

* Feature/makefile (#11)

* minor fixes

* makefile fixes - docker build

* minor fix in Makefile
Co-authored-by: Alex Haiut <alex@up9.com>

* Update Dockerfile, multi-runner.sh, and 31 more files...

* Update multi-runner.sh

* no message

* Update .dockerignore, Dockerfile, and 30 more files...

* Update cleaner.go, grpc_assembler.go, and 2 more files...

* start the pod with host network and privileged

* fix multi runner passive tapper command

* add HOST_MODE env var

* do not return true in the should tap function

* remove line in the end

* default value in api is input
fix description and pass the parameter in the multi runner script

* missing flag.parse

* no message

* fix image

* Create main.yml

* Update main.yml

* Update main.yml

* Update main.yml

* Update main.yml

* Update main.yml

* Update main.yml

* Update main.yml

* Update main.yml

* Update main.yml

* Update main.yml

* Update main.yml

* Update main.yml

* Update main.yml

* Update main.yml

* Update main.yml

* Update main.yml

* Update main.yml

* Update main.yml

* Update main.yml

* Small fixes - permission + har writing exception (#17)

* Select node by pod (#18)

* Select node by pod.

* Removed watch pod by regex. Irrelevant for now.

* Changed default image to develop:latest.

* Features/clifix (#19)

* makefile fixes - docker build

* readme update, CLI usage fix

* added chmod

Co-authored-by: Alex Haiut <alex@up9.com>

* meta information

* Only record traffic of the requested pod. Filtered by pod IP. (#21)

* fixed readme and reduced batch size to 5 (#22)

Co-authored-by: Alex Haiut <alex@up9.com>

* API and TAP in single process  (#24)

* no message
* no message

* CLI make --pod required flag and faster api image build (#25)

* makefile fixes - docker build

* readme update, CLI usage fix

* added chmod

* typo

* run example incorreect in makefile

* no message

* no message

* no message

Co-authored-by: Alex Haiut <alex@up9.com>

* Reduce delay between tap and UI - Skip dump to file (#26)

* Pass HARs between tap and api via channel.

* Fixed make docker commad.

* Various fixes.

* Added .DS_Store to .gitignore.

* Parse flags in Mizu main instead of in tap_output.go.

* Use channel to pass HAR by default instead of files.

* Infinite scroll (#28)

* no message

* infinite scroll + new ws implementation

* no message

* scrolling top

* fetch button

* more Backend changes

* fix go mod and sum

* mire fixes against develop

* unused code

* small ui refactor

Co-authored-by: Roee Gadot <roee.gadot@up9.com>

* Fix gRPC crash, display gRPC as base64, display gRPC URL and status code (#27)

* Added Method (POST) and URL (emtpy) to gRPC requests.

* Removed quickfix that skips writing HTTP/2 to HAR.

* Use HTTP/2 body to fill out http.Request and htt.Response.

* Make sure that in HARs request.postData.mimeType and response.content.mimeType are application/grpc in case of grpc.

* Comment.

* Add URL and status code for gRPC.

* Don't assume http scheme.

* Use http.Header.Set instead of manually acccessing the underlaying map.

* General stats api fix  (#29)

* refactor and validation

* Show gRPC as ASCII (#31)

* Moved try-catch up one block.

* Display grpc as ASCII.

* Better code in entries fetch endpoint (#30)

* no message
* no message

* Feature/UI/filters (#32)

* UI filters

* refactor

* Revert "refactor"

This reverts commit 70e7d4b6ac.

* remove recursive func

* CLI cleanup (#33)

* Moved cli root command to tap subcommand.

* tap subcommand works.

* Added view and fetch placeholders.

* Updated descriptions.

* Fixed indentation.

* Added versio subcommand.

* Removed version flag.

* gofmt.

* Changed pod from flag to arg.

* Commented out "all namespaces" flag.

* CLI cleanup 2 (#34)

* Renamed dashboard -> GUI/web interface.

* Commented out --quiet, removed unused config variables.

* Quiter output when calling unimplemented subcommands.

* Leftovers from PR #30 (#36)

Co-authored-by: up9-github <info@up9.com>
Co-authored-by: RamiBerm <54766858+RamiBerm@users.noreply.github.com>
Co-authored-by: Liraz Yehezkel <lirazy@up9.com>
Co-authored-by: Alex Haiut <alex@testr.io>
Co-authored-by: lirazyehezkel <61656597+lirazyehezkel@users.noreply.github.com>
Co-authored-by: Alex Haiut <alex@up9.com>
Co-authored-by: nimrod-up9 <59927337+nimrod-up9@users.noreply.github.com>
Co-authored-by: RamiBerm <rami.berman@up9.com>
Co-authored-by: Alex Haiut <alex.haiut@gmail.com>

* remove leftovers (#43)

* end of week - develop to master (#50)

* Provide cli version as git hash from makefile

* Update Makefile, version.go, and 3 more files...

* Update mizuRunner.go

* Update README.md, resolver.go, and 2 more files...

* Update provider.go

* Feature/UI/light theme (#44)

* light theme

* css polish

* unused code

* css

* text shadow

* footer style

* Update mizuRunner.go

* Handle nullable vars (#47)

* Decode gRPC body (#48)

* Decode grpc.

* Better variable names.

* Added protobuf-decoder dependency.

* Updated protobuf-decoder's version.

Co-authored-by: RamiBerm <rami.berman@up9.com>
Co-authored-by: RamiBerm <54766858+RamiBerm@users.noreply.github.com>
Co-authored-by: lirazyehezkel <61656597+lirazyehezkel@users.noreply.github.com>
Co-authored-by: nimrod-up9 <59927337+nimrod-up9@users.noreply.github.com>

* Fix mizu image parameter (#53)

* remove file

* mizu release 2021-06-21 (#79)

* Show pod name and namespace (#61)

* WIP

* Update main.go, consts.go, and 2 more files...

* Update messageSensitiveDataCleaner.go

* Update consts.go and messageSensitiveDataCleaner.go

* Update messageSensitiveDataCleaner.go

* Update main.go, consts.go, and 3 more files...

* WIP

* Update main.go, messageSensitiveDataCleaner.go, and 6 more files...

* Update main.go, messageSensitiveDataCleaner.go, and 3 more files...

* Update consts.go, messageSensitiveDataCleaner.go, and tap.go

* Update provider.go

* Update serializableRegexp.go

* Update tap.go

* TRA-3234 fetch with _source + no hard limit (#64)

* remove the HARD limit of 5000

* TRA-3299 Reduce footprint and Add Tolerances(#65)

* Use lib const for DNSClusterFirstWithHostNet.

* Whitespace.

* Break lines.

* Added affinity to pod names.

* Added tolerations to NoExecute and NoSchedule taints.

* Implementation of Mizu view command

* .

* .

* Update main.go and messageSensitiveDataCleaner.go

* Update main.go

* String and not pointers (#68)

* TRA-3318 - Cookies not null and fix har file names  (#69)

* no message

* TRA-3212 Passive-Tapper and Mizu share code (#70)

* Use log in tap package instead of fmt.

* Moved api/pkg/tap to root.

* Added go.mod and go.sum for tap.

* Added replace for shared.

* api uses tap module instead of tap package.

* Removed dependency of tap in shared by moving env var out of tap.

* Fixed compilation bugs.

* Fixed: Forgot to export struct field HostMode.

* Removed unused flag.

* Close har output channel when done.

* Moved websocket out of mizu and into passive-tapper.

* Send connection details over har output channel.

* Fixed compilation errors.

* Removed unused info from request response cache.

* Renamed connection -> connectionID.

* Fixed rename bug.

* Export setters and getters for filter ips and ports.

* Added tap dependency to Dockerfile.

* Uncomment error messages.

* Renamed `filterIpAddresses` -> `filterAuthorities`.

* Renamed ConnectionID -> ConnectionInfo.

* Fixed: Missed one replace.

* TRA-3342 Mizu/tap dump to har directory fails on Linux (#71)

* Instead of saving incomplete temp har files in a temp dir, save them in the output dir with a *.har.tmp suffix.

* API only loads har from *.har files (by extension).

* Add export entries endpoint for better up9 connect funcionality  (#72)

* no message
* no message
* no message

* Filter 'cookie' header

* Release action  (#73)

* Create main.yml

* Update main.yml

* Update main.yml

* Update main.yml

* Update main.yml

* trying new approach

* no message

* yaml error

* no message

* no message

* no message

* missing )

* no message

* no message

* remove main.yml and fix branches

* Create tag-temp.yaml

* Update tag-temp.yaml

* Update tag-temp.yaml

* no message

* no message

* no message

* no message

* no message

* no message

* no message

* #minor

* no message

* no message

* added checksum calc to CLI makefile

* fixed build error - created bin directory upfront

* using markdown for release text

* use separate checksum files

* fixed release readme

* #minor

* readme updated

Co-authored-by: Alex Haiut <alex@up9.com>

* TRA-3360 Fix: Mizu ignores -n namespace flag and records traffic from all pods (#75)

Do not tap pods in namespaces which were not requested.

* added apple/m1 binary, updated readme (#77)

Co-authored-by: Alex Haiut <alex@up9.com>

* Update README.md (#78)

Co-authored-by: lirazyehezkel <61656597+lirazyehezkel@users.noreply.github.com>
Co-authored-by: RamiBerm <rami.berman@up9.com>
Co-authored-by: RamiBerm <54766858+RamiBerm@users.noreply.github.com>
Co-authored-by: gadotroee <55343099+gadotroee@users.noreply.github.com>
Co-authored-by: nimrod-up9 <59927337+nimrod-up9@users.noreply.github.com>
Co-authored-by: Igor Gov <igor.govorov1@gmail.com>
Co-authored-by: Alex Haiut <alex@up9.com>

* temp fix - ignore agent image in config command (#185)

* removed duplicate har page header (#187)

* Hotfix - ignore not allowed set flags (#192)

#patch

* Add to periodic stats print in tapper (#221)

#patch

* Fixing bad conflict resolution

* some changes in the read me (#241)

change prerequisite to permissions and kubeconfig. These are more FYIs as Mizu requires very little prerequisites. 
Change the description to match getmizu.io

* added tapper count route and wait time for tappers in test (#226)

* Introducing API server provider (#243)

* added tap acceptance tests, fixed duplicate namespace problem (#244)

* added custom config path option (#247)

* Fixing call to analysis (#248)

* no message

* upgrade alpine base image (#413)

* Develop -> main  (#544)

* Add support of listening to multiple netns (#418)

* multiple netns listen - initial commit

* multiple netns listen - actual work

* remove redundant log line

* map /proc of host to tapper

* changing kubernetes provider again after big conflict

* revert node-sass version back to 5.0.0

* Rename host_source to hostSource

Co-authored-by: gadotroee <55343099+gadotroee@users.noreply.github.com>

* PR fixes - adding comment + typos + naming conventions

* go fmt + making procfs read only

* setns back to the original value after packet source initialized

Co-authored-by: gadotroee <55343099+gadotroee@users.noreply.github.com>

* TRA-3842 daemon acceptance tests (#429)

* Update tap_test.go and testsUtils.go

* Update tap_test.go

* Update testsUtils.go

* Update tap_test.go and testsUtils.go

* Update tap_test.go and testsUtils.go

* Update testsUtils.go

* Update tap_test.go

* gofmt

* TRA-3913 support mizu via expose service (#440)

* Update README.md, tapRunner.go, and 4 more files...

* Update testsUtils.go

* Update proxy.go

* Update README.md, testsUtils.go, and 3 more files...

* Update testsUtils.go and provider.go

* fix readme titles (#442)

* Auto close inactive issues  (#441)

* Migrate from SQLite to Basenine and introduce a new filtering syntax (#279)

* Fix the OOMKilled error by calling `debug.FreeOSMemory` periodically

* Remove `MAX_NUMBER_OF_GOROUTINES` environment variable

* Change the line

* Increase the default value of `TCP_STREAM_CHANNEL_TIMEOUT_MS` to `10000`

* Write the client and integrate to the new real-time database

* Refactor the WebSocket implementaiton for `/ws`

* Adapt the UI to the new filtering system

* Fix the rest of the issues in the UI

* Increase the buffer of the scanner

* Implement accessing single records

* Increase the buffer of another scanner

* Populate `Request` and `Response` fields of `MizuEntry`

* Add syntax highlighting for the query

* Add database to `Dockerfile`

* Fix some issues

* Update the `realtime_dbms` Git module commit hash

* Upgrade Gin version and print the query string

* Revert "Upgrade Gin version and print the query string"

This reverts commit aa09f904ee.

* Use WebSocket's itself to query instead of the query string

* Fix some errors related to conversion to HAR

* Fix the issues caused by the latest merge

* Fix the build error

* Fix PR validation GitHub workflow

* Replace the git submodule with latest Basenine version `0.1.0`

Remove `realtime_client.go` and use the official client library `github.com/up9inc/basenine/client/go` instead.

* Move Basenine host and port constants to `shared` module

* Reliably execute and wait for Basenine to become available

* Upgrade Basenine version

* Properly close WebSocket and data channel

* Fix the issues caused by the recent merge commit

* Clean up the TypeScript code

* Update `.gitignore`

* Limit the database size

* Add `Macros` method signature to `Dissector` interface and set the macros provided by the protocol extensions

* Run `go mod tidy` on `agent`

* Upgrade `github.com/up9inc/basenine/client/go` version

* Implement a mechanism to update the query using click events in the UI and use it for protocol macros

* Update the query on click to timestamps

* Fix some issues in the WebSocket and channel handling

* Update the query on clicks to status code

* Update the query on clicks to method, path and service

* Update the query on clicks to is outgoing, source and destination ports

* Add an API endpoint to validate the query against syntax errors

* Move the query background color state into `TrafficPage`

* Fix the logic in `setQuery`

* Display a toast message in case of a syntax error in the query

* Remove a call to `fmt.Printf`

* Upgrade Basenine version to `0.1.3`

* Fix an issue related to getting `MAX_ENTRIES_DB_BYTES` environment variable

* Have the `path` key in request details, in HTTP

* Rearrange the HTTP headers for the querying

* Do the same thing for `cookies` and `queryString`

* Update the query on click to table elements

Add the selectors for `TABLE` type representations in HTTP extension.

* Update the query on click to `bodySize` and `elapsedTime` in `EntryTitle`

* Add the selectors for `TABLE` type representations in AMQP extension

* Add the selectors for `TABLE` type representations in Kafka extension

* Add the selectors for `TABLE` type representations in Redis extension

* Define a struct in `tap/api.go` for the section representation data

* Add the selectors for `BODY` type representations

* Add `request.path` to the HTTP request details

* Change the summary string's field name from `path` to `summary`

* Introduce `queryable` CSS class for queryable UI elements and underline them on hover

* Instead of `N requests` at the bottom, make it `Displaying N results (queried X/Y)` and live update the values

Upgrade Basenine version to `0.2.0`.

* Verify the sha256sum of Basenine executable inside `Dockerfile`

* Pass the start time to web UI through WebSocket and always show the `EntriesList` footer

* Pipe the `stderr` of Basenine as well

* Fix the layout issues related to `CodeEditor` in the UI

* Use the correct `shasum` command in `Dockerfile`

* Upgrade Basenine version to `0.2.1`

* Limit the height of `CodeEditor` container

* Remove `Paused` enum `ConnectionStatus` in UI

* Fix the issue caused by the recent merge

* Add the filtering guide (cheatsheet)

* Update open cheatsheet button's title

* Update cheatsheet content

* Remove the old SQLite code, adapt the `--analyze` related code to Basenine

* Change the method signature of `NewEntry`

* Change the method signature of `Represent`

* Introduce `HTTPPair` field in `MizuEntry` specific to HTTP

* Remove `Entry`, `EntryId` and `EstimatedSizeBytes` fields from `MizuEntry`

Also remove the `getEstimatedEntrySizeBytes` method.

* Remove `gorm.io/gorm` dependency

* Remove unused `sensitiveDataFiltering` folder

* Increase the left margin of open cheatsheet button

* Add `overflow: auto` to the cheatsheet `Modal`

* Fix `GetEntry` method

* Fix the macro for gRPC

* Fix an interface conversion in case of AMQP

* Fix two more interface conversion errors in AMQP

* Make the `syncEntriesImpl` method blocking

* Fix a grammar mistake in the cheatsheet

* Adapt to the changes in the recent merge commit

* Improve the cheatsheet text

* Always display the timestamp in `en-US`

* Upgrade Basenine version to `0.2.2`

* Fix the order of closing Basenine connections and channels

* Don't close the Basenine channels at all

* Upgrade Basenine version to `0.2.3`

* Set the initial filter to `rlimit(100)`

* Make Basenine persistent

* Upgrade Basenine version to `0.2.4`

* Update `debug.Dockerfile`

* Fix a failing test

* Upgrade Basenine version to `0.2.5`

* Revert "Do not show play icon when disconnected (#428)"

This reverts commit 8af2e562f8.

* Upgrade Basenine version to `0.2.6`

* Make all non-informative things informative

* Make `100` a constant

* Use `===` in JavaScript no matter what

* Remove a forgotten `console.log`

* Add a comment and update the `query` in `syncEntriesImpl`

* Don't call `panic` in `GetEntry`

* Replace `panic` calls in `startBasenineServer` with `logger.Log.Panicf`

* Remove unnecessary `\n` characters in the logs

* Remove the `Reconnect` button (#444)

* Upgrade `github.com/up9inc/basenine/client/go` version (#446)

* Fix the `Analysis` button's style into its original state (#447)

* Fix the `Analysis` button's style into its original state

* Fix the MUI button style into its original state

* Fix the acceptance tests after the merger of #279 (#443)

* Enable acceptance tests

* Fix the acceptance tests

* Move `--headless` from `getDefaultCommandArgs` to `getDefaultTapCommandArgs`

* Fix rest of the failing acceptance tests

* Revert "Enable acceptance tests"

This reverts commit 3f919e865a.

* Revert "Revert "Enable acceptance tests""

This reverts commit c0bfe54b70.

* Ignore `--headless` in `mizu view`

* Make all non-informative things informative

* Remove `github.com/stretchr/testify` dependency from the acceptance tests

* Move the helper methods `waitTimeout` and `checkDBHasEntries` from `tap_test.go` to `testsUtils.go`

* Split `checkDBHasEntries` method into `getDBEntries` and `assertEntriesAtLeast` methods

* Revert "Revert "Revert "Enable acceptance tests"""

This reverts commit c13342671c.

* Revert "Revert "Revert "Revert "Enable acceptance tests""""

This reverts commit 0f8c436926.

* Make `getDBEntries` and `checkEntriesAtLeast` methods return errors instead

* Revert "Revert "Revert "Revert "Revert "Enable acceptance tests"""""

This reverts commit 643fdde009.

* Send the message into this WebSocket connection instead of all (#449)

* Fix the CSS issues in the cheatsheet modal (#448)

* Fix the CSS issues in the cheatsheet modal

* Change the Sass variable names

* moved headless to root config, use headless in view (#450)

* extend cleanup timeout to solve context timeout problem in dump logs (#453)

* Add link to exposing mizu wiki page in README (#455)

* changed logger debug mode to log level (#456)

* fixed acceptance test go sum (#458)

* Ignore `SNYK-JS-JSONSCHEMA-1920922` (#462)

Dependency tree:
`node-sass@5.0.0 > node-gyp@7.1.2 > request@2.88.2 > http-signature@1.2.0 > jsprim@1.4.1 > json-schema@0.2.3`

`node-sass` should fix it first.

* Optimize UI entry feed performance (#452)

* Optimize the React code for feeding the entries

By building `EntryItem` only once and updating the `entries` state on meta query messages.

* Upgrade `react-scrollable-feed-virtualized` version from `1.4.3` to `1.4.8`

* Fix the `isSelected` state

* Set the query text before deciding the background to prevent lags while typing

* Upgrade Basenine version from `0.2.6` to `0.2.7`

* Set the query background color only if the query is same after the HTTP request and use `useEffect` instead

* Upgrade Basenine version from `0.2.7` to `0.2.8`

* Use `CancelToken` of `axios` instead of trying to check the query state

* Turn `updateQuery` function into a state hook

* Update the macro for `http`

* Do the `source.cancel()` call in `axios.CancelToken`

* Reduce client-side logging

* Upgrade Basenine version from `0.2.8` to `0.2.9` (#465)

Fixes `limit` helper being not finished because of lack of meta updates.

* Set `response.bodySize` to `0` if it's negative (#466)

* Prevent `elapsedTime` to be negative (#467)

Also fix the `elapsedTime` for Redis.

* changes log format to be more readable (#463)

* Stop reduction of user agent header (#468)

* remove newline in logs, fixed logs time format (#469)

* TRA-3903 better health endpoint for daemon mode (#471)

* Update main.go, status_controller.go, and 2 more files...

* Update status_controller.go and mizuTapperSyncer.go

* fixed redact acceptance test (#472)

* Return `404` instead of `500` if the entry could not be found and display a toast message (#464)

* TRA-3903 add flag to disable pvc creation for daemon mode (#474)

* Update tapRunner.go and tapConfig.go

* Update tapConfig.go

* Revert "Update tapConfig.go"

This reverts commit 5c7c02c4ab.

* TRA-3903 - display targetted pods before waiting for all daemon resources to be created (#475)

* WIP

* Update tapRunner.go

* Update tapRunner.go

* Update the UI screenshots (#476)

* Update the UI screenshots

* Update `mizu-ui.png`

* TRA-3903 fix daemon mode in permission restricted configs (#473)

* Update tapRunner.go, permissions-all-namespaces-daemon.yaml, and 2 more files...

* Update tapRunner.go

* Update tapRunner.go and permissions-ns-daemon.yaml

* Update tapRunner.go

* Update tapRunner.go

* Update tapRunner.go

* TRA-3903 minor daemon mode refactor (#479)

* Update common.go and tapRunner.go

* Update common.go

* Don't omit the key-value pair if the value is `false` in `EntryTableSection` (#478)

* Sync entries in batches just as before (using `uploadIntervalSec` parameter) (#477)

* Sync entries in batches just as before (using `uploadIntervalSec` parameter)

* Replace `lastTimeSynced` value with `time.Time{}`

Since it will be overwritten by the very first iteration.

* Clear `focusedEntryId` state in case of a filter is applied (#482)

* Prevent the crash on client-side in case of `text` being undefined in `FancyTextDisplay` (#481)

* Prevent the crash on client-side in case of `text` being undefined in `FancyTextDisplay`

* Use `String(text)` instead

* Refactor watch pods to allow reusing watch wrapper (#470)

Currently shared/kubernetes/watch.go:FilteredWatch only watches pods.
This PR makes it reusable for other types of resources.
This is done in preparation for watching k8s events.

* Show the source and destination IP in the entry feed (#485)

* Upgrade Basenine version from `0.2.9` to `0.2.10` (#484)

* Upgrade Basenine version from `0.2.9` to `0.2.10`

Fixes the issues in `limit` and `rlimit` helpers that occur when they are on the left operand of a binary expression.

* Upgrade the client hash to latest

* Remove unnecessary `tcpdump` dependency from `Dockerfile` (#491)

* Ignore gob files (#488)

* Ignore gob files

* Remove `*.db` from `.gitignore`

* Update README (#486)

* Add token validity check (#483)

* Add support to auto discover envoy processes (#459)

* discover envoy pids using cluster ips

* add istio flag to cli + rename mtls flag to istio

* add istio.md to docs

* Fixing typos

* Fix minor typos and grammer in docs

Co-authored-by: Nimrod Gilboa Markevich <nimrod@up9.com>

* Improving daemon documentation (#457)

* Some changes to the doc (#494)

* Warn pods not starting (#493)

Print warning event related to mizu k8s resources.
In non-daemon print to CLI. In Daemon print to API-Server logs.

* Remove `tap/tester/` directory (#489)

Co-authored-by: gadotroee <55343099+gadotroee@users.noreply.github.com>

* Disable IPv4 defragmentation and support IPv6 (#487)

* Remove the extra negation on `nodefrag` flag's value

* Support IPv4 fragmentation and IPv6 at the same time

* Re-enable `nodefrag` flag

* Make the `gRPC` and `HTTP/2` distinction (#492)

* Remove the extra negation on `nodefrag` flag's value

* Support IPv4 fragmentation and IPv6 at the same time

* Set `Method` and `StatusCode` fields correctly for `HTTP/2`

* Replace unnecessary `grpc` naming with `http2`

* Make the `gRPC` and `HTTP/2` distinction

* Fix the macros of `http` extension

* Fix the macros of other protocol extensions

* Update the method signature of `Represent`

* Fix the `HTTP/2` support

* Fix some minor issues

* Upgrade Basenine version from `0.2.10` to `0.2.11`

Sorts macros before expanding them and prioritize the long macros.

* Don't regex split the gRPC method name

* Re-enable `nodefrag` flag

* Remove `SetHostname` method in HTTP extension (#496)

* Remove prevPodPhase (#497)

prevPodPhase does not take into account the fact that there may be more
than one tapper pod. Therefore it is not clear what its value
represents. It is only used in a debug print. It is not worth the effort
to fix for that one debug print.

Co-authored-by: gadotroee <55343099+gadotroee@users.noreply.github.com>

* minor logging changes (#499)

Co-authored-by: gadotroee <55343099+gadotroee@users.noreply.github.com>

* Use one channel for events instead of three (#495)

Use one channel for events instead of three separate channels by event type

* Add response body to the error in case of failure (#503)

* add response body to the error in case of failure

* fix typo + make inline condition

* Remove local dev instruction from readme (#507)

* Rename `URL` field to `Target URI` in the UI to prevent confusion (#509)

* Add HTTP2 Over Cleartext (H2C) support (#510)

* Add HTTP2 Over Cleartext (H2C) support

* Remove a parameter which is a remnant of debugging

* Hide `Encoding` field if it's `undefined` or empty in the UI (#511)

* Show the `EntryItem` as `EntrySummary` in `EntryDetailed` (#506)

* Fix the selected entry behavior by propagating the `focusedEntryId` through WebSocket (before #452) TRA-3983 (#513)

* Revert the select entry behavior into its original state RACING! (before #452) [TRA-3983 alternative 3]

* Remove the remaining `forceSelect`(s)

* Add a missing `focusedEntryId` prop

* Fix the race condition

* Propagate the `focusedEntryId` through WebSocket to prevent racing

* Handle unexpected socket close and replace the default `rlimit(100)` filter with `leftOff(-1)` filter (#508)

* Handle unexpected socket close and replace the default `rlimit(100)` filter with `leftOff(-1)` filter

* Rename `dontClear` parameter to `resetEntriesBuffer` and remove negation

* Add `Queryable` component to show a green add circle icon for the queryable UI elements (#512)

* Add `Queryable` component to show a green circle and use it in `EntryViewLine`

* Refactor `Queryable` component

* Use the `Queryable` component `EntryDetailed`

* Use the `Queryable` component `Summary`

* Instead of passing the style to `Queryable`, pass the children components directly

* Make `useTooltip = true` by default in `Queryable`

* Refactor a lot of styling to achieve using `Queryable` in `Protocol` component

* Migrate the last queryable elements in `EntryListItem` to `Queryable` component

* Fix some of the styling issues

* Make horizontal `Protocol` `Queryable` too

* Remove unnecessary child constants

* Revert some of the changes in 2a93f365f5

* Fix rest of the styling issues

* Fix one more styling issue

* Update the screenshots and text in the cheatsheet according to the change

* Use `let` not `var`

* Add missing dependencies to the React hook

* Bring back `GetEntries` HTTP endpoint (#515)

* Bring back `GetEntries` HTTP endpoint

* Upgrade Basenine version from `0.2.12` to `0.2.13`

* Accept negative `leftOff` value

* Remove `max`es from the validations

* Make `timeoutMs` optional

* Update the route comment

* Add `EntriesResponse` struct

* Disable telemetry by env var MIZU_DISABLE_TELEMTRY (#517)

* Replace `privileged` with specific CAPABILITIES requests  (#514)

* Fix the styling of `Queryable` under `StatusCode` and `Summary` components (#519)

* Fix the CSS issue in `Queryable` inside `EntryViewLine` (#521)

* TRA-4017 Bring back `getOldEntries` method using fetch API and always start streaming from now (#518)

* Bring back `getOldEntries` method using fetch API

* Determine no more data on top based on `leftOff` value

* Remove `entriesBuffer` state

* Always open WebSocket with some `leftOff` value

* Rename `leftOff` state to `leftOffBottom`

* Don't set the `focusedEntryId` through WebSocket if the WebSocket is closed

* Call `setQueriedCurrent` with addition

* Close WebSocket upon reaching to top

* Open WebSocket upon snapping to bottom

* Close the WebSocket on snap broken event instead

* Set queried current value to zero upon filter submit

* Upgrade `react-scrollable-feed-virtualized` version and use `scrollToIndex` function

* Change the footer text format

* Improve no more data top logic

* Fix `closeWebSocket()` call logic in `onSnapBrokenEvent` and handle `data.meta` being `null` in `getOldEntries`

* Fix the issues around fetching old records

* Clean up `EntriesList.module.sass`

* Decrement initial `leftOffTop` value by `2`

* Fix the order of `incomingEntries` in `getOldEntries`

* Request `leftOffTop - 1` from `fetchEntries`

* Limit the front-end total entries fetched through WebSocket count to `10000`

* Lose the UI performance gain that's provided by #452

* Revert "Fix the selected entry behavior by propagating the `focusedEntryId` through WebSocket (before #452) TRA-3983 (#513)"

This reverts commit 873f252544.

* Fix the issues caused by 09371f141f

* Upgrade Basenine version from `0.2.13` to `0.2.14`

* Upgrade Basenine version from `0.2.14` to `0.2.15`

* Fix the condition of "Fetch old records" button visibility

* Upgrade Basenine version from `0.2.15` to `0.2.16` and fix the UI code related to fetching old records

* Make `newEntries` constant

* Add type switch for `Base` field of `MizuEntry` (#520)

* Disable version check for devs (#522)

* Report the platform in telemtry (#523)

Co-authored-by: Igor Gov <igor.govorov1@gmail.com>

* Include milliseconds information into the timestamps in the UI (#524)

* Include milliseconds information into the timestamps in the UI

* Upgrade Basenine version from `0.2.16` to `0.2.17`

* Increase the `width` of timestamp

* Fix the CSS issues in queryable vertical protocol element (#526)

* Remove unnecessary fields and split `service` into `src.name` and `dst.name` (#525)

* Remove unnecessary fields and split `service` into `src.name` and `dst.name`

* Don't fall back to IP address but instead display `[Unresolved]` text

* Fix the CSS issues in the plus icon position and replace the separator `->` text with `SwapHorizIcon`

* make description of mizu config options public (#527)

* Fix the glitch (#529)

* Fix the glitch

* Bring back the functionality to "Fetch old records" and "Snap to bottom" buttons

* Fix the CSS issue in `Queryable` component for `src.name` field on heading mode (#530)

* API server stores tappers status (#531)

* Decreased API server boot time (#536)

* Change the connection status text and the toggle connection behavior (#534)

* Update the "Started listening at" timestamp and `queriedTotal` state based on database truncation (#533)

* Send pod info to tapper (#532)

* Alert on acceptance tests failure (#537)

* Fix health tapper status count (#538)

* Fix: acceptance tests (#539)

* Fix a JavaScript error in case of `null` attribute and an interface conversion error in the API server (#540)

* Bringing back the pod watch api server events to make acceptance test more stable (#541)

* TRA-4060 fix proxying error (#542)

* TRA-4062 remove duplicate target pod print (#543)

* Report pods "isTapped" to FE (#535)

* Fix acceptance tests (after pods status request change) (#545)

Co-authored-by: David Levanon <dvdlevanon@gmail.com>
Co-authored-by: gadotroee <55343099+gadotroee@users.noreply.github.com>
Co-authored-by: RamiBerm <54766858+RamiBerm@users.noreply.github.com>
Co-authored-by: M. Mert Yıldıran <mehmet@up9.com>
Co-authored-by: RoyUP9 <87927115+RoyUP9@users.noreply.github.com>
Co-authored-by: Nimrod Gilboa Markevich <59927337+nimrod-up9@users.noreply.github.com>
Co-authored-by: Nimrod Gilboa Markevich <nimrod@up9.com>
Co-authored-by: Alon Girmonsky <1990761+alongir@users.noreply.github.com>
Co-authored-by: Igor Gov <igor.govorov1@gmail.com>
Co-authored-by: Alex Haiut <alex@up9.com>

* no message

* Implement counters per operation

* Add service total counters

* Fix the new method usage

* Fix compile

* Output counters into description

* Add svc-level description

* Auto-rearrange the imports

* Add hits/s

Co-authored-by: gadotroee <55343099+gadotroee@users.noreply.github.com>
Co-authored-by: up9-github <info@up9.com>
Co-authored-by: RamiBerm <54766858+RamiBerm@users.noreply.github.com>
Co-authored-by: Liraz Yehezkel <lirazy@up9.com>
Co-authored-by: Alex Haiut <alex@testr.io>
Co-authored-by: lirazyehezkel <61656597+lirazyehezkel@users.noreply.github.com>
Co-authored-by: Alex Haiut <alex@up9.com>
Co-authored-by: nimrod-up9 <59927337+nimrod-up9@users.noreply.github.com>
Co-authored-by: RamiBerm <rami.berman@up9.com>
Co-authored-by: Alex Haiut <alex.haiut@gmail.com>
Co-authored-by: Roee Gadot <roee.gadot@up9.com>
Co-authored-by: Igor Gov <igor.govorov1@gmail.com>
Co-authored-by: Igor Gov <iggvrv@gmail.com>
Co-authored-by: RoyUP9 <87927115+RoyUP9@users.noreply.github.com>
Co-authored-by: Alon Girmonsky <1990761+alongir@users.noreply.github.com>
Co-authored-by: Roy Island <roy@up9.com>
Co-authored-by: David Levanon <dvdlevanon@gmail.com>
Co-authored-by: M. Mert Yıldıran <mehmet@up9.com>
Co-authored-by: Nimrod Gilboa Markevich <nimrod@up9.com>
2022-02-02 10:36:48 +02:00
Igor Gov
fd64c1bb14
Refactor mizuserver module rename (#742)
Co-authored-by: Igor Gov <igor.govorov1@gmail.com>
2022-02-01 17:19:24 +02:00
Igor Gov
602225bb36
Adding go lint to more modules (#738) 2022-02-01 12:08:55 +02:00
Igor Gov
0f6c56986f
Technical depth: Adding Go linter to CI (#734) 2022-02-01 08:47:26 +02:00
gadotroee
4be7164f20
Fix routing in frontend not working (#723) 2022-01-31 10:49:44 +02:00
Igor Gov
d0ef6c9f97
Remove debug logs from agent (#719) 2022-01-30 12:36:01 +02:00
Igor Gov
0f3dd66d2d
Experimental feature: elastic exporter (#713) 2022-01-30 09:22:13 +02:00
Igor Gov
5536e5bb44
Fixing minor bugs and remove unused dependency (#714) 2022-01-30 08:51:17 +02:00
Andrey Pokhilko
d011478a74
OAS: series of small improvements (#700) 2022-01-27 16:17:21 +02:00
Andrey Pokhilko
0c56c0f541
Have own HAR library code (#695) 2022-01-26 10:57:18 +02:00
Gustavo Massaneiro
843ac722c9
[TRA-4189] Traffic Volume in GB for General Status and telemetry (#696) 2022-01-26 10:49:42 +02:00
M. Mert Yıldıran
a9a61edd50
Add ARM64 and cross-compilation support to the agent image (#659)
* modified Dockerfile to work for both amd64 (Intel) and arm64 (M1)

* added changelog

* Update `Dockerfile` to have `ARCH` build argument

* Remove `docs/CHANGES.md`

* Upgrade the Basenine version from `v0.3.0` to `v0.4.6`

* Update `publish.yml` to have `ARCH` build argument

* Switch `BasenineImageRepo` to Docker Hub

* Have separate build arguments for `ARCH` and `GOARCH`

* Upgrade the Basenine version from `v0.4.6` to `v0.4.10`

* Oops forgot to update the 10th duplicated shell script

* Fix the oopsie and reduce duplications

* Fix `Dockerfile`

* Fix the incompatibility issue between Go plugins and gold linker in Alpine inside `Dockerfile`

* Fix `asm: xxhash_amd64.s:120: when dynamic linking, R15 is clobbered by a global variable access` error

* Update `Dockerfile` to have cross-compilation on an AMD64 machine

Also revert changes in the shell scripts

* Delete `debug.Dockerfile`

* Create a custom base (`debian:buster-slim` based) image for the shipped image

* Replace `mertyildiran/debian-pcap` with `up9inc/debian-pcap`

* Upgrade Basenine version to `v0.4.12`

* Use `debian:stable-slim` as the base

* Fix the indentation in the `Dockerfile`

* Update `publish.yml`

* Enable `publish.yml` for `feature/multiarch_build` branch

* Tag correctly and set `ARCH` Docker argument

* Remove the lines that are forgotten to be removed from the shell scripts

* Add `MizuAgentImageRepo` constant and use it as default `AgentImage` value

* Bring back `Set up Cloud SDK` step to `Build the CLI and publish` job

* Build ARM64 CLI for Linux as well

* Revert "Enable `publish.yml` for `feature/multiarch_build` branch"

This reverts commit d30be4c1f0.

* Revert Go 1.17 upgrade

* Remove `build_extensions_debug.sh` as well

* Make the `Dockerfile` to compile the agent statically

* Statically link the protocol extensions

* Fix `Dockerfile`

* Bring back `-s -w` flags

* Verify the signatures of the downloads in `dockcross/linux-arm64-musl`

* Revert modifications in some shell scripts

* Make the `BUILDARCH` and `TARGETARCH` separation in the `Dockerfile`

* Separate cross-compilation builder image into a separate repo named `up9inc/linux-arm64-musl-go-libpcap`

* Fill the shell script and specify the tag for `dockcross/linux-arm64-musl`

* Remove the unnecessary dependencies from `builder-native-base`

* Improve the comments in the `Dockerfile`

* Upgrade Basenine version to `v0.4.13`

* Fix `Dockerfile`

* Revert "Revert "Enable `publish.yml` for `feature/multiarch_build` branch""

This reverts commit 303e466bdc.

* Revert "Revert "Revert "Enable `publish.yml` for `feature/multiarch_build` branch"""

This reverts commit 0fe252bbdb.

* Remove `push-docker-debug` from the `Makefile`

* Rename `publish.yml` to `release.yml`

Co-authored-by: Alex Haiut <alex@up9.com>
2022-01-25 21:24:50 +03:00
RamiBerm
83c9194703
TRA-4202 role management (#688)
* WIP

* wip

* Update keto.yml, socket_routes.go, and 12 more files...

* fixes and docs

* Update api.js

* Update auth.go and api.js

* Update user_role_provider.go

* Update config_routes.go and api.js

* Update consts.go
2022-01-25 14:25:24 +02:00
RoyUP9
d216c64154
Added support of ui split build (#682) 2022-01-24 10:34:50 +02:00
RamiBerm
50c89e6245
Revert "TRA-4157 fix ws auth (#669)" (#671)
This reverts commit 676e50b0b1.
2022-01-20 16:29:23 +02:00
RamiBerm
676e50b0b1
TRA-4157 fix ws auth (#669)
* Update socket_routes.go, user_controller.go, and 2 more files...

* Update user_controller.go

* Switch to http-only cookies for more security
2022-01-20 14:10:25 +02:00
gadotroee
27dee4e09b
TRA-4193 - Try port forward if proxy is not available (#662) 2022-01-20 11:33:00 +02:00
Gustavo Massaneiro
d5fd2ff1da
Service Map (#623)
* debug builds and gcflags

* update dockerfile for debug

* service map routes and controller

* service map graph structure

* service map interface and new methods

* adding service map edges from mizu entries

* new service map count methods

* implementing the status endpoint

* ServiceMapResponse and ServiceMapEdge models

* service map get endpoint logic

* reset logic and endpoint

* fixed service map get status

* improvements to graph node structure

* front-end implementation and service map buttons

* new render endpoint to render the graph in real time

* spinner sass

* new ServiceMapModal component

* testing react-force-graph-2d lib

* Improvements to service map graph structure, added node id and updated edge source/destination type

* Revert "testing react-force-graph-2d lib"

This reverts commit 1153938386.

* testing react-graph-vis lib

* updated to work with react-graph-vis lib

* removed render endpoint

* go mod tidy

* serviceMap config flag

* using the serviceMap config flag

* passing mizu config to service map as a dependency

* service map tests

* Removed print functions

* finished service map tests

* new service property

* service map controller tests

* moved service map reset button to service map modal
reset closes the modal

* service map modal refresh button and logic

* reset button resets data and refresh

* service map modal close button

* node size/edge size based on the count value
edge color based on protocol

* nodes and edges shadow

* enabled physics to avoid node overlap, changed kafka protocol color to dark green

* showing edges count values and fixed bidirectional edges overlap

* go mod tidy

* removed console.log

* Using the destination node protocol instead of the source node protocol

* Revert "debug builds and gcflags"
Addressed by #624 and #626

This reverts commit 17ecaece3e.

* Revert "update dockerfile for debug"
Addressed by #635

This reverts commit 5dfc15b140.

* using the entire tap Protocol struct instead of only the protocol name

* using the backend protocol background color for node colors

* fixed test, the node list order can change

* re-factoring to get 100% coverage

* using protocol colors just for edges

* re-factored service map to use TCP Entry data. Node key is the entry ip-address instead of the name

* fallback to ip-address when entry name is unresolved

* re-factored front-end

* adjustment to main div style

* added support for multiple protocols for the same edge

* using the item protocol instead of the extension variable

* fixed controller tests

* displaying service name and ip-address on graph nodes

* fixed service map test, we cannot guarantee the slice order

* auth middleware

* created a new pkg for the service map

* re-factoring

* re-factored front-end

* reverting the import order as previous

* Aligning with other UI feature flags handling

* we don't need to get the status anymore, we have window["isServiceMapEnabled"]

* small adjustments

* renamed from .tsx to .ts

* button styles and minor improvements

* moved service map modal from trafficPage to app component

Co-authored-by: Igor Gov <igor.govorov1@gmail.com>
2022-01-19 15:27:12 -03:00
Igor Gov
e15eb71b77
Fix: no panic on failure to sync entries to up9.app (#648) 2022-01-16 11:50:40 +02:00
Igor Gov
8963630e9e
Experimental feature: OAS Generator (#632) 2022-01-13 09:34:55 +02:00
Igor Gov
0e5611b7e9
FE setting experimental feature flags (#633) 2022-01-12 18:05:12 +02:00
RoyUP9
26a9c31d1e
Extracted agent status to consistent volume (#628) 2022-01-12 16:03:50 +02:00
RoyUP9
b96542a8ed
Refactor to agent status (#622) 2022-01-11 20:01:39 +02:00
RoyUP9
a55f51f0e7
Extracted tap config to consistent volume (#617) 2022-01-11 13:44:41 +02:00
M. Mert Yıldıran
80e881fee2
Upgrade Basenine to 0.3.0, do a refactor to enable redact helper and update the cheatsheet (#614)
* Upgrade Basenine version from `0.2.26` to `0.3.0`

* Remove `Summarize` method from `Dissector` interface and refactor the data structures in `tap/api/api.go`

* Rename `MizuEntry` to `Entry` and `BaseEntryDetails` to `BaseEntry`

* Populate `ContractStatus` field as well

* Update the cheatsheet

* Upgrade the Basenine version in the helm chart as well

* Remove a forgoten `console.log` call
2022-01-11 12:51:30 +03:00
Nimrod Gilboa Markevich
5a044875d3
Rename Istio to service mesh (#605)
- Rename --istio flag to the more general --service-mesh
- Rename internal variables, consts and structures to reflect this conceptual change
- Update the docs accordingly
2022-01-09 13:21:14 +02:00
RoyUP9
c49c344c2a
Added kubernetes provider singleton (#599) 2022-01-09 10:50:58 +02:00
M. Mert Yıldıran
e3e9681110
Move Basenine binary into a separate container (#603)
* Move Basenine binary into a separate container

* Set `WorkingDir` to `shared.DataDirPath` in the `basenine` container

* Use `consts.go` to set the Basenine image and port

* Bring back the `net-wait-go` usage to prevent startup failures
2022-01-09 11:18:34 +03:00
RamiBerm
2110afc514
TRA-4075 fix logout (#595) 2022-01-06 14:01:14 +02:00
RoyUP9
2c4a5d06ab
Init tapped namespaces map (#593) 2022-01-06 13:21:58 +02:00
RamiBerm
5d36d9184d
fix enterprise cors (#592) 2022-01-06 12:39:03 +02:00
RamiBerm
63122cb0a7
TRA-4147 simpler kratos password policy (#590) 2022-01-06 12:15:46 +02:00
RoyUP9
b88bdb90f6
Fixed standalone config, small refactor (#589) 2022-01-06 12:04:58 +02:00
RamiBerm
833d08bb40
TRA-4075 integrate kratos user management (#583)
* WIP

* WIP

* WIP

* WIP

* Update App.tsx and Header.tsx

* Update createResources.go, provider.go, and 2 more files...

* WIP

* fix eof newlines

* Fix ts imports, add readiness probe to kratos to prevent mizu being used while kratos isnt ready

* cleaned code

* fix install create namespace

* Update package-lock.json

* Update provider.go

* Update provider.go

* Update provider.go

* Update install_controller.go

* Update kratos.yml

* Update start.sh

* Update provider.go

* Update provider.go

* Update main.go, socket_routes.go, and 8 more files...

* Update App.tsx

* Update installRunner.go

* Update App.tsx
2022-01-05 17:50:27 +02:00
RoyUP9
8e20ca797b
Added endpoint for getting tapped namespaces (#587) 2022-01-05 11:15:42 +02:00
RoyUP9
bf68689212
renamed standalone route to config route (#586) 2022-01-04 18:46:03 +02:00
RoyUP9
02b2cbaa03
Added update config route for install mode (#581) 2022-01-04 17:27:14 +02:00
RoyUP9
482036182b
Added setting ui mode - enterprise or normal (#575) 2021-12-30 14:58:53 +02:00
RoyUP9
de046c65b0
Mizu install mode (#566) 2021-12-28 15:44:37 +02:00
Igor Gov
b039c2abad
Fix: [EKS] server could not find the requested resourceevents.events.k8s.io (#565) 2021-12-28 13:24:51 +02:00
RoyUP9
15895d5947
Changed sync tappers to start only when sync tappers config is true (#562) 2021-12-27 17:19:00 +02:00
RamiBerm
2da7c0f0ed
TRA-4065 fix mizu tapping (#561)
* Update main.go and passive_tapper.go

* Update main.go
2021-12-27 15:11:36 +02:00
RamiBerm
cfe9e863b7
TRA-4065 support inflight tap target update (#556)
* WIP

* WIP

* Update main.go

* Update main.go and passive_tapper.go

* Update passive_tapper.go

* Update passive_tapper.go

* Update passive_tapper.go

* Update passive_tapper.go
2021-12-27 11:50:34 +02:00
gadotroee
2d78785558
Fix acceptance tests (after pods status request change) (#545) 2021-12-19 13:46:14 +02:00
Igor Gov
cba0c682e5
Report pods "isTapped" to FE (#535) 2021-12-19 13:03:53 +02:00
Igor Gov
397d3931ad
Fix health tapper status count (#538) 2021-12-16 12:39:45 +02:00
David Levanon
ab029f4394
Send pod info to tapper (#532) 2021-12-16 10:51:03 +02:00
M. Mert Yıldıran
a06d5cfbde
Update the "Started listening at" timestamp and queriedTotal state based on database truncation (#533) 2021-12-16 10:46:50 +02:00
Igor Gov
90c210452d
API server stores tappers status (#531) 2021-12-15 14:52:49 +02:00
M. Mert Yıldıran
4e17ac5654
Remove unnecessary fields and split service into src.name and dst.name (#525)
* Remove unnecessary fields and split `service` into `src.name` and `dst.name`

* Don't fall back to IP address but instead display `[Unresolved]` text

* Fix the CSS issues in the plus icon position and replace the separator `->` text with `SwapHorizIcon`
2021-12-14 11:36:02 +03:00
M. Mert Yıldıran
1ef3778051
Add type switch for Base field of MizuEntry (#520) 2021-12-08 16:53:57 +03:00
M. Mert Yıldıran
9f1e311689
TRA-4017 Bring back getOldEntries method using fetch API and always start streaming from now (#518)
* Bring back `getOldEntries` method using fetch API

* Determine no more data on top based on `leftOff` value

* Remove `entriesBuffer` state

* Always open WebSocket with some `leftOff` value

* Rename `leftOff` state to `leftOffBottom`

* Don't set the `focusedEntryId` through WebSocket if the WebSocket is closed

* Call `setQueriedCurrent` with addition

* Close WebSocket upon reaching to top

* Open WebSocket upon snapping to bottom

* Close the WebSocket on snap broken event instead

* Set queried current value to zero upon filter submit

* Upgrade `react-scrollable-feed-virtualized` version and use `scrollToIndex` function

* Change the footer text format

* Improve no more data top logic

* Fix `closeWebSocket()` call logic in `onSnapBrokenEvent` and handle `data.meta` being `null` in `getOldEntries`

* Fix the issues around fetching old records

* Clean up `EntriesList.module.sass`

* Decrement initial `leftOffTop` value by `2`

* Fix the order of `incomingEntries` in `getOldEntries`

* Request `leftOffTop - 1` from `fetchEntries`

* Limit the front-end total entries fetched through WebSocket count to `10000`

* Lose the UI performance gain that's provided by #452

* Revert "Fix the selected entry behavior by propagating the `focusedEntryId` through WebSocket (before #452) TRA-3983 (#513)"

This reverts commit 873f252544.

* Fix the issues caused by 09371f141f

* Upgrade Basenine version from `0.2.13` to `0.2.14`

* Upgrade Basenine version from `0.2.14` to `0.2.15`

* Fix the condition of "Fetch old records" button visibility

* Upgrade Basenine version from `0.2.15` to `0.2.16` and fix the UI code related to fetching old records

* Make `newEntries` constant
2021-12-08 15:19:35 +03:00
M. Mert Yıldıran
c47959dbd8
Bring back GetEntries HTTP endpoint (#515)
* Bring back `GetEntries` HTTP endpoint

* Upgrade Basenine version from `0.2.12` to `0.2.13`

* Accept negative `leftOff` value

* Remove `max`es from the validations

* Make `timeoutMs` optional

* Update the route comment

* Add `EntriesResponse` struct
2021-12-01 11:55:13 +03:00
M. Mert Yıldıran
b745f65971
Handle unexpected socket close and replace the default rlimit(100) filter with leftOff(-1) filter (#508)
* Handle unexpected socket close and replace the default `rlimit(100)` filter with `leftOff(-1)` filter

* Rename `dontClear` parameter to `resetEntriesBuffer` and remove negation
2021-11-30 16:30:18 +03:00
M. Mert Yıldıran
873f252544
Fix the selected entry behavior by propagating the focusedEntryId through WebSocket (before #452) TRA-3983 (#513)
* Revert the select entry behavior into its original state RACING! (before #452) [TRA-3983 alternative 3]

* Remove the remaining `forceSelect`(s)

* Add a missing `focusedEntryId` prop

* Fix the race condition

* Propagate the `focusedEntryId` through WebSocket to prevent racing
2021-11-30 15:27:10 +03:00
Nimrod Gilboa Markevich
1c18eb1b84
Use one channel for events instead of three (#495)
Use one channel for events instead of three separate channels by event type
2021-11-23 15:06:27 +02:00
M. Mert Yıldıran
12ca3d8779
Make the gRPC and HTTP/2 distinction (#492)
* Remove the extra negation on `nodefrag` flag's value

* Support IPv4 fragmentation and IPv6 at the same time

* Set `Method` and `StatusCode` fields correctly for `HTTP/2`

* Replace unnecessary `grpc` naming with `http2`

* Make the `gRPC` and `HTTP/2` distinction

* Fix the macros of `http` extension

* Fix the macros of other protocol extensions

* Update the method signature of `Represent`

* Fix the `HTTP/2` support

* Fix some minor issues

* Upgrade Basenine version from `0.2.10` to `0.2.11`

Sorts macros before expanding them and prioritize the long macros.

* Don't regex split the gRPC method name

* Re-enable `nodefrag` flag
2021-11-22 17:46:35 +03:00
Nimrod Gilboa Markevich
b1ad2efb96
Warn pods not starting (#493)
Print warning event related to mizu k8s resources.
In non-daemon print to CLI. In Daemon print to API-Server logs.
2021-11-22 15:30:10 +02:00
David Levanon
6caa94f08f
Add support to auto discover envoy processes (#459)
* discover envoy pids using cluster ips

* add istio flag to cli + rename mtls flag to istio

* add istio.md to docs

* Fixing typos

* Fix minor typos and grammer in docs

Co-authored-by: Nimrod Gilboa Markevich <nimrod@up9.com>
2021-11-21 15:45:07 +02:00
M. Mert Yıldıran
a849aae94c
Upgrade Basenine version from 0.2.9 to 0.2.10 (#484)
* Upgrade Basenine version from `0.2.9` to `0.2.10`

Fixes the issues in `limit` and `rlimit` helpers that occur when they are on the left operand of a binary expression.

* Upgrade the client hash to latest
2021-11-19 18:57:19 +03:00
M. Mert Yıldıran
a13fec3dae
Sync entries in batches just as before (using uploadIntervalSec parameter) (#477)
* Sync entries in batches just as before (using `uploadIntervalSec` parameter)

* Replace `lastTimeSynced` value with `time.Time{}`

Since it will be overwritten by the very first iteration.
2021-11-17 15:16:49 +03:00
M. Mert Yıldıran
7f265dc4c5
Return 404 instead of 500 if the entry could not be found and display a toast message (#464) 2021-11-16 17:13:07 +03:00
RamiBerm
89836d8d75
TRA-3903 better health endpoint for daemon mode (#471)
* Update main.go, status_controller.go, and 2 more files...

* Update status_controller.go and mizuTapperSyncer.go
2021-11-16 15:44:27 +02:00
RoyUP9
763f72a640
remove newline in logs, fixed logs time format (#469) 2021-11-16 12:07:48 +02:00
M. Mert Yıldıran
618cb3a409
Optimize UI entry feed performance (#452)
* Optimize the React code for feeding the entries

By building `EntryItem` only once and updating the `entries` state on meta query messages.

* Upgrade `react-scrollable-feed-virtualized` version from `1.4.3` to `1.4.8`

* Fix the `isSelected` state

* Set the query text before deciding the background to prevent lags while typing

* Upgrade Basenine version from `0.2.6` to `0.2.7`

* Set the query background color only if the query is same after the HTTP request and use `useEffect` instead

* Upgrade Basenine version from `0.2.7` to `0.2.8`

* Use `CancelToken` of `axios` instead of trying to check the query state

* Turn `updateQuery` function into a state hook

* Update the macro for `http`

* Do the `source.cancel()` call in `axios.CancelToken`

* Reduce client-side logging
2021-11-15 17:32:05 +03:00
RoyUP9
14a5fe11e7
changed logger debug mode to log level (#456) 2021-11-14 12:21:48 +02:00
M. Mert Yıldıran
ed8d36cdad
Send the message into this WebSocket connection instead of all (#449) 2021-11-11 11:10:42 +03:00
M. Mert Yıldıran
81c25f0bd4
Upgrade github.com/up9inc/basenine/client/go version (#446) 2021-11-10 17:09:40 +03:00
M. Mert Yıldıran
d2fe3f6620
Migrate from SQLite to Basenine and introduce a new filtering syntax (#279)
* Fix the OOMKilled error by calling `debug.FreeOSMemory` periodically

* Remove `MAX_NUMBER_OF_GOROUTINES` environment variable

* Change the line

* Increase the default value of `TCP_STREAM_CHANNEL_TIMEOUT_MS` to `10000`

* Write the client and integrate to the new real-time database

* Refactor the WebSocket implementaiton for `/ws`

* Adapt the UI to the new filtering system

* Fix the rest of the issues in the UI

* Increase the buffer of the scanner

* Implement accessing single records

* Increase the buffer of another scanner

* Populate `Request` and `Response` fields of `MizuEntry`

* Add syntax highlighting for the query

* Add database to `Dockerfile`

* Fix some issues

* Update the `realtime_dbms` Git module commit hash

* Upgrade Gin version and print the query string

* Revert "Upgrade Gin version and print the query string"

This reverts commit aa09f904ee.

* Use WebSocket's itself to query instead of the query string

* Fix some errors related to conversion to HAR

* Fix the issues caused by the latest merge

* Fix the build error

* Fix PR validation GitHub workflow

* Replace the git submodule with latest Basenine version `0.1.0`

Remove `realtime_client.go` and use the official client library `github.com/up9inc/basenine/client/go` instead.

* Move Basenine host and port constants to `shared` module

* Reliably execute and wait for Basenine to become available

* Upgrade Basenine version

* Properly close WebSocket and data channel

* Fix the issues caused by the recent merge commit

* Clean up the TypeScript code

* Update `.gitignore`

* Limit the database size

* Add `Macros` method signature to `Dissector` interface and set the macros provided by the protocol extensions

* Run `go mod tidy` on `agent`

* Upgrade `github.com/up9inc/basenine/client/go` version

* Implement a mechanism to update the query using click events in the UI and use it for protocol macros

* Update the query on click to timestamps

* Fix some issues in the WebSocket and channel handling

* Update the query on clicks to status code

* Update the query on clicks to method, path and service

* Update the query on clicks to is outgoing, source and destination ports

* Add an API endpoint to validate the query against syntax errors

* Move the query background color state into `TrafficPage`

* Fix the logic in `setQuery`

* Display a toast message in case of a syntax error in the query

* Remove a call to `fmt.Printf`

* Upgrade Basenine version to `0.1.3`

* Fix an issue related to getting `MAX_ENTRIES_DB_BYTES` environment variable

* Have the `path` key in request details, in HTTP

* Rearrange the HTTP headers for the querying

* Do the same thing for `cookies` and `queryString`

* Update the query on click to table elements

Add the selectors for `TABLE` type representations in HTTP extension.

* Update the query on click to `bodySize` and `elapsedTime` in `EntryTitle`

* Add the selectors for `TABLE` type representations in AMQP extension

* Add the selectors for `TABLE` type representations in Kafka extension

* Add the selectors for `TABLE` type representations in Redis extension

* Define a struct in `tap/api.go` for the section representation data

* Add the selectors for `BODY` type representations

* Add `request.path` to the HTTP request details

* Change the summary string's field name from `path` to `summary`

* Introduce `queryable` CSS class for queryable UI elements and underline them on hover

* Instead of `N requests` at the bottom, make it `Displaying N results (queried X/Y)` and live update the values

Upgrade Basenine version to `0.2.0`.

* Verify the sha256sum of Basenine executable inside `Dockerfile`

* Pass the start time to web UI through WebSocket and always show the `EntriesList` footer

* Pipe the `stderr` of Basenine as well

* Fix the layout issues related to `CodeEditor` in the UI

* Use the correct `shasum` command in `Dockerfile`

* Upgrade Basenine version to `0.2.1`

* Limit the height of `CodeEditor` container

* Remove `Paused` enum `ConnectionStatus` in UI

* Fix the issue caused by the recent merge

* Add the filtering guide (cheatsheet)

* Update open cheatsheet button's title

* Update cheatsheet content

* Remove the old SQLite code, adapt the `--analyze` related code to Basenine

* Change the method signature of `NewEntry`

* Change the method signature of `Represent`

* Introduce `HTTPPair` field in `MizuEntry` specific to HTTP

* Remove `Entry`, `EntryId` and `EstimatedSizeBytes` fields from `MizuEntry`

Also remove the `getEstimatedEntrySizeBytes` method.

* Remove `gorm.io/gorm` dependency

* Remove unused `sensitiveDataFiltering` folder

* Increase the left margin of open cheatsheet button

* Add `overflow: auto` to the cheatsheet `Modal`

* Fix `GetEntry` method

* Fix the macro for gRPC

* Fix an interface conversion in case of AMQP

* Fix two more interface conversion errors in AMQP

* Make the `syncEntriesImpl` method blocking

* Fix a grammar mistake in the cheatsheet

* Adapt to the changes in the recent merge commit

* Improve the cheatsheet text

* Always display the timestamp in `en-US`

* Upgrade Basenine version to `0.2.2`

* Fix the order of closing Basenine connections and channels

* Don't close the Basenine channels at all

* Upgrade Basenine version to `0.2.3`

* Set the initial filter to `rlimit(100)`

* Make Basenine persistent

* Upgrade Basenine version to `0.2.4`

* Update `debug.Dockerfile`

* Fix a failing test

* Upgrade Basenine version to `0.2.5`

* Revert "Do not show play icon when disconnected (#428)"

This reverts commit 8af2e562f8.

* Upgrade Basenine version to `0.2.6`

* Make all non-informative things informative

* Make `100` a constant

* Use `===` in JavaScript no matter what

* Remove a forgotten `console.log`

* Add a comment and update the `query` in `syncEntriesImpl`

* Don't call `panic` in `GetEntry`

* Replace `panic` calls in `startBasenineServer` with `logger.Log.Panicf`

* Remove unnecessary `\n` characters in the logs
2021-11-09 19:54:48 +03:00
David Levanon
8a90f02161
Add support of listening to multiple netns (#418)
* multiple netns listen - initial commit

* multiple netns listen - actual work

* remove redundant log line

* map /proc of host to tapper

* changing kubernetes provider again after big conflict

* revert node-sass version back to 5.0.0

* Rename host_source to hostSource

Co-authored-by: gadotroee <55343099+gadotroee@users.noreply.github.com>

* PR fixes - adding comment + typos + naming conventions

* go fmt + making procfs read only

* setns back to the original value after packet source initialized

Co-authored-by: gadotroee <55343099+gadotroee@users.noreply.github.com>
2021-11-07 16:00:59 +02:00
M. Mert Yıldıran
6811bd5050
Remove the ERROR level logs in loadOAS and let the caller handle them in INFO level (#434) 2021-11-07 12:40:57 +03:00
RamiBerm
a3ec5d147e
TRA-3842 daemon mode (#427)
* Update config.go, tapConfig.go, and models.go

* WIP

* Update go.sum

* Update tapRunner.go

* Update tap.go

* WIP

* WIP

* Update Dockerfile, main.go, and 2 more files...

* WIP

* Update utils.go, tapClusterResourceManagement.go, and utils.go

* Merge branch 'develop'

* Update metadata_controller.go, utils.go, and 2 more files...

* Update main.go, utils.go, and tapRunner.go

* Update tapRunner.go

* Update config.go, config.go, and models.go

* Update main.go, main.go, and stats_provider_test.go

* Update provider.go

* bug fixes

* Update main.go, metadata_controller.go, and 13 more files...

* Update metadata_controller.go, status_controller.go, and 4 more files...

* Update main.go, config.go, and 3 more files...

* Update tapRunner.go

* Update config.go, stats_provider_test.go, and consts.go
2021-11-04 11:46:45 +02:00
RamiBerm
655626bc42
TRA-3868 move tapped pod watch and tapper updating to shared (#416)
* WIP

* WIP

* WIP

* WIP

* WIP

* Update tapRunner.go and k8sTapManager.go

* Update cleanRunner.go, common.go, and 8 more files...

* Update common.go, tapConfig.go, and 2 more files...

* Update config.go, config.go, and 5 more files...

* Update tapRunner.go, config.go, and 7 more files...

* Update cleanRunner.go, logs.go, and 2 more files...

* Update k8sTapManager.go, provider.go, and watch.go

* Update go.sum, go.mod, and go.sum

* Update go.mod and go.sum

* Update go.mod, go.sum, and 2 more files...

* Revert "Update go.mod, go.sum, and 2 more files..."

This reverts commit 8140311349.

* Update funcWrappers.go, tapRunner.go, and 4 more files...

* Update main.go, tapRunner.go, and mizuTapperSyncer.go
2021-11-01 14:12:32 +02:00
RamiBerm
35dbd5fde2
TRA-3860 create main configmap for agent and tappers (#410)
* WIP

* Update options.go and serializable_regexp.go

* Update go.sum, go.sum, and 4 more files...

* Update go.sum, go.sum, and 4 more files...

* Update config.go and serializable_regexp.go

* Update config.go, config.json, and test.go

* Update tapRunner.go and provider.go

* Update provider.go

* Update tapRunner.go and provider.go

* Update config.json and test.go

* Update contract_validation.go, config.go, and 2 more files...

* Update main.go

* Update rulesHTTP.go

* Update config.go, size_enforcer.go, and 5 more files...

* Update config.go and config.go

Co-authored-by: Rami Berman <rami.berman@up9.com>
2021-10-31 15:29:05 +02:00
Igor Gov
2c29903910
Run snyk on all project with ignore current issues (#415) 2021-10-28 21:59:42 +03:00
RoyUP9
e3ead981ec
fixed upload entries (#407) 2021-10-28 11:20:45 +03:00
RoyUP9
4f6da91d74
fixed naming of latency to response time (#388) 2021-10-21 12:45:17 +03:00
RoyUP9
b6db64d868
fixed sync entries text (#383) 2021-10-20 12:25:31 +03:00
RamiBerm
160ae77145
TRA-3811 fix service resolving (#382)
Co-authored-by: Rami Berman <rami.berman@up9.com>
2021-10-20 11:50:39 +03:00