Commit Graph

368 Commits

Author SHA1 Message Date
Nimrod Gilboa Markevich
4c97316c02
Remove prevPodPhase (#497)
prevPodPhase does not take into account the fact that there may be more
than one tapper pod. Therefore it is not clear what its value
represents. It is only used in a debug print. It is not worth the effort
to fix for that one debug print.

Co-authored-by: gadotroee <55343099+gadotroee@users.noreply.github.com>
2021-11-23 10:03:36 +02:00
Nimrod Gilboa Markevich
b1ad2efb96
Warn pods not starting (#493)
Print warning event related to mizu k8s resources.
In non-daemon print to CLI. In Daemon print to API-Server logs.
2021-11-22 15:30:10 +02:00
David Levanon
6caa94f08f
Add support to auto discover envoy processes (#459)
* discover envoy pids using cluster ips

* add istio flag to cli + rename mtls flag to istio

* add istio.md to docs

* Fixing typos

* Fix minor typos and grammer in docs

Co-authored-by: Nimrod Gilboa Markevich <nimrod@up9.com>
2021-11-21 15:45:07 +02:00
RoyUP9
b77ea63f42
Add token validity check (#483) 2021-11-21 15:14:02 +02:00
Nimrod Gilboa Markevich
2e75834dd0
Refactor watch pods to allow reusing watch wrapper (#470)
Currently shared/kubernetes/watch.go:FilteredWatch only watches pods.
This PR makes it reusable for other types of resources.
This is done in preparation for watching k8s events.
2021-11-18 11:53:11 +02:00
RamiBerm
18be46809e
TRA-3903 minor daemon mode refactor (#479)
* Update common.go and tapRunner.go

* Update common.go
2021-11-17 11:18:08 +02:00
RamiBerm
b7f7daa05c
TRA-3903 fix daemon mode in permission restricted configs (#473)
* Update tapRunner.go, permissions-all-namespaces-daemon.yaml, and 2 more files...

* Update tapRunner.go

* Update tapRunner.go and permissions-ns-daemon.yaml

* Update tapRunner.go

* Update tapRunner.go

* Update tapRunner.go
2021-11-17 11:14:43 +02:00
RamiBerm
36077a9985
TRA-3903 - display targetted pods before waiting for all daemon resources to be created (#475)
* WIP

* Update tapRunner.go

* Update tapRunner.go
2021-11-16 17:53:38 +02:00
RamiBerm
51e0dd8ba9
TRA-3903 add flag to disable pvc creation for daemon mode (#474)
* Update tapRunner.go and tapConfig.go

* Update tapConfig.go

* Revert "Update tapConfig.go"

This reverts commit 5c7c02c4ab.
2021-11-16 17:11:47 +02:00
RoyUP9
763f72a640
remove newline in logs, fixed logs time format (#469) 2021-11-16 12:07:48 +02:00
RoyUP9
14a5fe11e7
changed logger debug mode to log level (#456) 2021-11-14 12:21:48 +02:00
RoyUP9
3e132905ce
extend cleanup timeout to solve context timeout problem in dump logs (#453) 2021-11-11 14:30:35 +02:00
RoyUP9
ea0b3fb34e
moved headless to root config, use headless in view (#450) 2021-11-11 12:11:02 +02:00
M. Mert Yıldıran
1ee8fb6292
Fix the acceptance tests after the merger of #279 (#443)
* Enable acceptance tests

* Fix the acceptance tests

* Move `--headless` from `getDefaultCommandArgs` to `getDefaultTapCommandArgs`

* Fix rest of the failing acceptance tests

* Revert "Enable acceptance tests"

This reverts commit 3f919e865a.

* Revert "Revert "Enable acceptance tests""

This reverts commit c0bfe54b70.

* Ignore `--headless` in `mizu view`

* Make all non-informative things informative

* Remove `github.com/stretchr/testify` dependency from the acceptance tests

* Move the helper methods `waitTimeout` and `checkDBHasEntries` from `tap_test.go` to `testsUtils.go`

* Split `checkDBHasEntries` method into `getDBEntries` and `assertEntriesAtLeast` methods

* Revert "Revert "Revert "Enable acceptance tests"""

This reverts commit c13342671c.

* Revert "Revert "Revert "Revert "Enable acceptance tests""""

This reverts commit 0f8c436926.

* Make `getDBEntries` and `checkEntriesAtLeast` methods return errors instead

* Revert "Revert "Revert "Revert "Revert "Enable acceptance tests"""""

This reverts commit 643fdde009.
2021-11-10 18:14:04 +03:00
M. Mert Yıldıran
d2fe3f6620
Migrate from SQLite to Basenine and introduce a new filtering syntax (#279)
* Fix the OOMKilled error by calling `debug.FreeOSMemory` periodically

* Remove `MAX_NUMBER_OF_GOROUTINES` environment variable

* Change the line

* Increase the default value of `TCP_STREAM_CHANNEL_TIMEOUT_MS` to `10000`

* Write the client and integrate to the new real-time database

* Refactor the WebSocket implementaiton for `/ws`

* Adapt the UI to the new filtering system

* Fix the rest of the issues in the UI

* Increase the buffer of the scanner

* Implement accessing single records

* Increase the buffer of another scanner

* Populate `Request` and `Response` fields of `MizuEntry`

* Add syntax highlighting for the query

* Add database to `Dockerfile`

* Fix some issues

* Update the `realtime_dbms` Git module commit hash

* Upgrade Gin version and print the query string

* Revert "Upgrade Gin version and print the query string"

This reverts commit aa09f904ee.

* Use WebSocket's itself to query instead of the query string

* Fix some errors related to conversion to HAR

* Fix the issues caused by the latest merge

* Fix the build error

* Fix PR validation GitHub workflow

* Replace the git submodule with latest Basenine version `0.1.0`

Remove `realtime_client.go` and use the official client library `github.com/up9inc/basenine/client/go` instead.

* Move Basenine host and port constants to `shared` module

* Reliably execute and wait for Basenine to become available

* Upgrade Basenine version

* Properly close WebSocket and data channel

* Fix the issues caused by the recent merge commit

* Clean up the TypeScript code

* Update `.gitignore`

* Limit the database size

* Add `Macros` method signature to `Dissector` interface and set the macros provided by the protocol extensions

* Run `go mod tidy` on `agent`

* Upgrade `github.com/up9inc/basenine/client/go` version

* Implement a mechanism to update the query using click events in the UI and use it for protocol macros

* Update the query on click to timestamps

* Fix some issues in the WebSocket and channel handling

* Update the query on clicks to status code

* Update the query on clicks to method, path and service

* Update the query on clicks to is outgoing, source and destination ports

* Add an API endpoint to validate the query against syntax errors

* Move the query background color state into `TrafficPage`

* Fix the logic in `setQuery`

* Display a toast message in case of a syntax error in the query

* Remove a call to `fmt.Printf`

* Upgrade Basenine version to `0.1.3`

* Fix an issue related to getting `MAX_ENTRIES_DB_BYTES` environment variable

* Have the `path` key in request details, in HTTP

* Rearrange the HTTP headers for the querying

* Do the same thing for `cookies` and `queryString`

* Update the query on click to table elements

Add the selectors for `TABLE` type representations in HTTP extension.

* Update the query on click to `bodySize` and `elapsedTime` in `EntryTitle`

* Add the selectors for `TABLE` type representations in AMQP extension

* Add the selectors for `TABLE` type representations in Kafka extension

* Add the selectors for `TABLE` type representations in Redis extension

* Define a struct in `tap/api.go` for the section representation data

* Add the selectors for `BODY` type representations

* Add `request.path` to the HTTP request details

* Change the summary string's field name from `path` to `summary`

* Introduce `queryable` CSS class for queryable UI elements and underline them on hover

* Instead of `N requests` at the bottom, make it `Displaying N results (queried X/Y)` and live update the values

Upgrade Basenine version to `0.2.0`.

* Verify the sha256sum of Basenine executable inside `Dockerfile`

* Pass the start time to web UI through WebSocket and always show the `EntriesList` footer

* Pipe the `stderr` of Basenine as well

* Fix the layout issues related to `CodeEditor` in the UI

* Use the correct `shasum` command in `Dockerfile`

* Upgrade Basenine version to `0.2.1`

* Limit the height of `CodeEditor` container

* Remove `Paused` enum `ConnectionStatus` in UI

* Fix the issue caused by the recent merge

* Add the filtering guide (cheatsheet)

* Update open cheatsheet button's title

* Update cheatsheet content

* Remove the old SQLite code, adapt the `--analyze` related code to Basenine

* Change the method signature of `NewEntry`

* Change the method signature of `Represent`

* Introduce `HTTPPair` field in `MizuEntry` specific to HTTP

* Remove `Entry`, `EntryId` and `EstimatedSizeBytes` fields from `MizuEntry`

Also remove the `getEstimatedEntrySizeBytes` method.

* Remove `gorm.io/gorm` dependency

* Remove unused `sensitiveDataFiltering` folder

* Increase the left margin of open cheatsheet button

* Add `overflow: auto` to the cheatsheet `Modal`

* Fix `GetEntry` method

* Fix the macro for gRPC

* Fix an interface conversion in case of AMQP

* Fix two more interface conversion errors in AMQP

* Make the `syncEntriesImpl` method blocking

* Fix a grammar mistake in the cheatsheet

* Adapt to the changes in the recent merge commit

* Improve the cheatsheet text

* Always display the timestamp in `en-US`

* Upgrade Basenine version to `0.2.2`

* Fix the order of closing Basenine connections and channels

* Don't close the Basenine channels at all

* Upgrade Basenine version to `0.2.3`

* Set the initial filter to `rlimit(100)`

* Make Basenine persistent

* Upgrade Basenine version to `0.2.4`

* Update `debug.Dockerfile`

* Fix a failing test

* Upgrade Basenine version to `0.2.5`

* Revert "Do not show play icon when disconnected (#428)"

This reverts commit 8af2e562f8.

* Upgrade Basenine version to `0.2.6`

* Make all non-informative things informative

* Make `100` a constant

* Use `===` in JavaScript no matter what

* Remove a forgotten `console.log`

* Add a comment and update the `query` in `syncEntriesImpl`

* Don't call `panic` in `GetEntry`

* Replace `panic` calls in `startBasenineServer` with `logger.Log.Panicf`

* Remove unnecessary `\n` characters in the logs
2021-11-09 19:54:48 +03:00
M. Mert Yıldıran
4009386d82
Set a default value for SEM_VER build-time variable in Dockerfile (#435)
Also fix a runtime error that happens when the API server's version is not a valid SemVer.
2021-11-07 12:36:02 +03:00
RamiBerm
a20f83597c
fix rbac error crash (#432) 2021-11-04 15:44:05 +02:00
RamiBerm
9a9e5fda0a
Fix clean again (#431)
* Revert "Fix mizu clean (#430)"

This reverts commit 57cd7a365b.

* Update cleanRunner.go and tapRunner.go
2021-11-04 14:47:48 +02:00
RamiBerm
57cd7a365b
Fix mizu clean (#430) 2021-11-04 14:11:08 +02:00
RamiBerm
a3ec5d147e
TRA-3842 daemon mode (#427)
* Update config.go, tapConfig.go, and models.go

* WIP

* Update go.sum

* Update tapRunner.go

* Update tap.go

* WIP

* WIP

* Update Dockerfile, main.go, and 2 more files...

* WIP

* Update utils.go, tapClusterResourceManagement.go, and utils.go

* Merge branch 'develop'

* Update metadata_controller.go, utils.go, and 2 more files...

* Update main.go, utils.go, and tapRunner.go

* Update tapRunner.go

* Update config.go, config.go, and models.go

* Update main.go, main.go, and stats_provider_test.go

* Update provider.go

* bug fixes

* Update main.go, metadata_controller.go, and 13 more files...

* Update metadata_controller.go, status_controller.go, and 4 more files...

* Update main.go, config.go, and 3 more files...

* Update tapRunner.go

* Update config.go, stats_provider_test.go, and consts.go
2021-11-04 11:46:45 +02:00
RamiBerm
655626bc42
TRA-3868 move tapped pod watch and tapper updating to shared (#416)
* WIP

* WIP

* WIP

* WIP

* WIP

* Update tapRunner.go and k8sTapManager.go

* Update cleanRunner.go, common.go, and 8 more files...

* Update common.go, tapConfig.go, and 2 more files...

* Update config.go, config.go, and 5 more files...

* Update tapRunner.go, config.go, and 7 more files...

* Update cleanRunner.go, logs.go, and 2 more files...

* Update k8sTapManager.go, provider.go, and watch.go

* Update go.sum, go.mod, and go.sum

* Update go.mod and go.sum

* Update go.mod, go.sum, and 2 more files...

* Revert "Update go.mod, go.sum, and 2 more files..."

This reverts commit 8140311349.

* Update funcWrappers.go, tapRunner.go, and 4 more files...

* Update main.go, tapRunner.go, and mizuTapperSyncer.go
2021-11-01 14:12:32 +02:00
RamiBerm
35dbd5fde2
TRA-3860 create main configmap for agent and tappers (#410)
* WIP

* Update options.go and serializable_regexp.go

* Update go.sum, go.sum, and 4 more files...

* Update go.sum, go.sum, and 4 more files...

* Update config.go and serializable_regexp.go

* Update config.go, config.json, and test.go

* Update tapRunner.go and provider.go

* Update provider.go

* Update tapRunner.go and provider.go

* Update config.json and test.go

* Update contract_validation.go, config.go, and 2 more files...

* Update main.go

* Update rulesHTTP.go

* Update config.go, size_enforcer.go, and 5 more files...

* Update config.go and config.go

Co-authored-by: Rami Berman <rami.berman@up9.com>
2021-10-31 15:29:05 +02:00
RoyUP9
f069fdaae1
changed ask confirmation to be asked only once (#409) 2021-10-28 14:55:33 +03:00
gadotroee
d478a7ec43
TRA-3867 proxy host from config (#411) 2021-10-28 14:53:28 +03:00
gadotroee
af96e55f61
running proxy on 0.0.0.0 (#408) 2021-10-28 11:44:29 +03:00
RoyUP9
fbee4454e4
added k8s server version check (#404) 2021-10-27 09:35:30 +03:00
RamiBerm
e9e16551ad
TRA-3850 mizu clean command + fix debug deploy (#403)
* debug mizu

* Update clean.go, cleanRunner.go, and tapRunner.go

* Update tapRunner.go

* Update cleanRunner.go

* Update tapRunner.go

Co-authored-by: Rami Berman <rami.berman@up9.com>
2021-10-27 09:28:54 +03:00
RoyUP9
779257b864
added alert and error when there is a proxy before k8b server (#402) 2021-10-26 16:57:56 +03:00
RoyUP9
4e50e17d81
build m1 and windows (#395) 2021-10-24 15:10:46 +03:00
Igor Gov
991eb2ab16
Revert "TRA-3828 added build for Mac/Apple M1 and Windows (#392)" (#393)
This reverts commit f0db3b81a8.
2021-10-24 12:27:05 +03:00
Alex Haiut
f0db3b81a8
TRA-3828 added build for Mac/Apple M1 and Windows (#392) 2021-10-24 11:56:12 +03:00
M. Mert Yıldıran
29ba963c48
Remove github.com/romana/rlog dependency completely (#374)
* Remove `github.com/romana/rlog` dependency completely

* Comment out all the unnecessary logging in the protocol extensions

* Remove commented out all the unnecessary logging lines

* Remove two more lines related to logging
2021-10-19 16:13:03 +03:00
RamiBerm
0473181f0a
TRA-3803 handle k8s watch timeouts (#372)
* Update watch.go and debounce.go

* Update debounce.go

* Update watch.go

* Update watch.go

* Update watch.go

* Update watch.go

* Update watch.go

Co-authored-by: Rami <rami@rami-work>
2021-10-19 14:41:37 +03:00
M. Mert Yıldıran
145e7cda01
Add OAS contract monitoring support (#325)
* Add OAS contract monitoring support

* Pass the contract failure reason to UI

* Fix the issues related to contract validation

* Fix rest of the issues in the UI

* Add documentation related to contract monitoring feature

* Fix a typo in the docs

* Unmarshal to `HTTPRequestResponsePair` only if the OAS validation is enabled

* Fix an issue caused by the merge commit

* Slightly change the logic in the `validateOAS` method

Change the `contractText` value to `No Breaches` or `Breach` and make the text `white-space: nowrap`.

* Retrieve and display the failure reason for both request and response

Also display the content of the contract/OAS file in the UI.

* Display the OAS under `CONTRACT` tab with syntax highlighting

Also fix the styling in the entry feed.

* Remove `EnforcePolicyFileDeprecated` constant

* Log the other errors as well

* Get context from caller instead

* Define a type for the contract status and make its values enum-like

* Remove an unnecessary `if` statement

* Validate OAS in the CLI before passing it to Agent

* Get rid of the `github.com/ghodss/yaml` dependency in `loadOAS` by using `LoadFromData`

* Fix an artifact from the merge conflict
2021-10-19 14:24:22 +03:00
M. Mert Yıldıran
b7ff076571
Set the default log level for Agent to INFO and raise it to DEBUG if dump-logs=true is provided (#373)
* Set the default log level for Agent to `INFO` and change it to `DEBUG` if `dump-logs=true` is provided

* Remove `Trace` method and replace its calls with `Debug`

* Export logging levels from `logger` by defining functions

* Revert "Export logging levels from `logger` by defining functions"

This reverts commit e554e40f4a.

* Run `go mod tidy` on agent

* Define a method named `determineLogLevel`
2021-10-19 14:22:20 +03:00
M. Mert Yıldıran
167b17dfd2
Move the 8899 integer and string literals into a const named DefaultApiServerPort in shared (#367) 2021-10-17 15:28:33 +03:00
M. Mert Yıldıran
147e812edb
Replace all rlog occurrences with the shared logger (#350)
* Replace all `rlog` occurrences with the shared logger

* Use the same log format in `InitLoggerStderrOnly` as well

* Convert one more `log.Fatal` to `logger.Log.Errorf` as well in the `cli`

* Replace `log.` occurrences with `logger.Log.` in `agent`

* Fix `cannot use err (type error)`

* Change the logging level to `DEBUG`

* Replace an `Errorf` with `Fatal`

* Add informative message
2021-10-17 12:15:30 +03:00
M. Mert Yıldıran
91196bb306
Add readiness and liveness probes to API server (#365)
* Add readiness and liveness probes to API server

* Use `intstr.FromInt(8899)` instead
2021-10-17 11:40:18 +03:00
M. Mert Yıldıran
b30b62ef77
Move cli/logger to shared, and refactor all its usages in cli (#349)
* Move `cli/logger` to `shared`, and refactor all its usages in `cli`

* Remove indirect for `op/go-logging` in `shared`
2021-10-14 10:18:01 +03:00
RoyUP9
26788bb3a6
organize routes (#348) 2021-10-13 17:31:15 +03:00
RoyUP9
2706cd4d50
api server remove unused env vars (#347) 2021-10-13 14:14:14 +03:00
RoyUP9
b40104b74c
changed sync entries to start on startup (#344) 2021-10-13 11:48:42 +03:00
RoyUP9
837e35255b
auth status route to api server (#342) 2021-10-12 11:03:58 +03:00
RoyUP9
04c0f8cbcd
tap to workspace (#315) 2021-10-11 15:42:41 +03:00
RoyUP9
ba6b5c868c
added semver isvalid check in version update checker (#338) 2021-10-11 11:32:41 +03:00
RoyUP9
9d378ed75b
refactor login (#339) 2021-10-11 11:31:12 +03:00
M. Mert Yıldıran
77710cc411
Format the strings in watchTapperPod method (#331) 2021-10-07 21:06:17 +03:00
RoyUP9
8b8c4609ce
renamed upload entries to sync entries (#330) 2021-10-07 18:33:14 +03:00
RoyUP9
14b616a856
Connecting Mizu to the application (#317) 2021-10-07 17:28:28 +03:00
RoyUP9
82d603c0fd
fixed version update http route (#329) 2021-10-07 17:26:38 +03:00
RoyUP9
f1a2ee7fb4
removed enforce policy file deprecated flag (#328) 2021-10-07 11:08:48 +03:00
Igor Gov
fd48cc6d87
Renaming ignored user agents var (#320) 2021-10-06 13:52:30 +03:00
RoyUP9
111d000c12
added interface conversion check (#318) 2021-10-06 13:38:32 +03:00
RoyUP9
9c98a4c2b1
Revert "Connecting Mizu to the application (#313)" (#316) 2021-10-06 10:41:23 +03:00
RoyUP9
d2d4ed5aee
Connecting Mizu to the application (#313) 2021-10-05 16:35:16 +03:00
Igor Gov
30fce5d765
Supporting Mizu view from given url (#312)
* Supporting Mizu view from given url
2021-10-05 12:24:50 +03:00
Igor Gov
90040798b8
Adding additional error handling to api server watch (#311) 2021-09-30 11:49:31 +03:00
M. Mert Yıldıran
9eecddddd5
Start the tapper after the API server is ready (#309) 2021-09-30 11:22:07 +03:00
M. Mert Yıldıran
cc49e815d6
Watch the tapper pod after starting it (#310)
* Watch the tapper pod after starting it

* Improve the logic in `watchTapperPod` method
2021-09-30 09:32:27 +03:00
Selton Fiuza
c26eb843e3
[refactor/TRA-3693] type:latency to slo and latency field to response-time (#282)
* type:latency to slo and latency field to response-time

* remove comment from import

* Friendly message on ignored rules and format

* formatting

* change conditional to catch negative values and ignore it

* Fix Bug Alon Reported

* sliceUtils to shared
2021-09-29 11:51:03 -03:00
M. Mert Yıldıran
cdf1c39a52
Omit the RULES tab if the policy rules feature is inactive (#303)
* Omit the `RULES` tab if the policy rules feature is inactive (WIP)

* Propagate the boolean value `isRulesEnabled` from file read error to UI

* Remove the debug log
2021-09-25 18:15:54 +03:00
Nimrod Gilboa Markevich
9212c195b4
Improve cloud resources cleanup (#215) 2021-09-23 20:51:37 +03:00
RoyUP9
f164e54fee
changed test rules config to readonly (#301) 2021-09-23 09:13:17 +03:00
RoyUP9
5dacd41ba9
renamed traffic-validation to traffic-validation-file (#296) 2021-09-22 11:21:43 +03:00
Igor Gov
749bee6d55
Using rlog in agent and tapper & removing unreferenced code (#289)
* .
2021-09-22 06:24:19 +03:00
Selton Fiuza
b50eced489
[Refactor/TRA-3692] rename test rules to traffic validation (#281) 2021-09-19 14:47:19 +03:00
M. Mert Yıldıran
5392475486
Fix the issues related to sensitive data filtering feature (#285)
* Run acceptance tests on pull request

* Take `options.DisableRedaction` into account

* Log `defaultTapConfig`

* Pass the `SENSITIVE_DATA_FILTERING_OPTIONS` to tapper daemon set too

* Revert "Run acceptance tests on pull request"

This reverts commit ad79f1418f.
2021-09-19 13:33:34 +03:00
M. Mert Yıldıran
9fa9b67328
Bring back the sensitive data filtering feature (#280)
* Bring back the sensitive data filtering feature

* Add `// global` comment
2021-09-18 20:13:59 +03:00
RoyUP9
d5b01347df
fixed crash when channel is closed (#273) 2021-09-14 11:53:47 +03:00
RoyUP9
6b52458642
removed exit if browser not supported (#270) 2021-09-12 16:09:04 +03:00
M. Mert Yıldıran
7cc077c8a0
Fix the memory exhaustion by optimizing max. AMQP message size and GOGC (#257)
* Permanently resolve the memory exhaustion in AMQP

Introduce;
- `MEMORY_PROFILING_DUMP_PATH`
- `MEMORY_PROFILING_TIME_INTERVAL`
environment variables and make `startMemoryProfiler` method more parameterized.

* Fix a leak in HTTP

* Revert "Fix a leak in HTTP"

This reverts commit 9d46820ff3.

* Set maximum AMQP message size to 16MB

* Set `GOGC` to 12800

* Remove some commented out lines and an unnecessary `else if`
2021-09-09 17:45:37 +03:00
Igor Gov
fae5f22d25
Adding a download command if new mizu version is detected (#267) 2021-09-08 22:28:07 +03:00
gadotroee
073b0b72d3
Remove fetch command (and direction) (#264) 2021-09-06 10:16:04 +03:00
M. Mert Yıldıran
366c1d0c6c
Refactor Mizu, define an extension API and add new protocols: AMQP, Kafka (#224)
* Separate HTTP related code into `extensions/http` as a Go plugin

* Move `extensions` folder into `tap` folder

* Move HTTP files into `tap/extensions/lib` for now

* Replace `orcaman/concurrent-map` with `sync.Map`

* Remove `grpc_assembler.go`

* Remove `github.com/up9inc/mizu/tap/extensions/http/lib`

* Add a build script to automatically build extensions from a known path and load them

* Start to define the extension API

* Implement the `run()` function for the TCP stream

* Add support of defining multiple ports to the extension API

* Set the extension name inside the extension

* Declare the `Dissect` function in the extension API

* Dissect HTTP request from inside the HTTP extension

* Make the distinction of outbound and inbound ports

* Dissect HTTP response from inside the HTTP extension

* Bring back the HTTP request-response pair matcher

* Return a `*api.RequestResponsePair` from the dissection

* Bring back the gRPC-HTTP/2 parser

* Fix the issues in `handleHTTP1ClientStream` and `handleHTTP1ServerStream`

* Call a function pointer to emit dissected data back to the `tap` package

* roee changes -
trying to fix agent to work with the "api" object) - ***still not working***

* small mistake in the conflicts

* Fix the issues that are introduced by the merge conflict

* Add `Emitter` interface to the API and send `OutputChannelItem`(s) to `OutputChannel`

* Fix the `HTTP1` handlers

* Set `ConnectionInfo` in HTTP handlers

* Fix the `Dockerfile` to build the extensions

* remove some unwanted code

* no message

* Re-enable `getStreamProps` function

* Migrate back from `gopacket/tcpassembly` to `gopacket/reassembly`

* Introduce `HTTPPayload` struct and `HTTPPayloader` interface to `MarshalJSON()` all the data structures that are returned by the HTTP protocol

* Read `socketHarOutChannel` instead of `filteredHarChannel`

* Connect `OutputChannelItem` to the last WebSocket means that finally the web UI started to work again

* Add `.env.example` to React app

* Marshal and unmarshal `*http.Request`, `*http.Response` pairs

* Move `loadExtensions` into `main.go` and map extensions into `extensionsMap`

* Add `Summarize()` method to the `Dissector` interface

* Add `Analyze` method to the `Dissector` interface and `MizuEntry` to the extension API

* Add `Protocol` struct and make it effect the UI

* Refactor `BaseEntryDetails` struct and display the source and destination ports in the UI

* Display the protocol name inside the details layout

* Add `Represent` method to the `Dissector` interface and manipulate the UI through this method

* Make the protocol color affect the details layout color and write protocol abbreviation vertically

* Remove everything HTTP related from the `tap` package and make the extension system fully functional

* Fix the TypeScript warnings

* Bring in the files related AMQP into `amqp` directory

* Add `--nodefrag` flag to the tapper and bring in the main AMQP code

* Implement the AMQP `BasicPublish` and fix some issues in the UI when the response payload is missing

* Implement `representBasicPublish` method

* Fix several minor issues

* Implement the AMQP `BasicDeliver`

* Implement the AMQP `QueueDeclare`

* Implement the AMQP `ExchangeDeclare`

* Implement the AMQP `ConnectionStart`

* Implement the AMQP `ConnectionClose`

* Implement the AMQP `QueueBind`

* Implement the AMQP `BasicConsume`

* Fix an issue in `ConnectionStart`

* Fix a linter error

* Bring in the files related Kafka into `kafka` directory

* Fix the build errors in Kafka Go files

* Implement `Dissect` method of Kafka and adapt request-response pair matcher to asynchronous client-server stream

* Do the "Is reversed?" checked inside `getStreamProps` and fix an issue in Kafka `Dissect` method

* Implement `Analyze`, `Summarize` methods of Kafka

* Implement the representations for Kafka `Metadata`, `RequestHeader` and `ResponseHeader`

* Refactor the AMQP and Kafka implementations to create the summary string only inside the `Analyze` method

* Implement the representations for Kafka `ApiVersions`

* Implement the representations for Kafka `Produce`

* Implement the representations for Kafka `Fetch`

* Implement the representations for Kafka `ListOffsets`, `CreateTopics` and `DeleteTopics`

* Fix the encoding of AMQP `BasicPublish` and `BasicDeliver` body

* Remove the unnecessary logging

* Remove more logging

* Introduce `Version` field to `Protocol` struct for dynamically switching the HTTP protocol to HTTP/2

* Fix the issues in analysis and representation of HTTP/2 (gRPC) protocol

* Fix the issues in summary section of details layout for HTTP/2 (gRPC) protocol

* Fix the read errors that freezes the sniffer in HTTP and Kafka

* Fix the issues in HTTP POST data

* Fix one more issue in HTTP POST data

* Fix an infinite loop in Kafka

* Fix another freezing issue in Kafka

* Revert "UI Infra - Support multiple entry types + refactoring (#211)"

This reverts commit f74a52d4dc.

* Fix more issues that are introduced by the merge

* Fix the status code in the summary section

* adding the cleaner again (why we removed it?).
add TODO: on the extension loop .

* fix dockerfile (remove deleting .env file) - it is found in dockerignore and fails to build if the file not exists

* fix GetEntrties ("/entries" endpoint) - working with "tapApi.BaseEntryDetail" (moved from shared)

* Fix an issue in the UI summary section

* Refactor the protocol payload structs

* Fix a log message in the passive tapper

* Adapt `APP_PORTS` environment variable to the new extension system and change its format to `APP_PORTS='{"http": ["8001"]}' `

* Revert "fix dockerfile (remove deleting .env file) - it is found in dockerignore and fails to build if the file not exists"

This reverts commit 4f514ae1f4.

* Bring in the necessary changes from f74a52d4dc

* Open the API server URL in the web browser as soon as Mizu is ready

* Make the TCP reader consists of a single Go routine (instead of two) and try to dissect in both client and server mode by rewinding

* Swap `TcpID` without overwriting it

* Sort extension by priority

* Try to dissect with looping through all the extensions

* fix getStreamProps function.
(it should be passed from CLI as it was before).

* Turn TCP reader back into two Goroutines (client and server)

* typo

* Learn `isClient` from the TCP stream

* Set `viewer` style `overflow: "auto"`

* Fix the memory leaks in AMQP and Kafka dissectors

* Revert some of the changes in be7c65eb6d

* Remove `allExtensionPorts` since it's no longer needed

* Remove `APP_PORTS` since it's no longer needed

* Fix all of the minor issues in the React code

* Check Kafka header size and fail-fast

* Break the dissectors loop upon a successful dissection

* Don't break the dissector loop. Protocols might collide

* Improve the HTTP request-response counter (still not perfect)

* Make the HTTP request-response counter perfect

* Revert "Revert some of the changes in be7c65eb6d3fb657a059707da3ca559937e59739"

This reverts commit 08e7d786d8.

* Bring back `filterItems` and `isHealthCheckByUserAgent` functions

* Remove some development artifacts

* remove unused and commented lines that are not relevant

* Fix the performance in TCP stream factory. Make it create two `tcpReader`(s) per extension

* Change a log to debug

* Make `*api.CounterPair` a field of `tcpReader`

* Set `isTapTarget` to always `true` again since `filterAuthorities` implementation has problems

* Remove a variable that's only used for logging even though not introduced by this branch

* Bring back the `NumberOfRules` field of `ApplicableRules` struct

* Remove the unused `NewEntry` function

* Move `k8sResolver == nil` check to a more appropriate place

* default healthChecksUserAgentHeaders should be empty array (like the default config value)

* remove spam console.log

* Rules button cause app to crash (access the service via incorrect property)

* Ignore all .env* files in docker build.

* Better caching in dockerfile: only copy go.mod before go mod download.

* Check for errors while loading an extension

* Add a comment about why `Protocol` is not a pointer

* Bring back the call to `deleteOlderThan`

* Remove the `nil` check

* Reduce the maximum allowed AMQP message from 128MB to 1MB

* Fix an error that only occurs when a Kafka broker is initiating

* Revert the change in b2abd7b990

* Fix the service name resolution in all protocols

* Remove the `anydirection` flag and fix the issue in `filterAuthorities`

* Pass `sync.Map` by reference to `deleteOlderThan` method

* Fix the packet capture issue in standalone mode that's introduced by the removal of `anydirection`

* Temporarily resolve the memory exhaustion in AMQP

* Fix a nil pointer dereference error

* Fix the CLI build error

* Fix a memory leak that's identified by `pprof`

Co-authored-by: Roee Gadot <roee.gadot@up9.com>
Co-authored-by: Nimrod Gilboa Markevich <nimrod@up9.com>
2021-09-02 14:34:06 +03:00
RoyUP9
17fa163ee3
added proxy logs, added events logs (#254) 2021-09-01 15:30:37 +03:00
RoyUP9
e25e7925b6
fixed version blocking (#251) 2021-08-30 15:11:14 +03:00
RoyUP9
80237c8090
fixed error on invalid config path (#250) 2021-08-30 11:43:44 +03:00
Igor Gov
a310953f05
Fixing call to analysis (#248) 2021-08-26 15:55:05 +03:00
RoyUP9
a9e92b60f5
added custom config path option (#247) 2021-08-26 13:50:41 +03:00
RoyUP9
35e40cd230
added tap acceptance tests, fixed duplicate namespace problem (#244) 2021-08-26 09:56:18 +03:00
Igor Gov
2575ad722a
Introducing API server provider (#243) 2021-08-22 11:41:38 +03:00
Igor Gov
6dd0ef1268
Adding user friendly message in view command before sleeping (#239) 2021-08-19 12:22:18 +03:00
Alex Haiut
83cfaed1a3
updated readme for release (#237) 2021-08-19 11:47:02 +03:00
RoyUP9
667f0dc87d
fixed namespace restricted validation (#235) 2021-08-19 11:33:48 +03:00
Igor Gov
a34c2fc0dc
Adding version check to all commands execution (#236) 2021-08-19 11:33:20 +03:00
RoyUP9
7f9fd82c0e
fixed panic when using invalid kube config path (#231) 2021-08-19 10:59:31 +03:00
Igor Gov
a9b5eba9d4
Fix: View command fail sporadically (#228) 2021-08-19 09:44:43 +03:00
RoyUP9
80201224c6
telemetry machine id (#230) 2021-08-19 09:44:23 +03:00
RoyUP9
bf27e94003
fixed version check, removed duplicate kube config, fix flags warning, fixed log of invalid config (#227) 2021-08-18 18:10:47 +03:00
RoyUP9
db1f4458c5
Introducing acceptance test (#222) 2021-08-18 10:22:45 +03:00
RoyUP9
b4f3b2c540
fixed test coverage (#218) 2021-08-15 14:22:49 +03:00
RoyUP9
a427534605
tests refactor (#216) 2021-08-15 12:30:34 +03:00
RoyUP9
f9677dbaa1
added resources to config (#208) 2021-08-12 16:33:32 +03:00
RoyUP9
0afab6c068
added set hierarchy, removed allowed set flags (#205) 2021-08-12 16:01:33 +03:00
Igor Gov
1d1b62ec4f
Improving log dump feature logs (#207) 2021-08-12 09:32:35 +03:00
RoyUP9
c8e5886a96
added telemetry api calls (#201) 2021-08-11 15:57:41 +03:00
Igor Gov
7b73004e85
Cli pkg refactor (2) (#200) 2021-08-11 09:56:03 +03:00
Igor Gov
0409eb239d
Report telemetry on develop and main branches (#195) 2021-08-10 18:10:02 +03:00
RoyUP9
d705ae3eb6
added support of slice in set, removed support of allowed set flags (#191) 2021-08-10 16:16:58 +03:00
RoyUP9
c53b2148d1
add readonly tag (#190) 2021-08-10 09:51:35 +03:00
RoyUP9
4406919565
added test workflow, added test for contains func (#184) 2021-08-09 16:04:00 +03:00
gadotroee
413fb5b3f5
Add option to supply user agents to ignore via config (#173) 2021-08-09 12:27:13 +03:00
RoyUP9
e36c146979
temp fix - ignore agent image in config command (#186) 2021-08-09 12:17:01 +03:00
Nimrod Gilboa Markevich
1cf9c29ef0
Remove hardump flag (#183)
Removed hardump flag and made it the default (and only) behavior.
2021-08-08 17:31:45 +03:00
RoyUP9
dfea8884d4
Adding 'configuration' section in readme (#180) 2021-08-08 14:05:15 +03:00
Igor Gov
d34dacbbe2
View command - moving version check after proxy creation (#177) 2021-08-08 12:26:58 +03:00
Nimrod Gilboa Markevich
0595df8b87
Adds Namespace-Restricted Mode to README. (#178) 2021-08-08 12:23:11 +03:00
Nimrod Gilboa Markevich
ebbe6458a8
Do not tap pods whose names start with "mizu-". (#176) 2021-08-08 10:51:39 +03:00
Igor Gov
7f2021c312
Several fixes for the release (#175) 2021-08-08 10:32:21 +03:00
RoyUP9
824945141a
fixed config parsing of int and uint (#172) 2021-08-05 21:45:18 +03:00
Igor Gov
0244f12167
Fixes (#171) 2021-08-05 19:29:06 +03:00
RoyUP9
60533a9591
added allowed set flag (#169) 2021-08-05 14:23:16 +03:00
Igor Gov
90f0f603c7
Support getting logs in ns restricted mode (#168) 2021-08-05 12:12:01 +03:00
RoyUP9
683d199774
added support of multiple namespaces (#167) 2021-08-05 11:19:29 +03:00
Igor Gov
fa632b49a7
Introducing mizu logs dump & Log prints alignment in API server using rlog (#165) 2021-08-05 11:01:08 +03:00
Nimrod Gilboa Markevich
04579eb03c
Namespace restricted mode (#147) 2021-08-05 10:28:31 +03:00
Selton Fiuza
dea223bfe1
Feature/tra 3349 validation rules merged with develop (#148)
* Implemented validation rules, based on: https://up9.atlassian.net/browse/TRA-3349

* Color on Entry based on rules

* Background red/green based on rules

* Change flag --validation-rules to --test-rules

* rules tab UI updated

* rules tab font and background-color is changed for objects

* Merged with develop

* Fixed compilation issues.

* Renamed fullEntry -> harEntry where appropriate.

* Change green/red logic

* Update models.go

* Fix latency bug and alignment

* Merge Conflicts fix

* Working after merge

* Working on Nimrod comments

* Resolving conflicts

* Resolving conflicts

* Resolving conflicts

* Nimrod Comments pt.3

* Log Error on configmap creation if the user doesn't have permission.

* Checking configmap permission to ignore --test-rules

* Revert time for mizu to get ready

* Nimrod comments pt 4 && merge develop pt3

* Nimrod comments pt 4 && merge develop pt3

* Const rulePolicyPath and filename

Co-authored-by: Neim <elezin9@gmail.com>
Co-authored-by: nimrod-up9 <nimrod@up9.com>
2021-08-04 09:21:36 -03:00
Igor Gov
d18f1f8316
Tapped pods report via endpoint instead of web socket (#164) 2021-08-04 10:41:33 +03:00
Igor Gov
f9202900ee
No warning when mizu rbac exists (#163) 2021-08-04 08:41:00 +03:00
Igor Gov
9e34662511
Adding logs and fixing several issues (#162)
* Config grooming and several general fixes
2021-08-04 08:18:07 +03:00
RoyUP9
69a9deab4b
Feature/tra 3474 config refactor (#155) 2021-08-03 17:23:52 +03:00
RamiBerm
f9396e01ca
TRA-3415 ignore unready pods (#160)
* Update tapRunner.go and provider.go

* Update tapRunner.go

* Update tapRunner.go

* Update tapRunner.go

* Update tapRunner.go and provider.go

Co-authored-by: RamiBerm <rami.berman@up9.com>
2021-08-03 15:02:31 +03:00
Nimrod Gilboa Markevich
2d5b170406
Added a default SEM_VER (#159)
* Set default SEM_VER to 0.0.0.

* Build docker with both latest and semver tags.

* Use docker tag instead of building twice.

* Build and tag in one command.
2021-08-03 14:35:02 +03:00
Nimrod Gilboa Markevich
dc59fb6931
Added/edited description to Makefile commands. (#157)
* Added missing description to Makefile commands.

* Uppercase.
2021-08-03 11:41:02 +03:00
Igor Gov
793bb97e51
Print Mizu URL only after the watched pods (#156)
* Print Mizu URL after the watched pods
2021-08-03 08:54:10 +03:00
gadotroee
8d8310ee02
Revert "feature/TRA_3427_demo_mode (#150)" (#151)
This reverts commit 71eff5ea04.
2021-07-29 21:06:41 +03:00
RoyUP9
0824524d62
Add telemetry to config (#152) 2021-07-29 11:02:09 +03:00
Selton Fiuza
71eff5ea04
feature/TRA_3427_demo_mode (#150)
* Demo Mode MVP

* messages improve

* downloading based on the OS

* downloading based on the OS

* downloading based on the OS

* Modeler keep running

* A lot of revisions comes now

* Fix color
2021-07-28 14:50:15 -03:00
Igor Gov
ffa34039b1
Fix fetch telemetry report (#145) 2021-07-27 15:10:22 +03:00
nimrod-up9
d888706e1e
Show that the default answer to analysis prompt is yes. (#144) 2021-07-27 14:52:43 +03:00
Igor Gov
0566f63d72
Remove redundant '\n' at the end of each log (#142) 2021-07-27 13:10:40 +03:00
gadotroee
6d49339e29
Fix log to debug (#139) 2021-07-26 14:04:48 +03:00
gadotroee
58f0de4d4e
WriteFile fix(#138) 2021-07-26 12:00:36 +03:00
gadotroee
f175480f65
Adding (basic) configuration (#135) 2021-07-26 11:23:35 +03:00
Igor Gov
f64ee23c74
Introducing new logger, logging debug to file and info to stderr (#134)
* Introducing new logger to file debug and info to stderr
2021-07-25 10:08:37 +03:00
nimrod-up9
803681a239
Renamed collector, aggregator to api server, api folder to agent (#133)
* Renamed aggregator -> apiServer.

* Format errors with container names.

* Renamed collector -> apiServer.

* Rephrased help messages.

* Moved api -> agent.

* Continue renameing api -> agent in Makefile and Dockerfiles.
2021-07-22 17:17:17 +03:00
Igor Gov
ac358be877
Making kube config errors more user friendly (#132)
* Making kube config errors more user friendly
2021-07-22 15:26:28 +03:00
nimrod-up9
2996c1a4bc
Install Mizu in a dedicated namespace (#123)
* Use "mizu" namespace instead of "default". Create and delete as necessary.

* Wait until namespace is deleted.

* Distinguish between timeout and other errors.

* Sorted consts.

* k8s provider gets the names of Mizu serviceaccount, clusterrole and clusterrolebindings from caller.

* Renames.

* Remove non-namespaced mizu resources when finished: clusterrole and clusterrolebindings.

* Don't wait for namespace deletion if it was already deleted.

* When watching pods, check for cancellation before reading from channels.

* Allow user to cancel resource deletion and to cancel the wait.

* Increased cleanup timeout.

* go mod tidy.

* Ignore cli build products.

* Print err.

* Don't delete clusterrole and clusterrolebinding if we do not have permissions.

* Added roles list in README.

* Added clusterrole and clusterrolebindings examples.
2021-07-22 14:26:12 +03:00
Igor Gov
e42c4f8648
Cleaning unused code (#129)
* Cleaning unused code
2021-07-21 17:07:32 +03:00
Igor Gov
7d5ed601df
Check if newer version exists in Github (#128)
* Check if newer version exists in Github
2021-07-21 16:36:47 +03:00
RamiBerm
77ced2a46d
Merge pull request #126 from up9inc/disable_redact_flag
Add option to disable sensitive data redaction by flag and fix CPU leaks
2021-07-21 11:20:36 +03:00
Igor Gov
360a4ea562 . 2021-07-21 11:06:42 +03:00
Igor Gov
7fb85df3ac Add version artifact 2021-07-21 10:52:37 +03:00
RamiBerm
80e97e7f7e Update main.go, tap.go, and 4 more files... 2021-07-21 10:49:27 +03:00
Igor Gov
4a030c02f7 revert 2021-07-20 16:57:38 +03:00
Igor Gov
ea5054866d . 2021-07-20 16:55:32 +03:00
Igor Gov
a11e8f730e . 2021-07-20 16:54:51 +03:00
Igor Gov
1e66ebd8b3 . 2021-07-20 16:51:52 +03:00
Igor Gov
3be0c9ecd9 Adding telemetry reports 2021-07-20 16:47:20 +03:00
RamiBerm
1d24188a02 Update tapRunner.go 2021-07-20 09:58:08 +03:00