Commit Graph

74 Commits

Author SHA1 Message Date
Nimrod Gilboa Markevich
ab38f4c011
Add profiling tools (#1087)
* Add gin-contrib/pprof dependency

* Run pprof server on agent with --profiler flag

* Add --profiler flag to cli

* Fix error message

* Print cpu usage percentage

* measure cpu of current pid instead of globaly on the system

* Add scripts to plot performance

* Plot packetsCount in analysis

* Concat to DataFrame

* Plot in turbo colorscheme

* Make COLORMAP const

* Fix rss units

* Reduce code repetition by adding function for plotting

* Allow grouping based on filenames

* Temporary: Marked with comments where to disable code for experiments

* Add newline at end of file

* Add tap.cpuprofile flag. Change memprofile flag to tap.memprofile

* create tapper modes for debugging using env vars

* Fix rss plot units (MB instead of bytes)

* Remove comment

* Add info to plot script

* Remove tap.cpumemprofile. Rename tap.memprofile to memprofile

* Remove unused import

* Remove whitespaces

Co-authored-by: M. Mert Yıldıran <mehmet@up9.com>

* Remove whitespaces

Co-authored-by: M. Mert Yıldıran <mehmet@up9.com>

* Remove whitespaces

Co-authored-by: M. Mert Yıldıran <mehmet@up9.com>

* Remove whitespaces

Co-authored-by: M. Mert Yıldıran <mehmet@up9.com>

* Remove whitespaces

Co-authored-by: M. Mert Yıldıran <mehmet@up9.com>

* Remove whitespaces

Co-authored-by: M. Mert Yıldıran <mehmet@up9.com>

* Rename debug env vars

* Create package for debug env vars, read each env var once

* Run go mod tidy

* Increment MatchedPairs before emitting

* Only count cores once

* Count virtual and physical cores

* Add dbgctl replace in cli

* Fix lint: Check return values

* Add tap/dbgctl to test-lint make rule

* Replace tap/dbgctl in all modules

* #run_acceptance_tests

* Copy dbgctl module to docker image

* Debug/profile tapper benchmark (#1093)

* add mizu debug env to avoid all extensions

* add readme + run_tapper_benchmark.sh

* temporary change branch name

* fix readme

* fix MIZU_BENCHMARK_CLIENTS_COUNT env

* change tap target to tcp stream

* track live tcp streams

* pr fixes

* rename tapperPacketsCount to ignored_packets_count

* change mizu tapper to mizu debugg

Co-authored-by: David Levanon <dvdlevanon@gmail.com>
Co-authored-by: M. Mert Yıldıran <mehmet@up9.com>
2022-05-18 15:42:13 +03:00
M. Mert Yıldıran
e1ad302c29
Make logger a separate module such that don't depend on shared module as a whole for logging (#1047)
* Make `logger` a separate module such that don't depend on `shared` module as a whole for logging

* Update `Dockerfile`
2022-04-27 22:26:27 +03:00
M. Mert Yıldıran
3b063c3bb5
Add the ability to set the insertion filter into CLI (#843)
* Add the ability to set the insertion filter into CLI

* Resolve the conflict

* Upgrade Basenine to `v0.5.0` and check the insertion filter against syntax errors inside CLI

* Upgrade Basenine to `v0.5.1`

* Update the sums
2022-03-02 12:51:32 +03:00
David Levanon
87ef469e25
Add tls tapper (#683)
* initial tls tapper commit

* add tls flag to mizu cli

* support ssl_read_ex/ssl_write_ex

* use hostproc to find libssl

* auto discover tls processes

* support libssl1.0

* recompile ebpf with old clang/llvm

* Update tap/passive_tapper.go

Co-authored-by: M. Mert Yıldıran <mehmet@up9.com>

* Update tap/tlstapper/tls_poller.go

Co-authored-by: M. Mert Yıldıran <mehmet@up9.com>

* Update tap/tlstapper/tls_poller.go

Co-authored-by: M. Mert Yıldıran <mehmet@up9.com>

* Update tap/tlstapper/tls_poller.go

Co-authored-by: M. Mert Yıldıran <mehmet@up9.com>

* Update tap/tlstapper/tls_poller.go

Co-authored-by: M. Mert Yıldıran <mehmet@up9.com>

* Update tap/tlstapper/tls_poller.go

Co-authored-by: M. Mert Yıldıran <mehmet@up9.com>

* Update tap/tlstapper/tls_poller.go

Co-authored-by: M. Mert Yıldıran <mehmet@up9.com>

* Update tap/tlstapper/tls_poller.go

Co-authored-by: M. Mert Yıldıran <mehmet@up9.com>

* upgrade ebpf go lib

* handling big tls messages

* fixing max buffer size in ebpf

* remove unused import

* fix linter issues

* minor pr fixes

* compile with old clang

* fix cgroup file format

* pr fixes + cgroup extract enhance

* fix linter

* adding indirect ebpf dep to agent go.mod

* adding ebpf docker builder

* minor pr fixes

* add req resp matcher to dissect

* rename ssl hooks to ssl hooks structs

* move to alpine, use local copy of mizu instead of git, add readme

* use global req resp mather for tls

Co-authored-by: M. Mert Yıldıran <mehmet@up9.com>
Co-authored-by: gadotroee <55343099+gadotroee@users.noreply.github.com>
2022-02-16 15:34:51 +02:00
Igor Gov
0f6c56986f
Technical depth: Adding Go linter to CI (#734) 2022-02-01 08:47:26 +02:00
Gustavo Massaneiro
e30b52f528
[TRA-4190] ExecutionTime telemetry (#685) 2022-01-25 11:13:49 +02:00
Nimrod Gilboa Markevich
5a044875d3
Rename Istio to service mesh (#605)
- Rename --istio flag to the more general --service-mesh
- Rename internal variables, consts and structures to reflect this conceptual change
- Update the docs accordingly
2022-01-09 13:21:14 +02:00
RoyUP9
de046c65b0
Mizu install mode (#566) 2021-12-28 15:44:37 +02:00
David Levanon
6caa94f08f
Add support to auto discover envoy processes (#459)
* discover envoy pids using cluster ips

* add istio flag to cli + rename mtls flag to istio

* add istio.md to docs

* Fixing typos

* Fix minor typos and grammer in docs

Co-authored-by: Nimrod Gilboa Markevich <nimrod@up9.com>
2021-11-21 15:45:07 +02:00
RoyUP9
b77ea63f42
Add token validity check (#483) 2021-11-21 15:14:02 +02:00
RoyUP9
ea0b3fb34e
moved headless to root config, use headless in view (#450) 2021-11-11 12:11:02 +02:00
M. Mert Yıldıran
1ee8fb6292
Fix the acceptance tests after the merger of #279 (#443)
* Enable acceptance tests

* Fix the acceptance tests

* Move `--headless` from `getDefaultCommandArgs` to `getDefaultTapCommandArgs`

* Fix rest of the failing acceptance tests

* Revert "Enable acceptance tests"

This reverts commit 3f919e865a.

* Revert "Revert "Enable acceptance tests""

This reverts commit c0bfe54b70.

* Ignore `--headless` in `mizu view`

* Make all non-informative things informative

* Remove `github.com/stretchr/testify` dependency from the acceptance tests

* Move the helper methods `waitTimeout` and `checkDBHasEntries` from `tap_test.go` to `testsUtils.go`

* Split `checkDBHasEntries` method into `getDBEntries` and `assertEntriesAtLeast` methods

* Revert "Revert "Revert "Enable acceptance tests"""

This reverts commit c13342671c.

* Revert "Revert "Revert "Revert "Enable acceptance tests""""

This reverts commit 0f8c436926.

* Make `getDBEntries` and `checkEntriesAtLeast` methods return errors instead

* Revert "Revert "Revert "Revert "Revert "Enable acceptance tests"""""

This reverts commit 643fdde009.
2021-11-10 18:14:04 +03:00
RamiBerm
a3ec5d147e
TRA-3842 daemon mode (#427)
* Update config.go, tapConfig.go, and models.go

* WIP

* Update go.sum

* Update tapRunner.go

* Update tap.go

* WIP

* WIP

* Update Dockerfile, main.go, and 2 more files...

* WIP

* Update utils.go, tapClusterResourceManagement.go, and utils.go

* Merge branch 'develop'

* Update metadata_controller.go, utils.go, and 2 more files...

* Update main.go, utils.go, and tapRunner.go

* Update tapRunner.go

* Update config.go, config.go, and models.go

* Update main.go, main.go, and stats_provider_test.go

* Update provider.go

* bug fixes

* Update main.go, metadata_controller.go, and 13 more files...

* Update metadata_controller.go, status_controller.go, and 4 more files...

* Update main.go, config.go, and 3 more files...

* Update tapRunner.go

* Update config.go, stats_provider_test.go, and consts.go
2021-11-04 11:46:45 +02:00
RoyUP9
f069fdaae1
changed ask confirmation to be asked only once (#409) 2021-10-28 14:55:33 +03:00
M. Mert Yıldıran
145e7cda01
Add OAS contract monitoring support (#325)
* Add OAS contract monitoring support

* Pass the contract failure reason to UI

* Fix the issues related to contract validation

* Fix rest of the issues in the UI

* Add documentation related to contract monitoring feature

* Fix a typo in the docs

* Unmarshal to `HTTPRequestResponsePair` only if the OAS validation is enabled

* Fix an issue caused by the merge commit

* Slightly change the logic in the `validateOAS` method

Change the `contractText` value to `No Breaches` or `Breach` and make the text `white-space: nowrap`.

* Retrieve and display the failure reason for both request and response

Also display the content of the contract/OAS file in the UI.

* Display the OAS under `CONTRACT` tab with syntax highlighting

Also fix the styling in the entry feed.

* Remove `EnforcePolicyFileDeprecated` constant

* Log the other errors as well

* Get context from caller instead

* Define a type for the contract status and make its values enum-like

* Remove an unnecessary `if` statement

* Validate OAS in the CLI before passing it to Agent

* Get rid of the `github.com/ghodss/yaml` dependency in `loadOAS` by using `LoadFromData`

* Fix an artifact from the merge conflict
2021-10-19 14:24:22 +03:00
M. Mert Yıldıran
b30b62ef77
Move cli/logger to shared, and refactor all its usages in cli (#349)
* Move `cli/logger` to `shared`, and refactor all its usages in `cli`

* Remove indirect for `op/go-logging` in `shared`
2021-10-14 10:18:01 +03:00
RoyUP9
b40104b74c
changed sync entries to start on startup (#344) 2021-10-13 11:48:42 +03:00
RoyUP9
04c0f8cbcd
tap to workspace (#315) 2021-10-11 15:42:41 +03:00
RoyUP9
f1a2ee7fb4
removed enforce policy file deprecated flag (#328) 2021-10-07 11:08:48 +03:00
RoyUP9
5dacd41ba9
renamed traffic-validation to traffic-validation-file (#296) 2021-09-22 11:21:43 +03:00
Selton Fiuza
b50eced489
[Refactor/TRA-3692] rename test rules to traffic validation (#281) 2021-09-19 14:47:19 +03:00
M. Mert Yıldıran
5392475486
Fix the issues related to sensitive data filtering feature (#285)
* Run acceptance tests on pull request

* Take `options.DisableRedaction` into account

* Log `defaultTapConfig`

* Pass the `SENSITIVE_DATA_FILTERING_OPTIONS` to tapper daemon set too

* Revert "Run acceptance tests on pull request"

This reverts commit ad79f1418f.
2021-09-19 13:33:34 +03:00
gadotroee
073b0b72d3
Remove fetch command (and direction) (#264) 2021-09-06 10:16:04 +03:00
RoyUP9
667f0dc87d
fixed namespace restricted validation (#235) 2021-08-19 11:33:48 +03:00
Igor Gov
7b73004e85
Cli pkg refactor (2) (#200) 2021-08-11 09:56:03 +03:00
RoyUP9
d705ae3eb6
added support of slice in set, removed support of allowed set flags (#191) 2021-08-10 16:16:58 +03:00
gadotroee
413fb5b3f5
Add option to supply user agents to ignore via config (#173) 2021-08-09 12:27:13 +03:00
RoyUP9
dfea8884d4
Adding 'configuration' section in readme (#180) 2021-08-08 14:05:15 +03:00
Igor Gov
7f2021c312
Several fixes for the release (#175) 2021-08-08 10:32:21 +03:00
RoyUP9
60533a9591
added allowed set flag (#169) 2021-08-05 14:23:16 +03:00
Igor Gov
90f0f603c7
Support getting logs in ns restricted mode (#168) 2021-08-05 12:12:01 +03:00
RoyUP9
683d199774
added support of multiple namespaces (#167) 2021-08-05 11:19:29 +03:00
Nimrod Gilboa Markevich
04579eb03c
Namespace restricted mode (#147) 2021-08-05 10:28:31 +03:00
Selton Fiuza
dea223bfe1
Feature/tra 3349 validation rules merged with develop (#148)
* Implemented validation rules, based on: https://up9.atlassian.net/browse/TRA-3349

* Color on Entry based on rules

* Background red/green based on rules

* Change flag --validation-rules to --test-rules

* rules tab UI updated

* rules tab font and background-color is changed for objects

* Merged with develop

* Fixed compilation issues.

* Renamed fullEntry -> harEntry where appropriate.

* Change green/red logic

* Update models.go

* Fix latency bug and alignment

* Merge Conflicts fix

* Working after merge

* Working on Nimrod comments

* Resolving conflicts

* Resolving conflicts

* Resolving conflicts

* Nimrod Comments pt.3

* Log Error on configmap creation if the user doesn't have permission.

* Checking configmap permission to ignore --test-rules

* Revert time for mizu to get ready

* Nimrod comments pt 4 && merge develop pt3

* Nimrod comments pt 4 && merge develop pt3

* Const rulePolicyPath and filename

Co-authored-by: Neim <elezin9@gmail.com>
Co-authored-by: nimrod-up9 <nimrod@up9.com>
2021-08-04 09:21:36 -03:00
RoyUP9
69a9deab4b
Feature/tra 3474 config refactor (#155) 2021-08-03 17:23:52 +03:00
nimrod-up9
d888706e1e
Show that the default answer to analysis prompt is yes. (#144) 2021-07-27 14:52:43 +03:00
Igor Gov
0566f63d72
Remove redundant '\n' at the end of each log (#142) 2021-07-27 13:10:40 +03:00
gadotroee
6d49339e29
Fix log to debug (#139) 2021-07-26 14:04:48 +03:00
gadotroee
f175480f65
Adding (basic) configuration (#135) 2021-07-26 11:23:35 +03:00
Igor Gov
f64ee23c74
Introducing new logger, logging debug to file and info to stderr (#134)
* Introducing new logger to file debug and info to stderr
2021-07-25 10:08:37 +03:00
nimrod-up9
803681a239
Renamed collector, aggregator to api server, api folder to agent (#133)
* Renamed aggregator -> apiServer.

* Format errors with container names.

* Renamed collector -> apiServer.

* Rephrased help messages.

* Moved api -> agent.

* Continue renameing api -> agent in Makefile and Dockerfiles.
2021-07-22 17:17:17 +03:00
RamiBerm
77ced2a46d
Merge pull request #126 from up9inc/disable_redact_flag
Add option to disable sensitive data redaction by flag and fix CPU leaks
2021-07-21 11:20:36 +03:00
RamiBerm
80e97e7f7e Update main.go, tap.go, and 4 more files... 2021-07-21 10:49:27 +03:00
Igor Gov
3be0c9ecd9 Adding telemetry reports 2021-07-20 16:47:20 +03:00
gadotroee
a0a9d74662
Versioning (#116)
Add versioning check
2021-07-15 16:19:29 +03:00
RamiBerm
1d6c176c7f Update tap.go 2021-07-15 11:10:16 +03:00
RamiBerm
3b9f5ee32f Update size_enforcer.go and tap.go 2021-07-15 10:41:29 +03:00
RamiBerm
6e14fa95a1 Update tap.go and go.sum 2021-07-15 09:16:01 +03:00
RamiBerm
cd34892943 Merge branch 'develop'
Conflicts:
	api/main.go
	cli/cmd/tap.go
2021-07-15 09:12:48 +03:00
RamiBerm
62b17c1822 Update tap.go 2021-07-15 09:08:43 +03:00