Commit Graph

77 Commits

Author SHA1 Message Date
Nimrod Gilboa Markevich
ab38f4c011
Add profiling tools (#1087)
* Add gin-contrib/pprof dependency

* Run pprof server on agent with --profiler flag

* Add --profiler flag to cli

* Fix error message

* Print cpu usage percentage

* measure cpu of current pid instead of globaly on the system

* Add scripts to plot performance

* Plot packetsCount in analysis

* Concat to DataFrame

* Plot in turbo colorscheme

* Make COLORMAP const

* Fix rss units

* Reduce code repetition by adding function for plotting

* Allow grouping based on filenames

* Temporary: Marked with comments where to disable code for experiments

* Add newline at end of file

* Add tap.cpuprofile flag. Change memprofile flag to tap.memprofile

* create tapper modes for debugging using env vars

* Fix rss plot units (MB instead of bytes)

* Remove comment

* Add info to plot script

* Remove tap.cpumemprofile. Rename tap.memprofile to memprofile

* Remove unused import

* Remove whitespaces

Co-authored-by: M. Mert Yıldıran <mehmet@up9.com>

* Remove whitespaces

Co-authored-by: M. Mert Yıldıran <mehmet@up9.com>

* Remove whitespaces

Co-authored-by: M. Mert Yıldıran <mehmet@up9.com>

* Remove whitespaces

Co-authored-by: M. Mert Yıldıran <mehmet@up9.com>

* Remove whitespaces

Co-authored-by: M. Mert Yıldıran <mehmet@up9.com>

* Remove whitespaces

Co-authored-by: M. Mert Yıldıran <mehmet@up9.com>

* Rename debug env vars

* Create package for debug env vars, read each env var once

* Run go mod tidy

* Increment MatchedPairs before emitting

* Only count cores once

* Count virtual and physical cores

* Add dbgctl replace in cli

* Fix lint: Check return values

* Add tap/dbgctl to test-lint make rule

* Replace tap/dbgctl in all modules

* #run_acceptance_tests

* Copy dbgctl module to docker image

* Debug/profile tapper benchmark (#1093)

* add mizu debug env to avoid all extensions

* add readme + run_tapper_benchmark.sh

* temporary change branch name

* fix readme

* fix MIZU_BENCHMARK_CLIENTS_COUNT env

* change tap target to tcp stream

* track live tcp streams

* pr fixes

* rename tapperPacketsCount to ignored_packets_count

* change mizu tapper to mizu debugg

Co-authored-by: David Levanon <dvdlevanon@gmail.com>
Co-authored-by: M. Mert Yıldıran <mehmet@up9.com>
2022-05-18 15:42:13 +03:00
David Levanon
a9de4f0bba
stop tapping self tapper traffic (#1083)
* stop tapping self tapper traffic

* run go mod tidy

* allow to explicitly ignore ports

* remove unused code

* remove shared from tap + go mod tidy

* move ignroe ports to tapper

* rename TapperPacketsCount to IgnoredPacketsCount

* don't check null - go is smart

* remove nil check
2022-05-18 15:13:10 +03:00
Igor Gov
74bd4b180f
Removing elastic dump experimental feature (#1086)
* Removing elastic dump experimental feature
2022-05-15 15:23:03 +03:00
RoyUP9
09a0fca2c2
Extracted insert to database functionality (#1082) 2022-05-15 09:19:33 +03:00
M. Mert Yıldıran
e1ad302c29
Make logger a separate module such that don't depend on shared module as a whole for logging (#1047)
* Make `logger` a separate module such that don't depend on `shared` module as a whole for logging

* Update `Dockerfile`
2022-04-27 22:26:27 +03:00
RoyUP9
d76f24edb8
Fixed tapper socket error on failure (#1029) 2022-04-20 14:53:07 +03:00
RoyUP9
d8fb8ff710
Fix for OAS reset not working (#978) 2022-04-07 18:14:03 +03:00
RamiBerm
76a6a77a14
Refactor ws (#961)
* Separate socket and basenine logic

* WIP

* Update socket_server_handlers.go

* Update socket_data_streamer.go and socket_server_handlers.go

* Update socket_server_handlers.go

* Merge branch 'develop' into refactor_ws
# Please enter a commit message to explain why this merge is necessary,
# especially if it merges an updated upstream into a topic branch.
#
# Lines starting with '#' will be ignored, and an empty message aborts
# the commit.

* empty commit for actions

* empty commit for actions

* commit for actions

* Revert "commit for actions"

This reverts commit 8ba2ecf7d3.

Co-authored-by: RoyUP9 <87927115+RoyUP9@users.noreply.github.com>
2022-04-04 17:33:53 +03:00
Andrey Pokhilko
97db24aeba
OAS: rework data feeding + sampleIDs (#917)
* Call OAS feeder

* Don't call old OAS code

* Rework calls

* Work on it

* Put back rules

* Make it compile

* start thinking of test

* Compiles

* Save

* Fixes

* Save

* Fixing

* Trying to fake conn

* add timeout

* Test timeout

* Fix tests

* Only build OAS for HTTP entries

* Remove some dead code

* Adding SampleIDs

* Cosmetics

* lint

* Revert rename

* Sample ID for content

* Cleanuo

* Add more sample IDs

* Checking hypothesis

* Move assignment place a bit

* Cosmetics

* Update test.yml

Co-authored-by: undera <undera@undera-old-desktop.home>
Co-authored-by: Igor Gov <iggvrv@gmail.com>
2022-03-30 11:14:25 +03:00
RamiBerm
63cf7ac34e
Refactor entries controller logic (#949)
* wip

* Update entries_controller.go and entries_provider.go

* Update entries_controller.go

* change entries provider into a struct + interface

* Update entries_provider.go

* Update entries_provider.go
2022-03-29 18:30:19 +03:00
Nimrod Gilboa Markevich
a5c35d7d90
Update tap targets over ws (#901)
Update tappers via websocket instead of by env var. This way the DaemonSet doesn't have to be applied just to notify the tappers that the tap targets changed. The number of tapper restarts is reduced. The DaemonSet still gets applied when there is a need to add/remove a tapper from a node.
2022-03-23 13:50:33 +02:00
M. Mert Yıldıran
774f07fccd
Add /db/flush and /db/reset API endpoints (#896)
* Add `/db/flush` and `/db/reset` API endpoints

* Handle the unmarshalling errors better in the WebSocket

* Handle Basenine connection error better in the WebSocket

* Upgrade to Basenine `v0.6.5`

* Fix the duplicated `StartTime` state

Co-authored-by: gadotroee <55343099+gadotroee@users.noreply.github.com>
2022-03-21 15:54:36 +02:00
RamiBerm
237002ef29
Dependency injection for oas servicemap (#895)
* Update main.go, main.go, and 3 more files...

* WIP

* Update main.go, oas_controller.go, and 3 more files...

* Update main.go, oas_generator.go, and servicemap.go

* Update loader.go and resolver.go

* Update oas_generator.go

* Update oas_generator.go, specgen_test.go, and 3 more files...

* Update service_map_controller_test.go

* Update oas_controller_test.go
2022-03-16 17:21:50 +02:00
RoyUP9
edbe4ab00b
Removed all code related to install (#892) 2022-03-15 16:41:42 +02:00
RoyUP9
8217ac3ed0
Switched echo to return json (#874) 2022-03-03 12:23:09 +02:00
M. Mert Yıldıran
3b063c3bb5
Add the ability to set the insertion filter into CLI (#843)
* Add the ability to set the insertion filter into CLI

* Resolve the conflict

* Upgrade Basenine to `v0.5.0` and check the insertion filter against syntax errors inside CLI

* Upgrade Basenine to `v0.5.1`

* Update the sums
2022-03-02 12:51:32 +03:00
Igor Gov
371e513249
Remove config dependency from basenine init (#846)
* Remove config dependency from basenine init
2022-02-23 10:06:19 +02:00
Igor Gov
97cce32e3f
Fix: service map component aware of agent config (#845)
* Fix: Service map component aware of mizu config
2022-02-23 09:35:05 +02:00
Igor Gov
d2e91b4ffa
Fix: tapper tries to load agent config map (#844) 2022-02-23 09:20:19 +02:00
Igor Gov
58a17897cf
Agent logs: output to stderr errors and above (#838)
* Agent logs: output to stderr errors and above
2022-02-22 14:02:51 +02:00
RamiBerm
145004fe43
TRA-4263 revert extensible routing (#774)
* Update main.go, extensions.go, and 2 more files...

* Update config_routes.go, entries_routes.go, and 7 more files...
2022-02-08 17:48:30 +02:00
RamiBerm
0a4674ea7c
TRA-4263 extensibility refactor (#770)
* Refactor routers

* refactor extension loading

* refactor server creation code

* Update main.go
2022-02-08 14:59:56 +02:00
Igor Gov
fd64c1bb14
Refactor mizuserver module rename (#742)
Co-authored-by: Igor Gov <igor.govorov1@gmail.com>
2022-02-01 17:19:24 +02:00
gadotroee
4be7164f20
Fix routing in frontend not working (#723) 2022-01-31 10:49:44 +02:00
Igor Gov
d0ef6c9f97
Remove debug logs from agent (#719) 2022-01-30 12:36:01 +02:00
Igor Gov
0f3dd66d2d
Experimental feature: elastic exporter (#713) 2022-01-30 09:22:13 +02:00
M. Mert Yıldıran
a9a61edd50
Add ARM64 and cross-compilation support to the agent image (#659)
* modified Dockerfile to work for both amd64 (Intel) and arm64 (M1)

* added changelog

* Update `Dockerfile` to have `ARCH` build argument

* Remove `docs/CHANGES.md`

* Upgrade the Basenine version from `v0.3.0` to `v0.4.6`

* Update `publish.yml` to have `ARCH` build argument

* Switch `BasenineImageRepo` to Docker Hub

* Have separate build arguments for `ARCH` and `GOARCH`

* Upgrade the Basenine version from `v0.4.6` to `v0.4.10`

* Oops forgot to update the 10th duplicated shell script

* Fix the oopsie and reduce duplications

* Fix `Dockerfile`

* Fix the incompatibility issue between Go plugins and gold linker in Alpine inside `Dockerfile`

* Fix `asm: xxhash_amd64.s:120: when dynamic linking, R15 is clobbered by a global variable access` error

* Update `Dockerfile` to have cross-compilation on an AMD64 machine

Also revert changes in the shell scripts

* Delete `debug.Dockerfile`

* Create a custom base (`debian:buster-slim` based) image for the shipped image

* Replace `mertyildiran/debian-pcap` with `up9inc/debian-pcap`

* Upgrade Basenine version to `v0.4.12`

* Use `debian:stable-slim` as the base

* Fix the indentation in the `Dockerfile`

* Update `publish.yml`

* Enable `publish.yml` for `feature/multiarch_build` branch

* Tag correctly and set `ARCH` Docker argument

* Remove the lines that are forgotten to be removed from the shell scripts

* Add `MizuAgentImageRepo` constant and use it as default `AgentImage` value

* Bring back `Set up Cloud SDK` step to `Build the CLI and publish` job

* Build ARM64 CLI for Linux as well

* Revert "Enable `publish.yml` for `feature/multiarch_build` branch"

This reverts commit d30be4c1f0.

* Revert Go 1.17 upgrade

* Remove `build_extensions_debug.sh` as well

* Make the `Dockerfile` to compile the agent statically

* Statically link the protocol extensions

* Fix `Dockerfile`

* Bring back `-s -w` flags

* Verify the signatures of the downloads in `dockcross/linux-arm64-musl`

* Revert modifications in some shell scripts

* Make the `BUILDARCH` and `TARGETARCH` separation in the `Dockerfile`

* Separate cross-compilation builder image into a separate repo named `up9inc/linux-arm64-musl-go-libpcap`

* Fill the shell script and specify the tag for `dockcross/linux-arm64-musl`

* Remove the unnecessary dependencies from `builder-native-base`

* Improve the comments in the `Dockerfile`

* Upgrade Basenine version to `v0.4.13`

* Fix `Dockerfile`

* Revert "Revert "Enable `publish.yml` for `feature/multiarch_build` branch""

This reverts commit 303e466bdc.

* Revert "Revert "Revert "Enable `publish.yml` for `feature/multiarch_build` branch"""

This reverts commit 0fe252bbdb.

* Remove `push-docker-debug` from the `Makefile`

* Rename `publish.yml` to `release.yml`

Co-authored-by: Alex Haiut <alex@up9.com>
2022-01-25 21:24:50 +03:00
RoyUP9
d216c64154
Added support of ui split build (#682) 2022-01-24 10:34:50 +02:00
Gustavo Massaneiro
d5fd2ff1da
Service Map (#623)
* debug builds and gcflags

* update dockerfile for debug

* service map routes and controller

* service map graph structure

* service map interface and new methods

* adding service map edges from mizu entries

* new service map count methods

* implementing the status endpoint

* ServiceMapResponse and ServiceMapEdge models

* service map get endpoint logic

* reset logic and endpoint

* fixed service map get status

* improvements to graph node structure

* front-end implementation and service map buttons

* new render endpoint to render the graph in real time

* spinner sass

* new ServiceMapModal component

* testing react-force-graph-2d lib

* Improvements to service map graph structure, added node id and updated edge source/destination type

* Revert "testing react-force-graph-2d lib"

This reverts commit 1153938386.

* testing react-graph-vis lib

* updated to work with react-graph-vis lib

* removed render endpoint

* go mod tidy

* serviceMap config flag

* using the serviceMap config flag

* passing mizu config to service map as a dependency

* service map tests

* Removed print functions

* finished service map tests

* new service property

* service map controller tests

* moved service map reset button to service map modal
reset closes the modal

* service map modal refresh button and logic

* reset button resets data and refresh

* service map modal close button

* node size/edge size based on the count value
edge color based on protocol

* nodes and edges shadow

* enabled physics to avoid node overlap, changed kafka protocol color to dark green

* showing edges count values and fixed bidirectional edges overlap

* go mod tidy

* removed console.log

* Using the destination node protocol instead of the source node protocol

* Revert "debug builds and gcflags"
Addressed by #624 and #626

This reverts commit 17ecaece3e.

* Revert "update dockerfile for debug"
Addressed by #635

This reverts commit 5dfc15b140.

* using the entire tap Protocol struct instead of only the protocol name

* using the backend protocol background color for node colors

* fixed test, the node list order can change

* re-factoring to get 100% coverage

* using protocol colors just for edges

* re-factored service map to use TCP Entry data. Node key is the entry ip-address instead of the name

* fallback to ip-address when entry name is unresolved

* re-factored front-end

* adjustment to main div style

* added support for multiple protocols for the same edge

* using the item protocol instead of the extension variable

* fixed controller tests

* displaying service name and ip-address on graph nodes

* fixed service map test, we cannot guarantee the slice order

* auth middleware

* created a new pkg for the service map

* re-factoring

* re-factored front-end

* reverting the import order as previous

* Aligning with other UI feature flags handling

* we don't need to get the status anymore, we have window["isServiceMapEnabled"]

* small adjustments

* renamed from .tsx to .ts

* button styles and minor improvements

* moved service map modal from trafficPage to app component

Co-authored-by: Igor Gov <igor.govorov1@gmail.com>
2022-01-19 15:27:12 -03:00
Igor Gov
e15eb71b77
Fix: no panic on failure to sync entries to up9.app (#648) 2022-01-16 11:50:40 +02:00
Igor Gov
8963630e9e
Experimental feature: OAS Generator (#632) 2022-01-13 09:34:55 +02:00
Igor Gov
0e5611b7e9
FE setting experimental feature flags (#633) 2022-01-12 18:05:12 +02:00
M. Mert Yıldıran
e3e9681110
Move Basenine binary into a separate container (#603)
* Move Basenine binary into a separate container

* Set `WorkingDir` to `shared.DataDirPath` in the `basenine` container

* Use `consts.go` to set the Basenine image and port

* Bring back the `net-wait-go` usage to prevent startup failures
2022-01-09 11:18:34 +03:00
RamiBerm
833d08bb40
TRA-4075 integrate kratos user management (#583)
* WIP

* WIP

* WIP

* WIP

* Update App.tsx and Header.tsx

* Update createResources.go, provider.go, and 2 more files...

* WIP

* fix eof newlines

* Fix ts imports, add readiness probe to kratos to prevent mizu being used while kratos isnt ready

* cleaned code

* fix install create namespace

* Update package-lock.json

* Update provider.go

* Update provider.go

* Update provider.go

* Update install_controller.go

* Update kratos.yml

* Update start.sh

* Update provider.go

* Update provider.go

* Update main.go, socket_routes.go, and 8 more files...

* Update App.tsx

* Update installRunner.go

* Update App.tsx
2022-01-05 17:50:27 +02:00
RoyUP9
bf68689212
renamed standalone route to config route (#586) 2022-01-04 18:46:03 +02:00
RoyUP9
02b2cbaa03
Added update config route for install mode (#581) 2022-01-04 17:27:14 +02:00
RoyUP9
482036182b
Added setting ui mode - enterprise or normal (#575) 2021-12-30 14:58:53 +02:00
RoyUP9
de046c65b0
Mizu install mode (#566) 2021-12-28 15:44:37 +02:00
RoyUP9
15895d5947
Changed sync tappers to start only when sync tappers config is true (#562) 2021-12-27 17:19:00 +02:00
RamiBerm
2da7c0f0ed
TRA-4065 fix mizu tapping (#561)
* Update main.go and passive_tapper.go

* Update main.go
2021-12-27 15:11:36 +02:00
RamiBerm
cfe9e863b7
TRA-4065 support inflight tap target update (#556)
* WIP

* WIP

* Update main.go

* Update main.go and passive_tapper.go

* Update passive_tapper.go

* Update passive_tapper.go

* Update passive_tapper.go

* Update passive_tapper.go
2021-12-27 11:50:34 +02:00
gadotroee
2d78785558
Fix acceptance tests (after pods status request change) (#545) 2021-12-19 13:46:14 +02:00
Igor Gov
cba0c682e5
Report pods "isTapped" to FE (#535) 2021-12-19 13:03:53 +02:00
David Levanon
ab029f4394
Send pod info to tapper (#532) 2021-12-16 10:51:03 +02:00
Igor Gov
90c210452d
API server stores tappers status (#531) 2021-12-15 14:52:49 +02:00
Nimrod Gilboa Markevich
1c18eb1b84
Use one channel for events instead of three (#495)
Use one channel for events instead of three separate channels by event type
2021-11-23 15:06:27 +02:00
Nimrod Gilboa Markevich
b1ad2efb96
Warn pods not starting (#493)
Print warning event related to mizu k8s resources.
In non-daemon print to CLI. In Daemon print to API-Server logs.
2021-11-22 15:30:10 +02:00
David Levanon
6caa94f08f
Add support to auto discover envoy processes (#459)
* discover envoy pids using cluster ips

* add istio flag to cli + rename mtls flag to istio

* add istio.md to docs

* Fixing typos

* Fix minor typos and grammer in docs

Co-authored-by: Nimrod Gilboa Markevich <nimrod@up9.com>
2021-11-21 15:45:07 +02:00
RamiBerm
89836d8d75
TRA-3903 better health endpoint for daemon mode (#471)
* Update main.go, status_controller.go, and 2 more files...

* Update status_controller.go and mizuTapperSyncer.go
2021-11-16 15:44:27 +02:00
RoyUP9
763f72a640
remove newline in logs, fixed logs time format (#469) 2021-11-16 12:07:48 +02:00