Commit Graph

1146 Commits

Author SHA1 Message Date
M. Mert Yıldıran
2ad4838cf5
Add Go crypto/tls eBPF tracer for TLS connections (#1120)
* Run `go generate tls_tapper.go`

* Add `golang_uprobes.c`

* Add Golang hooks and offsets

* Add `golangConnection` struct and implement `pollGolangReadWrite` method

* Upgrade `github.com/cilium/ebpf` version to `v0.8.1`

* Fix the linter error

* Move map related stuff to `maps.h` and run `go generate tls_tapper.go`

* Remove unused parameter

* Add an environment variable to test Golang locally

* Replace `Libssl` occurrences with `Ssllib` for consistency

* Fix exe path finding

* Temporarily disable OpenSSL

* Fix the mixed offsets and dissection preparation

* Change the read symbol from `net/http.(*persistConn).Read` to `crypto/tls.(*Conn).Read`

* Remove `len` and `cap` fields

* Fix the indent

* Fix the read data address

* Make `golang_dial_writes` key `__u64` and include the PID

* Fix the read data address one more time

* Temporarily disable the PCAP capture

* Add a uprobe for `net/http.(*gzipReader).Read` to read chunked HTTP response body

* Cancel `golang_crypto_tls_read_uprobe` if it's a gzip read

* Make hash map names more meaningful

* Pass the connection address from `write` to `gzip` through a common address between `gzip` and `dial`

* Fix the probed line number links

* Add `golangReader` struct and implement its `Read` method

* Have a single counter pair and request response matcher per Golang connection

* Add `MIZU_GLOBAL_GOLANG_PATH` environment variable

* `NULL` terminate the bytes with `unix.ByteSliceToString`

* Temporarily reject the gzip chunks

* Add malformed TODOs

* Revert "`NULL` terminate the bytes with `unix.ByteSliceToString`"

This reverts commit 7ee7ef7e44.

* Bring back `len` and `cap` fields

* Set `len` and `cap` in `golang_net_http_gzipreader_read_uprobe` as well

* Remove two `TODO`s

* Fix the `key_gzip` offsets

* Compress if it's gzip chunk (probably wrong!)

* Revert "Compress if it's gzip chunk (probably wrong!)"

This reverts commit 094a7c3da4.

* Remove `golang_net_http_gzipreader_read_uprobe`

* Read constant 4KiB

* Use constant read length

* Get the correct len of bytes (saw the second entry)

* Set all buffer sizes to `CHUNK_SIZE`

* Remove a `TODO`

* Revert "Temporarily disable the PCAP capture"

This reverts commit a2da15ef2d.

* Update `golang_crypto_tls_read_uprobe`

* Set the `reader` field of `tlsStream` to fix a `nil pointer dereference` error

* Don't export any fields of `golangConnection`

* Close the reader when we drop the connection

* Add a tracepoint for `sys_enter_close` to detect socket closes

* Rename `socket` struct to `golang_socket`

* Call `should_tap` in Golang uprobes

* Add `log_error` calls

* Revert "Temporarily disable OpenSSL"

This reverts commit f54d9a453f.

* Fix linter

* Revert "Revert "Temporarily disable OpenSSL""

This reverts commit 2433d867af.

* Change `golang_read_writes` map type from `BPF_RINGBUF` to `BPF_PERF_OUTPUT`

* Rename `golang_read_write` to `golang_event`

* Define an error

* Add comments

* Revert "Revert "Revert "Temporarily disable OpenSSL"""

This reverts commit e5a1de9c71.

* Fix `pollGolang`

* Revert "Revert "Revert "Revert "Temporarily disable OpenSSL""""

This reverts commit 6e1bd5d4f3.

* Fix `panic: send on closed channel`

* Revert "Revert "Revert "Revert "Revert "Temporarily disable OpenSSL"""""

This reverts commit 57d0584655.

* Use `findLibraryByPid`

* Revert "Revert "Revert "Revert "Revert "Revert "Temporarily disable OpenSSL""""""

This reverts commit 46f3d290b0.

* Revert "Revert "Revert "Revert "Revert "Revert "Revert "Temporarily disable OpenSSL"""""""

This reverts commit 775c833c06.

* Log tapping Golang

* Fix `Poll`

* Refactor `golang_net_http_dialconn_uprobe`

* Remove an excess error check

* Fix `can only use path@version syntax with 'go get' and 'go install' in module-aware mode` error in `tap/tlstapper/bpf-builder/build.sh`

* Unify Golang and OpenSSL under a single perf event buffer and `tls_chunk` struct

* Generate `tlsTapperChunkType` type (enum) as well

* Use kernel page size for the `sys_closes` perf buffer

* Fix the linter error

* Fix `MIZU_GLOBAL_GOLANG_PID` environment variable's functionality

* Rely on tracepoints for file descriptor retrieval in Golang implementation

* Remove the unnecessary changes

* Move common functions into `common.c`

* Declare `lookup_ssl_info` function to reduce duplication

* Fix linter

* Add comments and TODOs

* Remove `MIZU_GLOBAL_GOLANG_PATH` environment variable

* Update the object files

* Fix indentation

* Update object files

* Add `go_abi_internal.h`

* Fix `lookup_ssl_info`

* Convert indentation to spaces

* Add header guard comment

* Add more comments

* Find the `ret` instructions using Capstone Engine and `uprobe` the `return` statements

* Implement `get_fd_from_tcp_conn` function

* Separate SSL contexts to OpenSSL and Go

* Move `get_count_bytes` from `common.c` to `openssl_uprobes.c`

* Rename everything contains Golang to Go

* Reduce duplication in `go_uprobes.c`

* Update the comments

* Install Capstone in CI and Docker native builds

* Update `devops/install-capstone.sh`

* Add Capstone to AArch64 cross-compilation target

* Fix some of the issues on ARM64

* Delete the map element in `_ex_urpobe`

* Remove an unsued `LOG_` macro

* Rename `aquynh` to `capstone-engine`

* Add comment

* Revert "Fix some of the issues on ARM64"

This reverts commit 0b3eceddf4.

* Revert "Revert "Fix some of the issues on ARM64""

This reverts commit 681534ada1.

* Update object files

* Remove unnecessary return

* Increase timeout

* #run_acceptance_tests

* #run_acceptance_tests

* Fix the `arm64v8` sourced builds

* #run_acceptance_tests
2022-06-12 14:14:41 +03:00
lirazyehezkel
e41488ef3e
TRA-4276 Folder structure refactor (#1131)
* mizu-common folder structure refactor

* no message

* change images import from absolute to relative

* relative sass imports

* no message

* remove tgz

* spinner style
2022-06-12 11:20:47 +03:00
AmitUp9
533fb71bf4
TRA-4596_Update material UI to v5 (#1132)
* install @mui v5 and change imports

* upgrade MUI v5 in mizu community

* package-lock of mizu community and ui-common

* fix ui-common path

* #run_acceptance_tests

* remove comment

* cr fixes
2022-06-08 15:32:34 +03:00
RoyUP9
6f8aad83e6
Minikube start with more memory&cpu (#1130) 2022-06-06 17:53:12 +03:00
leon-up9
6e6bcec77e
Ui/TRA-4586_add-insertion-filter-to-settings-page (#1127)
* working query input

* removed prop

* splited to diffrent events

* export alias

* PR comments

Co-authored-by: Leon <>
2022-06-06 16:40:23 +03:00
leon-up9
71db792a4e
list takes 100% from parent (#1129)
Co-authored-by: Leon <>
2022-06-06 16:26:07 +03:00
gadotroee
f7f61c1217
Fix cypress files to cypress version 10 (#1124) 2022-06-06 11:13:20 +03:00
AmitUp9
696501fa11
remove poweredBy by display none the css calss (#1126) 2022-06-02 14:31:34 +03:00
gadotroee
415b5e08fd
Use Specific cypress version in the acceptance tests (#1125) 2022-06-02 13:21:44 +03:00
AmitUp9
7810f6defb
TRA_4565 - npm packages major updates (#1123)
* Major npm packages update

* npm i command added to dockerfile

* globally install npm new version

* remove the packages lock file copy

* package-lock commit and dockerfile revert changes

* space removal

Co-authored-by: Roee Gadot <roee.gadot@up9.com>
2022-06-02 11:22:37 +03:00
gadotroee
2aeac6c9e6
Remove analyze feature (#1121) 2022-06-01 10:17:14 +03:00
AmitUp9
dc241218bf
minor changes updated (#1122) 2022-05-31 15:08:15 +03:00
leon-up9
02b3672e09
added map object (#1119)
Co-authored-by: Leon <>
2022-05-30 13:13:07 +03:00
RoyUP9
45b368b33e
Updated the install command UX (#1118) 2022-05-30 11:18:46 +03:00
Igor Gov
8f64fdaa61
Alert to slack if release action fails (#1117)
* Trigger mizu ent stg deployment on release
2022-05-29 15:16:45 +03:00
AmitUp9
7edb0b153b
added condition that if no selector wouldn't be tooltip also (#1114) 2022-05-26 15:19:00 +03:00
AmitUp9
569a687fdf
Line numbers counter func added to utils (#1112)
* #run_acceptance_tests

* added new line to the end of the file

* #run_acceptance_tests

* linter fix

* more linter issues fix
2022-05-25 13:44:50 +03:00
AmitUp9
11e8b5eb65
TRA_4437- disable line numbers checkbox when only one line in content (#1109)
* number of lines state added

* added useEffect to update to showLineNubers state dynamically

* small cr fixes
2022-05-24 15:07:40 +03:00
AmitUp9
3901f3f3fe
change icon style css to move the icon so it can be clicked (#1111) 2022-05-24 14:16:27 +03:00
Nimrod Gilboa Markevich
2f1cc21fcb
Change redact to opt in (#1104) 2022-05-24 14:12:37 +03:00
leon-up9
433253a27b
font & padding change (#1108)
Co-authored-by: Leon <>
2022-05-24 12:09:13 +03:00
RoyUP9
00cc94fbe5
Changed OAS generator to get entries by push (#1103) 2022-05-22 14:53:48 +03:00
Nimrod Gilboa Markevich
8feef78ab1
Ignore mizu traffic in performance tests (#1102) 2022-05-22 12:08:22 +03:00
Nimrod Gilboa Markevich
992abc99bc
Disable redaction in performance tests (#1101) 2022-05-22 12:05:24 +03:00
Nimrod Gilboa Markevich
486d0b1088
Support tapper profiling on macOS (#1098) 2022-05-19 15:11:46 +03:00
David Levanon
f61a02d288
fix heap and goroutines metrics (#1099) 2022-05-19 14:55:13 +03:00
M. Mert Yıldıran
03694e57c0
Fix checkFilterByMethod in acceptance tests (#1094)
* Fix `checkFilterByMethod` in acceptance tests

* #run_acceptance_tests

* Add `numberOfRecords` parameter to test specs #run_acceptance_tests

* Fix the values #run_acceptance_tests

* Fix the values #run_acceptance_tests

* Fix #run_acceptance_tests

* #run_acceptance_tests

* Reduce value duplication #run_acceptance_tests

Co-authored-by: gadotroee <55343099+gadotroee@users.noreply.github.com>
Co-authored-by: Roee Gadot <roee.gadot@up9.com>
2022-05-19 13:37:50 +03:00
gadotroee
1760afda2b
Add tolerations to api server pod (#1035) 2022-05-19 10:46:36 +03:00
M. Mert Yıldıran
522e2cc3da
Call SetProtocol in AMQP faster and remove GetProtocol method (#1097)
* Call `SetProtocol` in AMQP faster and remove `GetProtocol` method

* #run_acceptance_tests

* Remove the unused fields from the test mocks #run_acceptance_tests
2022-05-19 10:07:40 +03:00
Nimrod Gilboa Markevich
ab38f4c011
Add profiling tools (#1087)
* Add gin-contrib/pprof dependency

* Run pprof server on agent with --profiler flag

* Add --profiler flag to cli

* Fix error message

* Print cpu usage percentage

* measure cpu of current pid instead of globaly on the system

* Add scripts to plot performance

* Plot packetsCount in analysis

* Concat to DataFrame

* Plot in turbo colorscheme

* Make COLORMAP const

* Fix rss units

* Reduce code repetition by adding function for plotting

* Allow grouping based on filenames

* Temporary: Marked with comments where to disable code for experiments

* Add newline at end of file

* Add tap.cpuprofile flag. Change memprofile flag to tap.memprofile

* create tapper modes for debugging using env vars

* Fix rss plot units (MB instead of bytes)

* Remove comment

* Add info to plot script

* Remove tap.cpumemprofile. Rename tap.memprofile to memprofile

* Remove unused import

* Remove whitespaces

Co-authored-by: M. Mert Yıldıran <mehmet@up9.com>

* Remove whitespaces

Co-authored-by: M. Mert Yıldıran <mehmet@up9.com>

* Remove whitespaces

Co-authored-by: M. Mert Yıldıran <mehmet@up9.com>

* Remove whitespaces

Co-authored-by: M. Mert Yıldıran <mehmet@up9.com>

* Remove whitespaces

Co-authored-by: M. Mert Yıldıran <mehmet@up9.com>

* Remove whitespaces

Co-authored-by: M. Mert Yıldıran <mehmet@up9.com>

* Rename debug env vars

* Create package for debug env vars, read each env var once

* Run go mod tidy

* Increment MatchedPairs before emitting

* Only count cores once

* Count virtual and physical cores

* Add dbgctl replace in cli

* Fix lint: Check return values

* Add tap/dbgctl to test-lint make rule

* Replace tap/dbgctl in all modules

* #run_acceptance_tests

* Copy dbgctl module to docker image

* Debug/profile tapper benchmark (#1093)

* add mizu debug env to avoid all extensions

* add readme + run_tapper_benchmark.sh

* temporary change branch name

* fix readme

* fix MIZU_BENCHMARK_CLIENTS_COUNT env

* change tap target to tcp stream

* track live tcp streams

* pr fixes

* rename tapperPacketsCount to ignored_packets_count

* change mizu tapper to mizu debugg

Co-authored-by: David Levanon <dvdlevanon@gmail.com>
Co-authored-by: M. Mert Yıldıran <mehmet@up9.com>
2022-05-18 15:42:13 +03:00
David Levanon
a9de4f0bba
stop tapping self tapper traffic (#1083)
* stop tapping self tapper traffic

* run go mod tidy

* allow to explicitly ignore ports

* remove unused code

* remove shared from tap + go mod tidy

* move ignroe ports to tapper

* rename TapperPacketsCount to IgnoredPacketsCount

* don't check null - go is smart

* remove nil check
2022-05-18 15:13:10 +03:00
lirazyehezkel
948af518b5
Fix double info icon in tapping status bar (#1095) 2022-05-18 14:40:20 +03:00
M. Mert Yıldıran
73448b514e
Fix checkFilter method in the acceptance tests (#1092)
* Fix `checkFilter` method in the acceptance tests

* #run_acceptance_tests

* Remove duplicate assertion #run_acceptance_tests
2022-05-17 14:41:08 +03:00
M. Mert Yıldıran
fc194354bc
Fix a nil pointer dereference error that occurs in tcpReader's Read method (#1090)
* Fix a `nil pointer dereference` error that occurs in `tcpReader`'s `Read` method

* #run_acceptance_tests

* #run_acceptance_tests

* Revert "Fix a `nil pointer dereference` error that occurs in `tcpReader`'s `Read` method"

This reverts commit ccef6cb393.

* Fix the race condition using locks #run_acceptance_tests
2022-05-17 13:13:50 +03:00
M. Mert Yıldıran
8418802c7e
Handle the wait for fetch in acceptance tests better (#1088)
* Handle the wait for fetch in acceptance tests better

* #run_acceptance_tests

* Fix the error #run_acceptance_tests

* Fix `waitForFetchAndPause` and `checkFilter` #run_acceptance_tests

* Fix the tests #run_acceptance_tests
2022-05-17 11:01:47 +03:00
M. Mert Yıldıran
bfa834e840
Spawn only two Goroutines per TCP stream (#1062)
* Spawn only two Goroutines per TCP stream

* Fix the linter error

* Use `isProtocolIdentified` method instead

* Fix the `Read` method of `tcpReader`

* Remove unnecessary `append`

* Copy to buffer only a message is received

* Remove `exhaustBuffer` field and add `rewind` function

* Rename `buffer` field to `pastData`

* Update tap/tcp_reader.go

Co-authored-by: Nimrod Gilboa Markevich <59927337+nimrod-up9@users.noreply.github.com>

* Use `copy` instead of assignment

* No lint

* #run_acceptance_tests

* Fix `rewind` #run_acceptance_tests

* Fix the buffering algorithm #run_acceptance_tests

* Add `TODO`

* Fix the problems in AMQP and Kafka #run_acceptance_tests

* Use `*bytes.Buffer` instead of `[]api.TcpReaderDataMsg` #run_acceptance_tests

* Have a single `*bytes.Buffer`

* Revert "Have a single `*bytes.Buffer`"

This reverts commit fad96a288a.

* Revert "Use `*bytes.Buffer` instead of `[]api.TcpReaderDataMsg` #run_acceptance_tests"

This reverts commit 0fc70bffe2.

* Fix the early timing out issue #run_acceptance_tests

* Remove `NewBytes()` method

* Update the `NewTcpReader` method signature #run_acceptance_tests

* #run_acceptance_tests

* #run_acceptance_tests

* #run_acceptance_tests

Co-authored-by: Nimrod Gilboa Markevich <59927337+nimrod-up9@users.noreply.github.com>
2022-05-16 16:06:36 +03:00
David Levanon
5c012641a5
Feature/limit fd to address map (#1078)
* limit fd to address map

* limit fd to address map

Co-authored-by: gadotroee <55343099+gadotroee@users.noreply.github.com>
2022-05-16 12:32:26 +03:00
Igor Gov
74bd4b180f
Removing elastic dump experimental feature (#1086)
* Removing elastic dump experimental feature
2022-05-15 15:23:03 +03:00
RoyUP9
8ea2dabb34
Added send to socket error validation (#1085) 2022-05-15 14:44:48 +03:00
leon-up9
366d34b8d0
Ui/fix/selectList-sticky-header-fix (#1084)
* selectList sticky header

* selectList changed & servicemap adapted

* ignore eslint

Co-authored-by: Leon <>
Co-authored-by: AmitUp9 <96980485+AmitUp9@users.noreply.github.com>
2022-05-15 14:39:09 +03:00
M. Mert Yıldıran
5fc3e38c1a
Fix the Kafka ApiKey query and add ApiKeyName field (human-readable ApiKey) (#1080)
* Fix the Kafka `ApiKey` query and add `ApiKeyName` field (human-readable `ApiKey`)

* Update the dataset for Kafka unit tests

* #run_acceptance_tests
2022-05-15 09:42:32 +03:00
RoyUP9
09a0fca2c2
Extracted insert to database functionality (#1082) 2022-05-15 09:19:33 +03:00
M. Mert Yıldıran
0437586908
Replace the gRPC reference link with a better one (#1081) 2022-05-14 19:43:43 +03:00
M. Mert Yıldıran
f8181ccb07
Remove ReassemblyStream interface (duplicate of gopacket/reassembly.Stream) (#1079) 2022-05-14 18:37:16 +03:00
leon-up9
414e5cfe5a
match selectlist ui (#1077)
* insert filter and header to selectlist

* handle single select

* rename search var

* font size changed
2022-05-11 19:51:12 +03:00
RoyUP9
2fac0009ea
Fixed oas query (#1076) 2022-05-11 15:36:46 +03:00
M. Mert Yıldıran
36d59ede07
Fix acceptance tests failures caused by fetch 50 and a race in the tests (#1074)
* #run_acceptance_tests

* Wait 4 seconds after filter submission #run_acceptance_tests

* Revert "Wait 4 seconds after filter submission #run_acceptance_tests"

This reverts commit 3c20ccfcec.

* Upgrade Basenine version to `v0.8.2` #run_acceptance_tests

* #run_acceptance_tests

* Revert "Revert "Wait 4 seconds after filter submission #run_acceptance_tests""

This reverts commit f0c595150d.

* Reduce the delay to 2 seconds #run_acceptance_tests

* Use `cy.wait()` instead of `setTimeout`

* #run_acceptance_tests

* Increase it to 4 seconds #run_acceptance_tests

* #run_acceptance_tests

* Wait for a second and pause the stream to preserve the DOM #run_acceptance_tests

* Wait in a better place #run_acceptance_tests

* Wait `pause-icon` to disappear #run_acceptance_tests

* Wait one more second #run_acceptance_tests

* Fix `setup.sh`

* #run_acceptance_tests

* Fix the place of `npm i` call

* #run_acceptance_tests

* Fix the other workflow as well #run_acceptance_tests

* Fix the workflows once more

* #run_acceptance_tests

* Don't ignore `*/cypress/support` #run_acceptance_tests

* Do `npm i` in `setup.sh` #run_acceptance_tests

* Wait for at least 4 seconds #run_acceptance_tests

* #run_acceptance_tests

* Fix #run_acceptance_tests

* Comment #run_acceptance_tests

* Fix `{alt+enter}` to `{ctrl+enter}` and update the `checkFilter` and `deeperCheck` method signatures #run_acceptance_tests

* #run_acceptance_tests

* #run_acceptance_tests

* Extract it to a function #run_acceptance_tests

* Remove `cypress-wait-until`

* Revert `.gitignore` #run_acceptance_tests
2022-05-11 13:14:01 +03:00
AmitUp9
ac53508ad7
TRA-4513_grpc heading overlap request size (#1075)
* screen resolution changes

* fix window width to not overlap between headlines
2022-05-11 09:42:47 +03:00
leon-up9
e24c18254c
UI/servicemap-dynamic-protocol-filters (#1072)
* filters visble on click

* servicemap layout

* headers added
Icon changed

* flex removed , we have max size for filter

* selection only from edges

* caption changed

* removed duplicated code

* changes

Co-authored-by: Leon <>
2022-05-10 13:49:46 +03:00
leon-up9
b2830f133f
UI/Service-map-GUI-improvements (#1055)
* filters visble on click

* servicemap layout

* headers added
Icon changed

* flex removed , we have max size for filter

* added mixins

* search input caption

* PR commemts

* changed colors to variables

* lint fix
small buttom style fix

Co-authored-by: Leon <>
2022-05-10 13:27:41 +03:00