Commit Graph

156 Commits

Author SHA1 Message Date
Andrey Pohilko
0477fd5030 Add codegen placeholders 2022-05-24 11:14:04 +01:00
RoyUP9
00cc94fbe5
Changed OAS generator to get entries by push (#1103) 2022-05-22 14:53:48 +03:00
Nimrod Gilboa Markevich
ab38f4c011
Add profiling tools (#1087)
* Add gin-contrib/pprof dependency

* Run pprof server on agent with --profiler flag

* Add --profiler flag to cli

* Fix error message

* Print cpu usage percentage

* measure cpu of current pid instead of globaly on the system

* Add scripts to plot performance

* Plot packetsCount in analysis

* Concat to DataFrame

* Plot in turbo colorscheme

* Make COLORMAP const

* Fix rss units

* Reduce code repetition by adding function for plotting

* Allow grouping based on filenames

* Temporary: Marked with comments where to disable code for experiments

* Add newline at end of file

* Add tap.cpuprofile flag. Change memprofile flag to tap.memprofile

* create tapper modes for debugging using env vars

* Fix rss plot units (MB instead of bytes)

* Remove comment

* Add info to plot script

* Remove tap.cpumemprofile. Rename tap.memprofile to memprofile

* Remove unused import

* Remove whitespaces

Co-authored-by: M. Mert Yıldıran <mehmet@up9.com>

* Remove whitespaces

Co-authored-by: M. Mert Yıldıran <mehmet@up9.com>

* Remove whitespaces

Co-authored-by: M. Mert Yıldıran <mehmet@up9.com>

* Remove whitespaces

Co-authored-by: M. Mert Yıldıran <mehmet@up9.com>

* Remove whitespaces

Co-authored-by: M. Mert Yıldıran <mehmet@up9.com>

* Remove whitespaces

Co-authored-by: M. Mert Yıldıran <mehmet@up9.com>

* Rename debug env vars

* Create package for debug env vars, read each env var once

* Run go mod tidy

* Increment MatchedPairs before emitting

* Only count cores once

* Count virtual and physical cores

* Add dbgctl replace in cli

* Fix lint: Check return values

* Add tap/dbgctl to test-lint make rule

* Replace tap/dbgctl in all modules

* #run_acceptance_tests

* Copy dbgctl module to docker image

* Debug/profile tapper benchmark (#1093)

* add mizu debug env to avoid all extensions

* add readme + run_tapper_benchmark.sh

* temporary change branch name

* fix readme

* fix MIZU_BENCHMARK_CLIENTS_COUNT env

* change tap target to tcp stream

* track live tcp streams

* pr fixes

* rename tapperPacketsCount to ignored_packets_count

* change mizu tapper to mizu debugg

Co-authored-by: David Levanon <dvdlevanon@gmail.com>
Co-authored-by: M. Mert Yıldıran <mehmet@up9.com>
2022-05-18 15:42:13 +03:00
Igor Gov
74bd4b180f
Removing elastic dump experimental feature (#1086)
* Removing elastic dump experimental feature
2022-05-15 15:23:03 +03:00
RoyUP9
8ea2dabb34
Added send to socket error validation (#1085) 2022-05-15 14:44:48 +03:00
RoyUP9
09a0fca2c2
Extracted insert to database functionality (#1082) 2022-05-15 09:19:33 +03:00
RoyUP9
2fac0009ea
Fixed oas query (#1076) 2022-05-11 15:36:46 +03:00
M. Mert Yıldıran
eef0ee8023
Fetch N number of records in M milliseconds timeout before streaming the records (#1056)
* Fetch N number of records in M milliseconds timeout before streaming the records

* Implement the functionality inside socket data streamer

* Reverse the `fetchData` slice

* #run_acceptance_tests

* Trying to fix the tests.
#run_acceptance_tests

* javascript compilation error.

* #run_acceptance_tests

* Name the method better

* Upgrade Basenine version to `v0.8.0`

* Fix some issues related to `Fetch`

* Upgrade the Basenine version in `Dockerfile` as well

* Remove underscore from the parameter name

* Parameterize fetch timeout ms

Co-authored-by: gadotroee <55343099+gadotroee@users.noreply.github.com>
Co-authored-by: Roee Gadot <roee.gadot@up9.com>
2022-05-09 19:11:27 +03:00
RoyUP9
4c0aeb8146
Fixed default mime type to empty string instead of text/html (#1071) 2022-05-08 16:52:50 +03:00
gadotroee
2f89690da6
Remove go-cache unused dependency (#1065) 2022-05-02 13:24:28 +03:00
M. Mert Yıldıran
e1ad302c29
Make logger a separate module such that don't depend on shared module as a whole for logging (#1047)
* Make `logger` a separate module such that don't depend on `shared` module as a whole for logging

* Update `Dockerfile`
2022-04-27 22:26:27 +03:00
RoyUP9
6a15bf1aed
Fixed OAS showing unresolved services (#1037) 2022-04-24 11:28:50 +03:00
Igor Gov
2de57a6630
Fix: websocket timeouts when many traffic etries sent (#1038) 2022-04-24 11:23:59 +03:00
M. Mert Yıldıran
9715bb046b
GetEntry unmarshaling error indicates an error message sent by the database server (#1020) 2022-04-19 16:23:23 +03:00
David Levanon
1213162b85
Add kube namespace to tls (TRA-4443) (#1013)
* add namespace to tls - initial commit
* add tls namespace to mizu entry
2022-04-18 16:12:51 +03:00
M. Mert Yıldıran
189c158150
Remove TLSWarning React component (#1014) 2022-04-18 15:21:25 +03:00
M. Mert Yıldıran
c5006e5f57
TRA-4501 Don't panic in case of a Basenine connection error and try to reconnect with 3 seconds intervals (#1012)
* Don't panic in case of a Basenine connection error and try to reconnect with 3 seconds intervals

* More improvements
2022-04-18 11:00:43 +03:00
M. Mert Yıldıran
d7fcf273c0
TRA-4494 Remove all non-functional OutboundLink code that was providing /status/recentTLSLinks endpoint (#1008)
* Remove non-critical TLS detected log that causes `slice bounds out of range` error

* Remove all non-functional `OutboundLink` code that was providing `/status/recentTLSLinks` endpoint

* Fix more unused code
2022-04-17 19:01:21 +03:00
gadotroee
3ed9bc1e0d
Upgrade Basenine version to v0.7.1 (#984)
* Basenine MongoDB mess

* Fix more

* Fix the `mongodb` container arguments

* Add Basenine ARM64 binary

* Make the changes related to `leftOff` becoming a string value

* Make `leftOffTop` state string

* Handle `CloseConnection` in `Fetch`

* Upgrade Basenine to `0.7.0`

* Revert the changes in `package.json` and `package-lock.json`

* Fix the `Dockerfile`

* Remove the binaries

* Increase the Basenine up deadline to 20 seconds

* Revert the changes in `shared/kubernetes/provider.go`

* Fix the OAS generator tests

* Protect from race condition

* Fix mutexes

* Fix unlock

* Fix logging data types

* Try to stabilize the tests

* Remove the `replace` statement

* revert the change the done in 2899414f2b to not change the leftOff

* Change `leftOffBottom` empty string default value to `latest`

* Upgrade Basenine to `0.7.1`

* Handle the Basenine client library errors better

* Use `DEFAULT_QUERY` constant

* Remove `min=-1`

* Replace some `Errorf`s with `Panicf`s

* Remove the closure in `runGenerator` method

* Remove an unnecessary check

Co-authored-by: M. Mert Yildiran <mehmet@up9.com>
Co-authored-by: Andrey Pokhilko <apc4@ya.ru>
Co-authored-by: undera <undera@undera-old-desktop.home>
Co-authored-by: AmitUp9 <96980485+AmitUp9@users.noreply.github.com>
2022-04-13 11:28:48 +03:00
RoyUP9
bb425fa6e2
Fixed service map unresolved bug (#986) 2022-04-11 14:37:53 +03:00
RoyUP9
d8fb8ff710
Fix for OAS reset not working (#978) 2022-04-07 18:14:03 +03:00
Nimrod Gilboa Markevich
f344bd2633
Make minor changes to OasGenerator (#977)
* Added log message
* Remove Reset function from OasGenerator interface, use Stop+Start instead
* SetEntriesQuery returns a bool stating whether the query changed
2022-04-07 10:46:30 +03:00
RoyUP9
cf5c03d45c
Fixed service map returning nil values (#975) 2022-04-06 13:12:38 +03:00
Nimrod Gilboa Markevich
491da24c63
Add ability to set query in OAS Generator (#964) 2022-04-06 11:54:55 +03:00
RamiBerm
76a6a77a14
Refactor ws (#961)
* Separate socket and basenine logic

* WIP

* Update socket_server_handlers.go

* Update socket_data_streamer.go and socket_server_handlers.go

* Update socket_server_handlers.go

* Merge branch 'develop' into refactor_ws
# Please enter a commit message to explain why this merge is necessary,
# especially if it merges an updated upstream into a topic branch.
#
# Lines starting with '#' will be ignored, and an empty message aborts
# the commit.

* empty commit for actions

* empty commit for actions

* commit for actions

* Revert "commit for actions"

This reverts commit 8ba2ecf7d3.

Co-authored-by: RoyUP9 <87927115+RoyUP9@users.noreply.github.com>
2022-04-04 17:33:53 +03:00
Igor Gov
6a7fad430c
Adding resolved prop to service map node (#959)
* Adding resolved prop to service map node

* fixing tests
2022-04-03 15:32:21 +03:00
Nimrod Gilboa Markevich
a49443f101
Set the entry namespace to the source namespace if the destination is not resolved (#950)
* Set the entry namespace to the source namespace if the destination is not resolved
* Overwrite src namespace with dst namespace only if dst non-empty
2022-03-30 15:40:21 +03:00
Andrey Pokhilko
97db24aeba
OAS: rework data feeding + sampleIDs (#917)
* Call OAS feeder

* Don't call old OAS code

* Rework calls

* Work on it

* Put back rules

* Make it compile

* start thinking of test

* Compiles

* Save

* Fixes

* Save

* Fixing

* Trying to fake conn

* add timeout

* Test timeout

* Fix tests

* Only build OAS for HTTP entries

* Remove some dead code

* Adding SampleIDs

* Cosmetics

* lint

* Revert rename

* Sample ID for content

* Cleanuo

* Add more sample IDs

* Checking hypothesis

* Move assignment place a bit

* Cosmetics

* Update test.yml

Co-authored-by: undera <undera@undera-old-desktop.home>
Co-authored-by: Igor Gov <iggvrv@gmail.com>
2022-03-30 11:14:25 +03:00
RamiBerm
63cf7ac34e
Refactor entries controller logic (#949)
* wip

* Update entries_controller.go and entries_provider.go

* Update entries_controller.go

* change entries provider into a struct + interface

* Update entries_provider.go

* Update entries_provider.go
2022-03-29 18:30:19 +03:00
Nimrod Gilboa Markevich
a5c35d7d90
Update tap targets over ws (#901)
Update tappers via websocket instead of by env var. This way the DaemonSet doesn't have to be applied just to notify the tappers that the tap targets changed. The number of tapper restarts is reduced. The DaemonSet still gets applied when there is a need to add/remove a tapper from a node.
2022-03-23 13:50:33 +02:00
M. Mert Yıldıran
308fa78955
TRA-4383 Calculate request and response sizes and display them instead of BodySize field (#897)
* Define `ReadProgress` struct and update `Dissector` interface such that the `bufio.Reader` progress can be learned on item emitting

* Display the `requestSize` and `responseSize` fields in the UI

* Update the tests

* publish ui-common version 1.0.130 and bump to this version in ui/package.json file

Co-authored-by: gadotroee <55343099+gadotroee@users.noreply.github.com>
Co-authored-by: Roee Gadot <roee.gadot@up9.com>
2022-03-21 19:34:59 +02:00
M. Mert Yıldıran
774f07fccd
Add /db/flush and /db/reset API endpoints (#896)
* Add `/db/flush` and `/db/reset` API endpoints

* Handle the unmarshalling errors better in the WebSocket

* Handle Basenine connection error better in the WebSocket

* Upgrade to Basenine `v0.6.5`

* Fix the duplicated `StartTime` state

Co-authored-by: gadotroee <55343099+gadotroee@users.noreply.github.com>
2022-03-21 15:54:36 +02:00
RamiBerm
237002ef29
Dependency injection for oas servicemap (#895)
* Update main.go, main.go, and 3 more files...

* WIP

* Update main.go, oas_controller.go, and 3 more files...

* Update main.go, oas_generator.go, and servicemap.go

* Update loader.go and resolver.go

* Update oas_generator.go

* Update oas_generator.go, specgen_test.go, and 3 more files...

* Update service_map_controller_test.go

* Update oas_controller_test.go
2022-03-16 17:21:50 +02:00
RoyUP9
edbe4ab00b
Removed all code related to install (#892) 2022-03-15 16:41:42 +02:00
M. Mert Yıldıran
dd430c31d5
Always derive the summary and method fields from the entry in the database on read (#877)
* Always derive the summary and method fields from the entry in the database on read

* Update the expected JSONs in the protocol unit tests

* Add test cases for `Summarize` method

* Remove unused `GetEntry` method, `DataUnmarshaler` struct and `UnmarshalData` method

* Temporarily enable the acceptance tests

* Temporarily disable Slack notification on failure

* Update the Cypress tests

* Fix an issue in Redis

* Fix a typo and the Cypress tests

* Revert "Temporarily disable Slack notification on failure"

This reverts commit cad1901ea4.

* Revert "Temporarily enable the acceptance tests"

This reverts commit bad7706c9b.
2022-03-06 15:41:36 +03:00
M. Mert Yıldıran
3b063c3bb5
Add the ability to set the insertion filter into CLI (#843)
* Add the ability to set the insertion filter into CLI

* Resolve the conflict

* Upgrade Basenine to `v0.5.0` and check the insertion filter against syntax errors inside CLI

* Upgrade Basenine to `v0.5.1`

* Update the sums
2022-03-02 12:51:32 +03:00
RamiBerm
346e904e77
Support stopping oas/servicemesh/telemetry in flight (#869)
* Update oas_generator.go and servicemap.go

* Update oas_generator.go

* Update esClient.go

* Update servicemap.go
2022-03-02 09:49:17 +02:00
Andrey Pokhilko
817c17ad32
OAS: Generate parameter patterns (#837)
* Figure out decimal pattern

* More pre-loaded

* Tune predefined spec

* Prefix and suffix detected

* Implement entry compacting

* God bless unit tests

* Fix a bug

* REsearch it more

* Write back test file

* Working on merging pathObjs

* Working on merging ops

* Commit

* Roll back some changes

* Fix test

* Fixed sorting

* Accept test file

* adding end line in the end of the file

Co-authored-by: Igor Gov <iggvrv@gmail.com>
2022-02-28 17:40:05 +03:00
Igor Gov
371e513249
Remove config dependency from basenine init (#846)
* Remove config dependency from basenine init
2022-02-23 10:06:19 +02:00
Igor Gov
97cce32e3f
Fix: service map component aware of agent config (#845)
* Fix: Service map component aware of mizu config
2022-02-23 09:35:05 +02:00
M. Mert Yıldıran
73f3e448cf
Fix the issue in the WebSocket that's introduced by #819 (#839) 2022-02-22 15:27:50 +03:00
Igor Gov
58a17897cf
Agent logs: output to stderr errors and above (#838)
* Agent logs: output to stderr errors and above
2022-02-22 14:02:51 +02:00
M. Mert Yıldıran
cf3106f636
Silence the logging that comes from Basenine and protocol dissectors (#835) 2022-02-21 18:08:16 +03:00
Andrey Pokhilko
a553a1b683
OAS: use resolved service names (#827)
* OAS service names to be resolved

* fix test
2022-02-21 17:20:59 +03:00
M. Mert Yıldıran
27fa0afb72
TRA-4331 Implement full data streaming over WebSocket (#819)
* Implement full data streaming over WebSocket

* Fix the linting error

* Make the empty being the criteria

* Use a label to break the nested loop
2022-02-17 17:01:44 +03:00
David Levanon
87ef469e25
Add tls tapper (#683)
* initial tls tapper commit

* add tls flag to mizu cli

* support ssl_read_ex/ssl_write_ex

* use hostproc to find libssl

* auto discover tls processes

* support libssl1.0

* recompile ebpf with old clang/llvm

* Update tap/passive_tapper.go

Co-authored-by: M. Mert Yıldıran <mehmet@up9.com>

* Update tap/tlstapper/tls_poller.go

Co-authored-by: M. Mert Yıldıran <mehmet@up9.com>

* Update tap/tlstapper/tls_poller.go

Co-authored-by: M. Mert Yıldıran <mehmet@up9.com>

* Update tap/tlstapper/tls_poller.go

Co-authored-by: M. Mert Yıldıran <mehmet@up9.com>

* Update tap/tlstapper/tls_poller.go

Co-authored-by: M. Mert Yıldıran <mehmet@up9.com>

* Update tap/tlstapper/tls_poller.go

Co-authored-by: M. Mert Yıldıran <mehmet@up9.com>

* Update tap/tlstapper/tls_poller.go

Co-authored-by: M. Mert Yıldıran <mehmet@up9.com>

* Update tap/tlstapper/tls_poller.go

Co-authored-by: M. Mert Yıldıran <mehmet@up9.com>

* upgrade ebpf go lib

* handling big tls messages

* fixing max buffer size in ebpf

* remove unused import

* fix linter issues

* minor pr fixes

* compile with old clang

* fix cgroup file format

* pr fixes + cgroup extract enhance

* fix linter

* adding indirect ebpf dep to agent go.mod

* adding ebpf docker builder

* minor pr fixes

* add req resp matcher to dissect

* rename ssl hooks to ssl hooks structs

* move to alpine, use local copy of mizu instead of git, add readme

* use global req resp mather for tls

Co-authored-by: M. Mert Yıldıran <mehmet@up9.com>
Co-authored-by: gadotroee <55343099+gadotroee@users.noreply.github.com>
2022-02-16 15:34:51 +02:00
Gustavo Massaneiro
bb3ae1ef70
Service Map node key as entry.Name instead of entry.IP (#818) 2022-02-16 09:01:59 +02:00
RamiBerm
4197ec198c
Tag entries destination namespace (#803)
* Update main.go, socket_server_handlers.go, and 7 more files...

* Update cors.go

* omit empty namespace in json so tests pass

* fix indent
2022-02-15 10:51:38 +02:00
Igor Gov
88e3fba1ef
Fix: tapper status race condition (#788) 2022-02-10 12:13:40 +02:00
RamiBerm
145004fe43
TRA-4263 revert extensible routing (#774)
* Update main.go, extensions.go, and 2 more files...

* Update config_routes.go, entries_routes.go, and 7 more files...
2022-02-08 17:48:30 +02:00