Commit Graph

168 Commits

Author SHA1 Message Date
gadotroee
ec11b21b51
Add available protocols to the stats endpoint (colors for methods are coming from server) (#1184)
* add protocols array to the endpoint

* no message

* no message

* fix tests and small fix for the iteration

* fix the color of the protocol

* Get protocols list and method colors from server

* fix tests

* cr fixes

Co-authored-by: Amit Fainholts <amit@up9.com>
2022-07-05 15:30:39 +03:00
gadotroee
d6944d467c
Merge traffic stats endpoints to one and add auto interval logic (#1174) 2022-07-04 10:01:49 +03:00
lirazyehezkel
302333b4ae
TRA-4622 Remove rules feature UI (#1178)
* Removed policy rules (validation rules) feature

* updated test pcap

* Remove rules

* fix replay in rules

Co-authored-by: Roy Island <roy@up9.com>
Co-authored-by: RoyUP9 <87927115+RoyUP9@users.noreply.github.com>
Co-authored-by: Roee Gadot <roee.gadot@up9.com>
2022-07-03 11:32:23 +03:00
gadotroee
3a9236a381
Fix replay to be like the test (#1173)
Represent should get entry that Marshaled and UnMarshaled
2022-06-29 11:54:16 +03:00
gadotroee
01af6aa19c
Add reply endpoint for http (#1168) 2022-06-28 18:39:23 +03:00
RoyUP9
2df9fb49db
Changed entry protocol field from struct to unique string (#1167) 2022-06-26 17:08:09 +03:00
RoyUP9
1ebc51b45e
Remove OAS contract monitoring feature (#1163) 2022-06-26 12:22:25 +03:00
gadotroee
a96072623f
Fix problems that golang inspection reported (#1158) 2022-06-23 16:49:56 +03:00
gadotroee
8c187179b0
Adding tests to the bucket statistics (#1160) 2022-06-23 11:04:12 +03:00
gadotroee
9a40895e9c
Add timing endpoint (timeline data) for stats (#1157) 2022-06-21 14:33:04 +03:00
gadotroee
6f117d0a84
Accumulative stats for protocol and methods (#1144)
only data for pie chart currently
2022-06-19 16:59:35 +03:00
Andrey Pokhilko
7a823e89f1
Add option to limit example len (#1145)
Do not save OAS example response if it is over the limit.
SpecGen.MaxExampleLen is measured in bytes.
Limit=0 means "don't record any examples", while limit=-1 means "record all".

Limit is configured via the config file / --set: --set oas.max-example-len=100.
The default is 10KB.

This PR breaks the previous config. Moved oas bool (enable/disable OAS) to oas.enable.
Users who wish to disable OAS (or to explicitly enable it) should now to so with --set oas.enable=false instead of --set oas=false.

Co-authored-by: Nimrod Gilboa Markevich <nimrod@up9.com>
Co-authored-by: Nimrod Gilboa Markevich <59927337+nimrod-up9@users.noreply.github.com>
2022-06-16 14:18:42 +03:00
gadotroee
2aeac6c9e6
Remove analyze feature (#1121) 2022-06-01 10:17:14 +03:00
RoyUP9
00cc94fbe5
Changed OAS generator to get entries by push (#1103) 2022-05-22 14:53:48 +03:00
Nimrod Gilboa Markevich
ab38f4c011
Add profiling tools (#1087)
* Add gin-contrib/pprof dependency

* Run pprof server on agent with --profiler flag

* Add --profiler flag to cli

* Fix error message

* Print cpu usage percentage

* measure cpu of current pid instead of globaly on the system

* Add scripts to plot performance

* Plot packetsCount in analysis

* Concat to DataFrame

* Plot in turbo colorscheme

* Make COLORMAP const

* Fix rss units

* Reduce code repetition by adding function for plotting

* Allow grouping based on filenames

* Temporary: Marked with comments where to disable code for experiments

* Add newline at end of file

* Add tap.cpuprofile flag. Change memprofile flag to tap.memprofile

* create tapper modes for debugging using env vars

* Fix rss plot units (MB instead of bytes)

* Remove comment

* Add info to plot script

* Remove tap.cpumemprofile. Rename tap.memprofile to memprofile

* Remove unused import

* Remove whitespaces

Co-authored-by: M. Mert Yıldıran <mehmet@up9.com>

* Remove whitespaces

Co-authored-by: M. Mert Yıldıran <mehmet@up9.com>

* Remove whitespaces

Co-authored-by: M. Mert Yıldıran <mehmet@up9.com>

* Remove whitespaces

Co-authored-by: M. Mert Yıldıran <mehmet@up9.com>

* Remove whitespaces

Co-authored-by: M. Mert Yıldıran <mehmet@up9.com>

* Remove whitespaces

Co-authored-by: M. Mert Yıldıran <mehmet@up9.com>

* Rename debug env vars

* Create package for debug env vars, read each env var once

* Run go mod tidy

* Increment MatchedPairs before emitting

* Only count cores once

* Count virtual and physical cores

* Add dbgctl replace in cli

* Fix lint: Check return values

* Add tap/dbgctl to test-lint make rule

* Replace tap/dbgctl in all modules

* #run_acceptance_tests

* Copy dbgctl module to docker image

* Debug/profile tapper benchmark (#1093)

* add mizu debug env to avoid all extensions

* add readme + run_tapper_benchmark.sh

* temporary change branch name

* fix readme

* fix MIZU_BENCHMARK_CLIENTS_COUNT env

* change tap target to tcp stream

* track live tcp streams

* pr fixes

* rename tapperPacketsCount to ignored_packets_count

* change mizu tapper to mizu debugg

Co-authored-by: David Levanon <dvdlevanon@gmail.com>
Co-authored-by: M. Mert Yıldıran <mehmet@up9.com>
2022-05-18 15:42:13 +03:00
Igor Gov
74bd4b180f
Removing elastic dump experimental feature (#1086)
* Removing elastic dump experimental feature
2022-05-15 15:23:03 +03:00
RoyUP9
8ea2dabb34
Added send to socket error validation (#1085) 2022-05-15 14:44:48 +03:00
RoyUP9
09a0fca2c2
Extracted insert to database functionality (#1082) 2022-05-15 09:19:33 +03:00
RoyUP9
2fac0009ea
Fixed oas query (#1076) 2022-05-11 15:36:46 +03:00
M. Mert Yıldıran
eef0ee8023
Fetch N number of records in M milliseconds timeout before streaming the records (#1056)
* Fetch N number of records in M milliseconds timeout before streaming the records

* Implement the functionality inside socket data streamer

* Reverse the `fetchData` slice

* #run_acceptance_tests

* Trying to fix the tests.
#run_acceptance_tests

* javascript compilation error.

* #run_acceptance_tests

* Name the method better

* Upgrade Basenine version to `v0.8.0`

* Fix some issues related to `Fetch`

* Upgrade the Basenine version in `Dockerfile` as well

* Remove underscore from the parameter name

* Parameterize fetch timeout ms

Co-authored-by: gadotroee <55343099+gadotroee@users.noreply.github.com>
Co-authored-by: Roee Gadot <roee.gadot@up9.com>
2022-05-09 19:11:27 +03:00
RoyUP9
4c0aeb8146
Fixed default mime type to empty string instead of text/html (#1071) 2022-05-08 16:52:50 +03:00
gadotroee
2f89690da6
Remove go-cache unused dependency (#1065) 2022-05-02 13:24:28 +03:00
M. Mert Yıldıran
e1ad302c29
Make logger a separate module such that don't depend on shared module as a whole for logging (#1047)
* Make `logger` a separate module such that don't depend on `shared` module as a whole for logging

* Update `Dockerfile`
2022-04-27 22:26:27 +03:00
RoyUP9
6a15bf1aed
Fixed OAS showing unresolved services (#1037) 2022-04-24 11:28:50 +03:00
Igor Gov
2de57a6630
Fix: websocket timeouts when many traffic etries sent (#1038) 2022-04-24 11:23:59 +03:00
M. Mert Yıldıran
9715bb046b
GetEntry unmarshaling error indicates an error message sent by the database server (#1020) 2022-04-19 16:23:23 +03:00
David Levanon
1213162b85
Add kube namespace to tls (TRA-4443) (#1013)
* add namespace to tls - initial commit
* add tls namespace to mizu entry
2022-04-18 16:12:51 +03:00
M. Mert Yıldıran
189c158150
Remove TLSWarning React component (#1014) 2022-04-18 15:21:25 +03:00
M. Mert Yıldıran
c5006e5f57
TRA-4501 Don't panic in case of a Basenine connection error and try to reconnect with 3 seconds intervals (#1012)
* Don't panic in case of a Basenine connection error and try to reconnect with 3 seconds intervals

* More improvements
2022-04-18 11:00:43 +03:00
M. Mert Yıldıran
d7fcf273c0
TRA-4494 Remove all non-functional OutboundLink code that was providing /status/recentTLSLinks endpoint (#1008)
* Remove non-critical TLS detected log that causes `slice bounds out of range` error

* Remove all non-functional `OutboundLink` code that was providing `/status/recentTLSLinks` endpoint

* Fix more unused code
2022-04-17 19:01:21 +03:00
gadotroee
3ed9bc1e0d
Upgrade Basenine version to v0.7.1 (#984)
* Basenine MongoDB mess

* Fix more

* Fix the `mongodb` container arguments

* Add Basenine ARM64 binary

* Make the changes related to `leftOff` becoming a string value

* Make `leftOffTop` state string

* Handle `CloseConnection` in `Fetch`

* Upgrade Basenine to `0.7.0`

* Revert the changes in `package.json` and `package-lock.json`

* Fix the `Dockerfile`

* Remove the binaries

* Increase the Basenine up deadline to 20 seconds

* Revert the changes in `shared/kubernetes/provider.go`

* Fix the OAS generator tests

* Protect from race condition

* Fix mutexes

* Fix unlock

* Fix logging data types

* Try to stabilize the tests

* Remove the `replace` statement

* revert the change the done in 2899414f2b to not change the leftOff

* Change `leftOffBottom` empty string default value to `latest`

* Upgrade Basenine to `0.7.1`

* Handle the Basenine client library errors better

* Use `DEFAULT_QUERY` constant

* Remove `min=-1`

* Replace some `Errorf`s with `Panicf`s

* Remove the closure in `runGenerator` method

* Remove an unnecessary check

Co-authored-by: M. Mert Yildiran <mehmet@up9.com>
Co-authored-by: Andrey Pokhilko <apc4@ya.ru>
Co-authored-by: undera <undera@undera-old-desktop.home>
Co-authored-by: AmitUp9 <96980485+AmitUp9@users.noreply.github.com>
2022-04-13 11:28:48 +03:00
RoyUP9
bb425fa6e2
Fixed service map unresolved bug (#986) 2022-04-11 14:37:53 +03:00
RoyUP9
d8fb8ff710
Fix for OAS reset not working (#978) 2022-04-07 18:14:03 +03:00
Nimrod Gilboa Markevich
f344bd2633
Make minor changes to OasGenerator (#977)
* Added log message
* Remove Reset function from OasGenerator interface, use Stop+Start instead
* SetEntriesQuery returns a bool stating whether the query changed
2022-04-07 10:46:30 +03:00
RoyUP9
cf5c03d45c
Fixed service map returning nil values (#975) 2022-04-06 13:12:38 +03:00
Nimrod Gilboa Markevich
491da24c63
Add ability to set query in OAS Generator (#964) 2022-04-06 11:54:55 +03:00
RamiBerm
76a6a77a14
Refactor ws (#961)
* Separate socket and basenine logic

* WIP

* Update socket_server_handlers.go

* Update socket_data_streamer.go and socket_server_handlers.go

* Update socket_server_handlers.go

* Merge branch 'develop' into refactor_ws
# Please enter a commit message to explain why this merge is necessary,
# especially if it merges an updated upstream into a topic branch.
#
# Lines starting with '#' will be ignored, and an empty message aborts
# the commit.

* empty commit for actions

* empty commit for actions

* commit for actions

* Revert "commit for actions"

This reverts commit 8ba2ecf7d3.

Co-authored-by: RoyUP9 <87927115+RoyUP9@users.noreply.github.com>
2022-04-04 17:33:53 +03:00
Igor Gov
6a7fad430c
Adding resolved prop to service map node (#959)
* Adding resolved prop to service map node

* fixing tests
2022-04-03 15:32:21 +03:00
Nimrod Gilboa Markevich
a49443f101
Set the entry namespace to the source namespace if the destination is not resolved (#950)
* Set the entry namespace to the source namespace if the destination is not resolved
* Overwrite src namespace with dst namespace only if dst non-empty
2022-03-30 15:40:21 +03:00
Andrey Pokhilko
97db24aeba
OAS: rework data feeding + sampleIDs (#917)
* Call OAS feeder

* Don't call old OAS code

* Rework calls

* Work on it

* Put back rules

* Make it compile

* start thinking of test

* Compiles

* Save

* Fixes

* Save

* Fixing

* Trying to fake conn

* add timeout

* Test timeout

* Fix tests

* Only build OAS for HTTP entries

* Remove some dead code

* Adding SampleIDs

* Cosmetics

* lint

* Revert rename

* Sample ID for content

* Cleanuo

* Add more sample IDs

* Checking hypothesis

* Move assignment place a bit

* Cosmetics

* Update test.yml

Co-authored-by: undera <undera@undera-old-desktop.home>
Co-authored-by: Igor Gov <iggvrv@gmail.com>
2022-03-30 11:14:25 +03:00
RamiBerm
63cf7ac34e
Refactor entries controller logic (#949)
* wip

* Update entries_controller.go and entries_provider.go

* Update entries_controller.go

* change entries provider into a struct + interface

* Update entries_provider.go

* Update entries_provider.go
2022-03-29 18:30:19 +03:00
Nimrod Gilboa Markevich
a5c35d7d90
Update tap targets over ws (#901)
Update tappers via websocket instead of by env var. This way the DaemonSet doesn't have to be applied just to notify the tappers that the tap targets changed. The number of tapper restarts is reduced. The DaemonSet still gets applied when there is a need to add/remove a tapper from a node.
2022-03-23 13:50:33 +02:00
M. Mert Yıldıran
308fa78955
TRA-4383 Calculate request and response sizes and display them instead of BodySize field (#897)
* Define `ReadProgress` struct and update `Dissector` interface such that the `bufio.Reader` progress can be learned on item emitting

* Display the `requestSize` and `responseSize` fields in the UI

* Update the tests

* publish ui-common version 1.0.130 and bump to this version in ui/package.json file

Co-authored-by: gadotroee <55343099+gadotroee@users.noreply.github.com>
Co-authored-by: Roee Gadot <roee.gadot@up9.com>
2022-03-21 19:34:59 +02:00
M. Mert Yıldıran
774f07fccd
Add /db/flush and /db/reset API endpoints (#896)
* Add `/db/flush` and `/db/reset` API endpoints

* Handle the unmarshalling errors better in the WebSocket

* Handle Basenine connection error better in the WebSocket

* Upgrade to Basenine `v0.6.5`

* Fix the duplicated `StartTime` state

Co-authored-by: gadotroee <55343099+gadotroee@users.noreply.github.com>
2022-03-21 15:54:36 +02:00
RamiBerm
237002ef29
Dependency injection for oas servicemap (#895)
* Update main.go, main.go, and 3 more files...

* WIP

* Update main.go, oas_controller.go, and 3 more files...

* Update main.go, oas_generator.go, and servicemap.go

* Update loader.go and resolver.go

* Update oas_generator.go

* Update oas_generator.go, specgen_test.go, and 3 more files...

* Update service_map_controller_test.go

* Update oas_controller_test.go
2022-03-16 17:21:50 +02:00
RoyUP9
edbe4ab00b
Removed all code related to install (#892) 2022-03-15 16:41:42 +02:00
M. Mert Yıldıran
dd430c31d5
Always derive the summary and method fields from the entry in the database on read (#877)
* Always derive the summary and method fields from the entry in the database on read

* Update the expected JSONs in the protocol unit tests

* Add test cases for `Summarize` method

* Remove unused `GetEntry` method, `DataUnmarshaler` struct and `UnmarshalData` method

* Temporarily enable the acceptance tests

* Temporarily disable Slack notification on failure

* Update the Cypress tests

* Fix an issue in Redis

* Fix a typo and the Cypress tests

* Revert "Temporarily disable Slack notification on failure"

This reverts commit cad1901ea4.

* Revert "Temporarily enable the acceptance tests"

This reverts commit bad7706c9b.
2022-03-06 15:41:36 +03:00
M. Mert Yıldıran
3b063c3bb5
Add the ability to set the insertion filter into CLI (#843)
* Add the ability to set the insertion filter into CLI

* Resolve the conflict

* Upgrade Basenine to `v0.5.0` and check the insertion filter against syntax errors inside CLI

* Upgrade Basenine to `v0.5.1`

* Update the sums
2022-03-02 12:51:32 +03:00
RamiBerm
346e904e77
Support stopping oas/servicemesh/telemetry in flight (#869)
* Update oas_generator.go and servicemap.go

* Update oas_generator.go

* Update esClient.go

* Update servicemap.go
2022-03-02 09:49:17 +02:00
Andrey Pokhilko
817c17ad32
OAS: Generate parameter patterns (#837)
* Figure out decimal pattern

* More pre-loaded

* Tune predefined spec

* Prefix and suffix detected

* Implement entry compacting

* God bless unit tests

* Fix a bug

* REsearch it more

* Write back test file

* Working on merging pathObjs

* Working on merging ops

* Commit

* Roll back some changes

* Fix test

* Fixed sorting

* Accept test file

* adding end line in the end of the file

Co-authored-by: Igor Gov <iggvrv@gmail.com>
2022-02-28 17:40:05 +03:00