Commit Graph

234 Commits

Author SHA1 Message Date
M. Mert Yıldıran
9eecddddd5 Start the tapper after the API server is ready (#309) 2021-09-30 11:22:07 +03:00
M. Mert Yıldıran
cc49e815d6 Watch the tapper pod after starting it (#310)
* Watch the tapper pod after starting it

* Improve the logic in `watchTapperPod` method
2021-09-30 09:32:27 +03:00
Selton Fiuza
c26eb843e3 [refactor/TRA-3693] type:latency to slo and latency field to response-time (#282)
* type:latency to slo and latency field to response-time

* remove comment from import

* Friendly message on ignored rules and format

* formatting

* change conditional to catch negative values and ignore it

* Fix Bug Alon Reported

* sliceUtils to shared
2021-09-29 11:51:03 -03:00
Nimrod Gilboa Markevich
9212c195b4 Improve cloud resources cleanup (#215) 2021-09-23 20:51:37 +03:00
RoyUP9
5dacd41ba9 renamed traffic-validation to traffic-validation-file (#296) 2021-09-22 11:21:43 +03:00
Selton Fiuza
b50eced489 [Refactor/TRA-3692] rename test rules to traffic validation (#281) 2021-09-19 14:47:19 +03:00
M. Mert Yıldıran
5392475486 Fix the issues related to sensitive data filtering feature (#285)
* Run acceptance tests on pull request

* Take `options.DisableRedaction` into account

* Log `defaultTapConfig`

* Pass the `SENSITIVE_DATA_FILTERING_OPTIONS` to tapper daemon set too

* Revert "Run acceptance tests on pull request"

This reverts commit ad79f1418f.
2021-09-19 13:33:34 +03:00
M. Mert Yıldıran
9fa9b67328 Bring back the sensitive data filtering feature (#280)
* Bring back the sensitive data filtering feature

* Add `// global` comment
2021-09-18 20:13:59 +03:00
RoyUP9
d5b01347df fixed crash when channel is closed (#273) 2021-09-14 11:53:47 +03:00
RoyUP9
6b52458642 removed exit if browser not supported (#270) 2021-09-12 16:09:04 +03:00
gadotroee
073b0b72d3 Remove fetch command (and direction) (#264) 2021-09-06 10:16:04 +03:00
M. Mert Yıldıran
366c1d0c6c Refactor Mizu, define an extension API and add new protocols: AMQP, Kafka (#224)
* Separate HTTP related code into `extensions/http` as a Go plugin

* Move `extensions` folder into `tap` folder

* Move HTTP files into `tap/extensions/lib` for now

* Replace `orcaman/concurrent-map` with `sync.Map`

* Remove `grpc_assembler.go`

* Remove `github.com/up9inc/mizu/tap/extensions/http/lib`

* Add a build script to automatically build extensions from a known path and load them

* Start to define the extension API

* Implement the `run()` function for the TCP stream

* Add support of defining multiple ports to the extension API

* Set the extension name inside the extension

* Declare the `Dissect` function in the extension API

* Dissect HTTP request from inside the HTTP extension

* Make the distinction of outbound and inbound ports

* Dissect HTTP response from inside the HTTP extension

* Bring back the HTTP request-response pair matcher

* Return a `*api.RequestResponsePair` from the dissection

* Bring back the gRPC-HTTP/2 parser

* Fix the issues in `handleHTTP1ClientStream` and `handleHTTP1ServerStream`

* Call a function pointer to emit dissected data back to the `tap` package

* roee changes -
trying to fix agent to work with the "api" object) - ***still not working***

* small mistake in the conflicts

* Fix the issues that are introduced by the merge conflict

* Add `Emitter` interface to the API and send `OutputChannelItem`(s) to `OutputChannel`

* Fix the `HTTP1` handlers

* Set `ConnectionInfo` in HTTP handlers

* Fix the `Dockerfile` to build the extensions

* remove some unwanted code

* no message

* Re-enable `getStreamProps` function

* Migrate back from `gopacket/tcpassembly` to `gopacket/reassembly`

* Introduce `HTTPPayload` struct and `HTTPPayloader` interface to `MarshalJSON()` all the data structures that are returned by the HTTP protocol

* Read `socketHarOutChannel` instead of `filteredHarChannel`

* Connect `OutputChannelItem` to the last WebSocket means that finally the web UI started to work again

* Add `.env.example` to React app

* Marshal and unmarshal `*http.Request`, `*http.Response` pairs

* Move `loadExtensions` into `main.go` and map extensions into `extensionsMap`

* Add `Summarize()` method to the `Dissector` interface

* Add `Analyze` method to the `Dissector` interface and `MizuEntry` to the extension API

* Add `Protocol` struct and make it effect the UI

* Refactor `BaseEntryDetails` struct and display the source and destination ports in the UI

* Display the protocol name inside the details layout

* Add `Represent` method to the `Dissector` interface and manipulate the UI through this method

* Make the protocol color affect the details layout color and write protocol abbreviation vertically

* Remove everything HTTP related from the `tap` package and make the extension system fully functional

* Fix the TypeScript warnings

* Bring in the files related AMQP into `amqp` directory

* Add `--nodefrag` flag to the tapper and bring in the main AMQP code

* Implement the AMQP `BasicPublish` and fix some issues in the UI when the response payload is missing

* Implement `representBasicPublish` method

* Fix several minor issues

* Implement the AMQP `BasicDeliver`

* Implement the AMQP `QueueDeclare`

* Implement the AMQP `ExchangeDeclare`

* Implement the AMQP `ConnectionStart`

* Implement the AMQP `ConnectionClose`

* Implement the AMQP `QueueBind`

* Implement the AMQP `BasicConsume`

* Fix an issue in `ConnectionStart`

* Fix a linter error

* Bring in the files related Kafka into `kafka` directory

* Fix the build errors in Kafka Go files

* Implement `Dissect` method of Kafka and adapt request-response pair matcher to asynchronous client-server stream

* Do the "Is reversed?" checked inside `getStreamProps` and fix an issue in Kafka `Dissect` method

* Implement `Analyze`, `Summarize` methods of Kafka

* Implement the representations for Kafka `Metadata`, `RequestHeader` and `ResponseHeader`

* Refactor the AMQP and Kafka implementations to create the summary string only inside the `Analyze` method

* Implement the representations for Kafka `ApiVersions`

* Implement the representations for Kafka `Produce`

* Implement the representations for Kafka `Fetch`

* Implement the representations for Kafka `ListOffsets`, `CreateTopics` and `DeleteTopics`

* Fix the encoding of AMQP `BasicPublish` and `BasicDeliver` body

* Remove the unnecessary logging

* Remove more logging

* Introduce `Version` field to `Protocol` struct for dynamically switching the HTTP protocol to HTTP/2

* Fix the issues in analysis and representation of HTTP/2 (gRPC) protocol

* Fix the issues in summary section of details layout for HTTP/2 (gRPC) protocol

* Fix the read errors that freezes the sniffer in HTTP and Kafka

* Fix the issues in HTTP POST data

* Fix one more issue in HTTP POST data

* Fix an infinite loop in Kafka

* Fix another freezing issue in Kafka

* Revert "UI Infra - Support multiple entry types + refactoring (#211)"

This reverts commit f74a52d4dc.

* Fix more issues that are introduced by the merge

* Fix the status code in the summary section

* adding the cleaner again (why we removed it?).
add TODO: on the extension loop .

* fix dockerfile (remove deleting .env file) - it is found in dockerignore and fails to build if the file not exists

* fix GetEntrties ("/entries" endpoint) - working with "tapApi.BaseEntryDetail" (moved from shared)

* Fix an issue in the UI summary section

* Refactor the protocol payload structs

* Fix a log message in the passive tapper

* Adapt `APP_PORTS` environment variable to the new extension system and change its format to `APP_PORTS='{"http": ["8001"]}' `

* Revert "fix dockerfile (remove deleting .env file) - it is found in dockerignore and fails to build if the file not exists"

This reverts commit 4f514ae1f4.

* Bring in the necessary changes from f74a52d4dc

* Open the API server URL in the web browser as soon as Mizu is ready

* Make the TCP reader consists of a single Go routine (instead of two) and try to dissect in both client and server mode by rewinding

* Swap `TcpID` without overwriting it

* Sort extension by priority

* Try to dissect with looping through all the extensions

* fix getStreamProps function.
(it should be passed from CLI as it was before).

* Turn TCP reader back into two Goroutines (client and server)

* typo

* Learn `isClient` from the TCP stream

* Set `viewer` style `overflow: "auto"`

* Fix the memory leaks in AMQP and Kafka dissectors

* Revert some of the changes in be7c65eb6d

* Remove `allExtensionPorts` since it's no longer needed

* Remove `APP_PORTS` since it's no longer needed

* Fix all of the minor issues in the React code

* Check Kafka header size and fail-fast

* Break the dissectors loop upon a successful dissection

* Don't break the dissector loop. Protocols might collide

* Improve the HTTP request-response counter (still not perfect)

* Make the HTTP request-response counter perfect

* Revert "Revert some of the changes in be7c65eb6d3fb657a059707da3ca559937e59739"

This reverts commit 08e7d786d8.

* Bring back `filterItems` and `isHealthCheckByUserAgent` functions

* Remove some development artifacts

* remove unused and commented lines that are not relevant

* Fix the performance in TCP stream factory. Make it create two `tcpReader`(s) per extension

* Change a log to debug

* Make `*api.CounterPair` a field of `tcpReader`

* Set `isTapTarget` to always `true` again since `filterAuthorities` implementation has problems

* Remove a variable that's only used for logging even though not introduced by this branch

* Bring back the `NumberOfRules` field of `ApplicableRules` struct

* Remove the unused `NewEntry` function

* Move `k8sResolver == nil` check to a more appropriate place

* default healthChecksUserAgentHeaders should be empty array (like the default config value)

* remove spam console.log

* Rules button cause app to crash (access the service via incorrect property)

* Ignore all .env* files in docker build.

* Better caching in dockerfile: only copy go.mod before go mod download.

* Check for errors while loading an extension

* Add a comment about why `Protocol` is not a pointer

* Bring back the call to `deleteOlderThan`

* Remove the `nil` check

* Reduce the maximum allowed AMQP message from 128MB to 1MB

* Fix an error that only occurs when a Kafka broker is initiating

* Revert the change in b2abd7b990

* Fix the service name resolution in all protocols

* Remove the `anydirection` flag and fix the issue in `filterAuthorities`

* Pass `sync.Map` by reference to `deleteOlderThan` method

* Fix the packet capture issue in standalone mode that's introduced by the removal of `anydirection`

* Temporarily resolve the memory exhaustion in AMQP

* Fix a nil pointer dereference error

* Fix the CLI build error

* Fix a memory leak that's identified by `pprof`

Co-authored-by: Roee Gadot <roee.gadot@up9.com>
Co-authored-by: Nimrod Gilboa Markevich <nimrod@up9.com>
2021-09-02 14:34:06 +03:00
RoyUP9
17fa163ee3 added proxy logs, added events logs (#254) 2021-09-01 15:30:37 +03:00
RoyUP9
e25e7925b6 fixed version blocking (#251) 2021-08-30 15:11:14 +03:00
RoyUP9
a9e92b60f5 added custom config path option (#247) 2021-08-26 13:50:41 +03:00
RoyUP9
35e40cd230 added tap acceptance tests, fixed duplicate namespace problem (#244) 2021-08-26 09:56:18 +03:00
Igor Gov
2575ad722a Introducing API server provider (#243) 2021-08-22 11:41:38 +03:00
Igor Gov
6dd0ef1268 Adding user friendly message in view command before sleeping (#239) 2021-08-19 12:22:18 +03:00
RoyUP9
667f0dc87d fixed namespace restricted validation (#235) 2021-08-19 11:33:48 +03:00
Igor Gov
a34c2fc0dc Adding version check to all commands execution (#236) 2021-08-19 11:33:20 +03:00
RoyUP9
7f9fd82c0e fixed panic when using invalid kube config path (#231) 2021-08-19 10:59:31 +03:00
Igor Gov
a9b5eba9d4 Fix: View command fail sporadically (#228) 2021-08-19 09:44:43 +03:00
RoyUP9
bf27e94003 fixed version check, removed duplicate kube config, fix flags warning, fixed log of invalid config (#227) 2021-08-18 18:10:47 +03:00
RoyUP9
db1f4458c5 Introducing acceptance test (#222) 2021-08-18 10:22:45 +03:00
RoyUP9
f9677dbaa1 added resources to config (#208) 2021-08-12 16:33:32 +03:00
RoyUP9
c8e5886a96 added telemetry api calls (#201) 2021-08-11 15:57:41 +03:00
Igor Gov
7b73004e85 Cli pkg refactor (2) (#200) 2021-08-11 09:56:03 +03:00
RoyUP9
d705ae3eb6 added support of slice in set, removed support of allowed set flags (#191) 2021-08-10 16:16:58 +03:00
gadotroee
413fb5b3f5 Add option to supply user agents to ignore via config (#173) 2021-08-09 12:27:13 +03:00
RoyUP9
dfea8884d4 Adding 'configuration' section in readme (#180) 2021-08-08 14:05:15 +03:00
Igor Gov
d34dacbbe2 View command - moving version check after proxy creation (#177) 2021-08-08 12:26:58 +03:00
Nimrod Gilboa Markevich
ebbe6458a8 Do not tap pods whose names start with "mizu-". (#176) 2021-08-08 10:51:39 +03:00
Igor Gov
7f2021c312 Several fixes for the release (#175) 2021-08-08 10:32:21 +03:00
Igor Gov
0244f12167 Fixes (#171) 2021-08-05 19:29:06 +03:00
RoyUP9
60533a9591 added allowed set flag (#169) 2021-08-05 14:23:16 +03:00
Igor Gov
90f0f603c7 Support getting logs in ns restricted mode (#168) 2021-08-05 12:12:01 +03:00
RoyUP9
683d199774 added support of multiple namespaces (#167) 2021-08-05 11:19:29 +03:00
Igor Gov
fa632b49a7 Introducing mizu logs dump & Log prints alignment in API server using rlog (#165) 2021-08-05 11:01:08 +03:00
Nimrod Gilboa Markevich
04579eb03c Namespace restricted mode (#147) 2021-08-05 10:28:31 +03:00
Selton Fiuza
dea223bfe1 Feature/tra 3349 validation rules merged with develop (#148)
* Implemented validation rules, based on: https://up9.atlassian.net/browse/TRA-3349

* Color on Entry based on rules

* Background red/green based on rules

* Change flag --validation-rules to --test-rules

* rules tab UI updated

* rules tab font and background-color is changed for objects

* Merged with develop

* Fixed compilation issues.

* Renamed fullEntry -> harEntry where appropriate.

* Change green/red logic

* Update models.go

* Fix latency bug and alignment

* Merge Conflicts fix

* Working after merge

* Working on Nimrod comments

* Resolving conflicts

* Resolving conflicts

* Resolving conflicts

* Nimrod Comments pt.3

* Log Error on configmap creation if the user doesn't have permission.

* Checking configmap permission to ignore --test-rules

* Revert time for mizu to get ready

* Nimrod comments pt 4 && merge develop pt3

* Nimrod comments pt 4 && merge develop pt3

* Const rulePolicyPath and filename

Co-authored-by: Neim <elezin9@gmail.com>
Co-authored-by: nimrod-up9 <nimrod@up9.com>
2021-08-04 09:21:36 -03:00
Igor Gov
d18f1f8316 Tapped pods report via endpoint instead of web socket (#164) 2021-08-04 10:41:33 +03:00
Igor Gov
f9202900ee No warning when mizu rbac exists (#163) 2021-08-04 08:41:00 +03:00
Igor Gov
9e34662511 Adding logs and fixing several issues (#162)
* Config grooming and several general fixes
2021-08-04 08:18:07 +03:00
RoyUP9
69a9deab4b Feature/tra 3474 config refactor (#155) 2021-08-03 17:23:52 +03:00
RamiBerm
f9396e01ca TRA-3415 ignore unready pods (#160)
* Update tapRunner.go and provider.go

* Update tapRunner.go

* Update tapRunner.go

* Update tapRunner.go

* Update tapRunner.go and provider.go

Co-authored-by: RamiBerm <rami.berman@up9.com>
2021-08-03 15:02:31 +03:00
Igor Gov
793bb97e51 Print Mizu URL only after the watched pods (#156)
* Print Mizu URL after the watched pods
2021-08-03 08:54:10 +03:00
gadotroee
8d8310ee02 Revert "feature/TRA_3427_demo_mode (#150)" (#151)
This reverts commit 71eff5ea04.
2021-07-29 21:06:41 +03:00
Selton Fiuza
71eff5ea04 feature/TRA_3427_demo_mode (#150)
* Demo Mode MVP

* messages improve

* downloading based on the OS

* downloading based on the OS

* downloading based on the OS

* Modeler keep running

* A lot of revisions comes now

* Fix color
2021-07-28 14:50:15 -03:00
Igor Gov
ffa34039b1 Fix fetch telemetry report (#145) 2021-07-27 15:10:22 +03:00
nimrod-up9
d888706e1e Show that the default answer to analysis prompt is yes. (#144) 2021-07-27 14:52:43 +03:00