* Add gin-contrib/pprof dependency
* Run pprof server on agent with --profiler flag
* Add --profiler flag to cli
* Fix error message
* Print cpu usage percentage
* measure cpu of current pid instead of globaly on the system
* Add scripts to plot performance
* Plot packetsCount in analysis
* Concat to DataFrame
* Plot in turbo colorscheme
* Make COLORMAP const
* Fix rss units
* Reduce code repetition by adding function for plotting
* Allow grouping based on filenames
* Temporary: Marked with comments where to disable code for experiments
* Add newline at end of file
* Add tap.cpuprofile flag. Change memprofile flag to tap.memprofile
* create tapper modes for debugging using env vars
* Fix rss plot units (MB instead of bytes)
* Remove comment
* Add info to plot script
* Remove tap.cpumemprofile. Rename tap.memprofile to memprofile
* Remove unused import
* Remove whitespaces
Co-authored-by: M. Mert Yıldıran <mehmet@up9.com>
* Remove whitespaces
Co-authored-by: M. Mert Yıldıran <mehmet@up9.com>
* Remove whitespaces
Co-authored-by: M. Mert Yıldıran <mehmet@up9.com>
* Remove whitespaces
Co-authored-by: M. Mert Yıldıran <mehmet@up9.com>
* Remove whitespaces
Co-authored-by: M. Mert Yıldıran <mehmet@up9.com>
* Remove whitespaces
Co-authored-by: M. Mert Yıldıran <mehmet@up9.com>
* Rename debug env vars
* Create package for debug env vars, read each env var once
* Run go mod tidy
* Increment MatchedPairs before emitting
* Only count cores once
* Count virtual and physical cores
* Add dbgctl replace in cli
* Fix lint: Check return values
* Add tap/dbgctl to test-lint make rule
* Replace tap/dbgctl in all modules
* #run_acceptance_tests
* Copy dbgctl module to docker image
* Debug/profile tapper benchmark (#1093)
* add mizu debug env to avoid all extensions
* add readme + run_tapper_benchmark.sh
* temporary change branch name
* fix readme
* fix MIZU_BENCHMARK_CLIENTS_COUNT env
* change tap target to tcp stream
* track live tcp streams
* pr fixes
* rename tapperPacketsCount to ignored_packets_count
* change mizu tapper to mizu debugg
Co-authored-by: David Levanon <dvdlevanon@gmail.com>
Co-authored-by: M. Mert Yıldıran <mehmet@up9.com>
* Fix a `nil pointer dereference` error that occurs in `tcpReader`'s `Read` method
* #run_acceptance_tests
* #run_acceptance_tests
* Revert "Fix a `nil pointer dereference` error that occurs in `tcpReader`'s `Read` method"
This reverts commit ccef6cb393.
* Fix the race condition using locks #run_acceptance_tests
* Spawn only two Goroutines per TCP stream
* Fix the linter error
* Use `isProtocolIdentified` method instead
* Fix the `Read` method of `tcpReader`
* Remove unnecessary `append`
* Copy to buffer only a message is received
* Remove `exhaustBuffer` field and add `rewind` function
* Rename `buffer` field to `pastData`
* Update tap/tcp_reader.go
Co-authored-by: Nimrod Gilboa Markevich <59927337+nimrod-up9@users.noreply.github.com>
* Use `copy` instead of assignment
* No lint
* #run_acceptance_tests
* Fix `rewind` #run_acceptance_tests
* Fix the buffering algorithm #run_acceptance_tests
* Add `TODO`
* Fix the problems in AMQP and Kafka #run_acceptance_tests
* Use `*bytes.Buffer` instead of `[]api.TcpReaderDataMsg` #run_acceptance_tests
* Have a single `*bytes.Buffer`
* Revert "Have a single `*bytes.Buffer`"
This reverts commit fad96a288a.
* Revert "Use `*bytes.Buffer` instead of `[]api.TcpReaderDataMsg` #run_acceptance_tests"
This reverts commit 0fc70bffe2.
* Fix the early timing out issue #run_acceptance_tests
* Remove `NewBytes()` method
* Update the `NewTcpReader` method signature #run_acceptance_tests
* #run_acceptance_tests
* #run_acceptance_tests
* #run_acceptance_tests
Co-authored-by: Nimrod Gilboa Markevich <59927337+nimrod-up9@users.noreply.github.com>
* Remove `tcpStreamWrapper` struct
* Refactor `tap` module and move some of the code to `tap/api` module
* Move `TrafficFilteringOptions` struct to `shared` module
* Change the `Dissect` method signature to have `*TcpReader` as an argument
* Add `CloseOtherProtocolDissectors` method and use it to synchronously close the other protocol dissectors
* Run `go mod tidy` in `cli` module
* Rename `SuperIdentifier` struct to `ProtoIdentifier`
* Remove `SuperTimer` struct
* Bring back `CloseTimedoutTcpStreamChannels` method
* Run `go mod tidy` everywhere
* Remove `GOGC` environment variable from tapper
* Fix the tests
* Bring back `debug.FreeOSMemory()` call
* Make `CloseOtherProtocolDissectors` method mutexed
* Revert "Remove `GOGC` environment variable from tapper"
This reverts commit cfc2484bbb.
* Bring back the removed `checksum`, `nooptcheck` and `ignorefsmerr` flags
* Define a bunch of interfaces and don't export any new structs from `tap/api`
* Keep the interfaces in `tap/api` but move the structs to `tap/tcp`
* Fix the unit tests by depending on `github.com/up9inc/mizu/tap`
* Use the modified `tlsEmitter`
* Define `TlsChunk` interface and make `tlsReader` implement `TcpReader`
* Remove unused fields in `tlsReader`
* Define `ReassemblyStream` interface and separate `gopacket` specififc fields to `tcpReassemblyStream` struct
Such that make `tap/api` don't depend on `gopacket`
* Remove the unused fields
* Make `tlsPoller` implement `TcpStream` interface and remove the call to `NewTcpStreamDummy` method
* Remove unused fields from `tlsPoller`
* Remove almost all of the setter methods in `TcpReader` and `TcpStream` interface and remove `TlsChunk` interface
* Revert "Revert "Remove `GOGC` environment variable from tapper""
This reverts commit ab2b9a803b.
* Revert "Bring back `debug.FreeOSMemory()` call"
This reverts commit 1cce863bbb.
* Remove excess comment
* Fix acceptance tests (`logger` module) #run_acceptance_tests
* Bring back `github.com/patrickmn/go-cache`
* Fix `NewTcpStream` method signature
* Put `tcpReader` and `tcpStream` mocks into protocol dissectors to remove `github.com/up9inc/mizu/tap` dependency
* Fix AMQP tests
* Revert 960ba644cd
* Revert `go.mod` and `go.sum` files in protocol dissectors
* Fix the comment position
* Revert `AppStatsInst` change
* Fix indent
* Fix CLI build
* Fix linter error
* Fix error msg
* Revert some of the changes in `chunk.go`
* Remove non-critical TLS detected log that causes `slice bounds out of range` error
* Remove all non-functional `OutboundLink` code that was providing `/status/recentTLSLinks` endpoint
* Fix more unused code
Co-authored-by: M. Mert Yıldıran <mehmet@up9.com>
Co-authored-by: M. Mert Yıldıran <mehmet@up9.com>
Co-authored-by: Nimrod Gilboa Markevich <59927337+nimrod-up9@users.noreply.github.com>
* Define `ReadProgress` struct and update `Dissector` interface such that the `bufio.Reader` progress can be learned on item emitting
* Display the `requestSize` and `responseSize` fields in the UI
* Update the tests
* publish ui-common version 1.0.130 and bump to this version in ui/package.json file
Co-authored-by: gadotroee <55343099+gadotroee@users.noreply.github.com>
Co-authored-by: Roee Gadot <roee.gadot@up9.com>
* Define `Capture` type and expect it as an argument in `Dissect` method
* Add an indicator for the eBPF sourced entries
* Fix the Go lint error
* Fix the logic in the UI
* Update the expected JSONs
* Add TODOs
* Add `UndefinedCapture` constant
* Define `CaptureTypes` enum
* Create a new request-response matcher for each TCP stream
* Fix the `ident` formats in request-response matchers
* Don't sort the items in the HTTP tests
* Update tap/extensions/kafka/matcher.go
Co-authored-by: gadotroee <55343099+gadotroee@users.noreply.github.com>
* Temporarily change the bucket folder to the new expected
* Bring back the `deleteOlderThan` method
* Use `api.RequestResponseMatcher` instead of `interface{}` as type
* Use `api.RequestResponseMatcher` instead of `interface{}` as type (more)
* Update the key format comments
Co-authored-by: gadotroee <55343099+gadotroee@users.noreply.github.com>
* Close the hanging TCP message channels after a dynamically aligned timeout (base `10000` milliseconds)
* Bring back `source.Lazy`
* Add a one more `sync.Map.Delete` call
* Improve the formula by taking base Goroutine count into account
* Reduce duplication
* Include the dropped TCP streams count into the stats tracker and print a debug log whenever it happens
* Add `superIdentifier` field to `tcpStream` to check if it has identified
Also stop the other protocol dissectors if a TCP stream identified by a protocol.
* Take one step forward in fixing the channel closing issue (WIP)
Add `sync.Mutex` to `tcpReader` and make the loops reference based.
* Fix the channel closing issue
* Improve the accuracy of the formula, log better and multiply `baseStreamChannelTimeoutMs` by 100
* Remove `fmt.Printf`
* Replace `runtime.Gosched()` with `time.Sleep(1 * time.Millisecond)`
* Close the channels of other protocols in case of an identification
* Simplify the logic
* Replace the formula with hard timeout 5000 milliseconds and 4000 maximum number of Goroutines
* Separate HTTP related code into `extensions/http` as a Go plugin
* Move `extensions` folder into `tap` folder
* Move HTTP files into `tap/extensions/lib` for now
* Replace `orcaman/concurrent-map` with `sync.Map`
* Remove `grpc_assembler.go`
* Remove `github.com/up9inc/mizu/tap/extensions/http/lib`
* Add a build script to automatically build extensions from a known path and load them
* Start to define the extension API
* Implement the `run()` function for the TCP stream
* Add support of defining multiple ports to the extension API
* Set the extension name inside the extension
* Declare the `Dissect` function in the extension API
* Dissect HTTP request from inside the HTTP extension
* Make the distinction of outbound and inbound ports
* Dissect HTTP response from inside the HTTP extension
* Bring back the HTTP request-response pair matcher
* Return a `*api.RequestResponsePair` from the dissection
* Bring back the gRPC-HTTP/2 parser
* Fix the issues in `handleHTTP1ClientStream` and `handleHTTP1ServerStream`
* Call a function pointer to emit dissected data back to the `tap` package
* roee changes -
trying to fix agent to work with the "api" object) - ***still not working***
* small mistake in the conflicts
* Fix the issues that are introduced by the merge conflict
* Add `Emitter` interface to the API and send `OutputChannelItem`(s) to `OutputChannel`
* Fix the `HTTP1` handlers
* Set `ConnectionInfo` in HTTP handlers
* Fix the `Dockerfile` to build the extensions
* remove some unwanted code
* no message
* Re-enable `getStreamProps` function
* Migrate back from `gopacket/tcpassembly` to `gopacket/reassembly`
* Introduce `HTTPPayload` struct and `HTTPPayloader` interface to `MarshalJSON()` all the data structures that are returned by the HTTP protocol
* Read `socketHarOutChannel` instead of `filteredHarChannel`
* Connect `OutputChannelItem` to the last WebSocket means that finally the web UI started to work again
* Add `.env.example` to React app
* Marshal and unmarshal `*http.Request`, `*http.Response` pairs
* Move `loadExtensions` into `main.go` and map extensions into `extensionsMap`
* Add `Summarize()` method to the `Dissector` interface
* Add `Analyze` method to the `Dissector` interface and `MizuEntry` to the extension API
* Add `Protocol` struct and make it effect the UI
* Refactor `BaseEntryDetails` struct and display the source and destination ports in the UI
* Display the protocol name inside the details layout
* Add `Represent` method to the `Dissector` interface and manipulate the UI through this method
* Make the protocol color affect the details layout color and write protocol abbreviation vertically
* Remove everything HTTP related from the `tap` package and make the extension system fully functional
* Fix the TypeScript warnings
* Bring in the files related AMQP into `amqp` directory
* Add `--nodefrag` flag to the tapper and bring in the main AMQP code
* Implement the AMQP `BasicPublish` and fix some issues in the UI when the response payload is missing
* Implement `representBasicPublish` method
* Fix several minor issues
* Implement the AMQP `BasicDeliver`
* Implement the AMQP `QueueDeclare`
* Implement the AMQP `ExchangeDeclare`
* Implement the AMQP `ConnectionStart`
* Implement the AMQP `ConnectionClose`
* Implement the AMQP `QueueBind`
* Implement the AMQP `BasicConsume`
* Fix an issue in `ConnectionStart`
* Fix a linter error
* Bring in the files related Kafka into `kafka` directory
* Fix the build errors in Kafka Go files
* Implement `Dissect` method of Kafka and adapt request-response pair matcher to asynchronous client-server stream
* Do the "Is reversed?" checked inside `getStreamProps` and fix an issue in Kafka `Dissect` method
* Implement `Analyze`, `Summarize` methods of Kafka
* Implement the representations for Kafka `Metadata`, `RequestHeader` and `ResponseHeader`
* Refactor the AMQP and Kafka implementations to create the summary string only inside the `Analyze` method
* Implement the representations for Kafka `ApiVersions`
* Implement the representations for Kafka `Produce`
* Implement the representations for Kafka `Fetch`
* Implement the representations for Kafka `ListOffsets`, `CreateTopics` and `DeleteTopics`
* Fix the encoding of AMQP `BasicPublish` and `BasicDeliver` body
* Remove the unnecessary logging
* Remove more logging
* Introduce `Version` field to `Protocol` struct for dynamically switching the HTTP protocol to HTTP/2
* Fix the issues in analysis and representation of HTTP/2 (gRPC) protocol
* Fix the issues in summary section of details layout for HTTP/2 (gRPC) protocol
* Fix the read errors that freezes the sniffer in HTTP and Kafka
* Fix the issues in HTTP POST data
* Fix one more issue in HTTP POST data
* Fix an infinite loop in Kafka
* Fix another freezing issue in Kafka
* Revert "UI Infra - Support multiple entry types + refactoring (#211)"
This reverts commit f74a52d4dc.
* Fix more issues that are introduced by the merge
* Fix the status code in the summary section
* adding the cleaner again (why we removed it?).
add TODO: on the extension loop .
* fix dockerfile (remove deleting .env file) - it is found in dockerignore and fails to build if the file not exists
* fix GetEntrties ("/entries" endpoint) - working with "tapApi.BaseEntryDetail" (moved from shared)
* Fix an issue in the UI summary section
* Refactor the protocol payload structs
* Fix a log message in the passive tapper
* Adapt `APP_PORTS` environment variable to the new extension system and change its format to `APP_PORTS='{"http": ["8001"]}' `
* Revert "fix dockerfile (remove deleting .env file) - it is found in dockerignore and fails to build if the file not exists"
This reverts commit 4f514ae1f4.
* Bring in the necessary changes from f74a52d4dc
* Open the API server URL in the web browser as soon as Mizu is ready
* Make the TCP reader consists of a single Go routine (instead of two) and try to dissect in both client and server mode by rewinding
* Swap `TcpID` without overwriting it
* Sort extension by priority
* Try to dissect with looping through all the extensions
* fix getStreamProps function.
(it should be passed from CLI as it was before).
* Turn TCP reader back into two Goroutines (client and server)
* typo
* Learn `isClient` from the TCP stream
* Set `viewer` style `overflow: "auto"`
* Fix the memory leaks in AMQP and Kafka dissectors
* Revert some of the changes in be7c65eb6d
* Remove `allExtensionPorts` since it's no longer needed
* Remove `APP_PORTS` since it's no longer needed
* Fix all of the minor issues in the React code
* Check Kafka header size and fail-fast
* Break the dissectors loop upon a successful dissection
* Don't break the dissector loop. Protocols might collide
* Improve the HTTP request-response counter (still not perfect)
* Make the HTTP request-response counter perfect
* Revert "Revert some of the changes in be7c65eb6d3fb657a059707da3ca559937e59739"
This reverts commit 08e7d786d8.
* Bring back `filterItems` and `isHealthCheckByUserAgent` functions
* Remove some development artifacts
* remove unused and commented lines that are not relevant
* Fix the performance in TCP stream factory. Make it create two `tcpReader`(s) per extension
* Change a log to debug
* Make `*api.CounterPair` a field of `tcpReader`
* Set `isTapTarget` to always `true` again since `filterAuthorities` implementation has problems
* Remove a variable that's only used for logging even though not introduced by this branch
* Bring back the `NumberOfRules` field of `ApplicableRules` struct
* Remove the unused `NewEntry` function
* Move `k8sResolver == nil` check to a more appropriate place
* default healthChecksUserAgentHeaders should be empty array (like the default config value)
* remove spam console.log
* Rules button cause app to crash (access the service via incorrect property)
* Ignore all .env* files in docker build.
* Better caching in dockerfile: only copy go.mod before go mod download.
* Check for errors while loading an extension
* Add a comment about why `Protocol` is not a pointer
* Bring back the call to `deleteOlderThan`
* Remove the `nil` check
* Reduce the maximum allowed AMQP message from 128MB to 1MB
* Fix an error that only occurs when a Kafka broker is initiating
* Revert the change in b2abd7b990
* Fix the service name resolution in all protocols
* Remove the `anydirection` flag and fix the issue in `filterAuthorities`
* Pass `sync.Map` by reference to `deleteOlderThan` method
* Fix the packet capture issue in standalone mode that's introduced by the removal of `anydirection`
* Temporarily resolve the memory exhaustion in AMQP
* Fix a nil pointer dereference error
* Fix the CLI build error
* Fix a memory leak that's identified by `pprof`
Co-authored-by: Roee Gadot <roee.gadot@up9.com>
Co-authored-by: Nimrod Gilboa Markevich <nimrod@up9.com>
