* Call `SetProtocol` in AMQP faster and remove `GetProtocol` method
* #run_acceptance_tests
* Remove the unused fields from the test mocks #run_acceptance_tests
* Add gin-contrib/pprof dependency
* Run pprof server on agent with --profiler flag
* Add --profiler flag to cli
* Fix error message
* Print cpu usage percentage
* measure cpu of current pid instead of globaly on the system
* Add scripts to plot performance
* Plot packetsCount in analysis
* Concat to DataFrame
* Plot in turbo colorscheme
* Make COLORMAP const
* Fix rss units
* Reduce code repetition by adding function for plotting
* Allow grouping based on filenames
* Temporary: Marked with comments where to disable code for experiments
* Add newline at end of file
* Add tap.cpuprofile flag. Change memprofile flag to tap.memprofile
* create tapper modes for debugging using env vars
* Fix rss plot units (MB instead of bytes)
* Remove comment
* Add info to plot script
* Remove tap.cpumemprofile. Rename tap.memprofile to memprofile
* Remove unused import
* Remove whitespaces
Co-authored-by: M. Mert Yıldıran <mehmet@up9.com>
* Remove whitespaces
Co-authored-by: M. Mert Yıldıran <mehmet@up9.com>
* Remove whitespaces
Co-authored-by: M. Mert Yıldıran <mehmet@up9.com>
* Remove whitespaces
Co-authored-by: M. Mert Yıldıran <mehmet@up9.com>
* Remove whitespaces
Co-authored-by: M. Mert Yıldıran <mehmet@up9.com>
* Remove whitespaces
Co-authored-by: M. Mert Yıldıran <mehmet@up9.com>
* Rename debug env vars
* Create package for debug env vars, read each env var once
* Run go mod tidy
* Increment MatchedPairs before emitting
* Only count cores once
* Count virtual and physical cores
* Add dbgctl replace in cli
* Fix lint: Check return values
* Add tap/dbgctl to test-lint make rule
* Replace tap/dbgctl in all modules
* #run_acceptance_tests
* Copy dbgctl module to docker image
* Debug/profile tapper benchmark (#1093)
* add mizu debug env to avoid all extensions
* add readme + run_tapper_benchmark.sh
* temporary change branch name
* fix readme
* fix MIZU_BENCHMARK_CLIENTS_COUNT env
* change tap target to tcp stream
* track live tcp streams
* pr fixes
* rename tapperPacketsCount to ignored_packets_count
* change mizu tapper to mizu debugg
Co-authored-by: David Levanon <dvdlevanon@gmail.com>
Co-authored-by: M. Mert Yıldıran <mehmet@up9.com>
* Spawn only two Goroutines per TCP stream
* Fix the linter error
* Use `isProtocolIdentified` method instead
* Fix the `Read` method of `tcpReader`
* Remove unnecessary `append`
* Copy to buffer only a message is received
* Remove `exhaustBuffer` field and add `rewind` function
* Rename `buffer` field to `pastData`
* Update tap/tcp_reader.go
Co-authored-by: Nimrod Gilboa Markevich <59927337+nimrod-up9@users.noreply.github.com>
* Use `copy` instead of assignment
* No lint
* #run_acceptance_tests
* Fix `rewind` #run_acceptance_tests
* Fix the buffering algorithm #run_acceptance_tests
* Add `TODO`
* Fix the problems in AMQP and Kafka #run_acceptance_tests
* Use `*bytes.Buffer` instead of `[]api.TcpReaderDataMsg` #run_acceptance_tests
* Have a single `*bytes.Buffer`
* Revert "Have a single `*bytes.Buffer`"
This reverts commit fad96a288a.
* Revert "Use `*bytes.Buffer` instead of `[]api.TcpReaderDataMsg` #run_acceptance_tests"
This reverts commit 0fc70bffe2.
* Fix the early timing out issue #run_acceptance_tests
* Remove `NewBytes()` method
* Update the `NewTcpReader` method signature #run_acceptance_tests
* #run_acceptance_tests
* #run_acceptance_tests
* #run_acceptance_tests
Co-authored-by: Nimrod Gilboa Markevich <59927337+nimrod-up9@users.noreply.github.com>
* Add GraphQL detection
* Change the abbreviation to `GQL`
* Fix the `TestMacros` test case
* Try to fix the security issues reported by Snyk
* Run `go mod tidy` in `agent` directory
* Upgrade `golang.org/x/crypto`
* Downgrade `golang.org/x/crypto`
* Downgrade `golang.org/x/crypto`
* Downgrade `golang.org/x/crypto`
* Upgrade to `github.com/vektah/gqlparser/v2 v2.4.2`
* Run `go mod tidy`
* Replace `github.com/vektah/gqlparser/v2` with its fork `github.com/mertyildiran/gqlparser/v2` to fix the vulns
* Upgrade the `github.com/mertyildiran/gqlparser/v2 v2.4.6`
* Don't upgrade `golang.org/x/net`
* Fix `panic: interface conversion: api.RequestResponseMatcher is nil, not *http.requestResponseMatcher` error
Also fix the request-response matcher maps iteration in `clean()` method.
* Fix the mocks in the unit tests
* Remove unnecessary fields from `tlsPoller` and implement `SetProtocol` method
* Use concrete types in `tap` package
* Share the streams map with the TLS tapper
* Check interface conversion error
* Remove `tcpStreamWrapper` struct
* Refactor `tap` module and move some of the code to `tap/api` module
* Move `TrafficFilteringOptions` struct to `shared` module
* Change the `Dissect` method signature to have `*TcpReader` as an argument
* Add `CloseOtherProtocolDissectors` method and use it to synchronously close the other protocol dissectors
* Run `go mod tidy` in `cli` module
* Rename `SuperIdentifier` struct to `ProtoIdentifier`
* Remove `SuperTimer` struct
* Bring back `CloseTimedoutTcpStreamChannels` method
* Run `go mod tidy` everywhere
* Remove `GOGC` environment variable from tapper
* Fix the tests
* Bring back `debug.FreeOSMemory()` call
* Make `CloseOtherProtocolDissectors` method mutexed
* Revert "Remove `GOGC` environment variable from tapper"
This reverts commit cfc2484bbb.
* Bring back the removed `checksum`, `nooptcheck` and `ignorefsmerr` flags
* Define a bunch of interfaces and don't export any new structs from `tap/api`
* Keep the interfaces in `tap/api` but move the structs to `tap/tcp`
* Fix the unit tests by depending on `github.com/up9inc/mizu/tap`
* Use the modified `tlsEmitter`
* Define `TlsChunk` interface and make `tlsReader` implement `TcpReader`
* Remove unused fields in `tlsReader`
* Define `ReassemblyStream` interface and separate `gopacket` specififc fields to `tcpReassemblyStream` struct
Such that make `tap/api` don't depend on `gopacket`
* Remove the unused fields
* Make `tlsPoller` implement `TcpStream` interface and remove the call to `NewTcpStreamDummy` method
* Remove unused fields from `tlsPoller`
* Remove almost all of the setter methods in `TcpReader` and `TcpStream` interface and remove `TlsChunk` interface
* Revert "Revert "Remove `GOGC` environment variable from tapper""
This reverts commit ab2b9a803b.
* Revert "Bring back `debug.FreeOSMemory()` call"
This reverts commit 1cce863bbb.
* Remove excess comment
* Fix acceptance tests (`logger` module) #run_acceptance_tests
* Bring back `github.com/patrickmn/go-cache`
* Fix `NewTcpStream` method signature
* Put `tcpReader` and `tcpStream` mocks into protocol dissectors to remove `github.com/up9inc/mizu/tap` dependency
* Fix AMQP tests
* Revert 960ba644cd
* Revert `go.mod` and `go.sum` files in protocol dissectors
* Fix the comment position
* Revert `AppStatsInst` change
* Fix indent
* Fix CLI build
* Fix linter error
* Fix error msg
* Revert some of the changes in `chunk.go`
* Remove gRPC related modifications
* Remove gRPC status text related modifications as well
* Fixing gRPC vertical image
detect grpc when content type is 'application/grpc' as well (and not only from the grpc-status)
Co-authored-by: gadotroee <55343099+gadotroee@users.noreply.github.com>
* Define `ReadProgress` struct and update `Dissector` interface such that the `bufio.Reader` progress can be learned on item emitting
* Display the `requestSize` and `responseSize` fields in the UI
* Update the tests
* publish ui-common version 1.0.130 and bump to this version in ui/package.json file
Co-authored-by: gadotroee <55343099+gadotroee@users.noreply.github.com>
Co-authored-by: Roee Gadot <roee.gadot@up9.com>
* Define `Capture` type and expect it as an argument in `Dissect` method
* Add an indicator for the eBPF sourced entries
* Fix the Go lint error
* Fix the logic in the UI
* Update the expected JSONs
* Add TODOs
* Add `UndefinedCapture` constant
* Define `CaptureTypes` enum
* Always derive the summary and method fields from the entry in the database on read
* Update the expected JSONs in the protocol unit tests
* Add test cases for `Summarize` method
* Remove unused `GetEntry` method, `DataUnmarshaler` struct and `UnmarshalData` method
* Temporarily enable the acceptance tests
* Temporarily disable Slack notification on failure
* Update the Cypress tests
* Fix an issue in Redis
* Fix a typo and the Cypress tests
* Revert "Temporarily disable Slack notification on failure"
This reverts commit cad1901ea4.
* Revert "Temporarily enable the acceptance tests"
This reverts commit bad7706c9b.
* Add unit tests for Kafka dissector
* Return `io.EOF` if request or response header size is zero
* Sort the slice in `representMapAsTable`
* Remove the dead code
* Remove more dead code
* Remove more dead code
* Fix `dissector.Analyze` call
Co-authored-by: gadotroee <55343099+gadotroee@users.noreply.github.com>
* Add unit tests for AMQP dissector
* Always use UTC as the timezone in the timestamps
* Remove the dead code
* Remove more dead code
* Fix `dissector.Analyze` call
Co-authored-by: gadotroee <55343099+gadotroee@users.noreply.github.com>
* Create a new request-response matcher for each TCP stream
* Fix the `ident` formats in request-response matchers
* Don't sort the items in the HTTP tests
* Update tap/extensions/kafka/matcher.go
Co-authored-by: gadotroee <55343099+gadotroee@users.noreply.github.com>
* Temporarily change the bucket folder to the new expected
* Bring back the `deleteOlderThan` method
* Use `api.RequestResponseMatcher` instead of `interface{}` as type
* Use `api.RequestResponseMatcher` instead of `interface{}` as type (more)
* Update the key format comments
Co-authored-by: gadotroee <55343099+gadotroee@users.noreply.github.com>
* Add unit tests for HTTP dissector
* Ignore some fields on test environment
* Remove Git patches
* Don't have indent in the expected JSONs
* Fix more issues and update expected JSONs
* Refactor the test environment lookup
* Add a Makefile
* Include HTTP tests into the CI
* Fix the linting errors
* Fix another linting error
* Update the expected JSONs
* Sort `PostData.Params` params as well
* Move expected JSONs into `expect/dissect`
* Add `TestAnalyze` method
* Add `TestRepresent` method
* Add `TestRegister`, `TestMacros` and `TestPing` methods
* Test extensions first
* Remove expected JSONs
* Remove `bin` directory and update `Makefile` rules
* Update `.gitignore`
* Fix skipping download functionality in the Makefile
* Run `go mod tidy`
* Fix the race condition in the tests
* Revert "Test extensions first"
This reverts commit b8350cf139.
* Make `TEST_UPDATE` env lookup one-liner
* Update .github/workflows/test.yml
Co-authored-by: gadotroee <55343099+gadotroee@users.noreply.github.com>
* Add a newline
* Replace `ls` with `find`
Co-authored-by: gadotroee <55343099+gadotroee@users.noreply.github.com>
* Add a PCAP based testbed
* Fix typos
* Download PCAPs from the Google Cloud bucket
* Add a Python script to automate the PCAP testbed
* Dump the test suite into a file named `suite.json`
* Serialize entries separately
* Dissect individual TCP streams one by one through separate PCAP files
* Improve the reliability a little bit
* Ditch the individual TCP streams idea
* Fix some issues in Kafka
* Print the total number of packets and TCP streams
* Fix an interface conversion error in AMQP
* Print the total number of returning items from the dissectors
* Print the total number of returning items from the dissectors really
* Fix a possible race condition
* Do atomic increments just to be sure
* Print the total number of Redis `Dissect` calls
* Improve the request-response matching in Redis by including the TCP stream ID
* Update the request-response pair matching key format in HTTP and Kafka
* Rearrange the test suite
* Add more queries to the test suite
* Remove the debug prints
* Add the assertions
* Close the WebSocket connection faster
* Make `MIZU_TEST` enviroment variable a shared constant
* Add `test-lint` rule
* Fix several issues in Kafka
* Update the test suite
* Add more queries
* Fix the `test-lint` rule
* Exit only after PCAP EOF
* Add more queries
* Update `suite.json`
* Make the tests more stable
* Revert the bad changes
* Run `go mod tidy` on `tap/`
* modified Dockerfile to work for both amd64 (Intel) and arm64 (M1)
* added changelog
* Update `Dockerfile` to have `ARCH` build argument
* Remove `docs/CHANGES.md`
* Upgrade the Basenine version from `v0.3.0` to `v0.4.6`
* Update `publish.yml` to have `ARCH` build argument
* Switch `BasenineImageRepo` to Docker Hub
* Have separate build arguments for `ARCH` and `GOARCH`
* Upgrade the Basenine version from `v0.4.6` to `v0.4.10`
* Oops forgot to update the 10th duplicated shell script
* Fix the oopsie and reduce duplications
* Fix `Dockerfile`
* Fix the incompatibility issue between Go plugins and gold linker in Alpine inside `Dockerfile`
* Fix `asm: xxhash_amd64.s:120: when dynamic linking, R15 is clobbered by a global variable access` error
* Update `Dockerfile` to have cross-compilation on an AMD64 machine
Also revert changes in the shell scripts
* Delete `debug.Dockerfile`
* Create a custom base (`debian:buster-slim` based) image for the shipped image
* Replace `mertyildiran/debian-pcap` with `up9inc/debian-pcap`
* Upgrade Basenine version to `v0.4.12`
* Use `debian:stable-slim` as the base
* Fix the indentation in the `Dockerfile`
* Update `publish.yml`
* Enable `publish.yml` for `feature/multiarch_build` branch
* Tag correctly and set `ARCH` Docker argument
* Remove the lines that are forgotten to be removed from the shell scripts
* Add `MizuAgentImageRepo` constant and use it as default `AgentImage` value
* Bring back `Set up Cloud SDK` step to `Build the CLI and publish` job
* Build ARM64 CLI for Linux as well
* Revert "Enable `publish.yml` for `feature/multiarch_build` branch"
This reverts commit d30be4c1f0.
* Revert Go 1.17 upgrade
* Remove `build_extensions_debug.sh` as well
* Make the `Dockerfile` to compile the agent statically
* Statically link the protocol extensions
* Fix `Dockerfile`
* Bring back `-s -w` flags
* Verify the signatures of the downloads in `dockcross/linux-arm64-musl`
* Revert modifications in some shell scripts
* Make the `BUILDARCH` and `TARGETARCH` separation in the `Dockerfile`
* Separate cross-compilation builder image into a separate repo named `up9inc/linux-arm64-musl-go-libpcap`
* Fill the shell script and specify the tag for `dockcross/linux-arm64-musl`
* Remove the unnecessary dependencies from `builder-native-base`
* Improve the comments in the `Dockerfile`
* Upgrade Basenine version to `v0.4.13`
* Fix `Dockerfile`
* Revert "Revert "Enable `publish.yml` for `feature/multiarch_build` branch""
This reverts commit 303e466bdc.
* Revert "Revert "Revert "Enable `publish.yml` for `feature/multiarch_build` branch"""
This reverts commit 0fe252bbdb.
* Remove `push-docker-debug` from the `Makefile`
* Rename `publish.yml` to `release.yml`
Co-authored-by: Alex Haiut <alex@up9.com>
* Handle nested `topicData` in `representProduceRequest`
* Handle nested `topics` in `representCreateTopicsRequest` and `representCreateTopicsResponse`
* Handle nested `responses` in `representProduceResponse`
* Handle nested `topics` in `representFetchRequest` and nested `responses` in `representFetchResponse`
* Introduce `ignoreKeys` argument to `representMapAsTable` and ignore the keys based on that argument
* Bring back the `nil` checks
Support source IP resolving for HTTP traffic in Istio service mesh.
If X-Forwarded-For HTTP request header is present, replace the source address with the left-most address in X-Forwarded-For (earliest in Envoy implementation of this header).
This allows Mizu to resolve source IPs in Istio service meshes, if the use_remote_address option is on. Added instructions on how to turn it on.
* Upgrade Basenine version from `0.2.26` to `0.3.0`
* Remove `Summarize` method from `Dissector` interface and refactor the data structures in `tap/api/api.go`
* Rename `MizuEntry` to `Entry` and `BaseEntryDetails` to `BaseEntry`
* Populate `ContractStatus` field as well
* Update the cheatsheet
* Upgrade the Basenine version in the helm chart as well
* Remove a forgoten `console.log` call
* Remove unnecessary fields and split `service` into `src.name` and `dst.name`
* Don't fall back to IP address but instead display `[Unresolved]` text
* Fix the CSS issues in the plus icon position and replace the separator `->` text with `SwapHorizIcon`
* Remove the extra negation on `nodefrag` flag's value
* Support IPv4 fragmentation and IPv6 at the same time
* Set `Method` and `StatusCode` fields correctly for `HTTP/2`
* Replace unnecessary `grpc` naming with `http2`
* Make the `gRPC` and `HTTP/2` distinction
* Fix the macros of `http` extension
* Fix the macros of other protocol extensions
* Update the method signature of `Represent`
* Fix the `HTTP/2` support
* Fix some minor issues
* Upgrade Basenine version from `0.2.10` to `0.2.11`
Sorts macros before expanding them and prioritize the long macros.
* Don't regex split the gRPC method name
* Re-enable `nodefrag` flag
