Commit Graph

7 Commits

Author SHA1 Message Date
M. Mert Yıldıran
d3e6a69d82
Refactor tap module to achieve synchronously closing other protocol dissectors upon identification (#1026)
* Remove `tcpStreamWrapper` struct

* Refactor `tap` module and move some of the code to `tap/api` module

* Move `TrafficFilteringOptions` struct to `shared` module

* Change the `Dissect` method signature to have `*TcpReader` as an argument

* Add `CloseOtherProtocolDissectors` method and use it to synchronously close the other protocol dissectors

* Run `go mod tidy` in `cli` module

* Rename `SuperIdentifier` struct to `ProtoIdentifier`

* Remove `SuperTimer` struct

* Bring back `CloseTimedoutTcpStreamChannels` method

* Run `go mod tidy` everywhere

* Remove `GOGC` environment variable from tapper

* Fix the tests

* Bring back `debug.FreeOSMemory()` call

* Make `CloseOtherProtocolDissectors` method mutexed

* Revert "Remove `GOGC` environment variable from tapper"

This reverts commit cfc2484bbb.

* Bring back the removed `checksum`, `nooptcheck` and `ignorefsmerr` flags

* Define a bunch of interfaces and don't export any new structs from `tap/api`

* Keep the interfaces in `tap/api` but move the structs to `tap/tcp`

* Fix the unit tests by depending on `github.com/up9inc/mizu/tap`

* Use the modified `tlsEmitter`

* Define `TlsChunk` interface and make `tlsReader` implement `TcpReader`

* Remove unused fields in `tlsReader`

* Define `ReassemblyStream` interface and separate `gopacket` specififc fields to `tcpReassemblyStream` struct

Such that make `tap/api` don't depend on `gopacket`

* Remove the unused fields

* Make `tlsPoller` implement `TcpStream` interface and remove the call to `NewTcpStreamDummy` method

* Remove unused fields from `tlsPoller`

* Remove almost all of the setter methods in `TcpReader` and `TcpStream` interface and remove `TlsChunk` interface

* Revert "Revert "Remove `GOGC` environment variable from tapper""

This reverts commit ab2b9a803b.

* Revert "Bring back `debug.FreeOSMemory()` call"

This reverts commit 1cce863bbb.

* Remove excess comment

* Fix acceptance tests (`logger` module) #run_acceptance_tests

* Bring back `github.com/patrickmn/go-cache`

* Fix `NewTcpStream` method signature

* Put `tcpReader` and `tcpStream` mocks into protocol dissectors to remove `github.com/up9inc/mizu/tap` dependency

* Fix AMQP tests

* Revert 960ba644cd

* Revert `go.mod` and `go.sum` files in protocol dissectors

* Fix the comment position

* Revert `AppStatsInst` change

* Fix indent

* Fix CLI build

* Fix linter error

* Fix error msg

* Revert some of the changes in `chunk.go`
2022-04-28 17:19:14 +03:00
M. Mert Yıldıran
308fa78955
TRA-4383 Calculate request and response sizes and display them instead of BodySize field (#897)
* Define `ReadProgress` struct and update `Dissector` interface such that the `bufio.Reader` progress can be learned on item emitting

* Display the `requestSize` and `responseSize` fields in the UI

* Update the tests

* publish ui-common version 1.0.130 and bump to this version in ui/package.json file

Co-authored-by: gadotroee <55343099+gadotroee@users.noreply.github.com>
Co-authored-by: Roee Gadot <roee.gadot@up9.com>
2022-03-21 19:34:59 +02:00
M. Mert Yıldıran
5455220a3a
Add an indicator for the eBPF sourced entries (#886)
* Define `Capture` type and expect it as an argument in `Dissect` method

* Add an indicator for the  eBPF sourced entries

* Fix the Go lint error

* Fix the logic in the UI

* Update the expected JSONs

* Add TODOs

* Add `UndefinedCapture` constant

* Define `CaptureTypes` enum
2022-03-17 09:32:09 +03:00
M. Mert Yıldıran
041223b558
Move the request-response matcher's scope from global-level to TCP stream-level (#793)
* Create a new request-response matcher for each TCP stream

* Fix the `ident` formats in request-response matchers

* Don't sort the items in the HTTP tests

* Update tap/extensions/kafka/matcher.go

Co-authored-by: gadotroee <55343099+gadotroee@users.noreply.github.com>

* Temporarily change the bucket folder to the new expected

* Bring back the `deleteOlderThan` method

* Use `api.RequestResponseMatcher` instead of `interface{}` as type

* Use `api.RequestResponseMatcher` instead of `interface{}` as type (more)

* Update the key format comments

Co-authored-by: gadotroee <55343099+gadotroee@users.noreply.github.com>
2022-02-14 18:16:58 +03:00
M. Mert Yıldıran
a42a0cd0b9
Improve request-response matching and fix some interface conversion errors (#778)
* Add a PCAP based testbed

* Fix typos

* Download PCAPs from the Google Cloud bucket

* Add a Python script to automate the PCAP testbed

* Dump the test suite into a file named `suite.json`

* Serialize entries separately

* Dissect individual TCP streams one by one through separate PCAP files

* Improve the reliability a little bit

* Ditch the individual TCP streams idea

* Fix some issues in Kafka

* Print the total number of packets and TCP streams

* Fix an interface conversion error in AMQP

* Print the total number of returning items from the dissectors

* Print the total number of returning items from the dissectors really

* Fix a possible race condition

* Do atomic increments just to be sure

* Print the total number of Redis `Dissect` calls

* Improve the request-response matching in Redis by including the TCP stream ID

* Update the request-response pair matching key format in HTTP and Kafka

* Rearrange the test suite

* Add more queries to the test suite

* Remove the debug prints

* Add the assertions

* Close the WebSocket connection faster

* Make `MIZU_TEST` enviroment variable a shared constant

* Add `test-lint` rule

* Fix several issues in Kafka

* Update the test suite

* Add more queries

* Fix the `test-lint` rule

* Exit only after PCAP EOF

* Add more queries

* Update `suite.json`

* Make the tests more stable

* Revert the bad changes

* Run `go mod tidy` on `tap/`
2022-02-08 21:32:27 +03:00
M. Mert Yıldıran
a9a61edd50
Add ARM64 and cross-compilation support to the agent image (#659)
* modified Dockerfile to work for both amd64 (Intel) and arm64 (M1)

* added changelog

* Update `Dockerfile` to have `ARCH` build argument

* Remove `docs/CHANGES.md`

* Upgrade the Basenine version from `v0.3.0` to `v0.4.6`

* Update `publish.yml` to have `ARCH` build argument

* Switch `BasenineImageRepo` to Docker Hub

* Have separate build arguments for `ARCH` and `GOARCH`

* Upgrade the Basenine version from `v0.4.6` to `v0.4.10`

* Oops forgot to update the 10th duplicated shell script

* Fix the oopsie and reduce duplications

* Fix `Dockerfile`

* Fix the incompatibility issue between Go plugins and gold linker in Alpine inside `Dockerfile`

* Fix `asm: xxhash_amd64.s:120: when dynamic linking, R15 is clobbered by a global variable access` error

* Update `Dockerfile` to have cross-compilation on an AMD64 machine

Also revert changes in the shell scripts

* Delete `debug.Dockerfile`

* Create a custom base (`debian:buster-slim` based) image for the shipped image

* Replace `mertyildiran/debian-pcap` with `up9inc/debian-pcap`

* Upgrade Basenine version to `v0.4.12`

* Use `debian:stable-slim` as the base

* Fix the indentation in the `Dockerfile`

* Update `publish.yml`

* Enable `publish.yml` for `feature/multiarch_build` branch

* Tag correctly and set `ARCH` Docker argument

* Remove the lines that are forgotten to be removed from the shell scripts

* Add `MizuAgentImageRepo` constant and use it as default `AgentImage` value

* Bring back `Set up Cloud SDK` step to `Build the CLI and publish` job

* Build ARM64 CLI for Linux as well

* Revert "Enable `publish.yml` for `feature/multiarch_build` branch"

This reverts commit d30be4c1f0.

* Revert Go 1.17 upgrade

* Remove `build_extensions_debug.sh` as well

* Make the `Dockerfile` to compile the agent statically

* Statically link the protocol extensions

* Fix `Dockerfile`

* Bring back `-s -w` flags

* Verify the signatures of the downloads in `dockcross/linux-arm64-musl`

* Revert modifications in some shell scripts

* Make the `BUILDARCH` and `TARGETARCH` separation in the `Dockerfile`

* Separate cross-compilation builder image into a separate repo named `up9inc/linux-arm64-musl-go-libpcap`

* Fill the shell script and specify the tag for `dockcross/linux-arm64-musl`

* Remove the unnecessary dependencies from `builder-native-base`

* Improve the comments in the `Dockerfile`

* Upgrade Basenine version to `v0.4.13`

* Fix `Dockerfile`

* Revert "Revert "Enable `publish.yml` for `feature/multiarch_build` branch""

This reverts commit 303e466bdc.

* Revert "Revert "Revert "Enable `publish.yml` for `feature/multiarch_build` branch"""

This reverts commit 0fe252bbdb.

* Remove `push-docker-debug` from the `Makefile`

* Rename `publish.yml` to `release.yml`

Co-authored-by: Alex Haiut <alex@up9.com>
2022-01-25 21:24:50 +03:00
M. Mert Yıldıran
7b333556d0
Add Redis Serialization Protocol support (#290)
* Bring in the files

* Add request-response pair matcher for Redis

* Implement the `Represent` method

* Update `representGeneric` method signature

* Don't export `IntToByteArr`

* Remove unused `newRedisInputStream` method

* Return the errors as string

* Adapt to the latest change in the `develop`
2021-09-23 17:09:00 +03:00