Commit Graph

152 Commits

Author SHA1 Message Date
Nimrod Gilboa Markevich
cfdafecc4a Add flag to dump mem profile every minute 2022-05-16 15:51:30 +03:00
David Levanon
092bd5d22d stop tapping self tapper traffic 2022-05-16 15:46:25 +03:00
David Levanon
7515d18f88 create tapper modes for debugging using env vars 2022-05-15 14:42:21 +03:00
Nimrod Gilboa Markevich
f6f06b4b27 Add tap.cpuprofile flag. Change memprofile flag to tap.memprofile 2022-05-15 13:52:44 +03:00
Nimrod Gilboa Markevich
8c788742f8 Temporary: Marked with comments where to disable code for experiments 2022-05-15 13:26:41 +03:00
David Levanon
2a0ade6c62 measure cpu of current pid instead of globaly on the system 2022-05-12 19:36:11 +03:00
Nimrod Gilboa Markevich
25b180d6ff Print cpu usage percentage 2022-05-12 18:50:04 +03:00
RoyUP9
4c0aeb8146
Fixed default mime type to empty string instead of text/html (#1071) 2022-05-08 16:52:50 +03:00
David Levanon
57f8a8dca9
Feature/fix tls not listening (#1046)
* avoid chunks with invalid address

* tls tapper should distict between pids

* prettfy tls verbose log and tls key

* support tls from multi threads + duplicate calls to the same target

* introduce fdCache and user address pair as tls key

* remove unused comment

* fix merge conflicts

* use lru for fdcache

* pr fixes - renaming

* fix conflict issue
2022-05-02 21:33:26 +03:00
M. Mert Yıldıran
aead6cbc19
Bring back debug.FreeOSMemory() call to prevent OOMKilled(s) (#1063)
* Revert "Revert "Bring back `debug.FreeOSMemory()` call""

This reverts commit 69538cb7e3.

* Revert "Revert "Revert "Remove `GOGC` environment variable from tapper"""

This reverts commit 3d9f37ab4b.

* Revert "Revert "Revert "Revert "Remove `GOGC` environment variable from tapper""""

This reverts commit 58ff0607ff.

Co-authored-by: gadotroee <55343099+gadotroee@users.noreply.github.com>
2022-05-02 14:13:04 +03:00
M. Mert Yıldıran
3cbccccb8b
Add GraphQL detection (#980)
* Add GraphQL detection

* Change the abbreviation to `GQL`

* Fix the `TestMacros` test case

* Try to fix the security issues reported by Snyk

* Run `go mod tidy` in `agent` directory

* Upgrade `golang.org/x/crypto`

* Downgrade `golang.org/x/crypto`

* Downgrade `golang.org/x/crypto`

* Downgrade `golang.org/x/crypto`

* Upgrade to `github.com/vektah/gqlparser/v2 v2.4.2`

* Run `go mod tidy`

* Replace `github.com/vektah/gqlparser/v2` with its fork `github.com/mertyildiran/gqlparser/v2` to fix the vulns

* Upgrade the `github.com/mertyildiran/gqlparser/v2 v2.4.6`

* Don't upgrade `golang.org/x/net`
2022-05-02 11:06:01 +03:00
M. Mert Yıldıran
684c51686f
Remove the obsolete replace statements that are artifacts of a merge conflict (#1061) 2022-05-01 16:51:43 +03:00
M. Mert Yıldıran
1de50b0572
Fix the request-response matcher maps iteration in clean() method and share the streams map with the TLS tapper (#1059)
* Fix `panic: interface conversion: api.RequestResponseMatcher is nil, not *http.requestResponseMatcher` error

Also fix the request-response matcher maps iteration in `clean()` method.

* Fix the mocks in the unit tests

* Remove unnecessary fields from `tlsPoller` and implement `SetProtocol` method

* Use concrete types in `tap` package

* Share the streams map with the TLS tapper

* Check interface conversion error
2022-05-01 16:16:22 +03:00
David Levanon
cade960b9b
Fix tls + creating tls_stream (#1058) 2022-05-01 14:46:31 +03:00
M. Mert Yıldıran
d3e6a69d82
Refactor tap module to achieve synchronously closing other protocol dissectors upon identification (#1026)
* Remove `tcpStreamWrapper` struct

* Refactor `tap` module and move some of the code to `tap/api` module

* Move `TrafficFilteringOptions` struct to `shared` module

* Change the `Dissect` method signature to have `*TcpReader` as an argument

* Add `CloseOtherProtocolDissectors` method and use it to synchronously close the other protocol dissectors

* Run `go mod tidy` in `cli` module

* Rename `SuperIdentifier` struct to `ProtoIdentifier`

* Remove `SuperTimer` struct

* Bring back `CloseTimedoutTcpStreamChannels` method

* Run `go mod tidy` everywhere

* Remove `GOGC` environment variable from tapper

* Fix the tests

* Bring back `debug.FreeOSMemory()` call

* Make `CloseOtherProtocolDissectors` method mutexed

* Revert "Remove `GOGC` environment variable from tapper"

This reverts commit cfc2484bbb.

* Bring back the removed `checksum`, `nooptcheck` and `ignorefsmerr` flags

* Define a bunch of interfaces and don't export any new structs from `tap/api`

* Keep the interfaces in `tap/api` but move the structs to `tap/tcp`

* Fix the unit tests by depending on `github.com/up9inc/mizu/tap`

* Use the modified `tlsEmitter`

* Define `TlsChunk` interface and make `tlsReader` implement `TcpReader`

* Remove unused fields in `tlsReader`

* Define `ReassemblyStream` interface and separate `gopacket` specififc fields to `tcpReassemblyStream` struct

Such that make `tap/api` don't depend on `gopacket`

* Remove the unused fields

* Make `tlsPoller` implement `TcpStream` interface and remove the call to `NewTcpStreamDummy` method

* Remove unused fields from `tlsPoller`

* Remove almost all of the setter methods in `TcpReader` and `TcpStream` interface and remove `TlsChunk` interface

* Revert "Revert "Remove `GOGC` environment variable from tapper""

This reverts commit ab2b9a803b.

* Revert "Bring back `debug.FreeOSMemory()` call"

This reverts commit 1cce863bbb.

* Remove excess comment

* Fix acceptance tests (`logger` module) #run_acceptance_tests

* Bring back `github.com/patrickmn/go-cache`

* Fix `NewTcpStream` method signature

* Put `tcpReader` and `tcpStream` mocks into protocol dissectors to remove `github.com/up9inc/mizu/tap` dependency

* Fix AMQP tests

* Revert 960ba644cd

* Revert `go.mod` and `go.sum` files in protocol dissectors

* Fix the comment position

* Revert `AppStatsInst` change

* Fix indent

* Fix CLI build

* Fix linter error

* Fix error msg

* Revert some of the changes in `chunk.go`
2022-04-28 17:19:14 +03:00
M. Mert Yıldıran
e1ad302c29
Make logger a separate module such that don't depend on shared module as a whole for logging (#1047)
* Make `logger` a separate module such that don't depend on `shared` module as a whole for logging

* Update `Dockerfile`
2022-04-27 22:26:27 +03:00
David Levanon
b5c665b602
set capture time for every packet, so long living readers would be accurate (#1043) 2022-04-26 14:37:44 +03:00
David Levanon
fadd052f02
init tls entries capture time (#1042) 2022-04-25 13:42:52 +03:00
gadotroee
149e86d050
Run unit tests when tap/api changes (#1016)
* update test files paths
2022-04-19 08:51:32 +03:00
David Levanon
1213162b85
Add kube namespace to tls (TRA-4443) (#1013)
* add namespace to tls - initial commit
* add tls namespace to mizu entry
2022-04-18 16:12:51 +03:00
M. Mert Yıldıran
d7fcf273c0
TRA-4494 Remove all non-functional OutboundLink code that was providing /status/recentTLSLinks endpoint (#1008)
* Remove non-critical TLS detected log that causes `slice bounds out of range` error

* Remove all non-functional `OutboundLink` code that was providing `/status/recentTLSLinks` endpoint

* Fix more unused code
2022-04-17 19:01:21 +03:00
M. Mert Yıldıran
1a5378b64b
Increase the OOM risk in exchange of less idle CPU usage (#979)
* Increase the OOM risk in exchange of less idle CPU usage

* Read the interval from an environment variable named `CLOSE_TIMEDOUT_TCP_STREAM_CHANNELS_INTERVAL_MS`

* Log the `getCloseTimedoutTcpChannelsInterval` return value
2022-04-14 09:58:05 +03:00
David Levanon
32d473ea26
Add service mesh badge (#989)
Co-authored-by: M. Mert Yıldıran <mehmet@up9.com>
Co-authored-by: M. Mert Yıldıran <mehmet@up9.com>
Co-authored-by: Nimrod Gilboa Markevich <59927337+nimrod-up9@users.noreply.github.com>
2022-04-13 13:30:57 +03:00
M. Mert Yıldıran
97000293fd
Fix the unit tests of protocol extensions (#998) 2022-04-13 12:23:08 +03:00
gadotroee
3ed9bc1e0d
Upgrade Basenine version to v0.7.1 (#984)
* Basenine MongoDB mess

* Fix more

* Fix the `mongodb` container arguments

* Add Basenine ARM64 binary

* Make the changes related to `leftOff` becoming a string value

* Make `leftOffTop` state string

* Handle `CloseConnection` in `Fetch`

* Upgrade Basenine to `0.7.0`

* Revert the changes in `package.json` and `package-lock.json`

* Fix the `Dockerfile`

* Remove the binaries

* Increase the Basenine up deadline to 20 seconds

* Revert the changes in `shared/kubernetes/provider.go`

* Fix the OAS generator tests

* Protect from race condition

* Fix mutexes

* Fix unlock

* Fix logging data types

* Try to stabilize the tests

* Remove the `replace` statement

* revert the change the done in 2899414f2b to not change the leftOff

* Change `leftOffBottom` empty string default value to `latest`

* Upgrade Basenine to `0.7.1`

* Handle the Basenine client library errors better

* Use `DEFAULT_QUERY` constant

* Remove `min=-1`

* Replace some `Errorf`s with `Panicf`s

* Remove the closure in `runGenerator` method

* Remove an unnecessary check

Co-authored-by: M. Mert Yildiran <mehmet@up9.com>
Co-authored-by: Andrey Pokhilko <apc4@ya.ru>
Co-authored-by: undera <undera@undera-old-desktop.home>
Co-authored-by: AmitUp9 <96980485+AmitUp9@users.noreply.github.com>
2022-04-13 11:28:48 +03:00
David Levanon
86e5dcea19
Tap TCP connections even if the handshake was missed (#988)
Support long-living connections. This commit improves support for Linkerd which uses long-living connections.
2022-04-13 11:17:37 +03:00
M. Mert Yıldıran
bbb44dae79
Fix the unit tests of protocol extensions (#982) 2022-04-09 06:56:09 -07:00
M. Mert Yıldıran
72a1aba3e5
TRA-4410 Display namespace field in the UI (#974) 2022-04-08 21:16:25 +03:00
M. Mert Yıldıran
6575495fa5
Remove gRPC related modifications (#958)
* Remove gRPC related modifications

* Remove gRPC status text related modifications as well

* Fixing gRPC vertical image

detect grpc when content type is 'application/grpc' as well  (and not only from the grpc-status)

Co-authored-by: gadotroee <55343099+gadotroee@users.noreply.github.com>
2022-04-06 18:50:36 +03:00
RoyUP9
d99c632102
Fixed golint strings.Title is deprecated error (#971) 2022-04-04 18:06:22 +03:00
M. Mert Yıldıran
2bfc523bbc
Handle reflect.TypeOf returning nil case (#970) 2022-04-04 16:25:18 +03:00
David Levanon
27a73e21fb
Read from service mesh network namespaces upon update (#944) 2022-03-30 13:56:37 +03:00
David Levanon
590fa08c81
EBPF error handling 2022-03-28 14:19:06 +03:00
Nimrod Gilboa Markevich
319c3c7a8d
Initialize tapper before ws (#932) 2022-03-26 21:25:18 +03:00
Nimrod Gilboa Markevich
99667984d6
Update tap targets without reinitializing packet source manager (#925) 2022-03-24 15:39:20 +02:00
David Levanon
763b0e7362
quick tls update pods solution (#918)
Update TLS tappers when tapped pods are updated via WS.
2022-03-24 15:21:56 +02:00
Nimrod Gilboa Markevich
a5c35d7d90
Update tap targets over ws (#901)
Update tappers via websocket instead of by env var. This way the DaemonSet doesn't have to be applied just to notify the tappers that the tap targets changed. The number of tapper restarts is reduced. The DaemonSet still gets applied when there is a need to add/remove a tapper from a node.
2022-03-23 13:50:33 +02:00
David Levanon
12f46da5c6
Support TLS big buffers (#893) 2022-03-23 11:27:06 +02:00
M. Mert Yıldıran
308fa78955
TRA-4383 Calculate request and response sizes and display them instead of BodySize field (#897)
* Define `ReadProgress` struct and update `Dissector` interface such that the `bufio.Reader` progress can be learned on item emitting

* Display the `requestSize` and `responseSize` fields in the UI

* Update the tests

* publish ui-common version 1.0.130 and bump to this version in ui/package.json file

Co-authored-by: gadotroee <55343099+gadotroee@users.noreply.github.com>
Co-authored-by: Roee Gadot <roee.gadot@up9.com>
2022-03-21 19:34:59 +02:00
M. Mert Yıldıran
5455220a3a
Add an indicator for the eBPF sourced entries (#886)
* Define `Capture` type and expect it as an argument in `Dissect` method

* Add an indicator for the  eBPF sourced entries

* Fix the Go lint error

* Fix the logic in the UI

* Update the expected JSONs

* Add TODOs

* Add `UndefinedCapture` constant

* Define `CaptureTypes` enum
2022-03-17 09:32:09 +03:00
David Levanon
6979441422
tls missing addresses (#825)
* stream seen file descriptors from ebpf

* re-generate bpf object files

* fixing pr comments
2022-03-14 15:40:27 +02:00
David Levanon
9ec8347c6c
set bpf filter for pcap (#865)
* set bpf filter for pcap

* implement pod updating mechanism

* Update tap/source/netns_packet_source.go

* Update tap/source/netns_packet_source.go

* minor pr fixes

Co-authored-by: Nimrod Gilboa Markevich <59927337+nimrod-up9@users.noreply.github.com>
2022-03-14 15:35:49 +02:00
David Levanon
4159938cea
add minikube over virtualbox cgroup format (#882) 2022-03-09 11:19:11 +02:00
David Levanon
5614e153f3
compile ebpf objects with llvm version 11 (#880) 2022-03-06 17:45:13 +02:00
M. Mert Yıldıran
dd430c31d5
Always derive the summary and method fields from the entry in the database on read (#877)
* Always derive the summary and method fields from the entry in the database on read

* Update the expected JSONs in the protocol unit tests

* Add test cases for `Summarize` method

* Remove unused `GetEntry` method, `DataUnmarshaler` struct and `UnmarshalData` method

* Temporarily enable the acceptance tests

* Temporarily disable Slack notification on failure

* Update the Cypress tests

* Fix an issue in Redis

* Fix a typo and the Cypress tests

* Revert "Temporarily disable Slack notification on failure"

This reverts commit cad1901ea4.

* Revert "Temporarily enable the acceptance tests"

This reverts commit bad7706c9b.
2022-03-06 15:41:36 +03:00
M. Mert Yıldıran
cf3106f636
Silence the logging that comes from Basenine and protocol dissectors (#835) 2022-02-21 18:08:16 +03:00
David Levanon
87ef469e25
Add tls tapper (#683)
* initial tls tapper commit

* add tls flag to mizu cli

* support ssl_read_ex/ssl_write_ex

* use hostproc to find libssl

* auto discover tls processes

* support libssl1.0

* recompile ebpf with old clang/llvm

* Update tap/passive_tapper.go

Co-authored-by: M. Mert Yıldıran <mehmet@up9.com>

* Update tap/tlstapper/tls_poller.go

Co-authored-by: M. Mert Yıldıran <mehmet@up9.com>

* Update tap/tlstapper/tls_poller.go

Co-authored-by: M. Mert Yıldıran <mehmet@up9.com>

* Update tap/tlstapper/tls_poller.go

Co-authored-by: M. Mert Yıldıran <mehmet@up9.com>

* Update tap/tlstapper/tls_poller.go

Co-authored-by: M. Mert Yıldıran <mehmet@up9.com>

* Update tap/tlstapper/tls_poller.go

Co-authored-by: M. Mert Yıldıran <mehmet@up9.com>

* Update tap/tlstapper/tls_poller.go

Co-authored-by: M. Mert Yıldıran <mehmet@up9.com>

* Update tap/tlstapper/tls_poller.go

Co-authored-by: M. Mert Yıldıran <mehmet@up9.com>

* upgrade ebpf go lib

* handling big tls messages

* fixing max buffer size in ebpf

* remove unused import

* fix linter issues

* minor pr fixes

* compile with old clang

* fix cgroup file format

* pr fixes + cgroup extract enhance

* fix linter

* adding indirect ebpf dep to agent go.mod

* adding ebpf docker builder

* minor pr fixes

* add req resp matcher to dissect

* rename ssl hooks to ssl hooks structs

* move to alpine, use local copy of mizu instead of git, add readme

* use global req resp mather for tls

Co-authored-by: M. Mert Yıldıran <mehmet@up9.com>
Co-authored-by: gadotroee <55343099+gadotroee@users.noreply.github.com>
2022-02-16 15:34:51 +02:00
M. Mert Yıldıran
72df652f6b
Add unit tests for Redis dissector (#809)
* Add unit tests for Redis dissector

* Update `Makefile`

* Fix `dissector.Analyze` call

Co-authored-by: gadotroee <55343099+gadotroee@users.noreply.github.com>
2022-02-16 11:29:55 +02:00
M. Mert Yıldıran
c67675c138
Add unit tests for Kafka dissector (#807)
* Add unit tests for Kafka dissector

* Return `io.EOF` if request or response header size is zero

* Sort the slice in `representMapAsTable`

* Remove the dead code

* Remove more dead code

* Remove more dead code

* Fix `dissector.Analyze` call

Co-authored-by: gadotroee <55343099+gadotroee@users.noreply.github.com>
2022-02-16 11:18:33 +02:00
M. Mert Yıldıran
e8d2b7eb3c
Add unit tests for AMQP dissector (#806)
* Add unit tests for AMQP dissector

* Always use UTC as the timezone in the timestamps

* Remove the dead code

* Remove more dead code

* Fix `dissector.Analyze` call

Co-authored-by: gadotroee <55343099+gadotroee@users.noreply.github.com>
2022-02-16 11:07:37 +02:00