* Add gin-contrib/pprof dependency
* Run pprof server on agent with --profiler flag
* Add --profiler flag to cli
* Fix error message
* Print cpu usage percentage
* measure cpu of current pid instead of globaly on the system
* Add scripts to plot performance
* Plot packetsCount in analysis
* Concat to DataFrame
* Plot in turbo colorscheme
* Make COLORMAP const
* Fix rss units
* Reduce code repetition by adding function for plotting
* Allow grouping based on filenames
* Temporary: Marked with comments where to disable code for experiments
* Add newline at end of file
* Add tap.cpuprofile flag. Change memprofile flag to tap.memprofile
* create tapper modes for debugging using env vars
* Fix rss plot units (MB instead of bytes)
* Remove comment
* Add info to plot script
* Remove tap.cpumemprofile. Rename tap.memprofile to memprofile
* Remove unused import
* Remove whitespaces
Co-authored-by: M. Mert Yıldıran <mehmet@up9.com>
* Remove whitespaces
Co-authored-by: M. Mert Yıldıran <mehmet@up9.com>
* Remove whitespaces
Co-authored-by: M. Mert Yıldıran <mehmet@up9.com>
* Remove whitespaces
Co-authored-by: M. Mert Yıldıran <mehmet@up9.com>
* Remove whitespaces
Co-authored-by: M. Mert Yıldıran <mehmet@up9.com>
* Remove whitespaces
Co-authored-by: M. Mert Yıldıran <mehmet@up9.com>
* Rename debug env vars
* Create package for debug env vars, read each env var once
* Run go mod tidy
* Increment MatchedPairs before emitting
* Only count cores once
* Count virtual and physical cores
* Add dbgctl replace in cli
* Fix lint: Check return values
* Add tap/dbgctl to test-lint make rule
* Replace tap/dbgctl in all modules
* #run_acceptance_tests
* Copy dbgctl module to docker image
* Debug/profile tapper benchmark (#1093)
* add mizu debug env to avoid all extensions
* add readme + run_tapper_benchmark.sh
* temporary change branch name
* fix readme
* fix MIZU_BENCHMARK_CLIENTS_COUNT env
* change tap target to tcp stream
* track live tcp streams
* pr fixes
* rename tapperPacketsCount to ignored_packets_count
* change mizu tapper to mizu debugg
Co-authored-by: David Levanon <dvdlevanon@gmail.com>
Co-authored-by: M. Mert Yıldıran <mehmet@up9.com>
* stop tapping self tapper traffic
* run go mod tidy
* allow to explicitly ignore ports
* remove unused code
* remove shared from tap + go mod tidy
* move ignroe ports to tapper
* rename TapperPacketsCount to IgnoredPacketsCount
* don't check null - go is smart
* remove nil check
* Remove `tcpStreamWrapper` struct
* Refactor `tap` module and move some of the code to `tap/api` module
* Move `TrafficFilteringOptions` struct to `shared` module
* Change the `Dissect` method signature to have `*TcpReader` as an argument
* Add `CloseOtherProtocolDissectors` method and use it to synchronously close the other protocol dissectors
* Run `go mod tidy` in `cli` module
* Rename `SuperIdentifier` struct to `ProtoIdentifier`
* Remove `SuperTimer` struct
* Bring back `CloseTimedoutTcpStreamChannels` method
* Run `go mod tidy` everywhere
* Remove `GOGC` environment variable from tapper
* Fix the tests
* Bring back `debug.FreeOSMemory()` call
* Make `CloseOtherProtocolDissectors` method mutexed
* Revert "Remove `GOGC` environment variable from tapper"
This reverts commit cfc2484bbb.
* Bring back the removed `checksum`, `nooptcheck` and `ignorefsmerr` flags
* Define a bunch of interfaces and don't export any new structs from `tap/api`
* Keep the interfaces in `tap/api` but move the structs to `tap/tcp`
* Fix the unit tests by depending on `github.com/up9inc/mizu/tap`
* Use the modified `tlsEmitter`
* Define `TlsChunk` interface and make `tlsReader` implement `TcpReader`
* Remove unused fields in `tlsReader`
* Define `ReassemblyStream` interface and separate `gopacket` specififc fields to `tcpReassemblyStream` struct
Such that make `tap/api` don't depend on `gopacket`
* Remove the unused fields
* Make `tlsPoller` implement `TcpStream` interface and remove the call to `NewTcpStreamDummy` method
* Remove unused fields from `tlsPoller`
* Remove almost all of the setter methods in `TcpReader` and `TcpStream` interface and remove `TlsChunk` interface
* Revert "Revert "Remove `GOGC` environment variable from tapper""
This reverts commit ab2b9a803b.
* Revert "Bring back `debug.FreeOSMemory()` call"
This reverts commit 1cce863bbb.
* Remove excess comment
* Fix acceptance tests (`logger` module) #run_acceptance_tests
* Bring back `github.com/patrickmn/go-cache`
* Fix `NewTcpStream` method signature
* Put `tcpReader` and `tcpStream` mocks into protocol dissectors to remove `github.com/up9inc/mizu/tap` dependency
* Fix AMQP tests
* Revert 960ba644cd
* Revert `go.mod` and `go.sum` files in protocol dissectors
* Fix the comment position
* Revert `AppStatsInst` change
* Fix indent
* Fix CLI build
* Fix linter error
* Fix error msg
* Revert some of the changes in `chunk.go`
Co-authored-by: M. Mert Yıldıran <mehmet@up9.com>
Co-authored-by: M. Mert Yıldıran <mehmet@up9.com>
Co-authored-by: Nimrod Gilboa Markevich <59927337+nimrod-up9@users.noreply.github.com>
* discover envoy pids using cluster ips
* add istio flag to cli + rename mtls flag to istio
* add istio.md to docs
* Fixing typos
* Fix minor typos and grammer in docs
Co-authored-by: Nimrod Gilboa Markevich <nimrod@up9.com>
* introduce tcp_assembler and tcp_packet_source - the motivation is to … (#380)
* add passive-tapper main tester (#353)
* add passive-tapper main tester
* add errors to go.sum of mizu agent
* disable host mode for tester - to avoid filterAuthorities
* rename main to tester
* build extenssions as part of the tester launch
* add a README to the tester
* solving go.mod and .sum conflicts with addition of go-errors
* trivial warning fixes (#354)
* add passive-tapper main tester
* trivial warning fixes
* add errors to go.sum of mizu agent
* disable host mode for tester - to avoid filterAuthorities
* tcp streams map (#355)
* add passive-tapper main tester
* trivial warning fixes
* add errors to go.sum of mizu agent
* tcp streams map
* disable host mode for tester - to avoid filterAuthorities
* set tcp streams map for tcp stream factory
* change rlog to mizu logger
* errors map (#356)
* add passive-tapper main tester
* trivial warning fixes
* add errors to go.sum of mizu agent
* tcp streams map
* disable host mode for tester - to avoid filterAuthorities
* set tcp streams map for tcp stream factory
* errors map
* change int to uint - errorsmap
* change from int to uint
* Change errorsMap.nErrors to uint.
* change errors map to mizu logger instead of rlog
* init mizu logger in tester + fix errormap declaration
Co-authored-by: Nimrod Gilboa Markevich <nimrod@up9.com>
* move own ips to tcp stream factory (#358)
* add passive-tapper main tester
* trivial warning fixes
* add errors to go.sum of mizu agent
* tcp streams map
* disable host mode for tester - to avoid filterAuthorities
* set tcp streams map for tcp stream factory
* errors map
* move own ips to tcp stream factory
* Feature/tapper refactor i/move own ips to tcp stream factory (#379)
* add passive-tapper main tester
* trivial warning fixes
* add errors to go.sum of mizu agent
* tcp streams map
* disable host mode for tester - to avoid filterAuthorities
* set tcp streams map for tcp stream factory
* errors map
* move own ips to tcp stream factory
* fix ownips compilation issue
* introduce tcp_assembler and tcp_packet_source - the motivation is to split the actual packet sources from the assembler, so we can have a single thread for the assembly which is separated from packet source threads
* make struts private at this point - planning to move some packages to their own package so we can utilize encapsulation
* move context to tcp_assembly + fix error check of tcp source packet
* use param instead of gloab flag for ipdefrag
* fix a typo
* remove unused pid param
Co-authored-by: Nimrod Gilboa Markevich <nimrod@up9.com>
* extract stats functions out of the main tapping function (#381)
* add passive-tapper main tester (#353)
* add passive-tapper main tester
* add errors to go.sum of mizu agent
* disable host mode for tester - to avoid filterAuthorities
* rename main to tester
* build extenssions as part of the tester launch
* add a README to the tester
* solving go.mod and .sum conflicts with addition of go-errors
* trivial warning fixes (#354)
* add passive-tapper main tester
* trivial warning fixes
* add errors to go.sum of mizu agent
* disable host mode for tester - to avoid filterAuthorities
* tcp streams map (#355)
* add passive-tapper main tester
* trivial warning fixes
* add errors to go.sum of mizu agent
* tcp streams map
* disable host mode for tester - to avoid filterAuthorities
* set tcp streams map for tcp stream factory
* change rlog to mizu logger
* errors map (#356)
* add passive-tapper main tester
* trivial warning fixes
* add errors to go.sum of mizu agent
* tcp streams map
* disable host mode for tester - to avoid filterAuthorities
* set tcp streams map for tcp stream factory
* errors map
* change int to uint - errorsmap
* change from int to uint
* Change errorsMap.nErrors to uint.
* change errors map to mizu logger instead of rlog
* init mizu logger in tester + fix errormap declaration
Co-authored-by: Nimrod Gilboa Markevich <nimrod@up9.com>
* move own ips to tcp stream factory (#358)
* add passive-tapper main tester
* trivial warning fixes
* add errors to go.sum of mizu agent
* tcp streams map
* disable host mode for tester - to avoid filterAuthorities
* set tcp streams map for tcp stream factory
* errors map
* move own ips to tcp stream factory
* Feature/tapper refactor i/move own ips to tcp stream factory (#379)
* add passive-tapper main tester
* trivial warning fixes
* add errors to go.sum of mizu agent
* tcp streams map
* disable host mode for tester - to avoid filterAuthorities
* set tcp streams map for tcp stream factory
* errors map
* move own ips to tcp stream factory
* fix ownips compilation issue
* introduce tcp_assembler and tcp_packet_source - the motivation is to split the actual packet sources from the assembler, so we can have a single thread for the assembly which is separated from packet source threads
* make struts private at this point - planning to move some packages to their own package so we can utilize encapsulation
* extract stats functions out of the main tapping function
* move context to tcp_assembly + fix error check of tcp source packet
* use param instead of gloab flag for ipdefrag
Co-authored-by: Nimrod Gilboa Markevich <nimrod@up9.com>
* Feature/tapper refactor i/internal tapper stats (#384)
* add passive-tapper main tester (#353)
* add passive-tapper main tester
* add errors to go.sum of mizu agent
* disable host mode for tester - to avoid filterAuthorities
* rename main to tester
* build extenssions as part of the tester launch
* add a README to the tester
* solving go.mod and .sum conflicts with addition of go-errors
* trivial warning fixes (#354)
* add passive-tapper main tester
* trivial warning fixes
* add errors to go.sum of mizu agent
* disable host mode for tester - to avoid filterAuthorities
* tcp streams map (#355)
* add passive-tapper main tester
* trivial warning fixes
* add errors to go.sum of mizu agent
* tcp streams map
* disable host mode for tester - to avoid filterAuthorities
* set tcp streams map for tcp stream factory
* change rlog to mizu logger
* errors map (#356)
* add passive-tapper main tester
* trivial warning fixes
* add errors to go.sum of mizu agent
* tcp streams map
* disable host mode for tester - to avoid filterAuthorities
* set tcp streams map for tcp stream factory
* errors map
* change int to uint - errorsmap
* change from int to uint
* Change errorsMap.nErrors to uint.
* change errors map to mizu logger instead of rlog
* init mizu logger in tester + fix errormap declaration
Co-authored-by: Nimrod Gilboa Markevich <nimrod@up9.com>
* move own ips to tcp stream factory (#358)
* add passive-tapper main tester
* trivial warning fixes
* add errors to go.sum of mizu agent
* tcp streams map
* disable host mode for tester - to avoid filterAuthorities
* set tcp streams map for tcp stream factory
* errors map
* move own ips to tcp stream factory
* Feature/tapper refactor i/move own ips to tcp stream factory (#379)
* add passive-tapper main tester
* trivial warning fixes
* add errors to go.sum of mizu agent
* tcp streams map
* disable host mode for tester - to avoid filterAuthorities
* set tcp streams map for tcp stream factory
* errors map
* move own ips to tcp stream factory
* fix ownips compilation issue
* introduce tcp_assembler and tcp_packet_source - the motivation is to split the actual packet sources from the assembler, so we can have a single thread for the assembly which is separated from packet source threads
* make struts private at this point - planning to move some packages to their own package so we can utilize encapsulation
* extract stats functions out of the main tapping function
* move context to tcp_assembly + fix error check of tcp source packet
* use param instead of gloab flag for ipdefrag
* introduce internal tapper stats
* minor fixes for errors map and internal stats
Co-authored-by: Nimrod Gilboa Markevich <nimrod@up9.com>
* Feature/tapper refactor i/diagnose package (#386)
* add passive-tapper main tester (#353)
* add passive-tapper main tester
* add errors to go.sum of mizu agent
* disable host mode for tester - to avoid filterAuthorities
* rename main to tester
* build extenssions as part of the tester launch
* add a README to the tester
* solving go.mod and .sum conflicts with addition of go-errors
* trivial warning fixes (#354)
* add passive-tapper main tester
* trivial warning fixes
* add errors to go.sum of mizu agent
* disable host mode for tester - to avoid filterAuthorities
* tcp streams map (#355)
* add passive-tapper main tester
* trivial warning fixes
* add errors to go.sum of mizu agent
* tcp streams map
* disable host mode for tester - to avoid filterAuthorities
* set tcp streams map for tcp stream factory
* change rlog to mizu logger
* errors map (#356)
* add passive-tapper main tester
* trivial warning fixes
* add errors to go.sum of mizu agent
* tcp streams map
* disable host mode for tester - to avoid filterAuthorities
* set tcp streams map for tcp stream factory
* errors map
* change int to uint - errorsmap
* change from int to uint
* Change errorsMap.nErrors to uint.
* change errors map to mizu logger instead of rlog
* init mizu logger in tester + fix errormap declaration
Co-authored-by: Nimrod Gilboa Markevich <nimrod@up9.com>
* move own ips to tcp stream factory (#358)
* add passive-tapper main tester
* trivial warning fixes
* add errors to go.sum of mizu agent
* tcp streams map
* disable host mode for tester - to avoid filterAuthorities
* set tcp streams map for tcp stream factory
* errors map
* move own ips to tcp stream factory
* Feature/tapper refactor i/move own ips to tcp stream factory (#379)
* add passive-tapper main tester
* trivial warning fixes
* add errors to go.sum of mizu agent
* tcp streams map
* disable host mode for tester - to avoid filterAuthorities
* set tcp streams map for tcp stream factory
* errors map
* move own ips to tcp stream factory
* fix ownips compilation issue
* introduce tcp_assembler and tcp_packet_source - the motivation is to split the actual packet sources from the assembler, so we can have a single thread for the assembly which is separated from packet source threads
* make struts private at this point - planning to move some packages to their own package so we can utilize encapsulation
* extract stats functions out of the main tapping function
* move context to tcp_assembly + fix error check of tcp source packet
* use param instead of gloab flag for ipdefrag
* introduce internal tapper stats
* minor fixes for errors map and internal stats
* move errors map + app stats + internal stats + periodic tasks to diagnose package
* initialize tapper internal stats
Co-authored-by: Nimrod Gilboa Markevich <nimrod@up9.com>
* move tcp packet source to its packet (#387)
* add passive-tapper main tester (#353)
* add passive-tapper main tester
* add errors to go.sum of mizu agent
* disable host mode for tester - to avoid filterAuthorities
* rename main to tester
* build extenssions as part of the tester launch
* add a README to the tester
* solving go.mod and .sum conflicts with addition of go-errors
* trivial warning fixes (#354)
* add passive-tapper main tester
* trivial warning fixes
* add errors to go.sum of mizu agent
* disable host mode for tester - to avoid filterAuthorities
* tcp streams map (#355)
* add passive-tapper main tester
* trivial warning fixes
* add errors to go.sum of mizu agent
* tcp streams map
* disable host mode for tester - to avoid filterAuthorities
* set tcp streams map for tcp stream factory
* change rlog to mizu logger
* errors map (#356)
* add passive-tapper main tester
* trivial warning fixes
* add errors to go.sum of mizu agent
* tcp streams map
* disable host mode for tester - to avoid filterAuthorities
* set tcp streams map for tcp stream factory
* errors map
* change int to uint - errorsmap
* change from int to uint
* Change errorsMap.nErrors to uint.
* change errors map to mizu logger instead of rlog
* init mizu logger in tester + fix errormap declaration
Co-authored-by: Nimrod Gilboa Markevich <nimrod@up9.com>
* move own ips to tcp stream factory (#358)
* add passive-tapper main tester
* trivial warning fixes
* add errors to go.sum of mizu agent
* tcp streams map
* disable host mode for tester - to avoid filterAuthorities
* set tcp streams map for tcp stream factory
* errors map
* move own ips to tcp stream factory
* Feature/tapper refactor i/move own ips to tcp stream factory (#379)
* add passive-tapper main tester
* trivial warning fixes
* add errors to go.sum of mizu agent
* tcp streams map
* disable host mode for tester - to avoid filterAuthorities
* set tcp streams map for tcp stream factory
* errors map
* move own ips to tcp stream factory
* fix ownips compilation issue
* introduce tcp_assembler and tcp_packet_source - the motivation is to split the actual packet sources from the assembler, so we can have a single thread for the assembly which is separated from packet source threads
* make struts private at this point - planning to move some packages to their own package so we can utilize encapsulation
* extract stats functions out of the main tapping function
* move context to tcp_assembly + fix error check of tcp source packet
* use param instead of gloab flag for ipdefrag
* introduce internal tapper stats
* minor fixes for errors map and internal stats
* move errors map + app stats + internal stats + periodic tasks to diagnose package
* move tcp packet source to its packet
* initialize tapper internal stats
Co-authored-by: Nimrod Gilboa Markevich <nimrod@up9.com>
* Fix coding style
* Remove `tap/internal_stats.go`
* make channel between input and assembler blocking - to preserve the same behaviour we have before the refactor
Co-authored-by: Nimrod Gilboa Markevich <nimrod@up9.com>
Co-authored-by: M. Mert Yildiran <mehmet@up9.com>
