* add protocols array to the endpoint
* no message
* no message
* fix tests and small fix for the iteration
* fix the color of the protocol
* Get protocols list and method colors from server
* fix tests
* cr fixes
Co-authored-by: Amit Fainholts <amit@up9.com>
* Determine the Go ABI and get `goid` offset from DWARF
* Add `ABI` enum and morph the function according to the detected ABI
* Pass `goid` offset to an eBPF map to retrieve it in eBPF context
* Add `vmlinux.h` and implement `get_goid_from_thread_local_storage`
* Fix BPF verifier errors
* Update the comments
* Add `go_abi_0.h` and implement `ABI0` specific reads for `arm64`
* Upgrade `github.com/cilium/ebpf` to `v0.9.0`
* Add a comment
* Add macros for x86 specific parts
* Update `x86.o`
* Fix the map key type
* Add `user_pt_regs`
* Update arm64 object file
* Fix the version detection logic
* Add `getGStructOffset` method
* Define `goid_offsets`, `goid_offsets_map` structs and pass the offsets correctly
* Fix the `net.TCPConn` and buffer addresses for `ABI0`
* Remove comment
* Fix the issues for arm64 build
* Update x86.o
* Revert "Fix the issues for arm64 build"
This reverts commit 48b041b1b6.
* Revert `user_pt_regs`
* Add `vmlinux` directory
* Fix the `build.sh` and `Dockerfile`
* Add vmlinux_arm64.h
* Disable `get_goid_from_thread_local_storage` on ARM64 with a macro
* Update x86.o
* Update arm64.o
* x86
* arm64
* Fix the cross-compilation issue from x86 to arm64
* Fix the same thing for x86
* Use `BPF_CORE_READ` macro instead of `bpf_ringbuf_reserve` to support kernel versions older than 5.8
Also;
Add legacy version of thread_struct: thread_struct___v46
Build an additional object file for the kernel versions older than or equal to 4.6 and load them accordingly.
Add github.com/moby/moby
* Make #define directives more definitive
* Select the x86 and arm64 versions of `vmlinux.h` using macros
* Put `goid` offsets into the map before installing `uprobe`(s)
* arm64
* #run_acceptance_tests
* Remove a forgotten `fmt.Printf`
* Log the detected Linux kernel version
* Handle the Go `ABIInternal` differences on ARM64
* Log the Capstone version, arch and mode
* Upgrade Capstone from `4.0.2` to `5.0-rc2` to have ARM64 instructions fix
* Use the correct register on ARM64 for reading buffer length
* Fix the addresses on ARM64
* #run_acceptance_tests
* Update `x86.o`
* Update `arm64.o`
* #run_acceptance_tests
Do not save OAS example response if it is over the limit.
SpecGen.MaxExampleLen is measured in bytes.
Limit=0 means "don't record any examples", while limit=-1 means "record all".
Limit is configured via the config file / --set: --set oas.max-example-len=100.
The default is 10KB.
This PR breaks the previous config. Moved oas bool (enable/disable OAS) to oas.enable.
Users who wish to disable OAS (or to explicitly enable it) should now to so with --set oas.enable=false instead of --set oas=false.
Co-authored-by: Nimrod Gilboa Markevich <nimrod@up9.com>
Co-authored-by: Nimrod Gilboa Markevich <59927337+nimrod-up9@users.noreply.github.com>
* Run `go generate tls_tapper.go`
* Add `golang_uprobes.c`
* Add Golang hooks and offsets
* Add `golangConnection` struct and implement `pollGolangReadWrite` method
* Upgrade `github.com/cilium/ebpf` version to `v0.8.1`
* Fix the linter error
* Move map related stuff to `maps.h` and run `go generate tls_tapper.go`
* Remove unused parameter
* Add an environment variable to test Golang locally
* Replace `Libssl` occurrences with `Ssllib` for consistency
* Fix exe path finding
* Temporarily disable OpenSSL
* Fix the mixed offsets and dissection preparation
* Change the read symbol from `net/http.(*persistConn).Read` to `crypto/tls.(*Conn).Read`
* Remove `len` and `cap` fields
* Fix the indent
* Fix the read data address
* Make `golang_dial_writes` key `__u64` and include the PID
* Fix the read data address one more time
* Temporarily disable the PCAP capture
* Add a uprobe for `net/http.(*gzipReader).Read` to read chunked HTTP response body
* Cancel `golang_crypto_tls_read_uprobe` if it's a gzip read
* Make hash map names more meaningful
* Pass the connection address from `write` to `gzip` through a common address between `gzip` and `dial`
* Fix the probed line number links
* Add `golangReader` struct and implement its `Read` method
* Have a single counter pair and request response matcher per Golang connection
* Add `MIZU_GLOBAL_GOLANG_PATH` environment variable
* `NULL` terminate the bytes with `unix.ByteSliceToString`
* Temporarily reject the gzip chunks
* Add malformed TODOs
* Revert "`NULL` terminate the bytes with `unix.ByteSliceToString`"
This reverts commit 7ee7ef7e44.
* Bring back `len` and `cap` fields
* Set `len` and `cap` in `golang_net_http_gzipreader_read_uprobe` as well
* Remove two `TODO`s
* Fix the `key_gzip` offsets
* Compress if it's gzip chunk (probably wrong!)
* Revert "Compress if it's gzip chunk (probably wrong!)"
This reverts commit 094a7c3da4.
* Remove `golang_net_http_gzipreader_read_uprobe`
* Read constant 4KiB
* Use constant read length
* Get the correct len of bytes (saw the second entry)
* Set all buffer sizes to `CHUNK_SIZE`
* Remove a `TODO`
* Revert "Temporarily disable the PCAP capture"
This reverts commit a2da15ef2d.
* Update `golang_crypto_tls_read_uprobe`
* Set the `reader` field of `tlsStream` to fix a `nil pointer dereference` error
* Don't export any fields of `golangConnection`
* Close the reader when we drop the connection
* Add a tracepoint for `sys_enter_close` to detect socket closes
* Rename `socket` struct to `golang_socket`
* Call `should_tap` in Golang uprobes
* Add `log_error` calls
* Revert "Temporarily disable OpenSSL"
This reverts commit f54d9a453f.
* Fix linter
* Revert "Revert "Temporarily disable OpenSSL""
This reverts commit 2433d867af.
* Change `golang_read_writes` map type from `BPF_RINGBUF` to `BPF_PERF_OUTPUT`
* Rename `golang_read_write` to `golang_event`
* Define an error
* Add comments
* Revert "Revert "Revert "Temporarily disable OpenSSL"""
This reverts commit e5a1de9c71.
* Fix `pollGolang`
* Revert "Revert "Revert "Revert "Temporarily disable OpenSSL""""
This reverts commit 6e1bd5d4f3.
* Fix `panic: send on closed channel`
* Revert "Revert "Revert "Revert "Revert "Temporarily disable OpenSSL"""""
This reverts commit 57d0584655.
* Use `findLibraryByPid`
* Revert "Revert "Revert "Revert "Revert "Revert "Temporarily disable OpenSSL""""""
This reverts commit 46f3d290b0.
* Revert "Revert "Revert "Revert "Revert "Revert "Revert "Temporarily disable OpenSSL"""""""
This reverts commit 775c833c06.
* Log tapping Golang
* Fix `Poll`
* Refactor `golang_net_http_dialconn_uprobe`
* Remove an excess error check
* Fix `can only use path@version syntax with 'go get' and 'go install' in module-aware mode` error in `tap/tlstapper/bpf-builder/build.sh`
* Unify Golang and OpenSSL under a single perf event buffer and `tls_chunk` struct
* Generate `tlsTapperChunkType` type (enum) as well
* Use kernel page size for the `sys_closes` perf buffer
* Fix the linter error
* Fix `MIZU_GLOBAL_GOLANG_PID` environment variable's functionality
* Rely on tracepoints for file descriptor retrieval in Golang implementation
* Remove the unnecessary changes
* Move common functions into `common.c`
* Declare `lookup_ssl_info` function to reduce duplication
* Fix linter
* Add comments and TODOs
* Remove `MIZU_GLOBAL_GOLANG_PATH` environment variable
* Update the object files
* Fix indentation
* Update object files
* Add `go_abi_internal.h`
* Fix `lookup_ssl_info`
* Convert indentation to spaces
* Add header guard comment
* Add more comments
* Find the `ret` instructions using Capstone Engine and `uprobe` the `return` statements
* Implement `get_fd_from_tcp_conn` function
* Separate SSL contexts to OpenSSL and Go
* Move `get_count_bytes` from `common.c` to `openssl_uprobes.c`
* Rename everything contains Golang to Go
* Reduce duplication in `go_uprobes.c`
* Update the comments
* Install Capstone in CI and Docker native builds
* Update `devops/install-capstone.sh`
* Add Capstone to AArch64 cross-compilation target
* Fix some of the issues on ARM64
* Delete the map element in `_ex_urpobe`
* Remove an unsued `LOG_` macro
* Rename `aquynh` to `capstone-engine`
* Add comment
* Revert "Fix some of the issues on ARM64"
This reverts commit 0b3eceddf4.
* Revert "Revert "Fix some of the issues on ARM64""
This reverts commit 681534ada1.
* Update object files
* Remove unnecessary return
* Increase timeout
* #run_acceptance_tests
* #run_acceptance_tests
* Fix the `arm64v8` sourced builds
* #run_acceptance_tests
* Add gin-contrib/pprof dependency
* Run pprof server on agent with --profiler flag
* Add --profiler flag to cli
* Fix error message
* Print cpu usage percentage
* measure cpu of current pid instead of globaly on the system
* Add scripts to plot performance
* Plot packetsCount in analysis
* Concat to DataFrame
* Plot in turbo colorscheme
* Make COLORMAP const
* Fix rss units
* Reduce code repetition by adding function for plotting
* Allow grouping based on filenames
* Temporary: Marked with comments where to disable code for experiments
* Add newline at end of file
* Add tap.cpuprofile flag. Change memprofile flag to tap.memprofile
* create tapper modes for debugging using env vars
* Fix rss plot units (MB instead of bytes)
* Remove comment
* Add info to plot script
* Remove tap.cpumemprofile. Rename tap.memprofile to memprofile
* Remove unused import
* Remove whitespaces
Co-authored-by: M. Mert Yıldıran <mehmet@up9.com>
* Remove whitespaces
Co-authored-by: M. Mert Yıldıran <mehmet@up9.com>
* Remove whitespaces
Co-authored-by: M. Mert Yıldıran <mehmet@up9.com>
* Remove whitespaces
Co-authored-by: M. Mert Yıldıran <mehmet@up9.com>
* Remove whitespaces
Co-authored-by: M. Mert Yıldıran <mehmet@up9.com>
* Remove whitespaces
Co-authored-by: M. Mert Yıldıran <mehmet@up9.com>
* Rename debug env vars
* Create package for debug env vars, read each env var once
* Run go mod tidy
* Increment MatchedPairs before emitting
* Only count cores once
* Count virtual and physical cores
* Add dbgctl replace in cli
* Fix lint: Check return values
* Add tap/dbgctl to test-lint make rule
* Replace tap/dbgctl in all modules
* #run_acceptance_tests
* Copy dbgctl module to docker image
* Debug/profile tapper benchmark (#1093)
* add mizu debug env to avoid all extensions
* add readme + run_tapper_benchmark.sh
* temporary change branch name
* fix readme
* fix MIZU_BENCHMARK_CLIENTS_COUNT env
* change tap target to tcp stream
* track live tcp streams
* pr fixes
* rename tapperPacketsCount to ignored_packets_count
* change mizu tapper to mizu debugg
Co-authored-by: David Levanon <dvdlevanon@gmail.com>
Co-authored-by: M. Mert Yıldıran <mehmet@up9.com>
* stop tapping self tapper traffic
* run go mod tidy
* allow to explicitly ignore ports
* remove unused code
* remove shared from tap + go mod tidy
* move ignroe ports to tapper
* rename TapperPacketsCount to IgnoredPacketsCount
* don't check null - go is smart
* remove nil check
* #run_acceptance_tests
* Wait 4 seconds after filter submission #run_acceptance_tests
* Revert "Wait 4 seconds after filter submission #run_acceptance_tests"
This reverts commit 3c20ccfcec.
* Upgrade Basenine version to `v0.8.2` #run_acceptance_tests
* #run_acceptance_tests
* Revert "Revert "Wait 4 seconds after filter submission #run_acceptance_tests""
This reverts commit f0c595150d.
* Reduce the delay to 2 seconds #run_acceptance_tests
* Use `cy.wait()` instead of `setTimeout`
* #run_acceptance_tests
* Increase it to 4 seconds #run_acceptance_tests
* #run_acceptance_tests
* Wait for a second and pause the stream to preserve the DOM #run_acceptance_tests
* Wait in a better place #run_acceptance_tests
* Wait `pause-icon` to disappear #run_acceptance_tests
* Wait one more second #run_acceptance_tests
* Fix `setup.sh`
* #run_acceptance_tests
* Fix the place of `npm i` call
* #run_acceptance_tests
* Fix the other workflow as well #run_acceptance_tests
* Fix the workflows once more
* #run_acceptance_tests
* Don't ignore `*/cypress/support` #run_acceptance_tests
* Do `npm i` in `setup.sh` #run_acceptance_tests
* Wait for at least 4 seconds #run_acceptance_tests
* #run_acceptance_tests
* Fix #run_acceptance_tests
* Comment #run_acceptance_tests
* Fix `{alt+enter}` to `{ctrl+enter}` and update the `checkFilter` and `deeperCheck` method signatures #run_acceptance_tests
* #run_acceptance_tests
* #run_acceptance_tests
* Extract it to a function #run_acceptance_tests
* Remove `cypress-wait-until`
* Revert `.gitignore` #run_acceptance_tests
* Fetch N number of records in M milliseconds timeout before streaming the records
* Implement the functionality inside socket data streamer
* Reverse the `fetchData` slice
* #run_acceptance_tests
* Trying to fix the tests.
#run_acceptance_tests
* javascript compilation error.
* #run_acceptance_tests
* Name the method better
* Upgrade Basenine version to `v0.8.0`
* Fix some issues related to `Fetch`
* Upgrade the Basenine version in `Dockerfile` as well
* Remove underscore from the parameter name
* Parameterize fetch timeout ms
Co-authored-by: gadotroee <55343099+gadotroee@users.noreply.github.com>
Co-authored-by: Roee Gadot <roee.gadot@up9.com>
* avoid chunks with invalid address
* tls tapper should distict between pids
* prettfy tls verbose log and tls key
* support tls from multi threads + duplicate calls to the same target
* introduce fdCache and user address pair as tls key
* remove unused comment
* fix merge conflicts
* use lru for fdcache
* pr fixes - renaming
* fix conflict issue
* Add GraphQL detection
* Change the abbreviation to `GQL`
* Fix the `TestMacros` test case
* Try to fix the security issues reported by Snyk
* Run `go mod tidy` in `agent` directory
* Upgrade `golang.org/x/crypto`
* Downgrade `golang.org/x/crypto`
* Downgrade `golang.org/x/crypto`
* Downgrade `golang.org/x/crypto`
* Upgrade to `github.com/vektah/gqlparser/v2 v2.4.2`
* Run `go mod tidy`
* Replace `github.com/vektah/gqlparser/v2` with its fork `github.com/mertyildiran/gqlparser/v2` to fix the vulns
* Upgrade the `github.com/mertyildiran/gqlparser/v2 v2.4.6`
* Don't upgrade `golang.org/x/net`
* Remove non-critical TLS detected log that causes `slice bounds out of range` error
* Remove all non-functional `OutboundLink` code that was providing `/status/recentTLSLinks` endpoint
* Fix more unused code
* Basenine MongoDB mess
* Fix more
* Fix the `mongodb` container arguments
* Add Basenine ARM64 binary
* Make the changes related to `leftOff` becoming a string value
* Make `leftOffTop` state string
* Handle `CloseConnection` in `Fetch`
* Upgrade Basenine to `0.7.0`
* Revert the changes in `package.json` and `package-lock.json`
* Fix the `Dockerfile`
* Remove the binaries
* Increase the Basenine up deadline to 20 seconds
* Revert the changes in `shared/kubernetes/provider.go`
* Fix the OAS generator tests
* Protect from race condition
* Fix mutexes
* Fix unlock
* Fix logging data types
* Try to stabilize the tests
* Remove the `replace` statement
* revert the change the done in 2899414f2b to not change the leftOff
* Change `leftOffBottom` empty string default value to `latest`
* Upgrade Basenine to `0.7.1`
* Handle the Basenine client library errors better
* Use `DEFAULT_QUERY` constant
* Remove `min=-1`
* Replace some `Errorf`s with `Panicf`s
* Remove the closure in `runGenerator` method
* Remove an unnecessary check
Co-authored-by: M. Mert Yildiran <mehmet@up9.com>
Co-authored-by: Andrey Pokhilko <apc4@ya.ru>
Co-authored-by: undera <undera@undera-old-desktop.home>
Co-authored-by: AmitUp9 <96980485+AmitUp9@users.noreply.github.com>
* Added log message
* Remove Reset function from OasGenerator interface, use Stop+Start instead
* SetEntriesQuery returns a bool stating whether the query changed
