Do not save OAS example response if it is over the limit.
SpecGen.MaxExampleLen is measured in bytes.
Limit=0 means "don't record any examples", while limit=-1 means "record all".
Limit is configured via the config file / --set: --set oas.max-example-len=100.
The default is 10KB.
This PR breaks the previous config. Moved oas bool (enable/disable OAS) to oas.enable.
Users who wish to disable OAS (or to explicitly enable it) should now to so with --set oas.enable=false instead of --set oas=false.
Co-authored-by: Nimrod Gilboa Markevich <nimrod@up9.com>
Co-authored-by: Nimrod Gilboa Markevich <59927337+nimrod-up9@users.noreply.github.com>
* Run `go generate tls_tapper.go`
* Add `golang_uprobes.c`
* Add Golang hooks and offsets
* Add `golangConnection` struct and implement `pollGolangReadWrite` method
* Upgrade `github.com/cilium/ebpf` version to `v0.8.1`
* Fix the linter error
* Move map related stuff to `maps.h` and run `go generate tls_tapper.go`
* Remove unused parameter
* Add an environment variable to test Golang locally
* Replace `Libssl` occurrences with `Ssllib` for consistency
* Fix exe path finding
* Temporarily disable OpenSSL
* Fix the mixed offsets and dissection preparation
* Change the read symbol from `net/http.(*persistConn).Read` to `crypto/tls.(*Conn).Read`
* Remove `len` and `cap` fields
* Fix the indent
* Fix the read data address
* Make `golang_dial_writes` key `__u64` and include the PID
* Fix the read data address one more time
* Temporarily disable the PCAP capture
* Add a uprobe for `net/http.(*gzipReader).Read` to read chunked HTTP response body
* Cancel `golang_crypto_tls_read_uprobe` if it's a gzip read
* Make hash map names more meaningful
* Pass the connection address from `write` to `gzip` through a common address between `gzip` and `dial`
* Fix the probed line number links
* Add `golangReader` struct and implement its `Read` method
* Have a single counter pair and request response matcher per Golang connection
* Add `MIZU_GLOBAL_GOLANG_PATH` environment variable
* `NULL` terminate the bytes with `unix.ByteSliceToString`
* Temporarily reject the gzip chunks
* Add malformed TODOs
* Revert "`NULL` terminate the bytes with `unix.ByteSliceToString`"
This reverts commit 7ee7ef7e44.
* Bring back `len` and `cap` fields
* Set `len` and `cap` in `golang_net_http_gzipreader_read_uprobe` as well
* Remove two `TODO`s
* Fix the `key_gzip` offsets
* Compress if it's gzip chunk (probably wrong!)
* Revert "Compress if it's gzip chunk (probably wrong!)"
This reverts commit 094a7c3da4.
* Remove `golang_net_http_gzipreader_read_uprobe`
* Read constant 4KiB
* Use constant read length
* Get the correct len of bytes (saw the second entry)
* Set all buffer sizes to `CHUNK_SIZE`
* Remove a `TODO`
* Revert "Temporarily disable the PCAP capture"
This reverts commit a2da15ef2d.
* Update `golang_crypto_tls_read_uprobe`
* Set the `reader` field of `tlsStream` to fix a `nil pointer dereference` error
* Don't export any fields of `golangConnection`
* Close the reader when we drop the connection
* Add a tracepoint for `sys_enter_close` to detect socket closes
* Rename `socket` struct to `golang_socket`
* Call `should_tap` in Golang uprobes
* Add `log_error` calls
* Revert "Temporarily disable OpenSSL"
This reverts commit f54d9a453f.
* Fix linter
* Revert "Revert "Temporarily disable OpenSSL""
This reverts commit 2433d867af.
* Change `golang_read_writes` map type from `BPF_RINGBUF` to `BPF_PERF_OUTPUT`
* Rename `golang_read_write` to `golang_event`
* Define an error
* Add comments
* Revert "Revert "Revert "Temporarily disable OpenSSL"""
This reverts commit e5a1de9c71.
* Fix `pollGolang`
* Revert "Revert "Revert "Revert "Temporarily disable OpenSSL""""
This reverts commit 6e1bd5d4f3.
* Fix `panic: send on closed channel`
* Revert "Revert "Revert "Revert "Revert "Temporarily disable OpenSSL"""""
This reverts commit 57d0584655.
* Use `findLibraryByPid`
* Revert "Revert "Revert "Revert "Revert "Revert "Temporarily disable OpenSSL""""""
This reverts commit 46f3d290b0.
* Revert "Revert "Revert "Revert "Revert "Revert "Revert "Temporarily disable OpenSSL"""""""
This reverts commit 775c833c06.
* Log tapping Golang
* Fix `Poll`
* Refactor `golang_net_http_dialconn_uprobe`
* Remove an excess error check
* Fix `can only use path@version syntax with 'go get' and 'go install' in module-aware mode` error in `tap/tlstapper/bpf-builder/build.sh`
* Unify Golang and OpenSSL under a single perf event buffer and `tls_chunk` struct
* Generate `tlsTapperChunkType` type (enum) as well
* Use kernel page size for the `sys_closes` perf buffer
* Fix the linter error
* Fix `MIZU_GLOBAL_GOLANG_PID` environment variable's functionality
* Rely on tracepoints for file descriptor retrieval in Golang implementation
* Remove the unnecessary changes
* Move common functions into `common.c`
* Declare `lookup_ssl_info` function to reduce duplication
* Fix linter
* Add comments and TODOs
* Remove `MIZU_GLOBAL_GOLANG_PATH` environment variable
* Update the object files
* Fix indentation
* Update object files
* Add `go_abi_internal.h`
* Fix `lookup_ssl_info`
* Convert indentation to spaces
* Add header guard comment
* Add more comments
* Find the `ret` instructions using Capstone Engine and `uprobe` the `return` statements
* Implement `get_fd_from_tcp_conn` function
* Separate SSL contexts to OpenSSL and Go
* Move `get_count_bytes` from `common.c` to `openssl_uprobes.c`
* Rename everything contains Golang to Go
* Reduce duplication in `go_uprobes.c`
* Update the comments
* Install Capstone in CI and Docker native builds
* Update `devops/install-capstone.sh`
* Add Capstone to AArch64 cross-compilation target
* Fix some of the issues on ARM64
* Delete the map element in `_ex_urpobe`
* Remove an unsued `LOG_` macro
* Rename `aquynh` to `capstone-engine`
* Add comment
* Revert "Fix some of the issues on ARM64"
This reverts commit 0b3eceddf4.
* Revert "Revert "Fix some of the issues on ARM64""
This reverts commit 681534ada1.
* Update object files
* Remove unnecessary return
* Increase timeout
* #run_acceptance_tests
* #run_acceptance_tests
* Fix the `arm64v8` sourced builds
* #run_acceptance_tests
* Add gin-contrib/pprof dependency
* Run pprof server on agent with --profiler flag
* Add --profiler flag to cli
* Fix error message
* Print cpu usage percentage
* measure cpu of current pid instead of globaly on the system
* Add scripts to plot performance
* Plot packetsCount in analysis
* Concat to DataFrame
* Plot in turbo colorscheme
* Make COLORMAP const
* Fix rss units
* Reduce code repetition by adding function for plotting
* Allow grouping based on filenames
* Temporary: Marked with comments where to disable code for experiments
* Add newline at end of file
* Add tap.cpuprofile flag. Change memprofile flag to tap.memprofile
* create tapper modes for debugging using env vars
* Fix rss plot units (MB instead of bytes)
* Remove comment
* Add info to plot script
* Remove tap.cpumemprofile. Rename tap.memprofile to memprofile
* Remove unused import
* Remove whitespaces
Co-authored-by: M. Mert Yıldıran <mehmet@up9.com>
* Remove whitespaces
Co-authored-by: M. Mert Yıldıran <mehmet@up9.com>
* Remove whitespaces
Co-authored-by: M. Mert Yıldıran <mehmet@up9.com>
* Remove whitespaces
Co-authored-by: M. Mert Yıldıran <mehmet@up9.com>
* Remove whitespaces
Co-authored-by: M. Mert Yıldıran <mehmet@up9.com>
* Remove whitespaces
Co-authored-by: M. Mert Yıldıran <mehmet@up9.com>
* Rename debug env vars
* Create package for debug env vars, read each env var once
* Run go mod tidy
* Increment MatchedPairs before emitting
* Only count cores once
* Count virtual and physical cores
* Add dbgctl replace in cli
* Fix lint: Check return values
* Add tap/dbgctl to test-lint make rule
* Replace tap/dbgctl in all modules
* #run_acceptance_tests
* Copy dbgctl module to docker image
* Debug/profile tapper benchmark (#1093)
* add mizu debug env to avoid all extensions
* add readme + run_tapper_benchmark.sh
* temporary change branch name
* fix readme
* fix MIZU_BENCHMARK_CLIENTS_COUNT env
* change tap target to tcp stream
* track live tcp streams
* pr fixes
* rename tapperPacketsCount to ignored_packets_count
* change mizu tapper to mizu debugg
Co-authored-by: David Levanon <dvdlevanon@gmail.com>
Co-authored-by: M. Mert Yıldıran <mehmet@up9.com>
* stop tapping self tapper traffic
* run go mod tidy
* allow to explicitly ignore ports
* remove unused code
* remove shared from tap + go mod tidy
* move ignroe ports to tapper
* rename TapperPacketsCount to IgnoredPacketsCount
* don't check null - go is smart
* remove nil check
* #run_acceptance_tests
* Wait 4 seconds after filter submission #run_acceptance_tests
* Revert "Wait 4 seconds after filter submission #run_acceptance_tests"
This reverts commit 3c20ccfcec.
* Upgrade Basenine version to `v0.8.2` #run_acceptance_tests
* #run_acceptance_tests
* Revert "Revert "Wait 4 seconds after filter submission #run_acceptance_tests""
This reverts commit f0c595150d.
* Reduce the delay to 2 seconds #run_acceptance_tests
* Use `cy.wait()` instead of `setTimeout`
* #run_acceptance_tests
* Increase it to 4 seconds #run_acceptance_tests
* #run_acceptance_tests
* Wait for a second and pause the stream to preserve the DOM #run_acceptance_tests
* Wait in a better place #run_acceptance_tests
* Wait `pause-icon` to disappear #run_acceptance_tests
* Wait one more second #run_acceptance_tests
* Fix `setup.sh`
* #run_acceptance_tests
* Fix the place of `npm i` call
* #run_acceptance_tests
* Fix the other workflow as well #run_acceptance_tests
* Fix the workflows once more
* #run_acceptance_tests
* Don't ignore `*/cypress/support` #run_acceptance_tests
* Do `npm i` in `setup.sh` #run_acceptance_tests
* Wait for at least 4 seconds #run_acceptance_tests
* #run_acceptance_tests
* Fix #run_acceptance_tests
* Comment #run_acceptance_tests
* Fix `{alt+enter}` to `{ctrl+enter}` and update the `checkFilter` and `deeperCheck` method signatures #run_acceptance_tests
* #run_acceptance_tests
* #run_acceptance_tests
* Extract it to a function #run_acceptance_tests
* Remove `cypress-wait-until`
* Revert `.gitignore` #run_acceptance_tests
* Fetch N number of records in M milliseconds timeout before streaming the records
* Implement the functionality inside socket data streamer
* Reverse the `fetchData` slice
* #run_acceptance_tests
* Trying to fix the tests.
#run_acceptance_tests
* javascript compilation error.
* #run_acceptance_tests
* Name the method better
* Upgrade Basenine version to `v0.8.0`
* Fix some issues related to `Fetch`
* Upgrade the Basenine version in `Dockerfile` as well
* Remove underscore from the parameter name
* Parameterize fetch timeout ms
Co-authored-by: gadotroee <55343099+gadotroee@users.noreply.github.com>
Co-authored-by: Roee Gadot <roee.gadot@up9.com>
* avoid chunks with invalid address
* tls tapper should distict between pids
* prettfy tls verbose log and tls key
* support tls from multi threads + duplicate calls to the same target
* introduce fdCache and user address pair as tls key
* remove unused comment
* fix merge conflicts
* use lru for fdcache
* pr fixes - renaming
* fix conflict issue
* Add GraphQL detection
* Change the abbreviation to `GQL`
* Fix the `TestMacros` test case
* Try to fix the security issues reported by Snyk
* Run `go mod tidy` in `agent` directory
* Upgrade `golang.org/x/crypto`
* Downgrade `golang.org/x/crypto`
* Downgrade `golang.org/x/crypto`
* Downgrade `golang.org/x/crypto`
* Upgrade to `github.com/vektah/gqlparser/v2 v2.4.2`
* Run `go mod tidy`
* Replace `github.com/vektah/gqlparser/v2` with its fork `github.com/mertyildiran/gqlparser/v2` to fix the vulns
* Upgrade the `github.com/mertyildiran/gqlparser/v2 v2.4.6`
* Don't upgrade `golang.org/x/net`
* Remove non-critical TLS detected log that causes `slice bounds out of range` error
* Remove all non-functional `OutboundLink` code that was providing `/status/recentTLSLinks` endpoint
* Fix more unused code
* Basenine MongoDB mess
* Fix more
* Fix the `mongodb` container arguments
* Add Basenine ARM64 binary
* Make the changes related to `leftOff` becoming a string value
* Make `leftOffTop` state string
* Handle `CloseConnection` in `Fetch`
* Upgrade Basenine to `0.7.0`
* Revert the changes in `package.json` and `package-lock.json`
* Fix the `Dockerfile`
* Remove the binaries
* Increase the Basenine up deadline to 20 seconds
* Revert the changes in `shared/kubernetes/provider.go`
* Fix the OAS generator tests
* Protect from race condition
* Fix mutexes
* Fix unlock
* Fix logging data types
* Try to stabilize the tests
* Remove the `replace` statement
* revert the change the done in 2899414f2b to not change the leftOff
* Change `leftOffBottom` empty string default value to `latest`
* Upgrade Basenine to `0.7.1`
* Handle the Basenine client library errors better
* Use `DEFAULT_QUERY` constant
* Remove `min=-1`
* Replace some `Errorf`s with `Panicf`s
* Remove the closure in `runGenerator` method
* Remove an unnecessary check
Co-authored-by: M. Mert Yildiran <mehmet@up9.com>
Co-authored-by: Andrey Pokhilko <apc4@ya.ru>
Co-authored-by: undera <undera@undera-old-desktop.home>
Co-authored-by: AmitUp9 <96980485+AmitUp9@users.noreply.github.com>
* Added log message
* Remove Reset function from OasGenerator interface, use Stop+Start instead
* SetEntriesQuery returns a bool stating whether the query changed
* Separate socket and basenine logic
* WIP
* Update socket_server_handlers.go
* Update socket_data_streamer.go and socket_server_handlers.go
* Update socket_server_handlers.go
* Merge branch 'develop' into refactor_ws
# Please enter a commit message to explain why this merge is necessary,
# especially if it merges an updated upstream into a topic branch.
#
# Lines starting with '#' will be ignored, and an empty message aborts
# the commit.
* empty commit for actions
* empty commit for actions
* commit for actions
* Revert "commit for actions"
This reverts commit 8ba2ecf7d3.
Co-authored-by: RoyUP9 <87927115+RoyUP9@users.noreply.github.com>
* Set the entry namespace to the source namespace if the destination is not resolved
* Overwrite src namespace with dst namespace only if dst non-empty
* Call OAS feeder
* Don't call old OAS code
* Rework calls
* Work on it
* Put back rules
* Make it compile
* start thinking of test
* Compiles
* Save
* Fixes
* Save
* Fixing
* Trying to fake conn
* add timeout
* Test timeout
* Fix tests
* Only build OAS for HTTP entries
* Remove some dead code
* Adding SampleIDs
* Cosmetics
* lint
* Revert rename
* Sample ID for content
* Cleanuo
* Add more sample IDs
* Checking hypothesis
* Move assignment place a bit
* Cosmetics
* Update test.yml
Co-authored-by: undera <undera@undera-old-desktop.home>
Co-authored-by: Igor Gov <iggvrv@gmail.com>
Update tappers via websocket instead of by env var. This way the DaemonSet doesn't have to be applied just to notify the tappers that the tap targets changed. The number of tapper restarts is reduced. The DaemonSet still gets applied when there is a need to add/remove a tapper from a node.
* Define `ReadProgress` struct and update `Dissector` interface such that the `bufio.Reader` progress can be learned on item emitting
* Display the `requestSize` and `responseSize` fields in the UI
* Update the tests
* publish ui-common version 1.0.130 and bump to this version in ui/package.json file
Co-authored-by: gadotroee <55343099+gadotroee@users.noreply.github.com>
Co-authored-by: Roee Gadot <roee.gadot@up9.com>
* Add `/db/flush` and `/db/reset` API endpoints
* Handle the unmarshalling errors better in the WebSocket
* Handle Basenine connection error better in the WebSocket
* Upgrade to Basenine `v0.6.5`
* Fix the duplicated `StartTime` state
Co-authored-by: gadotroee <55343099+gadotroee@users.noreply.github.com>
* Upgrade to Basenine `v0.6.2`, add `xml` helper, make `xml` and `json` helpers available in `redact` helper
* URL encode the query
* Upgrade to Basenine `v0.6.3`
* Always derive the summary and method fields from the entry in the database on read
* Update the expected JSONs in the protocol unit tests
* Add test cases for `Summarize` method
* Remove unused `GetEntry` method, `DataUnmarshaler` struct and `UnmarshalData` method
* Temporarily enable the acceptance tests
* Temporarily disable Slack notification on failure
* Update the Cypress tests
* Fix an issue in Redis
* Fix a typo and the Cypress tests
* Revert "Temporarily disable Slack notification on failure"
This reverts commit cad1901ea4.
* Revert "Temporarily enable the acceptance tests"
This reverts commit bad7706c9b.
