* Handle nested `topicData` in `representProduceRequest`
* Handle nested `topics` in `representCreateTopicsRequest` and `representCreateTopicsResponse`
* Handle nested `responses` in `representProduceResponse`
* Handle nested `topics` in `representFetchRequest` and nested `responses` in `representFetchResponse`
* Introduce `ignoreKeys` argument to `representMapAsTable` and ignore the keys based on that argument
* Bring back the `nil` checks
Support source IP resolving for HTTP traffic in Istio service mesh.
If X-Forwarded-For HTTP request header is present, replace the source address with the left-most address in X-Forwarded-For (earliest in Envoy implementation of this header).
This allows Mizu to resolve source IPs in Istio service meshes, if the use_remote_address option is on. Added instructions on how to turn it on.
* Upgrade Basenine version from `0.2.26` to `0.3.0`
* Remove `Summarize` method from `Dissector` interface and refactor the data structures in `tap/api/api.go`
* Rename `MizuEntry` to `Entry` and `BaseEntryDetails` to `BaseEntry`
* Populate `ContractStatus` field as well
* Update the cheatsheet
* Upgrade the Basenine version in the helm chart as well
* Remove a forgoten `console.log` call
- Rename --istio flag to the more general --service-mesh
- Rename internal variables, consts and structures to reflect this conceptual change
- Update the docs accordingly
* Remove unnecessary fields and split `service` into `src.name` and `dst.name`
* Don't fall back to IP address but instead display `[Unresolved]` text
* Fix the CSS issues in the plus icon position and replace the separator `->` text with `SwapHorizIcon`
* Remove the extra negation on `nodefrag` flag's value
* Support IPv4 fragmentation and IPv6 at the same time
* Set `Method` and `StatusCode` fields correctly for `HTTP/2`
* Replace unnecessary `grpc` naming with `http2`
* Make the `gRPC` and `HTTP/2` distinction
* Fix the macros of `http` extension
* Fix the macros of other protocol extensions
* Update the method signature of `Represent`
* Fix the `HTTP/2` support
* Fix some minor issues
* Upgrade Basenine version from `0.2.10` to `0.2.11`
Sorts macros before expanding them and prioritize the long macros.
* Don't regex split the gRPC method name
* Re-enable `nodefrag` flag
* discover envoy pids using cluster ips
* add istio flag to cli + rename mtls flag to istio
* add istio.md to docs
* Fixing typos
* Fix minor typos and grammer in docs
Co-authored-by: Nimrod Gilboa Markevich <nimrod@up9.com>
* Optimize the React code for feeding the entries
By building `EntryItem` only once and updating the `entries` state on meta query messages.
* Upgrade `react-scrollable-feed-virtualized` version from `1.4.3` to `1.4.8`
* Fix the `isSelected` state
* Set the query text before deciding the background to prevent lags while typing
* Upgrade Basenine version from `0.2.6` to `0.2.7`
* Set the query background color only if the query is same after the HTTP request and use `useEffect` instead
* Upgrade Basenine version from `0.2.7` to `0.2.8`
* Use `CancelToken` of `axios` instead of trying to check the query state
* Turn `updateQuery` function into a state hook
* Update the macro for `http`
* Do the `source.cancel()` call in `axios.CancelToken`
* Reduce client-side logging
* Fix the OOMKilled error by calling `debug.FreeOSMemory` periodically
* Remove `MAX_NUMBER_OF_GOROUTINES` environment variable
* Change the line
* Increase the default value of `TCP_STREAM_CHANNEL_TIMEOUT_MS` to `10000`
* Write the client and integrate to the new real-time database
* Refactor the WebSocket implementaiton for `/ws`
* Adapt the UI to the new filtering system
* Fix the rest of the issues in the UI
* Increase the buffer of the scanner
* Implement accessing single records
* Increase the buffer of another scanner
* Populate `Request` and `Response` fields of `MizuEntry`
* Add syntax highlighting for the query
* Add database to `Dockerfile`
* Fix some issues
* Update the `realtime_dbms` Git module commit hash
* Upgrade Gin version and print the query string
* Revert "Upgrade Gin version and print the query string"
This reverts commit aa09f904ee.
* Use WebSocket's itself to query instead of the query string
* Fix some errors related to conversion to HAR
* Fix the issues caused by the latest merge
* Fix the build error
* Fix PR validation GitHub workflow
* Replace the git submodule with latest Basenine version `0.1.0`
Remove `realtime_client.go` and use the official client library `github.com/up9inc/basenine/client/go` instead.
* Move Basenine host and port constants to `shared` module
* Reliably execute and wait for Basenine to become available
* Upgrade Basenine version
* Properly close WebSocket and data channel
* Fix the issues caused by the recent merge commit
* Clean up the TypeScript code
* Update `.gitignore`
* Limit the database size
* Add `Macros` method signature to `Dissector` interface and set the macros provided by the protocol extensions
* Run `go mod tidy` on `agent`
* Upgrade `github.com/up9inc/basenine/client/go` version
* Implement a mechanism to update the query using click events in the UI and use it for protocol macros
* Update the query on click to timestamps
* Fix some issues in the WebSocket and channel handling
* Update the query on clicks to status code
* Update the query on clicks to method, path and service
* Update the query on clicks to is outgoing, source and destination ports
* Add an API endpoint to validate the query against syntax errors
* Move the query background color state into `TrafficPage`
* Fix the logic in `setQuery`
* Display a toast message in case of a syntax error in the query
* Remove a call to `fmt.Printf`
* Upgrade Basenine version to `0.1.3`
* Fix an issue related to getting `MAX_ENTRIES_DB_BYTES` environment variable
* Have the `path` key in request details, in HTTP
* Rearrange the HTTP headers for the querying
* Do the same thing for `cookies` and `queryString`
* Update the query on click to table elements
Add the selectors for `TABLE` type representations in HTTP extension.
* Update the query on click to `bodySize` and `elapsedTime` in `EntryTitle`
* Add the selectors for `TABLE` type representations in AMQP extension
* Add the selectors for `TABLE` type representations in Kafka extension
* Add the selectors for `TABLE` type representations in Redis extension
* Define a struct in `tap/api.go` for the section representation data
* Add the selectors for `BODY` type representations
* Add `request.path` to the HTTP request details
* Change the summary string's field name from `path` to `summary`
* Introduce `queryable` CSS class for queryable UI elements and underline them on hover
* Instead of `N requests` at the bottom, make it `Displaying N results (queried X/Y)` and live update the values
Upgrade Basenine version to `0.2.0`.
* Verify the sha256sum of Basenine executable inside `Dockerfile`
* Pass the start time to web UI through WebSocket and always show the `EntriesList` footer
* Pipe the `stderr` of Basenine as well
* Fix the layout issues related to `CodeEditor` in the UI
* Use the correct `shasum` command in `Dockerfile`
* Upgrade Basenine version to `0.2.1`
* Limit the height of `CodeEditor` container
* Remove `Paused` enum `ConnectionStatus` in UI
* Fix the issue caused by the recent merge
* Add the filtering guide (cheatsheet)
* Update open cheatsheet button's title
* Update cheatsheet content
* Remove the old SQLite code, adapt the `--analyze` related code to Basenine
* Change the method signature of `NewEntry`
* Change the method signature of `Represent`
* Introduce `HTTPPair` field in `MizuEntry` specific to HTTP
* Remove `Entry`, `EntryId` and `EstimatedSizeBytes` fields from `MizuEntry`
Also remove the `getEstimatedEntrySizeBytes` method.
* Remove `gorm.io/gorm` dependency
* Remove unused `sensitiveDataFiltering` folder
* Increase the left margin of open cheatsheet button
* Add `overflow: auto` to the cheatsheet `Modal`
* Fix `GetEntry` method
* Fix the macro for gRPC
* Fix an interface conversion in case of AMQP
* Fix two more interface conversion errors in AMQP
* Make the `syncEntriesImpl` method blocking
* Fix a grammar mistake in the cheatsheet
* Adapt to the changes in the recent merge commit
* Improve the cheatsheet text
* Always display the timestamp in `en-US`
* Upgrade Basenine version to `0.2.2`
* Fix the order of closing Basenine connections and channels
* Don't close the Basenine channels at all
* Upgrade Basenine version to `0.2.3`
* Set the initial filter to `rlimit(100)`
* Make Basenine persistent
* Upgrade Basenine version to `0.2.4`
* Update `debug.Dockerfile`
* Fix a failing test
* Upgrade Basenine version to `0.2.5`
* Revert "Do not show play icon when disconnected (#428)"
This reverts commit 8af2e562f8.
* Upgrade Basenine version to `0.2.6`
* Make all non-informative things informative
* Make `100` a constant
* Use `===` in JavaScript no matter what
* Remove a forgotten `console.log`
* Add a comment and update the `query` in `syncEntriesImpl`
* Don't call `panic` in `GetEntry`
* Replace `panic` calls in `startBasenineServer` with `logger.Log.Panicf`
* Remove unnecessary `\n` characters in the logs
* multiple netns listen - initial commit
* multiple netns listen - actual work
* remove redundant log line
* map /proc of host to tapper
* changing kubernetes provider again after big conflict
* revert node-sass version back to 5.0.0
* Rename host_source to hostSource
Co-authored-by: gadotroee <55343099+gadotroee@users.noreply.github.com>
* PR fixes - adding comment + typos + naming conventions
* go fmt + making procfs read only
* setns back to the original value after packet source initialized
Co-authored-by: gadotroee <55343099+gadotroee@users.noreply.github.com>
* introduce tcp_assembler and tcp_packet_source - the motivation is to … (#380)
* add passive-tapper main tester (#353)
* add passive-tapper main tester
* add errors to go.sum of mizu agent
* disable host mode for tester - to avoid filterAuthorities
* rename main to tester
* build extenssions as part of the tester launch
* add a README to the tester
* solving go.mod and .sum conflicts with addition of go-errors
* trivial warning fixes (#354)
* add passive-tapper main tester
* trivial warning fixes
* add errors to go.sum of mizu agent
* disable host mode for tester - to avoid filterAuthorities
* tcp streams map (#355)
* add passive-tapper main tester
* trivial warning fixes
* add errors to go.sum of mizu agent
* tcp streams map
* disable host mode for tester - to avoid filterAuthorities
* set tcp streams map for tcp stream factory
* change rlog to mizu logger
* errors map (#356)
* add passive-tapper main tester
* trivial warning fixes
* add errors to go.sum of mizu agent
* tcp streams map
* disable host mode for tester - to avoid filterAuthorities
* set tcp streams map for tcp stream factory
* errors map
* change int to uint - errorsmap
* change from int to uint
* Change errorsMap.nErrors to uint.
* change errors map to mizu logger instead of rlog
* init mizu logger in tester + fix errormap declaration
Co-authored-by: Nimrod Gilboa Markevich <nimrod@up9.com>
* move own ips to tcp stream factory (#358)
* add passive-tapper main tester
* trivial warning fixes
* add errors to go.sum of mizu agent
* tcp streams map
* disable host mode for tester - to avoid filterAuthorities
* set tcp streams map for tcp stream factory
* errors map
* move own ips to tcp stream factory
* Feature/tapper refactor i/move own ips to tcp stream factory (#379)
* add passive-tapper main tester
* trivial warning fixes
* add errors to go.sum of mizu agent
* tcp streams map
* disable host mode for tester - to avoid filterAuthorities
* set tcp streams map for tcp stream factory
* errors map
* move own ips to tcp stream factory
* fix ownips compilation issue
* introduce tcp_assembler and tcp_packet_source - the motivation is to split the actual packet sources from the assembler, so we can have a single thread for the assembly which is separated from packet source threads
* make struts private at this point - planning to move some packages to their own package so we can utilize encapsulation
* move context to tcp_assembly + fix error check of tcp source packet
* use param instead of gloab flag for ipdefrag
* fix a typo
* remove unused pid param
Co-authored-by: Nimrod Gilboa Markevich <nimrod@up9.com>
* extract stats functions out of the main tapping function (#381)
* add passive-tapper main tester (#353)
* add passive-tapper main tester
* add errors to go.sum of mizu agent
* disable host mode for tester - to avoid filterAuthorities
* rename main to tester
* build extenssions as part of the tester launch
* add a README to the tester
* solving go.mod and .sum conflicts with addition of go-errors
* trivial warning fixes (#354)
* add passive-tapper main tester
* trivial warning fixes
* add errors to go.sum of mizu agent
* disable host mode for tester - to avoid filterAuthorities
* tcp streams map (#355)
* add passive-tapper main tester
* trivial warning fixes
* add errors to go.sum of mizu agent
* tcp streams map
* disable host mode for tester - to avoid filterAuthorities
* set tcp streams map for tcp stream factory
* change rlog to mizu logger
* errors map (#356)
* add passive-tapper main tester
* trivial warning fixes
* add errors to go.sum of mizu agent
* tcp streams map
* disable host mode for tester - to avoid filterAuthorities
* set tcp streams map for tcp stream factory
* errors map
* change int to uint - errorsmap
* change from int to uint
* Change errorsMap.nErrors to uint.
* change errors map to mizu logger instead of rlog
* init mizu logger in tester + fix errormap declaration
Co-authored-by: Nimrod Gilboa Markevich <nimrod@up9.com>
* move own ips to tcp stream factory (#358)
* add passive-tapper main tester
* trivial warning fixes
* add errors to go.sum of mizu agent
* tcp streams map
* disable host mode for tester - to avoid filterAuthorities
* set tcp streams map for tcp stream factory
* errors map
* move own ips to tcp stream factory
* Feature/tapper refactor i/move own ips to tcp stream factory (#379)
* add passive-tapper main tester
* trivial warning fixes
* add errors to go.sum of mizu agent
* tcp streams map
* disable host mode for tester - to avoid filterAuthorities
* set tcp streams map for tcp stream factory
* errors map
* move own ips to tcp stream factory
* fix ownips compilation issue
* introduce tcp_assembler and tcp_packet_source - the motivation is to split the actual packet sources from the assembler, so we can have a single thread for the assembly which is separated from packet source threads
* make struts private at this point - planning to move some packages to their own package so we can utilize encapsulation
* extract stats functions out of the main tapping function
* move context to tcp_assembly + fix error check of tcp source packet
* use param instead of gloab flag for ipdefrag
Co-authored-by: Nimrod Gilboa Markevich <nimrod@up9.com>
* Feature/tapper refactor i/internal tapper stats (#384)
* add passive-tapper main tester (#353)
* add passive-tapper main tester
* add errors to go.sum of mizu agent
* disable host mode for tester - to avoid filterAuthorities
* rename main to tester
* build extenssions as part of the tester launch
* add a README to the tester
* solving go.mod and .sum conflicts with addition of go-errors
* trivial warning fixes (#354)
* add passive-tapper main tester
* trivial warning fixes
* add errors to go.sum of mizu agent
* disable host mode for tester - to avoid filterAuthorities
* tcp streams map (#355)
* add passive-tapper main tester
* trivial warning fixes
* add errors to go.sum of mizu agent
* tcp streams map
* disable host mode for tester - to avoid filterAuthorities
* set tcp streams map for tcp stream factory
* change rlog to mizu logger
* errors map (#356)
* add passive-tapper main tester
* trivial warning fixes
* add errors to go.sum of mizu agent
* tcp streams map
* disable host mode for tester - to avoid filterAuthorities
* set tcp streams map for tcp stream factory
* errors map
* change int to uint - errorsmap
* change from int to uint
* Change errorsMap.nErrors to uint.
* change errors map to mizu logger instead of rlog
* init mizu logger in tester + fix errormap declaration
Co-authored-by: Nimrod Gilboa Markevich <nimrod@up9.com>
* move own ips to tcp stream factory (#358)
* add passive-tapper main tester
* trivial warning fixes
* add errors to go.sum of mizu agent
* tcp streams map
* disable host mode for tester - to avoid filterAuthorities
* set tcp streams map for tcp stream factory
* errors map
* move own ips to tcp stream factory
* Feature/tapper refactor i/move own ips to tcp stream factory (#379)
* add passive-tapper main tester
* trivial warning fixes
* add errors to go.sum of mizu agent
* tcp streams map
* disable host mode for tester - to avoid filterAuthorities
* set tcp streams map for tcp stream factory
* errors map
* move own ips to tcp stream factory
* fix ownips compilation issue
* introduce tcp_assembler and tcp_packet_source - the motivation is to split the actual packet sources from the assembler, so we can have a single thread for the assembly which is separated from packet source threads
* make struts private at this point - planning to move some packages to their own package so we can utilize encapsulation
* extract stats functions out of the main tapping function
* move context to tcp_assembly + fix error check of tcp source packet
* use param instead of gloab flag for ipdefrag
* introduce internal tapper stats
* minor fixes for errors map and internal stats
Co-authored-by: Nimrod Gilboa Markevich <nimrod@up9.com>
* Feature/tapper refactor i/diagnose package (#386)
* add passive-tapper main tester (#353)
* add passive-tapper main tester
* add errors to go.sum of mizu agent
* disable host mode for tester - to avoid filterAuthorities
* rename main to tester
* build extenssions as part of the tester launch
* add a README to the tester
* solving go.mod and .sum conflicts with addition of go-errors
* trivial warning fixes (#354)
* add passive-tapper main tester
* trivial warning fixes
* add errors to go.sum of mizu agent
* disable host mode for tester - to avoid filterAuthorities
* tcp streams map (#355)
* add passive-tapper main tester
* trivial warning fixes
* add errors to go.sum of mizu agent
* tcp streams map
* disable host mode for tester - to avoid filterAuthorities
* set tcp streams map for tcp stream factory
* change rlog to mizu logger
* errors map (#356)
* add passive-tapper main tester
* trivial warning fixes
* add errors to go.sum of mizu agent
* tcp streams map
* disable host mode for tester - to avoid filterAuthorities
* set tcp streams map for tcp stream factory
* errors map
* change int to uint - errorsmap
* change from int to uint
* Change errorsMap.nErrors to uint.
* change errors map to mizu logger instead of rlog
* init mizu logger in tester + fix errormap declaration
Co-authored-by: Nimrod Gilboa Markevich <nimrod@up9.com>
* move own ips to tcp stream factory (#358)
* add passive-tapper main tester
* trivial warning fixes
* add errors to go.sum of mizu agent
* tcp streams map
* disable host mode for tester - to avoid filterAuthorities
* set tcp streams map for tcp stream factory
* errors map
* move own ips to tcp stream factory
* Feature/tapper refactor i/move own ips to tcp stream factory (#379)
* add passive-tapper main tester
* trivial warning fixes
* add errors to go.sum of mizu agent
* tcp streams map
* disable host mode for tester - to avoid filterAuthorities
* set tcp streams map for tcp stream factory
* errors map
* move own ips to tcp stream factory
* fix ownips compilation issue
* introduce tcp_assembler and tcp_packet_source - the motivation is to split the actual packet sources from the assembler, so we can have a single thread for the assembly which is separated from packet source threads
* make struts private at this point - planning to move some packages to their own package so we can utilize encapsulation
* extract stats functions out of the main tapping function
* move context to tcp_assembly + fix error check of tcp source packet
* use param instead of gloab flag for ipdefrag
* introduce internal tapper stats
* minor fixes for errors map and internal stats
* move errors map + app stats + internal stats + periodic tasks to diagnose package
* initialize tapper internal stats
Co-authored-by: Nimrod Gilboa Markevich <nimrod@up9.com>
* move tcp packet source to its packet (#387)
* add passive-tapper main tester (#353)
* add passive-tapper main tester
* add errors to go.sum of mizu agent
* disable host mode for tester - to avoid filterAuthorities
* rename main to tester
* build extenssions as part of the tester launch
* add a README to the tester
* solving go.mod and .sum conflicts with addition of go-errors
* trivial warning fixes (#354)
* add passive-tapper main tester
* trivial warning fixes
* add errors to go.sum of mizu agent
* disable host mode for tester - to avoid filterAuthorities
* tcp streams map (#355)
* add passive-tapper main tester
* trivial warning fixes
* add errors to go.sum of mizu agent
* tcp streams map
* disable host mode for tester - to avoid filterAuthorities
* set tcp streams map for tcp stream factory
* change rlog to mizu logger
* errors map (#356)
* add passive-tapper main tester
* trivial warning fixes
* add errors to go.sum of mizu agent
* tcp streams map
* disable host mode for tester - to avoid filterAuthorities
* set tcp streams map for tcp stream factory
* errors map
* change int to uint - errorsmap
* change from int to uint
* Change errorsMap.nErrors to uint.
* change errors map to mizu logger instead of rlog
* init mizu logger in tester + fix errormap declaration
Co-authored-by: Nimrod Gilboa Markevich <nimrod@up9.com>
* move own ips to tcp stream factory (#358)
* add passive-tapper main tester
* trivial warning fixes
* add errors to go.sum of mizu agent
* tcp streams map
* disable host mode for tester - to avoid filterAuthorities
* set tcp streams map for tcp stream factory
* errors map
* move own ips to tcp stream factory
* Feature/tapper refactor i/move own ips to tcp stream factory (#379)
* add passive-tapper main tester
* trivial warning fixes
* add errors to go.sum of mizu agent
* tcp streams map
* disable host mode for tester - to avoid filterAuthorities
* set tcp streams map for tcp stream factory
* errors map
* move own ips to tcp stream factory
* fix ownips compilation issue
* introduce tcp_assembler and tcp_packet_source - the motivation is to split the actual packet sources from the assembler, so we can have a single thread for the assembly which is separated from packet source threads
* make struts private at this point - planning to move some packages to their own package so we can utilize encapsulation
* extract stats functions out of the main tapping function
* move context to tcp_assembly + fix error check of tcp source packet
* use param instead of gloab flag for ipdefrag
* introduce internal tapper stats
* minor fixes for errors map and internal stats
* move errors map + app stats + internal stats + periodic tasks to diagnose package
* move tcp packet source to its packet
* initialize tapper internal stats
Co-authored-by: Nimrod Gilboa Markevich <nimrod@up9.com>
* Fix coding style
* Remove `tap/internal_stats.go`
* make channel between input and assembler blocking - to preserve the same behaviour we have before the refactor
Co-authored-by: Nimrod Gilboa Markevich <nimrod@up9.com>
Co-authored-by: M. Mert Yildiran <mehmet@up9.com>
* Remove `github.com/romana/rlog` dependency completely
* Comment out all the unnecessary logging in the protocol extensions
* Remove commented out all the unnecessary logging lines
* Remove two more lines related to logging
* Add OAS contract monitoring support
* Pass the contract failure reason to UI
* Fix the issues related to contract validation
* Fix rest of the issues in the UI
* Add documentation related to contract monitoring feature
* Fix a typo in the docs
* Unmarshal to `HTTPRequestResponsePair` only if the OAS validation is enabled
* Fix an issue caused by the merge commit
* Slightly change the logic in the `validateOAS` method
Change the `contractText` value to `No Breaches` or `Breach` and make the text `white-space: nowrap`.
* Retrieve and display the failure reason for both request and response
Also display the content of the contract/OAS file in the UI.
* Display the OAS under `CONTRACT` tab with syntax highlighting
Also fix the styling in the entry feed.
* Remove `EnforcePolicyFileDeprecated` constant
* Log the other errors as well
* Get context from caller instead
* Define a type for the contract status and make its values enum-like
* Remove an unnecessary `if` statement
* Validate OAS in the CLI before passing it to Agent
* Get rid of the `github.com/ghodss/yaml` dependency in `loadOAS` by using `LoadFromData`
* Fix an artifact from the merge conflict
* Set the default log level for Agent to `INFO` and change it to `DEBUG` if `dump-logs=true` is provided
* Remove `Trace` method and replace its calls with `Debug`
* Export logging levels from `logger` by defining functions
* Revert "Export logging levels from `logger` by defining functions"
This reverts commit e554e40f4a.
* Run `go mod tidy` on agent
* Define a method named `determineLogLevel`
* Omit the `RULES` tab if the policy rules feature is inactive (WIP)
* Propagate the boolean value `isRulesEnabled` from file read error to UI
* Remove the debug log
