* Remove the extra negation on `nodefrag` flag's value
* Support IPv4 fragmentation and IPv6 at the same time
* Set `Method` and `StatusCode` fields correctly for `HTTP/2`
* Replace unnecessary `grpc` naming with `http2`
* Make the `gRPC` and `HTTP/2` distinction
* Fix the macros of `http` extension
* Fix the macros of other protocol extensions
* Update the method signature of `Represent`
* Fix the `HTTP/2` support
* Fix some minor issues
* Upgrade Basenine version from `0.2.10` to `0.2.11`
Sorts macros before expanding them and prioritize the long macros.
* Don't regex split the gRPC method name
* Re-enable `nodefrag` flag
* discover envoy pids using cluster ips
* add istio flag to cli + rename mtls flag to istio
* add istio.md to docs
* Fixing typos
* Fix minor typos and grammer in docs
Co-authored-by: Nimrod Gilboa Markevich <nimrod@up9.com>
* Optimize the React code for feeding the entries
By building `EntryItem` only once and updating the `entries` state on meta query messages.
* Upgrade `react-scrollable-feed-virtualized` version from `1.4.3` to `1.4.8`
* Fix the `isSelected` state
* Set the query text before deciding the background to prevent lags while typing
* Upgrade Basenine version from `0.2.6` to `0.2.7`
* Set the query background color only if the query is same after the HTTP request and use `useEffect` instead
* Upgrade Basenine version from `0.2.7` to `0.2.8`
* Use `CancelToken` of `axios` instead of trying to check the query state
* Turn `updateQuery` function into a state hook
* Update the macro for `http`
* Do the `source.cancel()` call in `axios.CancelToken`
* Reduce client-side logging
* Fix the OOMKilled error by calling `debug.FreeOSMemory` periodically
* Remove `MAX_NUMBER_OF_GOROUTINES` environment variable
* Change the line
* Increase the default value of `TCP_STREAM_CHANNEL_TIMEOUT_MS` to `10000`
* Write the client and integrate to the new real-time database
* Refactor the WebSocket implementaiton for `/ws`
* Adapt the UI to the new filtering system
* Fix the rest of the issues in the UI
* Increase the buffer of the scanner
* Implement accessing single records
* Increase the buffer of another scanner
* Populate `Request` and `Response` fields of `MizuEntry`
* Add syntax highlighting for the query
* Add database to `Dockerfile`
* Fix some issues
* Update the `realtime_dbms` Git module commit hash
* Upgrade Gin version and print the query string
* Revert "Upgrade Gin version and print the query string"
This reverts commit aa09f904ee.
* Use WebSocket's itself to query instead of the query string
* Fix some errors related to conversion to HAR
* Fix the issues caused by the latest merge
* Fix the build error
* Fix PR validation GitHub workflow
* Replace the git submodule with latest Basenine version `0.1.0`
Remove `realtime_client.go` and use the official client library `github.com/up9inc/basenine/client/go` instead.
* Move Basenine host and port constants to `shared` module
* Reliably execute and wait for Basenine to become available
* Upgrade Basenine version
* Properly close WebSocket and data channel
* Fix the issues caused by the recent merge commit
* Clean up the TypeScript code
* Update `.gitignore`
* Limit the database size
* Add `Macros` method signature to `Dissector` interface and set the macros provided by the protocol extensions
* Run `go mod tidy` on `agent`
* Upgrade `github.com/up9inc/basenine/client/go` version
* Implement a mechanism to update the query using click events in the UI and use it for protocol macros
* Update the query on click to timestamps
* Fix some issues in the WebSocket and channel handling
* Update the query on clicks to status code
* Update the query on clicks to method, path and service
* Update the query on clicks to is outgoing, source and destination ports
* Add an API endpoint to validate the query against syntax errors
* Move the query background color state into `TrafficPage`
* Fix the logic in `setQuery`
* Display a toast message in case of a syntax error in the query
* Remove a call to `fmt.Printf`
* Upgrade Basenine version to `0.1.3`
* Fix an issue related to getting `MAX_ENTRIES_DB_BYTES` environment variable
* Have the `path` key in request details, in HTTP
* Rearrange the HTTP headers for the querying
* Do the same thing for `cookies` and `queryString`
* Update the query on click to table elements
Add the selectors for `TABLE` type representations in HTTP extension.
* Update the query on click to `bodySize` and `elapsedTime` in `EntryTitle`
* Add the selectors for `TABLE` type representations in AMQP extension
* Add the selectors for `TABLE` type representations in Kafka extension
* Add the selectors for `TABLE` type representations in Redis extension
* Define a struct in `tap/api.go` for the section representation data
* Add the selectors for `BODY` type representations
* Add `request.path` to the HTTP request details
* Change the summary string's field name from `path` to `summary`
* Introduce `queryable` CSS class for queryable UI elements and underline them on hover
* Instead of `N requests` at the bottom, make it `Displaying N results (queried X/Y)` and live update the values
Upgrade Basenine version to `0.2.0`.
* Verify the sha256sum of Basenine executable inside `Dockerfile`
* Pass the start time to web UI through WebSocket and always show the `EntriesList` footer
* Pipe the `stderr` of Basenine as well
* Fix the layout issues related to `CodeEditor` in the UI
* Use the correct `shasum` command in `Dockerfile`
* Upgrade Basenine version to `0.2.1`
* Limit the height of `CodeEditor` container
* Remove `Paused` enum `ConnectionStatus` in UI
* Fix the issue caused by the recent merge
* Add the filtering guide (cheatsheet)
* Update open cheatsheet button's title
* Update cheatsheet content
* Remove the old SQLite code, adapt the `--analyze` related code to Basenine
* Change the method signature of `NewEntry`
* Change the method signature of `Represent`
* Introduce `HTTPPair` field in `MizuEntry` specific to HTTP
* Remove `Entry`, `EntryId` and `EstimatedSizeBytes` fields from `MizuEntry`
Also remove the `getEstimatedEntrySizeBytes` method.
* Remove `gorm.io/gorm` dependency
* Remove unused `sensitiveDataFiltering` folder
* Increase the left margin of open cheatsheet button
* Add `overflow: auto` to the cheatsheet `Modal`
* Fix `GetEntry` method
* Fix the macro for gRPC
* Fix an interface conversion in case of AMQP
* Fix two more interface conversion errors in AMQP
* Make the `syncEntriesImpl` method blocking
* Fix a grammar mistake in the cheatsheet
* Adapt to the changes in the recent merge commit
* Improve the cheatsheet text
* Always display the timestamp in `en-US`
* Upgrade Basenine version to `0.2.2`
* Fix the order of closing Basenine connections and channels
* Don't close the Basenine channels at all
* Upgrade Basenine version to `0.2.3`
* Set the initial filter to `rlimit(100)`
* Make Basenine persistent
* Upgrade Basenine version to `0.2.4`
* Update `debug.Dockerfile`
* Fix a failing test
* Upgrade Basenine version to `0.2.5`
* Revert "Do not show play icon when disconnected (#428)"
This reverts commit 8af2e562f8.
* Upgrade Basenine version to `0.2.6`
* Make all non-informative things informative
* Make `100` a constant
* Use `===` in JavaScript no matter what
* Remove a forgotten `console.log`
* Add a comment and update the `query` in `syncEntriesImpl`
* Don't call `panic` in `GetEntry`
* Replace `panic` calls in `startBasenineServer` with `logger.Log.Panicf`
* Remove unnecessary `\n` characters in the logs
* multiple netns listen - initial commit
* multiple netns listen - actual work
* remove redundant log line
* map /proc of host to tapper
* changing kubernetes provider again after big conflict
* revert node-sass version back to 5.0.0
* Rename host_source to hostSource
Co-authored-by: gadotroee <55343099+gadotroee@users.noreply.github.com>
* PR fixes - adding comment + typos + naming conventions
* go fmt + making procfs read only
* setns back to the original value after packet source initialized
Co-authored-by: gadotroee <55343099+gadotroee@users.noreply.github.com>
* introduce tcp_assembler and tcp_packet_source - the motivation is to … (#380)
* add passive-tapper main tester (#353)
* add passive-tapper main tester
* add errors to go.sum of mizu agent
* disable host mode for tester - to avoid filterAuthorities
* rename main to tester
* build extenssions as part of the tester launch
* add a README to the tester
* solving go.mod and .sum conflicts with addition of go-errors
* trivial warning fixes (#354)
* add passive-tapper main tester
* trivial warning fixes
* add errors to go.sum of mizu agent
* disable host mode for tester - to avoid filterAuthorities
* tcp streams map (#355)
* add passive-tapper main tester
* trivial warning fixes
* add errors to go.sum of mizu agent
* tcp streams map
* disable host mode for tester - to avoid filterAuthorities
* set tcp streams map for tcp stream factory
* change rlog to mizu logger
* errors map (#356)
* add passive-tapper main tester
* trivial warning fixes
* add errors to go.sum of mizu agent
* tcp streams map
* disable host mode for tester - to avoid filterAuthorities
* set tcp streams map for tcp stream factory
* errors map
* change int to uint - errorsmap
* change from int to uint
* Change errorsMap.nErrors to uint.
* change errors map to mizu logger instead of rlog
* init mizu logger in tester + fix errormap declaration
Co-authored-by: Nimrod Gilboa Markevich <nimrod@up9.com>
* move own ips to tcp stream factory (#358)
* add passive-tapper main tester
* trivial warning fixes
* add errors to go.sum of mizu agent
* tcp streams map
* disable host mode for tester - to avoid filterAuthorities
* set tcp streams map for tcp stream factory
* errors map
* move own ips to tcp stream factory
* Feature/tapper refactor i/move own ips to tcp stream factory (#379)
* add passive-tapper main tester
* trivial warning fixes
* add errors to go.sum of mizu agent
* tcp streams map
* disable host mode for tester - to avoid filterAuthorities
* set tcp streams map for tcp stream factory
* errors map
* move own ips to tcp stream factory
* fix ownips compilation issue
* introduce tcp_assembler and tcp_packet_source - the motivation is to split the actual packet sources from the assembler, so we can have a single thread for the assembly which is separated from packet source threads
* make struts private at this point - planning to move some packages to their own package so we can utilize encapsulation
* move context to tcp_assembly + fix error check of tcp source packet
* use param instead of gloab flag for ipdefrag
* fix a typo
* remove unused pid param
Co-authored-by: Nimrod Gilboa Markevich <nimrod@up9.com>
* extract stats functions out of the main tapping function (#381)
* add passive-tapper main tester (#353)
* add passive-tapper main tester
* add errors to go.sum of mizu agent
* disable host mode for tester - to avoid filterAuthorities
* rename main to tester
* build extenssions as part of the tester launch
* add a README to the tester
* solving go.mod and .sum conflicts with addition of go-errors
* trivial warning fixes (#354)
* add passive-tapper main tester
* trivial warning fixes
* add errors to go.sum of mizu agent
* disable host mode for tester - to avoid filterAuthorities
* tcp streams map (#355)
* add passive-tapper main tester
* trivial warning fixes
* add errors to go.sum of mizu agent
* tcp streams map
* disable host mode for tester - to avoid filterAuthorities
* set tcp streams map for tcp stream factory
* change rlog to mizu logger
* errors map (#356)
* add passive-tapper main tester
* trivial warning fixes
* add errors to go.sum of mizu agent
* tcp streams map
* disable host mode for tester - to avoid filterAuthorities
* set tcp streams map for tcp stream factory
* errors map
* change int to uint - errorsmap
* change from int to uint
* Change errorsMap.nErrors to uint.
* change errors map to mizu logger instead of rlog
* init mizu logger in tester + fix errormap declaration
Co-authored-by: Nimrod Gilboa Markevich <nimrod@up9.com>
* move own ips to tcp stream factory (#358)
* add passive-tapper main tester
* trivial warning fixes
* add errors to go.sum of mizu agent
* tcp streams map
* disable host mode for tester - to avoid filterAuthorities
* set tcp streams map for tcp stream factory
* errors map
* move own ips to tcp stream factory
* Feature/tapper refactor i/move own ips to tcp stream factory (#379)
* add passive-tapper main tester
* trivial warning fixes
* add errors to go.sum of mizu agent
* tcp streams map
* disable host mode for tester - to avoid filterAuthorities
* set tcp streams map for tcp stream factory
* errors map
* move own ips to tcp stream factory
* fix ownips compilation issue
* introduce tcp_assembler and tcp_packet_source - the motivation is to split the actual packet sources from the assembler, so we can have a single thread for the assembly which is separated from packet source threads
* make struts private at this point - planning to move some packages to their own package so we can utilize encapsulation
* extract stats functions out of the main tapping function
* move context to tcp_assembly + fix error check of tcp source packet
* use param instead of gloab flag for ipdefrag
Co-authored-by: Nimrod Gilboa Markevich <nimrod@up9.com>
* Feature/tapper refactor i/internal tapper stats (#384)
* add passive-tapper main tester (#353)
* add passive-tapper main tester
* add errors to go.sum of mizu agent
* disable host mode for tester - to avoid filterAuthorities
* rename main to tester
* build extenssions as part of the tester launch
* add a README to the tester
* solving go.mod and .sum conflicts with addition of go-errors
* trivial warning fixes (#354)
* add passive-tapper main tester
* trivial warning fixes
* add errors to go.sum of mizu agent
* disable host mode for tester - to avoid filterAuthorities
* tcp streams map (#355)
* add passive-tapper main tester
* trivial warning fixes
* add errors to go.sum of mizu agent
* tcp streams map
* disable host mode for tester - to avoid filterAuthorities
* set tcp streams map for tcp stream factory
* change rlog to mizu logger
* errors map (#356)
* add passive-tapper main tester
* trivial warning fixes
* add errors to go.sum of mizu agent
* tcp streams map
* disable host mode for tester - to avoid filterAuthorities
* set tcp streams map for tcp stream factory
* errors map
* change int to uint - errorsmap
* change from int to uint
* Change errorsMap.nErrors to uint.
* change errors map to mizu logger instead of rlog
* init mizu logger in tester + fix errormap declaration
Co-authored-by: Nimrod Gilboa Markevich <nimrod@up9.com>
* move own ips to tcp stream factory (#358)
* add passive-tapper main tester
* trivial warning fixes
* add errors to go.sum of mizu agent
* tcp streams map
* disable host mode for tester - to avoid filterAuthorities
* set tcp streams map for tcp stream factory
* errors map
* move own ips to tcp stream factory
* Feature/tapper refactor i/move own ips to tcp stream factory (#379)
* add passive-tapper main tester
* trivial warning fixes
* add errors to go.sum of mizu agent
* tcp streams map
* disable host mode for tester - to avoid filterAuthorities
* set tcp streams map for tcp stream factory
* errors map
* move own ips to tcp stream factory
* fix ownips compilation issue
* introduce tcp_assembler and tcp_packet_source - the motivation is to split the actual packet sources from the assembler, so we can have a single thread for the assembly which is separated from packet source threads
* make struts private at this point - planning to move some packages to their own package so we can utilize encapsulation
* extract stats functions out of the main tapping function
* move context to tcp_assembly + fix error check of tcp source packet
* use param instead of gloab flag for ipdefrag
* introduce internal tapper stats
* minor fixes for errors map and internal stats
Co-authored-by: Nimrod Gilboa Markevich <nimrod@up9.com>
* Feature/tapper refactor i/diagnose package (#386)
* add passive-tapper main tester (#353)
* add passive-tapper main tester
* add errors to go.sum of mizu agent
* disable host mode for tester - to avoid filterAuthorities
* rename main to tester
* build extenssions as part of the tester launch
* add a README to the tester
* solving go.mod and .sum conflicts with addition of go-errors
* trivial warning fixes (#354)
* add passive-tapper main tester
* trivial warning fixes
* add errors to go.sum of mizu agent
* disable host mode for tester - to avoid filterAuthorities
* tcp streams map (#355)
* add passive-tapper main tester
* trivial warning fixes
* add errors to go.sum of mizu agent
* tcp streams map
* disable host mode for tester - to avoid filterAuthorities
* set tcp streams map for tcp stream factory
* change rlog to mizu logger
* errors map (#356)
* add passive-tapper main tester
* trivial warning fixes
* add errors to go.sum of mizu agent
* tcp streams map
* disable host mode for tester - to avoid filterAuthorities
* set tcp streams map for tcp stream factory
* errors map
* change int to uint - errorsmap
* change from int to uint
* Change errorsMap.nErrors to uint.
* change errors map to mizu logger instead of rlog
* init mizu logger in tester + fix errormap declaration
Co-authored-by: Nimrod Gilboa Markevich <nimrod@up9.com>
* move own ips to tcp stream factory (#358)
* add passive-tapper main tester
* trivial warning fixes
* add errors to go.sum of mizu agent
* tcp streams map
* disable host mode for tester - to avoid filterAuthorities
* set tcp streams map for tcp stream factory
* errors map
* move own ips to tcp stream factory
* Feature/tapper refactor i/move own ips to tcp stream factory (#379)
* add passive-tapper main tester
* trivial warning fixes
* add errors to go.sum of mizu agent
* tcp streams map
* disable host mode for tester - to avoid filterAuthorities
* set tcp streams map for tcp stream factory
* errors map
* move own ips to tcp stream factory
* fix ownips compilation issue
* introduce tcp_assembler and tcp_packet_source - the motivation is to split the actual packet sources from the assembler, so we can have a single thread for the assembly which is separated from packet source threads
* make struts private at this point - planning to move some packages to their own package so we can utilize encapsulation
* extract stats functions out of the main tapping function
* move context to tcp_assembly + fix error check of tcp source packet
* use param instead of gloab flag for ipdefrag
* introduce internal tapper stats
* minor fixes for errors map and internal stats
* move errors map + app stats + internal stats + periodic tasks to diagnose package
* initialize tapper internal stats
Co-authored-by: Nimrod Gilboa Markevich <nimrod@up9.com>
* move tcp packet source to its packet (#387)
* add passive-tapper main tester (#353)
* add passive-tapper main tester
* add errors to go.sum of mizu agent
* disable host mode for tester - to avoid filterAuthorities
* rename main to tester
* build extenssions as part of the tester launch
* add a README to the tester
* solving go.mod and .sum conflicts with addition of go-errors
* trivial warning fixes (#354)
* add passive-tapper main tester
* trivial warning fixes
* add errors to go.sum of mizu agent
* disable host mode for tester - to avoid filterAuthorities
* tcp streams map (#355)
* add passive-tapper main tester
* trivial warning fixes
* add errors to go.sum of mizu agent
* tcp streams map
* disable host mode for tester - to avoid filterAuthorities
* set tcp streams map for tcp stream factory
* change rlog to mizu logger
* errors map (#356)
* add passive-tapper main tester
* trivial warning fixes
* add errors to go.sum of mizu agent
* tcp streams map
* disable host mode for tester - to avoid filterAuthorities
* set tcp streams map for tcp stream factory
* errors map
* change int to uint - errorsmap
* change from int to uint
* Change errorsMap.nErrors to uint.
* change errors map to mizu logger instead of rlog
* init mizu logger in tester + fix errormap declaration
Co-authored-by: Nimrod Gilboa Markevich <nimrod@up9.com>
* move own ips to tcp stream factory (#358)
* add passive-tapper main tester
* trivial warning fixes
* add errors to go.sum of mizu agent
* tcp streams map
* disable host mode for tester - to avoid filterAuthorities
* set tcp streams map for tcp stream factory
* errors map
* move own ips to tcp stream factory
* Feature/tapper refactor i/move own ips to tcp stream factory (#379)
* add passive-tapper main tester
* trivial warning fixes
* add errors to go.sum of mizu agent
* tcp streams map
* disable host mode for tester - to avoid filterAuthorities
* set tcp streams map for tcp stream factory
* errors map
* move own ips to tcp stream factory
* fix ownips compilation issue
* introduce tcp_assembler and tcp_packet_source - the motivation is to split the actual packet sources from the assembler, so we can have a single thread for the assembly which is separated from packet source threads
* make struts private at this point - planning to move some packages to their own package so we can utilize encapsulation
* extract stats functions out of the main tapping function
* move context to tcp_assembly + fix error check of tcp source packet
* use param instead of gloab flag for ipdefrag
* introduce internal tapper stats
* minor fixes for errors map and internal stats
* move errors map + app stats + internal stats + periodic tasks to diagnose package
* move tcp packet source to its packet
* initialize tapper internal stats
Co-authored-by: Nimrod Gilboa Markevich <nimrod@up9.com>
* Fix coding style
* Remove `tap/internal_stats.go`
* make channel between input and assembler blocking - to preserve the same behaviour we have before the refactor
Co-authored-by: Nimrod Gilboa Markevich <nimrod@up9.com>
Co-authored-by: M. Mert Yildiran <mehmet@up9.com>
* Remove `github.com/romana/rlog` dependency completely
* Comment out all the unnecessary logging in the protocol extensions
* Remove commented out all the unnecessary logging lines
* Remove two more lines related to logging
* Add OAS contract monitoring support
* Pass the contract failure reason to UI
* Fix the issues related to contract validation
* Fix rest of the issues in the UI
* Add documentation related to contract monitoring feature
* Fix a typo in the docs
* Unmarshal to `HTTPRequestResponsePair` only if the OAS validation is enabled
* Fix an issue caused by the merge commit
* Slightly change the logic in the `validateOAS` method
Change the `contractText` value to `No Breaches` or `Breach` and make the text `white-space: nowrap`.
* Retrieve and display the failure reason for both request and response
Also display the content of the contract/OAS file in the UI.
* Display the OAS under `CONTRACT` tab with syntax highlighting
Also fix the styling in the entry feed.
* Remove `EnforcePolicyFileDeprecated` constant
* Log the other errors as well
* Get context from caller instead
* Define a type for the contract status and make its values enum-like
* Remove an unnecessary `if` statement
* Validate OAS in the CLI before passing it to Agent
* Get rid of the `github.com/ghodss/yaml` dependency in `loadOAS` by using `LoadFromData`
* Fix an artifact from the merge conflict
* Set the default log level for Agent to `INFO` and change it to `DEBUG` if `dump-logs=true` is provided
* Remove `Trace` method and replace its calls with `Debug`
* Export logging levels from `logger` by defining functions
* Revert "Export logging levels from `logger` by defining functions"
This reverts commit e554e40f4a.
* Run `go mod tidy` on agent
* Define a method named `determineLogLevel`
* Omit the `RULES` tab if the policy rules feature is inactive (WIP)
* Propagate the boolean value `isRulesEnabled` from file read error to UI
* Remove the debug log
* Bring in the files
* Add request-response pair matcher for Redis
* Implement the `Represent` method
* Update `representGeneric` method signature
* Don't export `IntToByteArr`
* Remove unused `newRedisInputStream` method
* Return the errors as string
* Adapt to the latest change in the `develop`
* Run acceptance tests on pull request
* Take `options.DisableRedaction` into account
* Log `defaultTapConfig`
* Pass the `SENSITIVE_DATA_FILTERING_OPTIONS` to tapper daemon set too
* Revert "Run acceptance tests on pull request"
This reverts commit ad79f1418f.
* Fix the OOMKilled error by calling `debug.FreeOSMemory` periodically
* Remove `MAX_NUMBER_OF_GOROUTINES` environment variable
* Change the line
* Increase the default value of `TCP_STREAM_CHANNEL_TIMEOUT_MS` to `10000`
* Fix rules
* Not reay, error on running
* Empty dissector Rules()
* almost working
* Finally, fixed
* undo changes on agent/pkg/utils/har.go
* fix not showing service on rules detail
* Update tap/api/api.go
Co-authored-by: gadotroee <55343099+gadotroee@users.noreply.github.com>
* Update agent/pkg/controllers/entries_controller.go
Co-authored-by: gadotroee <55343099+gadotroee@users.noreply.github.com>
* Update agent/pkg/controllers/entries_controller.go
Co-authored-by: gadotroee <55343099+gadotroee@users.noreply.github.com>
* unwrap Data
* Fix bug off using more than one latency rule that always get the first.
* fix json type, decoding base64 before unmarshal
* Run `go mod tidy` on `cli/go.sum`
* Fix the linting issues
* Remove a `FIXME` comment
* Remove a CSS rule
* Adapt `ruleNumberText` CSS class to the design language of the UI
* Fix an issue in the UI related to `rule.Latency` slipping out
* Removed unecessary codes.
Co-authored-by: gadotroee <55343099+gadotroee@users.noreply.github.com>
Co-authored-by: M. Mert Yildiran <mehmet@up9.com>
* Move `stats_tracker.go` into the extension API and increment `MatchedPairs` from inside the `Emit` method
* Replace multiple `sync.Mutex`(es) with low-level atomic memory primitives
* Make `ScrollableFeed` virtualized by replacing `react-scrollable-feed` with `react-scrollable-feed-virtualized`
* fix get new entries button
* Fix the not populated `Protocol` struct in case of `GetEntries` endpoint is called
Co-authored-by: Liraz Yehezkel <lirazy@up9.com>
* Close the hanging TCP message channels after a dynamically aligned timeout (base `10000` milliseconds)
* Bring back `source.Lazy`
* Add a one more `sync.Map.Delete` call
* Improve the formula by taking base Goroutine count into account
* Reduce duplication
* Include the dropped TCP streams count into the stats tracker and print a debug log whenever it happens
* Add `superIdentifier` field to `tcpStream` to check if it has identified
Also stop the other protocol dissectors if a TCP stream identified by a protocol.
* Take one step forward in fixing the channel closing issue (WIP)
Add `sync.Mutex` to `tcpReader` and make the loops reference based.
* Fix the channel closing issue
* Improve the accuracy of the formula, log better and multiply `baseStreamChannelTimeoutMs` by 100
* Remove `fmt.Printf`
* Replace `runtime.Gosched()` with `time.Sleep(1 * time.Millisecond)`
* Close the channels of other protocols in case of an identification
* Simplify the logic
* Replace the formula with hard timeout 5000 milliseconds and 4000 maximum number of Goroutines
* Permanently resolve the memory exhaustion in AMQP
Introduce;
- `MEMORY_PROFILING_DUMP_PATH`
- `MEMORY_PROFILING_TIME_INTERVAL`
environment variables and make `startMemoryProfiler` method more parameterized.
* Fix a leak in HTTP
* Revert "Fix a leak in HTTP"
This reverts commit 9d46820ff3.
* Set maximum AMQP message size to 16MB
* Set `GOGC` to 12800
* Remove some commented out lines and an unnecessary `else if`
* Separate HTTP related code into `extensions/http` as a Go plugin
* Move `extensions` folder into `tap` folder
* Move HTTP files into `tap/extensions/lib` for now
* Replace `orcaman/concurrent-map` with `sync.Map`
* Remove `grpc_assembler.go`
* Remove `github.com/up9inc/mizu/tap/extensions/http/lib`
* Add a build script to automatically build extensions from a known path and load them
* Start to define the extension API
* Implement the `run()` function for the TCP stream
* Add support of defining multiple ports to the extension API
* Set the extension name inside the extension
* Declare the `Dissect` function in the extension API
* Dissect HTTP request from inside the HTTP extension
* Make the distinction of outbound and inbound ports
* Dissect HTTP response from inside the HTTP extension
* Bring back the HTTP request-response pair matcher
* Return a `*api.RequestResponsePair` from the dissection
* Bring back the gRPC-HTTP/2 parser
* Fix the issues in `handleHTTP1ClientStream` and `handleHTTP1ServerStream`
* Call a function pointer to emit dissected data back to the `tap` package
* roee changes -
trying to fix agent to work with the "api" object) - ***still not working***
* small mistake in the conflicts
* Fix the issues that are introduced by the merge conflict
* Add `Emitter` interface to the API and send `OutputChannelItem`(s) to `OutputChannel`
* Fix the `HTTP1` handlers
* Set `ConnectionInfo` in HTTP handlers
* Fix the `Dockerfile` to build the extensions
* remove some unwanted code
* no message
* Re-enable `getStreamProps` function
* Migrate back from `gopacket/tcpassembly` to `gopacket/reassembly`
* Introduce `HTTPPayload` struct and `HTTPPayloader` interface to `MarshalJSON()` all the data structures that are returned by the HTTP protocol
* Read `socketHarOutChannel` instead of `filteredHarChannel`
* Connect `OutputChannelItem` to the last WebSocket means that finally the web UI started to work again
* Add `.env.example` to React app
* Marshal and unmarshal `*http.Request`, `*http.Response` pairs
* Move `loadExtensions` into `main.go` and map extensions into `extensionsMap`
* Add `Summarize()` method to the `Dissector` interface
* Add `Analyze` method to the `Dissector` interface and `MizuEntry` to the extension API
* Add `Protocol` struct and make it effect the UI
* Refactor `BaseEntryDetails` struct and display the source and destination ports in the UI
* Display the protocol name inside the details layout
* Add `Represent` method to the `Dissector` interface and manipulate the UI through this method
* Make the protocol color affect the details layout color and write protocol abbreviation vertically
* Remove everything HTTP related from the `tap` package and make the extension system fully functional
* Fix the TypeScript warnings
* Bring in the files related AMQP into `amqp` directory
* Add `--nodefrag` flag to the tapper and bring in the main AMQP code
* Implement the AMQP `BasicPublish` and fix some issues in the UI when the response payload is missing
* Implement `representBasicPublish` method
* Fix several minor issues
* Implement the AMQP `BasicDeliver`
* Implement the AMQP `QueueDeclare`
* Implement the AMQP `ExchangeDeclare`
* Implement the AMQP `ConnectionStart`
* Implement the AMQP `ConnectionClose`
* Implement the AMQP `QueueBind`
* Implement the AMQP `BasicConsume`
* Fix an issue in `ConnectionStart`
* Fix a linter error
* Bring in the files related Kafka into `kafka` directory
* Fix the build errors in Kafka Go files
* Implement `Dissect` method of Kafka and adapt request-response pair matcher to asynchronous client-server stream
* Do the "Is reversed?" checked inside `getStreamProps` and fix an issue in Kafka `Dissect` method
* Implement `Analyze`, `Summarize` methods of Kafka
* Implement the representations for Kafka `Metadata`, `RequestHeader` and `ResponseHeader`
* Refactor the AMQP and Kafka implementations to create the summary string only inside the `Analyze` method
* Implement the representations for Kafka `ApiVersions`
* Implement the representations for Kafka `Produce`
* Implement the representations for Kafka `Fetch`
* Implement the representations for Kafka `ListOffsets`, `CreateTopics` and `DeleteTopics`
* Fix the encoding of AMQP `BasicPublish` and `BasicDeliver` body
* Remove the unnecessary logging
* Remove more logging
* Introduce `Version` field to `Protocol` struct for dynamically switching the HTTP protocol to HTTP/2
* Fix the issues in analysis and representation of HTTP/2 (gRPC) protocol
* Fix the issues in summary section of details layout for HTTP/2 (gRPC) protocol
* Fix the read errors that freezes the sniffer in HTTP and Kafka
* Fix the issues in HTTP POST data
* Fix one more issue in HTTP POST data
* Fix an infinite loop in Kafka
* Fix another freezing issue in Kafka
* Revert "UI Infra - Support multiple entry types + refactoring (#211)"
This reverts commit f74a52d4dc.
* Fix more issues that are introduced by the merge
* Fix the status code in the summary section
* adding the cleaner again (why we removed it?).
add TODO: on the extension loop .
* fix dockerfile (remove deleting .env file) - it is found in dockerignore and fails to build if the file not exists
* fix GetEntrties ("/entries" endpoint) - working with "tapApi.BaseEntryDetail" (moved from shared)
* Fix an issue in the UI summary section
* Refactor the protocol payload structs
* Fix a log message in the passive tapper
* Adapt `APP_PORTS` environment variable to the new extension system and change its format to `APP_PORTS='{"http": ["8001"]}' `
* Revert "fix dockerfile (remove deleting .env file) - it is found in dockerignore and fails to build if the file not exists"
This reverts commit 4f514ae1f4.
* Bring in the necessary changes from f74a52d4dc
* Open the API server URL in the web browser as soon as Mizu is ready
* Make the TCP reader consists of a single Go routine (instead of two) and try to dissect in both client and server mode by rewinding
* Swap `TcpID` without overwriting it
* Sort extension by priority
* Try to dissect with looping through all the extensions
* fix getStreamProps function.
(it should be passed from CLI as it was before).
* Turn TCP reader back into two Goroutines (client and server)
* typo
* Learn `isClient` from the TCP stream
* Set `viewer` style `overflow: "auto"`
* Fix the memory leaks in AMQP and Kafka dissectors
* Revert some of the changes in be7c65eb6d
* Remove `allExtensionPorts` since it's no longer needed
* Remove `APP_PORTS` since it's no longer needed
* Fix all of the minor issues in the React code
* Check Kafka header size and fail-fast
* Break the dissectors loop upon a successful dissection
* Don't break the dissector loop. Protocols might collide
* Improve the HTTP request-response counter (still not perfect)
* Make the HTTP request-response counter perfect
* Revert "Revert some of the changes in be7c65eb6d3fb657a059707da3ca559937e59739"
This reverts commit 08e7d786d8.
* Bring back `filterItems` and `isHealthCheckByUserAgent` functions
* Remove some development artifacts
* remove unused and commented lines that are not relevant
* Fix the performance in TCP stream factory. Make it create two `tcpReader`(s) per extension
* Change a log to debug
* Make `*api.CounterPair` a field of `tcpReader`
* Set `isTapTarget` to always `true` again since `filterAuthorities` implementation has problems
* Remove a variable that's only used for logging even though not introduced by this branch
* Bring back the `NumberOfRules` field of `ApplicableRules` struct
* Remove the unused `NewEntry` function
* Move `k8sResolver == nil` check to a more appropriate place
* default healthChecksUserAgentHeaders should be empty array (like the default config value)
* remove spam console.log
* Rules button cause app to crash (access the service via incorrect property)
* Ignore all .env* files in docker build.
* Better caching in dockerfile: only copy go.mod before go mod download.
* Check for errors while loading an extension
* Add a comment about why `Protocol` is not a pointer
* Bring back the call to `deleteOlderThan`
* Remove the `nil` check
* Reduce the maximum allowed AMQP message from 128MB to 1MB
* Fix an error that only occurs when a Kafka broker is initiating
* Revert the change in b2abd7b990
* Fix the service name resolution in all protocols
* Remove the `anydirection` flag and fix the issue in `filterAuthorities`
* Pass `sync.Map` by reference to `deleteOlderThan` method
* Fix the packet capture issue in standalone mode that's introduced by the removal of `anydirection`
* Temporarily resolve the memory exhaustion in AMQP
* Fix a nil pointer dereference error
* Fix the CLI build error
* Fix a memory leak that's identified by `pprof`
Co-authored-by: Roee Gadot <roee.gadot@up9.com>
Co-authored-by: Nimrod Gilboa Markevich <nimrod@up9.com>
* Implemented validation rules, based on: https://up9.atlassian.net/browse/TRA-3349
* Color on Entry based on rules
* Background red/green based on rules
* Change flag --validation-rules to --test-rules
* rules tab UI updated
* rules tab font and background-color is changed for objects
* Merged with develop
* Fixed compilation issues.
* Renamed fullEntry -> harEntry where appropriate.
* Change green/red logic
* Update models.go
* Fix latency bug and alignment
* Merge Conflicts fix
* Working after merge
* Working on Nimrod comments
* Resolving conflicts
* Resolving conflicts
* Resolving conflicts
* Nimrod Comments pt.3
* Log Error on configmap creation if the user doesn't have permission.
* Checking configmap permission to ignore --test-rules
* Revert time for mizu to get ready
* Nimrod comments pt 4 && merge develop pt3
* Nimrod comments pt 4 && merge develop pt3
* Const rulePolicyPath and filename
Co-authored-by: Neim <elezin9@gmail.com>
Co-authored-by: nimrod-up9 <nimrod@up9.com>
* Update passive_tapper.go and tls_utils.go
* Update go.mod, go.sum, and 18 more files...
* go fmt
* Update http_reader.go, passive_tapper.go, and 3 more files...
* Update status_controller.go and status_provider.go
Co-authored-by: RamiBerm <rami.berman@up9.com>
* Renamed aggregator -> apiServer.
* Format errors with container names.
* Renamed collector -> apiServer.
* Rephrased help messages.
* Moved api -> agent.
* Continue renameing api -> agent in Makefile and Dockerfiles.
* Tap outgoing: If --anydirection flag is passed with HOST_MODE, tap by source IP.
* Moved ConnectionInfo from http_matcher to http_reader.
* Generalized shouldTap in stream factory to get more properties.
* tap reports IsOutgoing property of tcp connection.
* gofmt.
* CLI instructs tapper to tap outgoing connections.
* API saves IsOutgoing to DB and passes it to UI.
* Add a visual marker in the HAR list for outgoing messages.
* Fixed: Swapped src and dst.
* Resolver keeps a list of all ClusterIP services.
* Do not save HARs with destination ClusterIP services.
* CLI accepts flag that controls traffic direction.
* Indicate incoming/outgoing with icon instead of with border color.
* Fixed: Didn't filter messages to services in aggregator.
* Clearer syntax around the direction icon. Added title text.
* Fixed width around direction icon.
* Less repetition.
* Removed TODO.
* Renamed incoming -> ingoing.
* More verbose title text to image.
* Switched routine order for readability.
* Instead of saving incomplete temp har files in a temp dir, save them in the output dir with a *.har.tmp suffix.
* API only loads har from *.har files (by extension).
* Use log in tap package instead of fmt.
* Moved api/pkg/tap to root.
* Added go.mod and go.sum for tap.
* Added replace for shared.
* api uses tap module instead of tap package.
* Removed dependency of tap in shared by moving env var out of tap.
* Fixed compilation bugs.
* Fixed: Forgot to export struct field HostMode.
* Removed unused flag.
* Close har output channel when done.
* Moved websocket out of mizu and into passive-tapper.
* Send connection details over har output channel.
* Fixed compilation errors.
* Removed unused info from request response cache.
* Renamed connection -> connectionID.
* Fixed rename bug.
* Export setters and getters for filter ips and ports.
* Added tap dependency to Dockerfile.
* Uncomment error messages.
* Renamed `filterIpAddresses` -> `filterAuthorities`.
* Renamed ConnectionID -> ConnectionInfo.
* Fixed: Missed one replace.
