Commit Graph

79 Commits

Author SHA1 Message Date
Nimrod Gilboa Markevich
267a9343d9 Add --profiler flag to cli 2022-05-12 16:55:02 +03:00
M. Mert Yıldıran
e1ad302c29
Make logger a separate module such that don't depend on shared module as a whole for logging (#1047)
* Make `logger` a separate module such that don't depend on `shared` module as a whole for logging

* Update `Dockerfile`
2022-04-27 22:26:27 +03:00
M. Mert Yıldıran
80ebd7199d
Fix the CLI build error for Windows target (#1031) 2022-04-24 09:45:15 +03:00
Igor Gov
a3de34f544
Turn OAS feature ON by default (#927)
* Turn OAS feature ON by default
2022-03-24 20:40:16 +02:00
RoyUP9
cff5987ed4
Added check pre install (#905) 2022-03-21 17:19:04 +02:00
RoyUP9
482e5c8b69
Added check pull image flag (#899)
Co-authored-by: gadotroee <55343099+gadotroee@users.noreply.github.com>
2022-03-21 15:24:03 +02:00
RoyUP9
edbe4ab00b
Removed all code related to install (#892) 2022-03-15 16:41:42 +02:00
RoyUP9
f5fa9ff270
Added mizu install template (#884) 2022-03-09 17:52:55 +02:00
gadotroee
2c1aa9022b
Add option to specify k8s context (#878)
Co-authored-by: M. Mert Yildiran <mehmet@up9.com>
2022-03-05 19:15:33 +02:00
M. Mert Yıldıran
3b063c3bb5
Add the ability to set the insertion filter into CLI (#843)
* Add the ability to set the insertion filter into CLI

* Resolve the conflict

* Upgrade Basenine to `v0.5.0` and check the insertion filter against syntax errors inside CLI

* Upgrade Basenine to `v0.5.1`

* Update the sums
2022-03-02 12:51:32 +03:00
RoyUP9
30a85a4b92
Added tap pre check (#848) 2022-02-24 11:48:42 +02:00
David Levanon
87ef469e25
Add tls tapper (#683)
* initial tls tapper commit

* add tls flag to mizu cli

* support ssl_read_ex/ssl_write_ex

* use hostproc to find libssl

* auto discover tls processes

* support libssl1.0

* recompile ebpf with old clang/llvm

* Update tap/passive_tapper.go

Co-authored-by: M. Mert Yıldıran <mehmet@up9.com>

* Update tap/tlstapper/tls_poller.go

Co-authored-by: M. Mert Yıldıran <mehmet@up9.com>

* Update tap/tlstapper/tls_poller.go

Co-authored-by: M. Mert Yıldıran <mehmet@up9.com>

* Update tap/tlstapper/tls_poller.go

Co-authored-by: M. Mert Yıldıran <mehmet@up9.com>

* Update tap/tlstapper/tls_poller.go

Co-authored-by: M. Mert Yıldıran <mehmet@up9.com>

* Update tap/tlstapper/tls_poller.go

Co-authored-by: M. Mert Yıldıran <mehmet@up9.com>

* Update tap/tlstapper/tls_poller.go

Co-authored-by: M. Mert Yıldıran <mehmet@up9.com>

* Update tap/tlstapper/tls_poller.go

Co-authored-by: M. Mert Yıldıran <mehmet@up9.com>

* upgrade ebpf go lib

* handling big tls messages

* fixing max buffer size in ebpf

* remove unused import

* fix linter issues

* minor pr fixes

* compile with old clang

* fix cgroup file format

* pr fixes + cgroup extract enhance

* fix linter

* adding indirect ebpf dep to agent go.mod

* adding ebpf docker builder

* minor pr fixes

* add req resp matcher to dissect

* rename ssl hooks to ssl hooks structs

* move to alpine, use local copy of mizu instead of git, add readme

* use global req resp mather for tls

Co-authored-by: M. Mert Yıldıran <mehmet@up9.com>
Co-authored-by: gadotroee <55343099+gadotroee@users.noreply.github.com>
2022-02-16 15:34:51 +02:00
Alex Haiut
dfe63f2318
enabled service-map by default (#816) 2022-02-15 22:41:57 +02:00
Igor Gov
ad6fb844aa
Moving from Semver to major.patch versioning (#755)
* Moving from Semver to major.patch versions

* version check should work with new version forma

Co-authored-by: Igor Gov <igor.govorov1@gmail.com>
2022-02-06 15:01:36 +02:00
Igor Gov
0f6c56986f
Technical depth: Adding Go linter to CI (#734) 2022-02-01 08:47:26 +02:00
Alex Haiut
daf6b3db06
unhide svcmap + CHANGELOG (#724) 2022-01-31 17:20:36 +02:00
Igor Gov
83b657472b
Revert "Fix: show agent-image in config after generation (#717)" (#728) 2022-01-31 13:20:36 +02:00
Igor Gov
70cef9dc4b
Fix: show agent-image in config after generation (#717) 2022-01-30 09:25:56 +02:00
Igor Gov
0f3dd66d2d
Experimental feature: elastic exporter (#713) 2022-01-30 09:22:13 +02:00
M. Mert Yıldıran
7fa1a191a6
TRA-4235 Move Basenine binary into the same agent image but run it as a separate container (#702)
* TRA-4235 Revert "Move Basenine binary into a separate container"

* Deploy the same agent image as a separate container for Basenine

Co-authored-by: Igor Gov <iggvrv@gmail.com>
2022-01-27 11:40:26 +03:00
M. Mert Yıldıran
a9a61edd50
Add ARM64 and cross-compilation support to the agent image (#659)
* modified Dockerfile to work for both amd64 (Intel) and arm64 (M1)

* added changelog

* Update `Dockerfile` to have `ARCH` build argument

* Remove `docs/CHANGES.md`

* Upgrade the Basenine version from `v0.3.0` to `v0.4.6`

* Update `publish.yml` to have `ARCH` build argument

* Switch `BasenineImageRepo` to Docker Hub

* Have separate build arguments for `ARCH` and `GOARCH`

* Upgrade the Basenine version from `v0.4.6` to `v0.4.10`

* Oops forgot to update the 10th duplicated shell script

* Fix the oopsie and reduce duplications

* Fix `Dockerfile`

* Fix the incompatibility issue between Go plugins and gold linker in Alpine inside `Dockerfile`

* Fix `asm: xxhash_amd64.s:120: when dynamic linking, R15 is clobbered by a global variable access` error

* Update `Dockerfile` to have cross-compilation on an AMD64 machine

Also revert changes in the shell scripts

* Delete `debug.Dockerfile`

* Create a custom base (`debian:buster-slim` based) image for the shipped image

* Replace `mertyildiran/debian-pcap` with `up9inc/debian-pcap`

* Upgrade Basenine version to `v0.4.12`

* Use `debian:stable-slim` as the base

* Fix the indentation in the `Dockerfile`

* Update `publish.yml`

* Enable `publish.yml` for `feature/multiarch_build` branch

* Tag correctly and set `ARCH` Docker argument

* Remove the lines that are forgotten to be removed from the shell scripts

* Add `MizuAgentImageRepo` constant and use it as default `AgentImage` value

* Bring back `Set up Cloud SDK` step to `Build the CLI and publish` job

* Build ARM64 CLI for Linux as well

* Revert "Enable `publish.yml` for `feature/multiarch_build` branch"

This reverts commit d30be4c1f0.

* Revert Go 1.17 upgrade

* Remove `build_extensions_debug.sh` as well

* Make the `Dockerfile` to compile the agent statically

* Statically link the protocol extensions

* Fix `Dockerfile`

* Bring back `-s -w` flags

* Verify the signatures of the downloads in `dockcross/linux-arm64-musl`

* Revert modifications in some shell scripts

* Make the `BUILDARCH` and `TARGETARCH` separation in the `Dockerfile`

* Separate cross-compilation builder image into a separate repo named `up9inc/linux-arm64-musl-go-libpcap`

* Fill the shell script and specify the tag for `dockcross/linux-arm64-musl`

* Remove the unnecessary dependencies from `builder-native-base`

* Improve the comments in the `Dockerfile`

* Upgrade Basenine version to `v0.4.13`

* Fix `Dockerfile`

* Revert "Revert "Enable `publish.yml` for `feature/multiarch_build` branch""

This reverts commit 303e466bdc.

* Revert "Revert "Revert "Enable `publish.yml` for `feature/multiarch_build` branch"""

This reverts commit 0fe252bbdb.

* Remove `push-docker-debug` from the `Makefile`

* Rename `publish.yml` to `release.yml`

Co-authored-by: Alex Haiut <alex@up9.com>
2022-01-25 21:24:50 +03:00
RamiBerm
83c9194703
TRA-4202 role management (#688)
* WIP

* wip

* Update keto.yml, socket_routes.go, and 12 more files...

* fixes and docs

* Update api.js

* Update auth.go and api.js

* Update user_role_provider.go

* Update config_routes.go and api.js

* Update consts.go
2022-01-25 14:25:24 +02:00
gadotroee
6bab381280
Make kratos image configurable (#670) 2022-01-20 13:48:02 +02:00
gadotroee
27dee4e09b
TRA-4193 - Try port forward if proxy is not available (#662) 2022-01-20 11:33:00 +02:00
RoyUP9
ae1bcf4c0c
Added api server timeout env for install and tap (#647) 2022-01-16 11:48:22 +02:00
Igor Gov
0e5611b7e9
FE setting experimental feature flags (#633) 2022-01-12 18:05:12 +02:00
Igor Gov
bfbbc27e62
Adding experimental feature flags (#627) 2022-01-12 09:33:41 +02:00
Nimrod Gilboa Markevich
5a044875d3
Rename Istio to service mesh (#605)
- Rename --istio flag to the more general --service-mesh
- Rename internal variables, consts and structures to reflect this conceptual change
- Update the docs accordingly
2022-01-09 13:21:14 +02:00
RoyUP9
de046c65b0
Mizu install mode (#566) 2021-12-28 15:44:37 +02:00
RoyUP9
1e1b5f0c0f
move mizu agent config from config (#563) 2021-12-28 09:59:21 +02:00
RoyUP9
15895d5947
Changed sync tappers to start only when sync tappers config is true (#562) 2021-12-27 17:19:00 +02:00
David Levanon
6caa94f08f
Add support to auto discover envoy processes (#459)
* discover envoy pids using cluster ips

* add istio flag to cli + rename mtls flag to istio

* add istio.md to docs

* Fixing typos

* Fix minor typos and grammer in docs

Co-authored-by: Nimrod Gilboa Markevich <nimrod@up9.com>
2021-11-21 15:45:07 +02:00
RamiBerm
51e0dd8ba9
TRA-3903 add flag to disable pvc creation for daemon mode (#474)
* Update tapRunner.go and tapConfig.go

* Update tapConfig.go

* Revert "Update tapConfig.go"

This reverts commit 5c7c02c4ab.
2021-11-16 17:11:47 +02:00
RoyUP9
14a5fe11e7
changed logger debug mode to log level (#456) 2021-11-14 12:21:48 +02:00
RoyUP9
ea0b3fb34e
moved headless to root config, use headless in view (#450) 2021-11-11 12:11:02 +02:00
M. Mert Yıldıran
1ee8fb6292
Fix the acceptance tests after the merger of #279 (#443)
* Enable acceptance tests

* Fix the acceptance tests

* Move `--headless` from `getDefaultCommandArgs` to `getDefaultTapCommandArgs`

* Fix rest of the failing acceptance tests

* Revert "Enable acceptance tests"

This reverts commit 3f919e865a.

* Revert "Revert "Enable acceptance tests""

This reverts commit c0bfe54b70.

* Ignore `--headless` in `mizu view`

* Make all non-informative things informative

* Remove `github.com/stretchr/testify` dependency from the acceptance tests

* Move the helper methods `waitTimeout` and `checkDBHasEntries` from `tap_test.go` to `testsUtils.go`

* Split `checkDBHasEntries` method into `getDBEntries` and `assertEntriesAtLeast` methods

* Revert "Revert "Revert "Enable acceptance tests"""

This reverts commit c13342671c.

* Revert "Revert "Revert "Revert "Enable acceptance tests""""

This reverts commit 0f8c436926.

* Make `getDBEntries` and `checkEntriesAtLeast` methods return errors instead

* Revert "Revert "Revert "Revert "Revert "Enable acceptance tests"""""

This reverts commit 643fdde009.
2021-11-10 18:14:04 +03:00
M. Mert Yıldıran
d2fe3f6620
Migrate from SQLite to Basenine and introduce a new filtering syntax (#279)
* Fix the OOMKilled error by calling `debug.FreeOSMemory` periodically

* Remove `MAX_NUMBER_OF_GOROUTINES` environment variable

* Change the line

* Increase the default value of `TCP_STREAM_CHANNEL_TIMEOUT_MS` to `10000`

* Write the client and integrate to the new real-time database

* Refactor the WebSocket implementaiton for `/ws`

* Adapt the UI to the new filtering system

* Fix the rest of the issues in the UI

* Increase the buffer of the scanner

* Implement accessing single records

* Increase the buffer of another scanner

* Populate `Request` and `Response` fields of `MizuEntry`

* Add syntax highlighting for the query

* Add database to `Dockerfile`

* Fix some issues

* Update the `realtime_dbms` Git module commit hash

* Upgrade Gin version and print the query string

* Revert "Upgrade Gin version and print the query string"

This reverts commit aa09f904ee.

* Use WebSocket's itself to query instead of the query string

* Fix some errors related to conversion to HAR

* Fix the issues caused by the latest merge

* Fix the build error

* Fix PR validation GitHub workflow

* Replace the git submodule with latest Basenine version `0.1.0`

Remove `realtime_client.go` and use the official client library `github.com/up9inc/basenine/client/go` instead.

* Move Basenine host and port constants to `shared` module

* Reliably execute and wait for Basenine to become available

* Upgrade Basenine version

* Properly close WebSocket and data channel

* Fix the issues caused by the recent merge commit

* Clean up the TypeScript code

* Update `.gitignore`

* Limit the database size

* Add `Macros` method signature to `Dissector` interface and set the macros provided by the protocol extensions

* Run `go mod tidy` on `agent`

* Upgrade `github.com/up9inc/basenine/client/go` version

* Implement a mechanism to update the query using click events in the UI and use it for protocol macros

* Update the query on click to timestamps

* Fix some issues in the WebSocket and channel handling

* Update the query on clicks to status code

* Update the query on clicks to method, path and service

* Update the query on clicks to is outgoing, source and destination ports

* Add an API endpoint to validate the query against syntax errors

* Move the query background color state into `TrafficPage`

* Fix the logic in `setQuery`

* Display a toast message in case of a syntax error in the query

* Remove a call to `fmt.Printf`

* Upgrade Basenine version to `0.1.3`

* Fix an issue related to getting `MAX_ENTRIES_DB_BYTES` environment variable

* Have the `path` key in request details, in HTTP

* Rearrange the HTTP headers for the querying

* Do the same thing for `cookies` and `queryString`

* Update the query on click to table elements

Add the selectors for `TABLE` type representations in HTTP extension.

* Update the query on click to `bodySize` and `elapsedTime` in `EntryTitle`

* Add the selectors for `TABLE` type representations in AMQP extension

* Add the selectors for `TABLE` type representations in Kafka extension

* Add the selectors for `TABLE` type representations in Redis extension

* Define a struct in `tap/api.go` for the section representation data

* Add the selectors for `BODY` type representations

* Add `request.path` to the HTTP request details

* Change the summary string's field name from `path` to `summary`

* Introduce `queryable` CSS class for queryable UI elements and underline them on hover

* Instead of `N requests` at the bottom, make it `Displaying N results (queried X/Y)` and live update the values

Upgrade Basenine version to `0.2.0`.

* Verify the sha256sum of Basenine executable inside `Dockerfile`

* Pass the start time to web UI through WebSocket and always show the `EntriesList` footer

* Pipe the `stderr` of Basenine as well

* Fix the layout issues related to `CodeEditor` in the UI

* Use the correct `shasum` command in `Dockerfile`

* Upgrade Basenine version to `0.2.1`

* Limit the height of `CodeEditor` container

* Remove `Paused` enum `ConnectionStatus` in UI

* Fix the issue caused by the recent merge

* Add the filtering guide (cheatsheet)

* Update open cheatsheet button's title

* Update cheatsheet content

* Remove the old SQLite code, adapt the `--analyze` related code to Basenine

* Change the method signature of `NewEntry`

* Change the method signature of `Represent`

* Introduce `HTTPPair` field in `MizuEntry` specific to HTTP

* Remove `Entry`, `EntryId` and `EstimatedSizeBytes` fields from `MizuEntry`

Also remove the `getEstimatedEntrySizeBytes` method.

* Remove `gorm.io/gorm` dependency

* Remove unused `sensitiveDataFiltering` folder

* Increase the left margin of open cheatsheet button

* Add `overflow: auto` to the cheatsheet `Modal`

* Fix `GetEntry` method

* Fix the macro for gRPC

* Fix an interface conversion in case of AMQP

* Fix two more interface conversion errors in AMQP

* Make the `syncEntriesImpl` method blocking

* Fix a grammar mistake in the cheatsheet

* Adapt to the changes in the recent merge commit

* Improve the cheatsheet text

* Always display the timestamp in `en-US`

* Upgrade Basenine version to `0.2.2`

* Fix the order of closing Basenine connections and channels

* Don't close the Basenine channels at all

* Upgrade Basenine version to `0.2.3`

* Set the initial filter to `rlimit(100)`

* Make Basenine persistent

* Upgrade Basenine version to `0.2.4`

* Update `debug.Dockerfile`

* Fix a failing test

* Upgrade Basenine version to `0.2.5`

* Revert "Do not show play icon when disconnected (#428)"

This reverts commit 8af2e562f8.

* Upgrade Basenine version to `0.2.6`

* Make all non-informative things informative

* Make `100` a constant

* Use `===` in JavaScript no matter what

* Remove a forgotten `console.log`

* Add a comment and update the `query` in `syncEntriesImpl`

* Don't call `panic` in `GetEntry`

* Replace `panic` calls in `startBasenineServer` with `logger.Log.Panicf`

* Remove unnecessary `\n` characters in the logs
2021-11-09 19:54:48 +03:00
RamiBerm
a3ec5d147e
TRA-3842 daemon mode (#427)
* Update config.go, tapConfig.go, and models.go

* WIP

* Update go.sum

* Update tapRunner.go

* Update tap.go

* WIP

* WIP

* Update Dockerfile, main.go, and 2 more files...

* WIP

* Update utils.go, tapClusterResourceManagement.go, and utils.go

* Merge branch 'develop'

* Update metadata_controller.go, utils.go, and 2 more files...

* Update main.go, utils.go, and tapRunner.go

* Update tapRunner.go

* Update config.go, config.go, and models.go

* Update main.go, main.go, and stats_provider_test.go

* Update provider.go

* bug fixes

* Update main.go, metadata_controller.go, and 13 more files...

* Update metadata_controller.go, status_controller.go, and 4 more files...

* Update main.go, config.go, and 3 more files...

* Update tapRunner.go

* Update config.go, stats_provider_test.go, and consts.go
2021-11-04 11:46:45 +02:00
RamiBerm
655626bc42
TRA-3868 move tapped pod watch and tapper updating to shared (#416)
* WIP

* WIP

* WIP

* WIP

* WIP

* Update tapRunner.go and k8sTapManager.go

* Update cleanRunner.go, common.go, and 8 more files...

* Update common.go, tapConfig.go, and 2 more files...

* Update config.go, config.go, and 5 more files...

* Update tapRunner.go, config.go, and 7 more files...

* Update cleanRunner.go, logs.go, and 2 more files...

* Update k8sTapManager.go, provider.go, and watch.go

* Update go.sum, go.mod, and go.sum

* Update go.mod and go.sum

* Update go.mod, go.sum, and 2 more files...

* Revert "Update go.mod, go.sum, and 2 more files..."

This reverts commit 8140311349.

* Update funcWrappers.go, tapRunner.go, and 4 more files...

* Update main.go, tapRunner.go, and mizuTapperSyncer.go
2021-11-01 14:12:32 +02:00
RamiBerm
35dbd5fde2
TRA-3860 create main configmap for agent and tappers (#410)
* WIP

* Update options.go and serializable_regexp.go

* Update go.sum, go.sum, and 4 more files...

* Update go.sum, go.sum, and 4 more files...

* Update config.go and serializable_regexp.go

* Update config.go, config.json, and test.go

* Update tapRunner.go and provider.go

* Update provider.go

* Update tapRunner.go and provider.go

* Update config.json and test.go

* Update contract_validation.go, config.go, and 2 more files...

* Update main.go

* Update rulesHTTP.go

* Update config.go, size_enforcer.go, and 5 more files...

* Update config.go and config.go

Co-authored-by: Rami Berman <rami.berman@up9.com>
2021-10-31 15:29:05 +02:00
RoyUP9
f069fdaae1
changed ask confirmation to be asked only once (#409) 2021-10-28 14:55:33 +03:00
gadotroee
d478a7ec43
TRA-3867 proxy host from config (#411) 2021-10-28 14:53:28 +03:00
RoyUP9
779257b864
added alert and error when there is a proxy before k8b server (#402) 2021-10-26 16:57:56 +03:00
M. Mert Yıldıran
145e7cda01
Add OAS contract monitoring support (#325)
* Add OAS contract monitoring support

* Pass the contract failure reason to UI

* Fix the issues related to contract validation

* Fix rest of the issues in the UI

* Add documentation related to contract monitoring feature

* Fix a typo in the docs

* Unmarshal to `HTTPRequestResponsePair` only if the OAS validation is enabled

* Fix an issue caused by the merge commit

* Slightly change the logic in the `validateOAS` method

Change the `contractText` value to `No Breaches` or `Breach` and make the text `white-space: nowrap`.

* Retrieve and display the failure reason for both request and response

Also display the content of the contract/OAS file in the UI.

* Display the OAS under `CONTRACT` tab with syntax highlighting

Also fix the styling in the entry feed.

* Remove `EnforcePolicyFileDeprecated` constant

* Log the other errors as well

* Get context from caller instead

* Define a type for the contract status and make its values enum-like

* Remove an unnecessary `if` statement

* Validate OAS in the CLI before passing it to Agent

* Get rid of the `github.com/ghodss/yaml` dependency in `loadOAS` by using `LoadFromData`

* Fix an artifact from the merge conflict
2021-10-19 14:24:22 +03:00
M. Mert Yıldıran
b30b62ef77
Move cli/logger to shared, and refactor all its usages in cli (#349)
* Move `cli/logger` to `shared`, and refactor all its usages in `cli`

* Remove indirect for `op/go-logging` in `shared`
2021-10-14 10:18:01 +03:00
RoyUP9
04c0f8cbcd
tap to workspace (#315) 2021-10-11 15:42:41 +03:00
RoyUP9
9d378ed75b
refactor login (#339) 2021-10-11 11:31:12 +03:00
RoyUP9
8b8c4609ce
renamed upload entries to sync entries (#330) 2021-10-07 18:33:14 +03:00
RoyUP9
14b616a856
Connecting Mizu to the application (#317) 2021-10-07 17:28:28 +03:00
RoyUP9
f1a2ee7fb4
removed enforce policy file deprecated flag (#328) 2021-10-07 11:08:48 +03:00