github/kubeshark
1
0
Fork 0
mirror of https://github.com/kubeshark/kubeshark.git synced 2025-06-19 04:53:32 +00:00
Code Issues Packages Projects Releases Wiki Activity

Migrate from SQLite to Basenine and introduce a new filtering syntax (#279)

Browse Source 
* Fix the OOMKilled error by calling `debug.FreeOSMemory` periodically

* Remove `MAX_NUMBER_OF_GOROUTINES` environment variable

* Change the line

* Increase the default value of `TCP_STREAM_CHANNEL_TIMEOUT_MS` to `10000`

* Write the client and integrate to the new real-time database

* Refactor the WebSocket implementaiton for `/ws`

* Adapt the UI to the new filtering system

* Fix the rest of the issues in the UI

* Increase the buffer of the scanner

* Implement accessing single records

* Increase the buffer of another scanner

* Populate `Request` and `Response` fields of `MizuEntry`

* Add syntax highlighting for the query

* Add database to `Dockerfile`

* Fix some issues

* Update the `realtime_dbms` Git module commit hash

* Upgrade Gin version and print the query string

* Revert "Upgrade Gin version and print the query string"

This reverts commit aa09f904ee.

* Use WebSocket's itself to query instead of the query string

* Fix some errors related to conversion to HAR

* Fix the issues caused by the latest merge

* Fix the build error

* Fix PR validation GitHub workflow

* Replace the git submodule with latest Basenine version `0.1.0`

Remove `realtime_client.go` and use the official client library `github.com/up9inc/basenine/client/go` instead.

* Move Basenine host and port constants to `shared` module

* Reliably execute and wait for Basenine to become available

* Upgrade Basenine version

* Properly close WebSocket and data channel

* Fix the issues caused by the recent merge commit

* Clean up the TypeScript code

* Update `.gitignore`

* Limit the database size

* Add `Macros` method signature to `Dissector` interface and set the macros provided by the protocol extensions

* Run `go mod tidy` on `agent`

* Upgrade `github.com/up9inc/basenine/client/go` version

* Implement a mechanism to update the query using click events in the UI and use it for protocol macros

* Update the query on click to timestamps

* Fix some issues in the WebSocket and channel handling

* Update the query on clicks to status code

* Update the query on clicks to method, path and service

* Update the query on clicks to is outgoing, source and destination ports

* Add an API endpoint to validate the query against syntax errors

* Move the query background color state into `TrafficPage`

* Fix the logic in `setQuery`

* Display a toast message in case of a syntax error in the query

* Remove a call to `fmt.Printf`

* Upgrade Basenine version to `0.1.3`

* Fix an issue related to getting `MAX_ENTRIES_DB_BYTES` environment variable

* Have the `path` key in request details, in HTTP

* Rearrange the HTTP headers for the querying

* Do the same thing for `cookies` and `queryString`

* Update the query on click to table elements

Add the selectors for `TABLE` type representations in HTTP extension.

* Update the query on click to `bodySize` and `elapsedTime` in `EntryTitle`

* Add the selectors for `TABLE` type representations in AMQP extension

* Add the selectors for `TABLE` type representations in Kafka extension

* Add the selectors for `TABLE` type representations in Redis extension

* Define a struct in `tap/api.go` for the section representation data

* Add the selectors for `BODY` type representations

* Add `request.path` to the HTTP request details

* Change the summary string's field name from `path` to `summary`

* Introduce `queryable` CSS class for queryable UI elements and underline them on hover

* Instead of `N requests` at the bottom, make it `Displaying N results (queried X/Y)` and live update the values

Upgrade Basenine version to `0.2.0`.

* Verify the sha256sum of Basenine executable inside `Dockerfile`

* Pass the start time to web UI through WebSocket and always show the `EntriesList` footer

* Pipe the `stderr` of Basenine as well

* Fix the layout issues related to `CodeEditor` in the UI

* Use the correct `shasum` command in `Dockerfile`

* Upgrade Basenine version to `0.2.1`

* Limit the height of `CodeEditor` container

* Remove `Paused` enum `ConnectionStatus` in UI

* Fix the issue caused by the recent merge

* Add the filtering guide (cheatsheet)

* Update open cheatsheet button's title

* Update cheatsheet content

* Remove the old SQLite code, adapt the `--analyze` related code to Basenine

* Change the method signature of `NewEntry`

* Change the method signature of `Represent`

* Introduce `HTTPPair` field in `MizuEntry` specific to HTTP

* Remove `Entry`, `EntryId` and `EstimatedSizeBytes` fields from `MizuEntry`

Also remove the `getEstimatedEntrySizeBytes` method.

* Remove `gorm.io/gorm` dependency

* Remove unused `sensitiveDataFiltering` folder

* Increase the left margin of open cheatsheet button

* Add `overflow: auto` to the cheatsheet `Modal`

* Fix `GetEntry` method

* Fix the macro for gRPC

* Fix an interface conversion in case of AMQP

* Fix two more interface conversion errors in AMQP

* Make the `syncEntriesImpl` method blocking

* Fix a grammar mistake in the cheatsheet

* Adapt to the changes in the recent merge commit

* Improve the cheatsheet text

* Always display the timestamp in `en-US`

* Upgrade Basenine version to `0.2.2`

* Fix the order of closing Basenine connections and channels

* Don't close the Basenine channels at all

* Upgrade Basenine version to `0.2.3`

* Set the initial filter to `rlimit(100)`

* Make Basenine persistent

* Upgrade Basenine version to `0.2.4`

* Update `debug.Dockerfile`

* Fix a failing test

* Upgrade Basenine version to `0.2.5`

* Revert "Do not show play icon when disconnected (#428)"

This reverts commit 8af2e562f8.

* Upgrade Basenine version to `0.2.6`

* Make all non-informative things informative

* Make `100` a constant

* Use `===` in JavaScript no matter what

* Remove a forgotten `console.log`

* Add a comment and update the `query` in `syncEntriesImpl`

* Don't call `panic` in `GetEntry`

* Replace `panic` calls in `startBasenineServer` with `logger.Log.Panicf`

* Remove unnecessary `\n` characters in the logs
This commit is contained in:
M. Mert Yıldıran 2021-11-09 19:54:48 +03:00 committed by GitHub
parent 31d95c6557
commit d2fe3f6620
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
62 changed files with 3077 additions and 2327 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

3
.gitignore vendored
View File

@ -29,3 +29,6 @@ build
# pprof
pprof/*
# Database Files
*.bin

12
Dockerfile
View File

@ -13,7 +13,7 @@ FROM golang:1.16-alpine AS builder
# Set necessary environment variables needed for our image.
ENV CGO_ENABLED=1 GOOS=linux GOARCH=amd64
RUN apk add libpcap-dev gcc g++ make bash
RUN apk add libpcap-dev gcc g++ make bash perl-utils
# Move to agent working directory (/agent-build).
WORKDIR /app/agent-build
@ -24,7 +24,7 @@ COPY tap/go.mod tap/go.mod ../tap/
COPY tap/api/go.* ../tap/api/
RUN go mod download
# cheap trick to make the build faster (As long as go.mod wasn't changes)
RUN go list -f '{{.Path}}@{{.Version}}' -m all | sed 1d | grep -e 'go-cache' -e 'sqlite' | xargs go get
RUN go list -f '{{.Path}}@{{.Version}}' -m all | sed 1d | grep -e 'go-cache' | xargs go get
ARG COMMIT_HASH
ARG GIT_BRANCH
@ -41,16 +41,24 @@ RUN go build -ldflags="-s -w \
-X 'mizuserver/pkg/version.BuildTimestamp=${BUILD_TIMESTAMP}' \
-X 'mizuserver/pkg/version.SemVer=${SEM_VER}'" -o mizuagent .
# Download Basenine executable, verify the sha1sum and move it to a directory in $PATH
ADD https://github.com/up9inc/basenine/releases/download/v0.2.6/basenine_linux_amd64 ./basenine_linux_amd64
ADD https://github.com/up9inc/basenine/releases/download/v0.2.6/basenine_linux_amd64.sha256 ./basenine_linux_amd64.sha256
RUN shasum -a 256 -c basenine_linux_amd64.sha256
RUN chmod +x ./basenine_linux_amd64
COPY devops/build_extensions.sh ..
RUN cd .. && /bin/bash build_extensions.sh
FROM alpine:3.14
RUN apk add bash libpcap-dev tcpdump
WORKDIR /app
# Copy binary and config files from /build to root folder of scratch container.
COPY --from=builder ["/app/agent-build/mizuagent", "."]
COPY --from=builder ["/app/agent-build/basenine_linux_amd64", "/usr/local/bin/basenine"]
COPY --from=builder ["/app/agent/build/extensions", "extensions"]
COPY --from=site-build ["/app/ui-build/build", "site"]
RUN mkdir /app/data/

6
acceptanceTests/tap_test.go
View File

@ -472,7 +472,7 @@ func TestTapRedact(t *testing.T) {
entryPayload := entryRequest["payload"].(map[string]interface{})
entryDetails := entryPayload["details"].(map[string]interface{})
headers := entryDetails["headers"].([]interface{})
headers := entryDetails["_headers"].([]interface{})
for _, headerInterface := range headers {
header := headerInterface.(map[string]interface{})
if header["name"].(string) != "User-Agent" {
@ -587,7 +587,7 @@ func TestTapNoRedact(t *testing.T) {
entryPayload := entryRequest["payload"].(map[string]interface{})
entryDetails := entryPayload["details"].(map[string]interface{})
headers := entryDetails["headers"].([]interface{})
headers := entryDetails["_headers"].([]interface{})
for _, headerInterface := range headers {
header := headerInterface.(map[string]interface{})
if header["name"].(string) != "User-Agent" {
@ -808,7 +808,7 @@ func TestTapIgnoredUserAgents(t *testing.T) {
entryPayload := entryRequest["payload"].(map[string]interface{})
entryDetails := entryPayload["details"].(map[string]interface{})
entryHeaders := entryDetails["headers"].([]interface{})
entryHeaders := entryDetails["_headers"].([]interface{})
for _, headerInterface := range entryHeaders {
header := headerInterface.(map[string]interface{})
if header["name"].(string) != ignoredUserAgentCustomHeader {

7
agent/go.mod
View File

@ -3,8 +3,8 @@ module mizuserver
go 1.16
require (
github.com/antelman107/net-wait-go v0.0.0-20210623112055-cf684aebda7b
github.com/djherbis/atime v1.0.0
github.com/fsnotify/fsnotify v1.4.9
github.com/getkin/kin-openapi v0.76.0
github.com/gin-contrib/static v0.0.1
github.com/gin-gonic/gin v1.7.2
@ -16,13 +16,12 @@ require (
github.com/op/go-logging v0.0.0-20160315200505-970db520ece7
github.com/orcaman/concurrent-map v0.0.0-20210106121528-16402b402231
github.com/patrickmn/go-cache v2.1.0+incompatible
github.com/up9inc/basenine/client/go v0.0.0-20211106180626-0193408db715
github.com/up9inc/mizu/shared v0.0.0
github.com/up9inc/mizu/tap v0.0.0
github.com/up9inc/mizu/tap/api v0.0.0
github.com/yalp/jsonpath v0.0.0-20180802001716-5cc68e5049a0
go.mongodb.org/mongo-driver v1.7.1
gorm.io/driver/sqlite v1.1.4
gorm.io/gorm v1.21.8
golang.org/x/text v0.3.5 // indirect
k8s.io/api v0.21.2
k8s.io/apimachinery v0.21.2
k8s.io/client-go v0.21.2

76
agent/go.sum
View File

@ -52,6 +52,8 @@ github.com/alecthomas/template v0.0.0-20190718012654-fb15b899a751/go.mod h1:LOuy
github.com/alecthomas/units v0.0.0-20151022065526-2efee857e7cf/go.mod h1:ybxpYRFXyAe+OPACYpWeL0wqObRcbAqCMya13uyzqw0=
github.com/alecthomas/units v0.0.0-20190717042225-c3de453c63f4/go.mod h1:ybxpYRFXyAe+OPACYpWeL0wqObRcbAqCMya13uyzqw0=
github.com/andreyvit/diff v0.0.0-20170406064948-c7f18ee00883/go.mod h1:rCTlJbsFo29Kk6CurOXKm700vrz8f0KW0JNfpkRJY/8=
github.com/antelman107/net-wait-go v0.0.0-20210623112055-cf684aebda7b h1:8m+eVxVVDDyJFidv7Ck1OwqnDaQR6pTSRGlCC2Dnw0A=
github.com/antelman107/net-wait-go v0.0.0-20210623112055-cf684aebda7b/go.mod h1:+tQQjzrp2501Nd6JXrb9s/XsNvFK3ZbxOnCdQl/vDRo=
github.com/armon/circbuf v0.0.0-20150827004946-bbbad097214e/go.mod h1:3U/XgcO3hCbHZ8TKRvWD2dDTCfh9M9ya+I9JpbB7O8o=
github.com/armon/consul-api v0.0.0-20180202201655-eb2c6b5be1b6/go.mod h1:grANhF5doyWs3UAsr3K4I6qtAmlQcZDesFNEHPZAzj8=
github.com/armon/go-metrics v0.0.0-20180917152333-f0300d1749da/go.mod h1:Q73ZrmVTwzkszR9V5SSuryQ31EELlFMUz1kKyl939pY=
@ -94,7 +96,6 @@ github.com/djherbis/atime v1.0.0 h1:ySLvBAM0EvOGaX7TI4dAM5lWj+RdJUCKtGSEHN8SGBg=
github.com/djherbis/atime v1.0.0/go.mod h1:5W+KBIuTwVGcqjIfaTwt+KSYX1o6uep8dtevevQP/f8=
github.com/docker/distribution v2.7.1+incompatible/go.mod h1:J2gT2udsDAN96Uj4KfcMRqY0/ypR+oyYUYmja8H+y+w=
github.com/docker/go-units v0.3.3/go.mod h1:fgPhTUdO+D/Jk86RDLlptpiXQzgHJF7gydDDbaIK4Dk=
github.com/docker/go-units v0.4.0 h1:3uh0PgVws3nIA0Q+MwDC8yjEPf9zjRfZZWXZYDct3Tw=
github.com/docker/go-units v0.4.0/go.mod h1:fgPhTUdO+D/Jk86RDLlptpiXQzgHJF7gydDDbaIK4Dk=
github.com/docopt/docopt-go v0.0.0-20180111231733-ee0de3bc6815/go.mod h1:WwZ+bS3ebgob9U8Nd0kOddGdZWjyMGR8Wziv+TBNwSE=
github.com/elazarl/goproxy v0.0.0-20180725130230-947c36da3153/go.mod h1:/Zj4wYkgs4iZTTu3o/KG3Itv/qCCa8VVMlb3i9OVuzc=
@ -111,7 +112,6 @@ github.com/fatih/color v1.7.0/go.mod h1:Zm6kSWBoL9eyXnKyktHP6abPY2pDugNf5Kwzbycv
github.com/form3tech-oss/jwt-go v3.2.2+incompatible h1:TcekIExNqud5crz4xD2pavyTgWiPvpYe4Xau31I0PRk=
github.com/form3tech-oss/jwt-go v3.2.2+incompatible/go.mod h1:pbq4aXjuKjdthFRnoDwaVPLA+WlJuPGy+QneDUgJi2k=
github.com/fsnotify/fsnotify v1.4.7/go.mod h1:jwhsz4b93w/PPRr/qN1Yymfu8t87LnFCMoQvtojpjFo=
github.com/fsnotify/fsnotify v1.4.9 h1:hsms1Qyu0jgnwNXIxa+/V/PDsU6CfLf6CNO8H7IWoS4=
github.com/fsnotify/fsnotify v1.4.9/go.mod h1:znqG4EE+3YCdAaPaxE2ZRY/06pZUdp0tY4IgpuI1SZQ=
github.com/fvbommel/sortorder v1.0.1/go.mod h1:uk88iVf1ovNn1iLfgUVU2F9o5eO30ui720w+kxuqRs0=
github.com/getkin/kin-openapi v0.76.0 h1:j77zg3Ec+k+r+GA3d8hBoXpAc6KX9TbBPrwQGBIy2sY=
@ -195,31 +195,7 @@ github.com/go-playground/validator/v10 v10.4.1/go.mod h1:nlOn6nFhuKACm19sB/8EGNn
github.com/go-playground/validator/v10 v10.5.0 h1:X9rflw/KmpACwT8zdrm1upefpvdy6ur8d1kWyq6sg3E=
github.com/go-playground/validator/v10 v10.5.0/go.mod h1:xm76BBt941f7yWdGnI2DVPFFg1UK3YY04qifoXU3lOk=
github.com/go-stack/stack v1.8.0/go.mod h1:v0f6uXyyMGvRgIKkXu+yp6POWl0qKG85gN/melR3HDY=
github.com/gobuffalo/attrs v0.0.0-20190224210810-a9411de4debd/go.mod h1:4duuawTqi2wkkpB4ePgWMaai6/Kc6WEz83bhFwpHzj0=
github.com/gobuffalo/depgen v0.0.0-20190329151759-d478694a28d3/go.mod h1:3STtPUQYuzV0gBVOY3vy6CfMm/ljR4pABfrTeHNLHUY=
github.com/gobuffalo/depgen v0.1.0/go.mod h1:+ifsuy7fhi15RWncXQQKjWS9JPkdah5sZvtHc2RXGlg=
github.com/gobuffalo/envy v1.6.15/go.mod h1:n7DRkBerg/aorDM8kbduw5dN3oXGswK5liaSCx4T5NI=
github.com/gobuffalo/envy v1.7.0/go.mod h1:n7DRkBerg/aorDM8kbduw5dN3oXGswK5liaSCx4T5NI=
github.com/gobuffalo/flect v0.1.0/go.mod h1:d2ehjJqGOH/Kjqcoz+F7jHTBbmDb38yXA598Hb50EGs=
github.com/gobuffalo/flect v0.1.1/go.mod h1:8JCgGVbRjJhVgD6399mQr4fx5rRfGKVzFjbj6RE/9UI=
github.com/gobuffalo/flect v0.1.3/go.mod h1:8JCgGVbRjJhVgD6399mQr4fx5rRfGKVzFjbj6RE/9UI=
github.com/gobuffalo/genny v0.0.0-20190329151137-27723ad26ef9/go.mod h1:rWs4Z12d1Zbf19rlsn0nurr75KqhYp52EAGGxTbBhNk=
github.com/gobuffalo/genny v0.0.0-20190403191548-3ca520ef0d9e/go.mod h1:80lIj3kVJWwOrXWWMRzzdhW3DsrdjILVil/SFKBzF28=
github.com/gobuffalo/genny v0.1.0/go.mod h1:XidbUqzak3lHdS//TPu2OgiFB+51Ur5f7CSnXZ/JDvo=
github.com/gobuffalo/genny v0.1.1/go.mod h1:5TExbEyY48pfunL4QSXxlDOmdsD44RRq4mVZ0Ex28Xk=
github.com/gobuffalo/gitgen v0.0.0-20190315122116-cc086187d211/go.mod h1:vEHJk/E9DmhejeLeNt7UVvlSGv3ziL+djtTr3yyzcOw=
github.com/gobuffalo/gogen v0.0.0-20190315121717-8f38393713f5/go.mod h1:V9QVDIxsgKNZs6L2IYiGR8datgMhB577vzTDqypH360=
github.com/gobuffalo/gogen v0.1.0/go.mod h1:8NTelM5qd8RZ15VjQTFkAW6qOMx5wBbW4dSCS3BY8gg=
github.com/gobuffalo/gogen v0.1.1/go.mod h1:y8iBtmHmGc4qa3urIyo1shvOD8JftTtfcKi+71xfDNE=
github.com/gobuffalo/here v0.6.0/go.mod h1:wAG085dHOYqUpf+Ap+WOdrPTp5IYcDAs/x7PLa8Y5fM=
github.com/gobuffalo/logger v0.0.0-20190315122211-86e12af44bc2/go.mod h1:QdxcLw541hSGtBnhUc4gaNIXRjiDppFGaDqzbrBd3v8=
github.com/gobuffalo/mapi v1.0.1/go.mod h1:4VAGh89y6rVOvm5A8fKFxYG+wIW6LO1FMTG9hnKStFc=
github.com/gobuffalo/mapi v1.0.2/go.mod h1:4VAGh89y6rVOvm5A8fKFxYG+wIW6LO1FMTG9hnKStFc=
github.com/gobuffalo/packd v0.0.0-20190315124812-a385830c7fc0/go.mod h1:M2Juc+hhDXf/PnmBANFCqx4DM3wRbgDvnVWeG2RIxq4=
github.com/gobuffalo/packd v0.1.0/go.mod h1:M2Juc+hhDXf/PnmBANFCqx4DM3wRbgDvnVWeG2RIxq4=
github.com/gobuffalo/packr/v2 v2.0.9/go.mod h1:emmyGweYTm6Kdper+iywB6YK5YzuKchGtJQZ0Odn4pQ=
github.com/gobuffalo/packr/v2 v2.2.0/go.mod h1:CaAwI0GPIAv+5wKLtv8Afwl+Cm78K/I/VCm/3ptBN+0=
github.com/gobuffalo/syncx v0.0.0-20190224160051-33c29581e754/go.mod h1:HhnNqWY95UYwwW3uSASeV7vtgYkT2t16hJgV3AEPUpw=
github.com/gogo/protobuf v1.1.1/go.mod h1:r8qH/GZQm5c6nD/R0oafs1akxWv10x8SbQlK7atdtwQ=
github.com/gogo/protobuf v1.2.1/go.mod h1:hp+jE20tsWTFYpLwKvXlhS1hjn+gTNwPg2I6zVXpSg4=
github.com/gogo/protobuf v1.3.2 h1:Ov1cvc58UF3b5XjBnZv7+opcTcQFZebYjWzi34vdm4Q=
@ -251,7 +227,6 @@ github.com/golang/protobuf v1.4.1/go.mod h1:U8fpvMrcmy5pZrNK1lt4xCsGvpyWQ/VVv6QD
github.com/golang/protobuf v1.4.2/go.mod h1:oDoupMAO8OvCJWAcko0GGGIgR6R6ocIYbsSw735rRwI=
github.com/golang/protobuf v1.4.3 h1:JjCZWpVbqXDqFVmTfYWEVTMIYrL/NPdPSCHPJ0T/raM=
github.com/golang/protobuf v1.4.3/go.mod h1:oDoupMAO8OvCJWAcko0GGGIgR6R6ocIYbsSw735rRwI=
github.com/golang/snappy v0.0.1/go.mod h1:/XxbfmMg8lxefKM7IXC3fBNl/7bRcc72aCRzEWrmP2Q=
github.com/golangplus/testing v0.0.0-20180327235837-af21d9c3145e/go.mod h1:0AA//k/eakGydO4jKRoRL2j92ZKSzTgj9tclaCrvXHk=
github.com/google/btree v0.0.0-20180813153112-4030bb1f1f0c/go.mod h1:lNA+9X1NB3Zf8V7Ke586lFgjr2dZNuvo3lPJSGZ5JPQ=
github.com/google/btree v1.0.0/go.mod h1:lNA+9X1NB3Zf8V7Ke586lFgjr2dZNuvo3lPJSGZ5JPQ=
@ -321,12 +296,6 @@ github.com/ianlancetaylor/demangle v0.0.0-20181102032728-5e5cf60278f6/go.mod h1:
github.com/imdario/mergo v0.3.5 h1:JboBksRwiiAJWvIYJVo46AfV+IAIKZpfrSzVKj42R4Q=
github.com/imdario/mergo v0.3.5/go.mod h1:2EnlNZ0deacrJVfApfmtdGgDfMuh/nq6Ok1EcJh5FfA=
github.com/inconshreveable/mousetrap v1.0.0/go.mod h1:PxqpIevigyE2G7u3NXJIT2ANytuPF1OarO4DADm73n8=
github.com/jinzhu/inflection v1.0.0 h1:K317FqzuhWc8YvSVlFMCCUb36O/S9MCKRDI7QkRKD/E=
github.com/jinzhu/inflection v1.0.0/go.mod h1:h+uFLlag+Qp1Va5pdKtLDYj+kHp5pxUVkryuEj+Srlc=
github.com/jinzhu/now v1.1.1/go.mod h1:d3SSVoowX0Lcu0IBviAWJpolVfI5UJVZZ7cO71lE/z8=
github.com/jinzhu/now v1.1.2 h1:eVKgfIdy9b6zbWBMgFpfDPoAMifwSZagU9HmEU6zgiI=
github.com/jinzhu/now v1.1.2/go.mod h1:d3SSVoowX0Lcu0IBviAWJpolVfI5UJVZZ7cO71lE/z8=
github.com/joho/godotenv v1.3.0/go.mod h1:7hK45KPybAkOC6peb+G5yklZfMxEjkZhHbwpqxOKXbg=
github.com/jonboulle/clockwork v0.1.0/go.mod h1:Ii8DK3G1RaLaWxj9trq07+26W01tbo22gdxWY5EU2bo=
github.com/json-iterator/go v1.1.6/go.mod h1:+SdeFBvtyEkXs7REEP0seUULqWtbJapLOCVDaaPEHmU=
github.com/json-iterator/go v1.1.9/go.mod h1:KdQUCv79m/52Kvf8AW2vK1V8akMuk1QjK/uOdHXbAo4=
@ -336,14 +305,10 @@ github.com/jstemmer/go-junit-report v0.0.0-20190106144839-af01ea7f8024/go.mod h1
github.com/jstemmer/go-junit-report v0.9.1/go.mod h1:Brl9GWCQeLvo8nXZwPNNblvFj/XSXhF0NWZEnDohbsk=
github.com/jtolds/gls v4.20.0+incompatible/go.mod h1:QJZ7F/aHp+rZTRtaJ1ow/lLfFfVYBRgL+9YlvaHOwJU=
github.com/julienschmidt/httprouter v1.2.0/go.mod h1:SYymIcj16QtmaHHD7aYtjjsJG7VTCxuUUipMqKk8s4w=
github.com/karrick/godirwalk v1.8.0/go.mod h1:H5KPZjojv4lE+QYImBI8xVtrBRgYrIVsaRPx4tDPEn4=
github.com/karrick/godirwalk v1.10.3/go.mod h1:RoGL9dQei4vP9ilrpETWE8CLOZ1kiN0LhBygSwrAsHA=
github.com/kisielk/errcheck v1.1.0/go.mod h1:EZBBE59ingxPouuu3KfxchcWSUPOHkagtvWXihfKN4Q=
github.com/kisielk/errcheck v1.5.0/go.mod h1:pFxgyoBC7bSaBwPgfKdkLd5X25qrDl4LWUI2bnpBCr8=
github.com/kisielk/gotool v1.0.0/go.mod h1:XhKaO+MFFWcvkIS/tQcRk01m1F5IRFswLeQ+oQHNcck=
github.com/klauspost/compress v1.9.5/go.mod h1:RyIbtBH6LamlWaDj8nUwkbUhJ87Yi3uG0guNDohfE1A=
github.com/konsorten/go-windows-terminal-sequences v1.0.1/go.mod h1:T0+1ngSBFLxvqU3pZ+m/2kptfBszLMUkC4ZK/EgS/cQ=
github.com/konsorten/go-windows-terminal-sequences v1.0.2/go.mod h1:T0+1ngSBFLxvqU3pZ+m/2kptfBszLMUkC4ZK/EgS/cQ=
github.com/kr/logfmt v0.0.0-20140226030751-b84e30acd515/go.mod h1:+0opPa2QZZtGFBFZlji/RkVcI2GknAs/DXo4wKdlNEc=
github.com/kr/pretty v0.1.0/go.mod h1:dAy3ld7l9f0ibDNOQOHHMYYIIbhfbHSm3C4ZsoJORNo=
github.com/kr/pretty v0.2.0/go.mod h1:ipq/a2n7PKx3OHsz4KJII5eveXtPO4qwEXGdVfWzfnI=
@ -364,16 +329,12 @@ github.com/mailru/easyjson v0.0.0-20190614124828-94de47d64c63/go.mod h1:C1wdFJiN
github.com/mailru/easyjson v0.0.0-20190626092158-b2ccc519800e/go.mod h1:C1wdFJiN94OJF2b5HbByQZoLdCWB1Yqtg26g4irojpc=
github.com/mailru/easyjson v0.7.0 h1:aizVhC/NAAcKWb+5QsU1iNOZb4Yws5UO2I+aIprQITM=
github.com/mailru/easyjson v0.7.0/go.mod h1:KAzv3t3aY1NaHWoQz1+4F1ccyAH66Jk7yos7ldAVICs=
github.com/markbates/oncer v0.0.0-20181203154359-bf2de49a0be2/go.mod h1:Ld9puTsIW75CHf65OeIOkyKbteujpZVXDpWK6YGZbxE=
github.com/markbates/pkger v0.17.1/go.mod h1:0JoVlrol20BSywW79rN3kdFFsE5xYM+rSCQDXbLhiuI=
github.com/markbates/safe v1.0.1/go.mod h1:nAqgmRi7cY2nqMc92/bSEeQA+R4OheNU2T1kNSCBdG0=
github.com/mattn/go-colorable v0.0.9/go.mod h1:9vuHe8Xs5qXnSaW/c/ABM9alt+Vo+STaOChaDxuIBZU=
github.com/mattn/go-isatty v0.0.3/go.mod h1:M+lRXTBqGeGNdLjl/ufCoiOlB5xdOkqRJdNxMWT7Zi4=
github.com/mattn/go-isatty v0.0.12 h1:wuysRhFDzyxgEmMf5xjvJ2M9dZoWAXNNr5LSBS7uHXY=
github.com/mattn/go-isatty v0.0.12/go.mod h1:cbi8OIDigv2wuxKPP5vlRcQ1OAZbq2CE4Kysco4FUpU=
github.com/mattn/go-runewidth v0.0.7/go.mod h1:H031xJmbD/WCDINGzjvQ9THkh0rPKHF+m2gUSrubnMI=
github.com/mattn/go-sqlite3 v1.14.5 h1:1IdxlwTNazvbKJQSxoJ5/9ECbEeaTTyeU7sEAZ5KKTQ=
github.com/mattn/go-sqlite3 v1.14.5/go.mod h1:WVKg1VTActs4Qso6iwGbiFih2UIHo0ENGwNd0Lj+XmI=
github.com/matttproud/golang_protobuf_extensions v1.0.1/go.mod h1:D8He9yQNgCq6Z5Ld7szi9bcBfOoFv/3dc6xSMkL2PC0=
github.com/matttproud/golang_protobuf_extensions v1.0.2-0.20181231171920-c182affec369/go.mod h1:BSXmuO+STAnVfrANrmjBb36TMTDstsz7MSK+HVaYKv4=
github.com/miekg/dns v1.0.14/go.mod h1:W1PPwlIAgtquWBMBEV9nkV9Cazfe8ScdGz/Lj7v3Nrg=
@ -395,7 +356,6 @@ github.com/modern-go/reflect2 v0.0.0-20180701023420-4b7aa43c6742/go.mod h1:bx2lN
github.com/modern-go/reflect2 v1.0.1 h1:9f412s+6RmYXLWZSEzVVgPGK7C2PphHj5RJrvfx9AWI=
github.com/modern-go/reflect2 v1.0.1/go.mod h1:bx2lNnkwVCuqBIxFjflWJWanXIb3RllmbCylyMrvgv0=
github.com/monochromegane/go-gitignore v0.0.0-20200626010858-205db1a8cc00/go.mod h1:Pm3mSP3c5uWn86xMLZ5Sa7JB9GsEZySvHYXCTK4E9q4=
github.com/montanaflynn/stats v0.0.0-20171201202039-1bf9dbcd8cbe/go.mod h1:wL8QJuTMNUDYhXwkmfOly8iTdp5TEcJFWZD2D7SIkUc=
github.com/munnerz/goautoneg v0.0.0-20120707110453-a547fc61f48d/go.mod h1:+n7T8mK8HuQTcFwEeznm/DIxMOiR9yIdICNftLE1DvQ=
github.com/mwitkow/go-conntrack v0.0.0-20161129095857-cc309e4a2223/go.mod h1:qRWi+5nqEBWmkhHvq77mSJWrCKwh8bxhgT7d/eI7P4U=
github.com/mxk/go-flowrate v0.0.0-20140419014527-cca7078d478f h1:y5//uYreIhSUg3J1GEMiLbxo1LJaP8RfCpH6pymGZus=
@ -419,7 +379,6 @@ github.com/patrickmn/go-cache v2.1.0+incompatible h1:HRMgzkcYKYpi3C8ajMPV8OFXaaR
github.com/patrickmn/go-cache v2.1.0+incompatible/go.mod h1:3Qf8kWWT7OJRJbdiICTKqZju1ZixQ/KpMGzzAfe6+WQ=
github.com/pborman/uuid v1.2.0/go.mod h1:X/NO0urCmaxf9VXbdlT7C2Yzkj2IKimNn4k+gtPdI/k=
github.com/pelletier/go-toml v1.2.0/go.mod h1:5z9KED0ma1S8pY6P1sdut58dfprrGBbd/94hg7ilaic=
github.com/pelletier/go-toml v1.7.0/go.mod h1:vwGMzjaWMwyfHwgIBhI2YUM4fB6nL6lVAvS1LBMMhTE=
github.com/peterbourgon/diskv v2.0.1+incompatible/go.mod h1:uqqh8zWWbv1HBMNONnaR/tNboyR3/BZd58JJSHlUSCU=
github.com/pkg/errors v0.8.0/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0=
github.com/pkg/errors v0.8.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0=
@ -447,8 +406,6 @@ github.com/prometheus/procfs v0.1.3/go.mod h1:lV6e/gmhEcM9IjHGsFOCxxuZ+z1YqCvr4O
github.com/prometheus/procfs v0.2.0/go.mod h1:lV6e/gmhEcM9IjHGsFOCxxuZ+z1YqCvr4OA4YeYWdaU=
github.com/prometheus/tsdb v0.7.1/go.mod h1:qhTCs0VvXwvX/y3TZrWD7rabWM+ijKTux40TwIPHuXU=
github.com/rogpeppe/fastuuid v0.0.0-20150106093220-6724a57986af/go.mod h1:XWv6SoW27p1b0cqNHllgS5HIMJraePCO15w5zCzIWYg=
github.com/rogpeppe/go-internal v1.1.0/go.mod h1:M8bDsm7K2OlrFYOpmOWEs/qY81heoFRclV5y23lUDJ4=
github.com/rogpeppe/go-internal v1.2.2/go.mod h1:M8bDsm7K2OlrFYOpmOWEs/qY81heoFRclV5y23lUDJ4=
github.com/rogpeppe/go-internal v1.3.0/go.mod h1:M8bDsm7K2OlrFYOpmOWEs/qY81heoFRclV5y23lUDJ4=
github.com/russross/blackfriday v1.5.2/go.mod h1:JO/DiYxRf+HjHt06OyowR9PTA263kcR/rfWxYHBV53g=
github.com/russross/blackfriday/v2 v2.0.1/go.mod h1:+Rmxgy9KzJVeS9/2gXHxylqXiyQDYRxCVz55jmeOWTM=
@ -458,8 +415,6 @@ github.com/sergi/go-diff v1.0.0/go.mod h1:0CfEIISq7TuYL3j771MWULgwwjU+GofnZX9QAm
github.com/sergi/go-diff v1.1.0/go.mod h1:STckp+ISIX8hZLjrqAeVduY0gWCT9IjLuqbuNXdaHfM=
github.com/shurcooL/sanitized_anchor_name v1.0.0/go.mod h1:1NzhyTcUVG4SuEtjjoZeVRXNmyL/1OwPU0+IJeTBvfc=
github.com/sirupsen/logrus v1.2.0/go.mod h1:LxeOpSwHxABJmUn/MG1IvRgCAasNZTLOkJPxbbu5VWo=
github.com/sirupsen/logrus v1.4.0/go.mod h1:LxeOpSwHxABJmUn/MG1IvRgCAasNZTLOkJPxbbu5VWo=
github.com/sirupsen/logrus v1.4.1/go.mod h1:ni0Sbl8bgC9z8RoU9G6nDWqqs/fq4eDPysMBDgk/93Q=
github.com/sirupsen/logrus v1.4.2/go.mod h1:tLMulIdttU9McNUspp0xgXVQah82FyeX6MwdIuYE2rE=
github.com/smartystreets/assertions v0.0.0-20180927180507-b2de0cb4f26d/go.mod h1:OnSkiWE9lh6wB0YB77sQom3nweQdgAjqCqsofrRNTgc=
github.com/smartystreets/goconvey v1.6.4/go.mod h1:syvi0/a8iFYH4r/RixwvyeAJjdLS9QV7WQ/tjFTllLA=
@ -468,7 +423,6 @@ github.com/spaolacci/murmur3 v0.0.0-20180118202830-f09979ecbc72/go.mod h1:JwIasO
github.com/spf13/afero v1.1.2/go.mod h1:j4pytiNVoe2o6bmDsKpLACNPDBIoEAkihy7loJ1B0CQ=
github.com/spf13/afero v1.2.2/go.mod h1:9ZxEEn6pIJ8Rxe320qSDBk6AsU0r9pR7Q4OcevTdifk=
github.com/spf13/cast v1.3.0/go.mod h1:Qx5cxh0v+4UWYiBimWS+eyWzqEqokIECu5etghLkUJE=
github.com/spf13/cobra v0.0.3/go.mod h1:1l0Ry5zgKvJasoi3XT1TypsSe7PqH0Sj9dhYf7v3XqQ=
github.com/spf13/cobra v1.0.0/go.mod h1:/6GTrnGXV9HjY+aR4k0oJ5tcvakLuG6EuKReYlHNrgE=
github.com/spf13/cobra v1.1.1/go.mod h1:WnodtKOvamDL/PwE2M4iKs8aMDBZ5Q5klgD3qfVJQMI=
github.com/spf13/jwalterweatherman v1.0.0/go.mod h1:cQK4TGJAtQXfYWX+Ddv3mKDzgVb68N+wFjFa4jdeBTo=
@ -485,8 +439,9 @@ github.com/stretchr/testify v1.2.2/go.mod h1:a8OnRcib4nhh0OaRAV+Yts87kKdq0PP7pXf
github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI=
github.com/stretchr/testify v1.4.0/go.mod h1:j7eGeouHqKxXV5pUuKE4zz7dFj8WfuZ+81PSLYec5m4=
github.com/stretchr/testify v1.5.1/go.mod h1:5W2xD1RspED5o8YsWQXVCued0rvSQ+mT+I5cxcmMvtA=
github.com/stretchr/testify v1.6.1 h1:hDPOHmpOpP40lSULcqw7IrRb/u7w6RpDC9399XyoNd0=
github.com/stretchr/testify v1.6.1/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
github.com/stretchr/testify v1.7.0 h1:nwc3DEeHmmLAfoZucVR881uASk0Mfjw8xYJ99tb5CcY=
github.com/stretchr/testify v1.7.0/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
github.com/subosito/gotenv v1.2.0/go.mod h1:N0PQaV/YGNqwC0u51sEeR/aUtSLEXKX9iv69rRypqCw=
github.com/tidwall/pretty v1.0.0/go.mod h1:XNkn88O1ChpSDQmQeStsy+sBenx6DDtFZJxhVysOjyk=
github.com/tmc/grpc-websocket-proxy v0.0.0-20190109142713-0ad062ec5ee5/go.mod h1:ncp9v5uamzpCO7NfCPTXjqaC+bZgJeR0sMTm6dMHP7U=
@ -495,26 +450,22 @@ github.com/ugorji/go v1.1.7 h1:/68gy2h+1mWMrwZFeD1kQialdSzAb432dtpeJ42ovdo=
github.com/ugorji/go v1.1.7/go.mod h1:kZn38zHttfInRq0xu/PH0az30d+z6vm202qpg1oXVMw=
github.com/ugorji/go/codec v1.1.7 h1:2SvQaVZ1ouYrrKKwoSk2pzd4A9evlKJb9oTL+OaLUSs=
github.com/ugorji/go/codec v1.1.7/go.mod h1:Ax+UKWsSmolVDwsd+7N3ZtXu+yMGCf907BLYF3GoBXY=
github.com/up9inc/basenine/client/go v0.0.0-20211106180626-0193408db715 h1:3RNTMQZO/4g5gRn4R98cPwCjCrsMklmcOS0g+QwCh5c=
github.com/up9inc/basenine/client/go v0.0.0-20211106180626-0193408db715/go.mod h1:SvJGPoa/6erhUQV7kvHBwM/0x5LyO6XaG2lUaCaKiUI=
github.com/vektah/gqlparser v1.1.2/go.mod h1:1ycwN7Ij5njmMkPPAOaRFY4rET2Enx7IkVv3vaXspKw=
github.com/vishvananda/netns v0.0.0-20210104183010-2eb08e3e575f h1:p4VB7kIXpOQvVn1ZaTIVp+3vuYAXFe3OJEvjbUYJLaA=
github.com/vishvananda/netns v0.0.0-20210104183010-2eb08e3e575f/go.mod h1:DD4vA1DwXk04H54A1oHXtwZmA0grkVMdPxx/VGLCah0=
github.com/xdg-go/pbkdf2 v1.0.0/go.mod h1:jrpuAogTd400dnrH08LKmI/xc1MbPOebTwRqcT5RDeI=
github.com/xdg-go/scram v1.0.2/go.mod h1:1WAq6h33pAW+iRreB34OORO2Nf7qel3VV3fjBj+hCSs=
github.com/xdg-go/stringprep v1.0.2/go.mod h1:8F9zXuvzgwmyT5DUm4GUfZGDdT3W+LCvS6+da4O5kxM=
github.com/xiang90/probing v0.0.0-20190116061207-43a291ad63a2/go.mod h1:UETIi67q53MR2AWcXfiuqkDkRtnGDLqkBTpCHuJHxtU=
github.com/xlab/treeprint v0.0.0-20181112141820-a009c3971eca/go.mod h1:ce1O1j6UtZfjr22oyGxGLbauSBp2YVXpARAosm7dHBg=
github.com/xordataexchange/crypt v0.0.3-0.20170626215501-b2862e3d0a77/go.mod h1:aYKd//L2LvnjZzWKhF00oedf4jCCReLcmhLdhm1A27Q=
github.com/yalp/jsonpath v0.0.0-20180802001716-5cc68e5049a0 h1:6fRhSjgLCkTD3JnJxvaJ4Sj+TYblw757bqYgZaOq5ZY=
github.com/yalp/jsonpath v0.0.0-20180802001716-5cc68e5049a0/go.mod h1:/LWChgwKmvncFJFHJ7Gvn9wZArjbV5/FppcK2fKk/tI=
github.com/youmark/pkcs8 v0.0.0-20181117223130-1be2e3e5546d/go.mod h1:rHwXgn7JulP+udvsHwJoVG1YGAP6VLg4y9I5dyZdqmA=
github.com/yuin/goldmark v1.1.27/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74=
github.com/yuin/goldmark v1.2.1/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74=
go.etcd.io/bbolt v1.3.2/go.mod h1:IbVyRI1SCnLcuJnV2u8VeU0CEYM7e686BmAb1XKL+uU=
go.mongodb.org/mongo-driver v1.0.3/go.mod h1:u7ryQJ+DOzQmeO7zB6MHyr8jkEQvC8vH7qLUO4lqsUM=
go.mongodb.org/mongo-driver v1.1.1/go.mod h1:u7ryQJ+DOzQmeO7zB6MHyr8jkEQvC8vH7qLUO4lqsUM=
go.mongodb.org/mongo-driver v1.1.2/go.mod h1:u7ryQJ+DOzQmeO7zB6MHyr8jkEQvC8vH7qLUO4lqsUM=
go.mongodb.org/mongo-driver v1.7.1 h1:jwqTeEM3x6L9xDXrCxN0Hbg7vdGfPBOTIkr0+/LYZDA=
go.mongodb.org/mongo-driver v1.7.1/go.mod h1:Q4oFMbo1+MSNqICAdYMlC/zSTrwCogR4R8NzkI+yfU8=
go.opencensus.io v0.21.0/go.mod h1:mSImk1erAIZhrmZN+AvHh14ztQfjbGwt4TtuofqLduU=
go.opencensus.io v0.22.0/go.mod h1:+kGneAE2xo2IficOXnaByMWTGM9T73dGwxeWcUqIpI8=
go.opencensus.io v0.22.2/go.mod h1:yxeiOL68Rb0Xd1ddK5vPZ/oVn4vY4Ynel7k9FzqtOIw=
@ -527,13 +478,11 @@ golang.org/x/crypto v0.0.0-20180904163835-0709b304e793/go.mod h1:6SG95UA2DQfeDnf
golang.org/x/crypto v0.0.0-20181029021203-45a5f77698d3/go.mod h1:6SG95UA2DQfeDnfUPMdvaQW0Q7yPrPDi9nlGo2tz2b4=
golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w=
golang.org/x/crypto v0.0.0-20190320223903-b7391e95e576/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w=
golang.org/x/crypto v0.0.0-20190422162423-af44ce270edf/go.mod h1:WFFai1msRO1wXaEeE5yQxYXgSfI8pQAWXbQop6sCtWE=
golang.org/x/crypto v0.0.0-20190510104115-cbcb75029529/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI=
golang.org/x/crypto v0.0.0-20190605123033-f99c8df09eb5/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI=
golang.org/x/crypto v0.0.0-20190611184440-5c40567a22f8/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI=
golang.org/x/crypto v0.0.0-20190617133340-57b3e21c3d56/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI=
golang.org/x/crypto v0.0.0-20191011191535-87dc89f01550/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI=
golang.org/x/crypto v0.0.0-20200302210943-78000ba7a073/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto=
golang.org/x/crypto v0.0.0-20200622213623-75b288015ac9/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto=
golang.org/x/crypto v0.0.0-20201002170205-7f63de1d35b0/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto=
golang.org/x/crypto v0.0.0-20210220033148-5ea612d1eb83 h1:/ZScEX8SfEmUGRHs0gxpqteO5nfNW6axyZbBdw9A12g=
@ -611,7 +560,6 @@ golang.org/x/sync v0.0.0-20180314180146-1d60e4601c6f/go.mod h1:RxMgew5VJxzue5/jJ
golang.org/x/sync v0.0.0-20181108010431-42b317875d0f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20181221193216-37e7f081c4d4/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20190227155943-e225da77a7e6/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20190412183630-56d357773e84/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20190423024810-112230192c58/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20190911185100-cd5d95a43a6e/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20201020160332-67f06af15bc9/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
@ -625,13 +573,10 @@ golang.org/x/sys v0.0.0-20181116152217-5ac8a444bdc5/go.mod h1:STP8DvDyc/dI5b8T5h
golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
golang.org/x/sys v0.0.0-20190312061237-fead79001313/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20190321052220-f7bb7a8bee54/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20190403152447-81d4e9dc473e/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20190412213103-97732733099d/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20190419153524-e8e3143a4f4a/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20190422165155-953cdadca894/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20190502145724-3ef323f4f1fd/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20190507160741-ecd444e8653b/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20190531175056-4c3a928424d2/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20190606165138-5da285871e9c/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20190616124812-15dcb6c0061f/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20190624142023-c5567b49c5d0/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
@ -685,13 +630,9 @@ golang.org/x/tools v0.0.0-20190311212946-11955173bddd/go.mod h1:LCzVGOaR6xXOjkQ3
golang.org/x/tools v0.0.0-20190312151545-0bb0c0a6e846/go.mod h1:LCzVGOaR6xXOjkQ3onu1FJEFr0SW1gC7cKk1uF8kGRs=
golang.org/x/tools v0.0.0-20190312170243-e65039ee4138/go.mod h1:LCzVGOaR6xXOjkQ3onu1FJEFr0SW1gC7cKk1uF8kGRs=
golang.org/x/tools v0.0.0-20190328211700-ab21143f2384/go.mod h1:LCzVGOaR6xXOjkQ3onu1FJEFr0SW1gC7cKk1uF8kGRs=
golang.org/x/tools v0.0.0-20190329151228-23e29df326fe/go.mod h1:LCzVGOaR6xXOjkQ3onu1FJEFr0SW1gC7cKk1uF8kGRs=
golang.org/x/tools v0.0.0-20190416151739-9c9e1878f421/go.mod h1:LCzVGOaR6xXOjkQ3onu1FJEFr0SW1gC7cKk1uF8kGRs=
golang.org/x/tools v0.0.0-20190420181800-aa740d480789/go.mod h1:LCzVGOaR6xXOjkQ3onu1FJEFr0SW1gC7cKk1uF8kGRs=
golang.org/x/tools v0.0.0-20190425150028-36563e24a262/go.mod h1:RgjU9mgBXZiqYHBnxXauZ1Gv1EHHAz9KjViQ78xBX0Q=
golang.org/x/tools v0.0.0-20190506145303-2d16b83fe98c/go.mod h1:RgjU9mgBXZiqYHBnxXauZ1Gv1EHHAz9KjViQ78xBX0Q=
golang.org/x/tools v0.0.0-20190524140312-2c0ae7006135/go.mod h1:RgjU9mgBXZiqYHBnxXauZ1Gv1EHHAz9KjViQ78xBX0Q=
golang.org/x/tools v0.0.0-20190531172133-b3315ee88b7d/go.mod h1:/rFqwRUd4F7ZHNgwSSTFct+R/Kf4OFW1sUzUTQQTgfc=
golang.org/x/tools v0.0.0-20190606124116-d0a3d012864b/go.mod h1:/rFqwRUd4F7ZHNgwSSTFct+R/Kf4OFW1sUzUTQQTgfc=
golang.org/x/tools v0.0.0-20190614205625-5aca471b1d59/go.mod h1:/rFqwRUd4F7ZHNgwSSTFct+R/Kf4OFW1sUzUTQQTgfc=
golang.org/x/tools v0.0.0-20190617190820-da514acc4774/go.mod h1:/rFqwRUd4F7ZHNgwSSTFct+R/Kf4OFW1sUzUTQQTgfc=
@ -805,11 +746,6 @@ gopkg.in/yaml.v2 v2.4.0/go.mod h1:RDklbk79AGWmwhnvt/jBztapEOGDOx6ZbXqjP6csGnQ=
gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
gopkg.in/yaml.v3 v3.0.0-20210107192922-496545a6307b h1:h8qDotaEPuJATrMmW04NCwg7v22aHH28wwpauUhK9Oo=
gopkg.in/yaml.v3 v3.0.0-20210107192922-496545a6307b/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
gorm.io/driver/sqlite v1.1.4 h1:PDzwYE+sI6De2+mxAneV9Xs11+ZyKV6oxD3wDGkaNvM=
gorm.io/driver/sqlite v1.1.4/go.mod h1:mJCeTFr7+crvS+TRnWc5Z3UvwxUN1BGBLMrf5LA9DYw=
gorm.io/gorm v1.20.7/go.mod h1:0HFTzE/SqkGTzK6TlDPPQbAYCluiVvhzoA1+aVyzenw=
gorm.io/gorm v1.21.8 h1:2CEwZSzogdhsKPlJ9OvBKTdlWIpELXb6HbfLfMNhSYI=
gorm.io/gorm v1.21.8/go.mod h1:F+OptMscr0P2F2qU97WT1WimdH9GaQPoDW7AYd5i2Y0=
gotest.tools/v3 v3.0.2/go.mod h1:3SzNCllyD9/Y+b5r9JIKQ474KzkZyqLqEfYqMsX94Bk=
gotest.tools/v3 v3.0.3/go.mod h1:Z7Lb0S5l+klDB31fvDQX8ss/FlKDxtlFlw3Oa8Ymbl8=
honnef.co/go/tools v0.0.0-20190102054323-c2f93a96b099/go.mod h1:rf3lG4BRIbNafJWhAfAdb/ePZxsR/4RtNHQocxwk9r4=

73
agent/main.go
View File

@ -6,13 +6,10 @@ import (
"errors"
"flag"
"fmt"
"github.com/up9inc/mizu/shared/kubernetes"
"io/ioutil"
v1 "k8s.io/api/core/v1"
"mizuserver/pkg/api"
"mizuserver/pkg/config"
"mizuserver/pkg/controllers"
"mizuserver/pkg/database"
"mizuserver/pkg/models"
"mizuserver/pkg/providers"
"mizuserver/pkg/routes"
@ -20,6 +17,7 @@ import (
"mizuserver/pkg/utils"
"net/http"
"os"
"os/exec"
"os/signal"
"path"
"path/filepath"
@ -28,10 +26,15 @@ import (
"syscall"
"time"
"github.com/up9inc/mizu/shared/kubernetes"
v1 "k8s.io/api/core/v1"
"github.com/antelman107/net-wait-go/wait"
"github.com/gin-contrib/static"
"github.com/gin-gonic/gin"
"github.com/gorilla/websocket"
"github.com/op/go-logging"
basenine "github.com/up9inc/basenine/client/go"
"github.com/up9inc/mizu/shared"
"github.com/up9inc/mizu/shared/logger"
"github.com/up9inc/mizu/tap"
@ -49,10 +52,12 @@ var harsDir = flag.String("hars-dir", "", "Directory to read hars from")
var extensions []*tapApi.Extension // global
var extensionsMap map[string]*tapApi.Extension // global
var startTime int64
const (
socketConnectionRetries = 10
socketConnectionRetries = 10
socketConnectionRetryDelay = time.Second * 2
socketHandshakeTimeout = time.Second * 2
socketHandshakeTimeout = time.Second * 2
)
func main() {
@ -109,7 +114,8 @@ func main() {
go pipeTapChannelToSocket(socketConnection, filteredOutputItemsChannel)
} else if *apiServerMode {
database.InitDataBase(config.Config.AgentDatabasePath)
startBasenineServer(shared.BasenineHost, shared.BaseninePort)
startTime = time.Now().UnixNano() / int64(time.Millisecond)
api.StartResolving(*namespace)
outputItemsChannel := make(chan *tapApi.OutputChannelItem)
@ -142,6 +148,53 @@ func main() {
logger.Log.Info("Exiting")
}
func startBasenineServer(host string, port string) {
cmd := exec.Command("basenine", "-addr", host, "-port", port, "-persistent")
cmd.Dir = config.Config.AgentDatabasePath
cmd.Stdout = os.Stdout
cmd.Stderr = os.Stderr
err := cmd.Start()
if err != nil {
logger.Log.Panicf("Failed starting Basenine: %v", err)
}
if !wait.New(
wait.WithProto("tcp"),
wait.WithWait(200*time.Millisecond),
wait.WithBreak(50*time.Millisecond),
wait.WithDeadline(5*time.Second),
wait.WithDebug(true),
).Do([]string{fmt.Sprintf("%s:%s", host, port)}) {
logger.Log.Panicf("Basenine is not available: %v", err)
}
// Make a channel to gracefully exit Basenine.
channel := make(chan os.Signal)
signal.Notify(channel, os.Interrupt, syscall.SIGTERM)
// Handle the channel.
go func() {
<-channel
cmd.Process.Signal(syscall.SIGTERM)
}()
// Limit the database size to default 200MB
err = basenine.Limit(host, port, config.Config.MaxDBSizeBytes)
if err != nil {
logger.Log.Panicf("Error while limiting database size: %v", err)
}
for _, extension := range extensions {
macros := extension.Dissector.Macros()
for macro, expanded := range macros {
err = basenine.Macro(host, port, macro, expanded)
if err != nil {
logger.Log.Panicf("Error while adding a macro: %v", err)
}
}
}
}
func loadExtensions() {
dir, _ := filepath.Abs(filepath.Dir(os.Args[0]))
extensionsDir := path.Join(dir, "./extensions/")
@ -200,7 +253,8 @@ func hostApi(socketHarOutputChannel chan<- *tapApi.OutputChannelItem) {
app.Use(static.ServeRoot("/", "./site"))
app.Use(CORSMiddleware()) // This has to be called after the static middleware, does not work if its called before
api.WebSocketRoutes(app, &eventHandlers)
api.WebSocketRoutes(app, &eventHandlers, startTime)
routes.QueryRoutes(app)
routes.EntriesRoutes(app)
routes.MetadataRoutes(app)
routes.StatusRoutes(app)
@ -361,7 +415,7 @@ func dialSocketWithRetry(socketAddress string, retryAmount int, retryDelay time.
socketConnection, _, err := dialer.Dial(socketAddress, nil)
if err != nil {
if i < retryAmount {
logger.Log.Infof("socket connection to %s failed: %v, retrying %d out of %d in %d seconds...", socketAddress, err, i, retryAmount, retryDelay / time.Second)
logger.Log.Infof("socket connection to %s failed: %v, retrying %d out of %d in %d seconds...", socketAddress, err, i, retryAmount, retryDelay/time.Second)
time.Sleep(retryDelay)
}
} else {
@ -371,8 +425,7 @@ func dialSocketWithRetry(socketAddress string, retryAmount int, retryDelay time.
return nil, lastErr
}
func startMizuTapperSyncer(ctx context.Context) (*kubernetes.MizuTapperSyncer, error){
func startMizuTapperSyncer(ctx context.Context) (*kubernetes.MizuTapperSyncer, error) {
provider, err := kubernetes.NewProviderInCluster()
if err != nil {
return nil, err

48
agent/pkg/api/main.go
View File

@ -5,7 +5,6 @@ import (
"context"
"encoding/json"
"fmt"
"mizuserver/pkg/database"
"mizuserver/pkg/holder"
"mizuserver/pkg/providers"
"os"
@ -14,15 +13,16 @@ import (
"strings"
"time"
"go.mongodb.org/mongo-driver/bson/primitive"
"github.com/google/martian/har"
"github.com/up9inc/mizu/shared"
"github.com/up9inc/mizu/shared/logger"
tapApi "github.com/up9inc/mizu/tap/api"
"mizuserver/pkg/models"
"mizuserver/pkg/resolver"
"mizuserver/pkg/utils"
basenine "github.com/up9inc/basenine/client/go"
)
var k8sResolver *resolver.Resolver
@ -99,6 +99,12 @@ func startReadingChannel(outputItems <-chan *tapApi.OutputChannelItem, extension
panic("Channel of captured messages is nil")
}
connection, err := basenine.NewConnection(shared.BasenineHost, shared.BaseninePort)
if err != nil {
panic(err)
}
connection.InsertMode()
disableOASValidation := false
ctx := context.Background()
doc, contractContent, router, err := loadOAS(ctx)
@ -112,13 +118,13 @@ func startReadingChannel(outputItems <-chan *tapApi.OutputChannelItem, extension
extension := extensionsMap[item.Protocol.Name]
resolvedSource, resolvedDestionation := resolveIP(item.ConnectionInfo)
mizuEntry := extension.Dissector.Analyze(item, primitive.NewObjectID().Hex(), resolvedSource, resolvedDestionation)
mizuEntry := extension.Dissector.Analyze(item, resolvedSource, resolvedDestionation)
baseEntry := extension.Dissector.Summarize(mizuEntry)
mizuEntry.EstimatedSizeBytes = getEstimatedEntrySizeBytes(mizuEntry)
mizuEntry.Base = baseEntry
if extension.Protocol.Name == "http" {
if !disableOASValidation {
var httpPair tapApi.HTTPRequestResponsePair
json.Unmarshal([]byte(mizuEntry.Entry), &httpPair)
json.Unmarshal([]byte(mizuEntry.HTTPPair), &httpPair)
contract := handleOAS(ctx, doc, router, httpPair.Request.Payload.RawRequest, httpPair.Response.Payload.RawResponse, contractContent)
baseEntry.ContractStatus = contract.Status
@ -128,18 +134,18 @@ func startReadingChannel(outputItems <-chan *tapApi.OutputChannelItem, extension
mizuEntry.ContractContent = contract.Content
}
var pair tapApi.RequestResponsePair
json.Unmarshal([]byte(mizuEntry.Entry), &pair)
harEntry, err := utils.NewEntry(&pair)
harEntry, err := utils.NewEntry(mizuEntry.Request, mizuEntry.Response, mizuEntry.StartTime, mizuEntry.ElapsedTime)
if err == nil {
rules, _, _ := models.RunValidationRulesState(*harEntry, mizuEntry.Service)
baseEntry.Rules = rules
}
}
database.CreateEntry(mizuEntry)
baseEntryBytes, _ := models.CreateBaseEntryWebSocketMessage(baseEntry)
BroadcastToBrowserClients(baseEntryBytes)
data, err := json.Marshal(mizuEntry)
if err != nil {
panic(err)
}
connection.SendText(string(data))
}
}
@ -171,21 +177,3 @@ func CheckIsServiceIP(address string) bool {
}
return k8sResolver.CheckIsServiceIP(address)
}
// gives a rough estimate of the size this will take up in the db, good enough for maintaining db size limit accurately
func getEstimatedEntrySizeBytes(mizuEntry *tapApi.MizuEntry) int {
sizeBytes := len(mizuEntry.Entry)
sizeBytes += len(mizuEntry.EntryId)
sizeBytes += len(mizuEntry.Service)
sizeBytes += len(mizuEntry.Url)
sizeBytes += len(mizuEntry.Method)
sizeBytes += len(mizuEntry.RequestSenderIp)
sizeBytes += len(mizuEntry.ResolvedDestination)
sizeBytes += len(mizuEntry.ResolvedSource)
sizeBytes += 8 // Status bytes (sqlite integer is always 8 bytes)
sizeBytes += 8 // Timestamp bytes
sizeBytes += 8 // SizeBytes bytes
sizeBytes += 1 // IsOutgoing bytes
return sizeBytes
}

101
agent/pkg/api/socket_routes.go
View File

@ -1,13 +1,18 @@
package api
import (
"encoding/json"
"errors"
"fmt"
"mizuserver/pkg/models"
"net/http"
"sync"
"time"
"github.com/gin-gonic/gin"
"github.com/gorilla/websocket"
basenine "github.com/up9inc/basenine/client/go"
"github.com/up9inc/mizu/shared"
"github.com/up9inc/mizu/shared/debounce"
"github.com/up9inc/mizu/shared/logger"
)
@ -39,17 +44,17 @@ func init() {
connectedWebsockets = make(map[int]*SocketConnection, 0)
}
func WebSocketRoutes(app *gin.Engine, eventHandlers EventHandlers) {
func WebSocketRoutes(app *gin.Engine, eventHandlers EventHandlers, startTime int64) {
app.GET("/ws", func(c *gin.Context) {
websocketHandler(c.Writer, c.Request, eventHandlers, false)
websocketHandler(c.Writer, c.Request, eventHandlers, false, startTime)
})
app.GET("/wsTapper", func(c *gin.Context) {
websocketHandler(c.Writer, c.Request, eventHandlers, true)
websocketHandler(c.Writer, c.Request, eventHandlers, true, startTime)
})
}
func websocketHandler(w http.ResponseWriter, r *http.Request, eventHandlers EventHandlers, isTapper bool) {
conn, err := websocketUpgrader.Upgrade(w, r, nil)
func websocketHandler(w http.ResponseWriter, r *http.Request, eventHandlers EventHandlers, isTapper bool, startTime int64) {
ws, err := websocketUpgrader.Upgrade(w, r, nil)
if err != nil {
logger.Log.Errorf("Failed to set websocket upgrade: %v", err)
return
@ -59,23 +64,103 @@ func websocketHandler(w http.ResponseWriter, r *http.Request, eventHandlers Even
connectedWebsocketIdCounter++
socketId := connectedWebsocketIdCounter
connectedWebsockets[socketId] = &SocketConnection{connection: conn, lock: &sync.Mutex{}, eventHandlers: eventHandlers, isTapper: isTapper}
connectedWebsockets[socketId] = &SocketConnection{connection: ws, lock: &sync.Mutex{}, eventHandlers: eventHandlers, isTapper: isTapper}
websocketIdsLock.Unlock()
var connection *basenine.Connection
var isQuerySet bool
// `!isTapper` means it's a connection from the web UI
if !isTapper {
connection, err = basenine.NewConnection(shared.BasenineHost, shared.BaseninePort)
if err != nil {
panic(err)
}
}
data := make(chan []byte)
meta := make(chan []byte)
defer func() {
data <- []byte(basenine.CloseChannel)
meta <- []byte(basenine.CloseChannel)
connection.Close()
socketCleanup(socketId, connectedWebsockets[socketId])
}()
eventHandlers.WebSocketConnect(socketId, isTapper)
startTimeBytes, _ := models.CreateWebsocketStartTimeMessage(startTime)
BroadcastToBrowserClients(startTimeBytes)
for {
_, msg, err := conn.ReadMessage()
_, msg, err := ws.ReadMessage()
if err != nil {
logger.Log.Errorf("Error reading message, socket id: %d, error: %v", socketId, err)
break
}
eventHandlers.WebSocketMessage(socketId, msg)
if !isTapper && !isQuerySet {
query := string(msg)
err = basenine.Validate(shared.BasenineHost, shared.BaseninePort, query)
if err != nil {
toastBytes, _ := models.CreateWebsocketToastMessage(&models.ToastMessage{
Type: "error",
AutoClose: 5000,
Text: fmt.Sprintf("Syntax error: %s", err.Error()),
})
BroadcastToBrowserClients(toastBytes)
break
}
isQuerySet = true
handleDataChannel := func(c *basenine.Connection, data chan []byte) {
for {
bytes := <-data
if string(bytes) == basenine.CloseChannel {
return
}
var dataMap map[string]interface{}
err = json.Unmarshal(bytes, &dataMap)
base := dataMap["base"].(map[string]interface{})
base["id"] = uint(dataMap["id"].(float64))
baseEntryBytes, _ := models.CreateBaseEntryWebSocketMessage(base)
BroadcastToBrowserClients(baseEntryBytes)
}
}
handleMetaChannel := func(c *basenine.Connection, meta chan []byte) {
for {
bytes := <-meta
if string(bytes) == basenine.CloseChannel {
return
}
var metadata *basenine.Metadata
err = json.Unmarshal(bytes, &metadata)
if err != nil {
logger.Log.Debugf("Error recieving metadata: %v\n", err.Error())
}
metadataBytes, _ := models.CreateWebsocketQueryMetadataMessage(metadata)
BroadcastToBrowserClients(metadataBytes)
}
}
go handleDataChannel(connection, data)
go handleMetaChannel(connection, meta)
connection.Query(query, data, meta)
} else {
eventHandlers.WebSocketMessage(socketId, msg)
}
}
}

89
agent/pkg/controllers/entries_controller.go
View File

@ -2,14 +2,17 @@ package controllers
import (
"encoding/json"
"fmt"
"github.com/gin-gonic/gin"
tapApi "github.com/up9inc/mizu/tap/api"
"mizuserver/pkg/database"
"mizuserver/pkg/models"
"mizuserver/pkg/utils"
"mizuserver/pkg/validation"
"net/http"
"strconv"
"github.com/gin-gonic/gin"
basenine "github.com/up9inc/basenine/client/go"
"github.com/up9inc/mizu/shared"
"github.com/up9inc/mizu/shared/logger"
tapApi "github.com/up9inc/mizu/tap/api"
)
var extensionsMap map[string]*tapApi.Extension // global
@ -18,68 +21,36 @@ func InitExtensionsMap(ref map[string]*tapApi.Extension) {
extensionsMap = ref
}
func GetEntries(c *gin.Context) {
entriesFilter := &models.EntriesFilter{}
if err := c.BindQuery(entriesFilter); err != nil {
c.JSON(http.StatusBadRequest, err)
}
err := validation.Validate(entriesFilter)
func Error(c *gin.Context, err error) bool {
if err != nil {
c.JSON(http.StatusBadRequest, err)
logger.Log.Errorf("Error getting entry: %v", err)
c.Error(err)
c.AbortWithStatusJSON(http.StatusInternalServerError, gin.H{"error": true, "msg": err.Error()})
return true // signal that there was an error and the caller should return
}
order := database.OperatorToOrderMapping[entriesFilter.Operator]
operatorSymbol := database.OperatorToSymbolMapping[entriesFilter.Operator]
var entries []tapApi.MizuEntry
database.GetEntriesTable().
Order(fmt.Sprintf("timestamp %s", order)).
Where(fmt.Sprintf("timestamp %s %v", operatorSymbol, entriesFilter.Timestamp)).
Limit(entriesFilter.Limit).
Find(&entries)
if len(entries) > 0 && order == database.OrderDesc {
// the entries always order from oldest to newest - we should reverse
utils.ReverseSlice(entries)
}
baseEntries := make([]tapApi.BaseEntryDetails, 0)
for _, entry := range entries {
baseEntryDetails := tapApi.BaseEntryDetails{}
if err := models.GetEntry(&entry, &baseEntryDetails); err != nil {
continue
}
var pair tapApi.RequestResponsePair
json.Unmarshal([]byte(entry.Entry), &pair)
harEntry, err := utils.NewEntry(&pair)
if err == nil {
rules, _, _ := models.RunValidationRulesState(*harEntry, entry.Service)
baseEntryDetails.Rules = rules
}
baseEntries = append(baseEntries, baseEntryDetails)
}
c.JSON(http.StatusOK, baseEntries)
return false // no error, can continue
}
func GetEntry(c *gin.Context) {
var entryData tapApi.MizuEntry
database.GetEntriesTable().
Where(map[string]string{"entryId": c.Param("entryId")}).
First(&entryData)
id, _ := strconv.Atoi(c.Param("id"))
var entry tapApi.MizuEntry
bytes, err := basenine.Single(shared.BasenineHost, shared.BaseninePort, id)
if Error(c, err) {
return // exit
}
err = json.Unmarshal(bytes, &entry)
if Error(c, err) {
return // exit
}
extension := extensionsMap[entryData.ProtocolName]
protocol, representation, bodySize, _ := extension.Dissector.Represent(&entryData)
extension := extensionsMap[entry.Protocol.Name]
protocol, representation, bodySize, _ := extension.Dissector.Represent(entry.Protocol, entry.Request, entry.Response)
var rules []map[string]interface{}
var isRulesEnabled bool
if entryData.ProtocolName == "http" {
var pair tapApi.RequestResponsePair
json.Unmarshal([]byte(entryData.Entry), &pair)
harEntry, _ := utils.NewEntry(&pair)
_, rulesMatched, _isRulesEnabled := models.RunValidationRulesState(*harEntry, entryData.Service)
if entry.Protocol.Name == "http" {
harEntry, _ := utils.NewEntry(entry.Request, entry.Response, entry.StartTime, entry.ElapsedTime)
_, rulesMatched, _isRulesEnabled := models.RunValidationRulesState(*harEntry, entry.Service)
isRulesEnabled = _isRulesEnabled
inrec, _ := json.Marshal(rulesMatched)
json.Unmarshal(inrec, &rules)
@ -89,7 +60,7 @@ func GetEntry(c *gin.Context) {
Protocol: protocol,
Representation: string(representation),
BodySize: bodySize,
Data: entryData,
Data: entry,
Rules: rules,
IsRulesEnabled: isRulesEnabled,
})

31
agent/pkg/controllers/query_controller.go Normal file
View File

@ -0,0 +1,31 @@
package controllers
import (
"net/http"
"github.com/gin-gonic/gin"
basenine "github.com/up9inc/basenine/client/go"
"github.com/up9inc/mizu/shared"
)
type ValidateResponse struct {
Valid bool `json:"valid"`
Message string `json:"message"`
}
func PostValidate(c *gin.Context) {
query := c.PostForm("query")
valid := true
message := ""
err := basenine.Validate(shared.BasenineHost, shared.BaseninePort, query)
if err != nil {
valid = false
message = err.Error()
}
c.JSON(http.StatusOK, ValidateResponse{
Valid: valid,
Message: message,
})
}

78
agent/pkg/database/main.go
View File

@ -1,78 +0,0 @@
package database
import (
"fmt"
"mizuserver/pkg/utils"
"time"
"gorm.io/driver/sqlite"
"gorm.io/gorm"
"gorm.io/gorm/logger"
tapApi "github.com/up9inc/mizu/tap/api"
)
const (
OrderDesc = "desc"
OrderAsc = "asc"
LT = "lt"
GT = "gt"
TimeFormat = "2006-01-02 15:04:05.000000000"
)
var (
DB *gorm.DB
IsDBLocked = false
OperatorToSymbolMapping = map[string]string{
LT: "<",
GT: ">",
}
OperatorToOrderMapping = map[string]string{
LT: OrderDesc,
GT: OrderAsc,
}
)
var DBPath string
func GetEntriesTable() *gorm.DB {
return DB.Table("mizu_entries")
}
func CreateEntry(entry *tapApi.MizuEntry) {
if IsDBLocked {
return
}
GetEntriesTable().Create(entry)
}
func InitDataBase(databasePath string) *gorm.DB {
DBPath = databasePath
DB, _ = gorm.Open(sqlite.Open(databasePath), &gorm.Config{
Logger: &utils.TruncatingLogger{LogLevel: logger.Warn, SlowThreshold: 500 * time.Millisecond},
})
_ = DB.AutoMigrate(&tapApi.MizuEntry{}) // this will ensure table is created
go StartEnforcingDatabaseSize()
return DB
}
func GetEntriesFromDb(timeFrom time.Time, timeTo time.Time, protocolName *string) []tapApi.MizuEntry {
order := OrderDesc
protocolNameCondition := "1 = 1"
if protocolName != nil {
protocolNameCondition = fmt.Sprintf("protocolName = '%s'", *protocolName)
}
var entries []tapApi.MizuEntry
GetEntriesTable().
Where(protocolNameCondition).
Where(fmt.Sprintf("created_at BETWEEN '%s' AND '%s'", timeFrom.Format(TimeFormat), timeTo.Format(TimeFormat))).
Order(fmt.Sprintf("timestamp %s", order)).
Find(&entries)
if len(entries) > 0 {
// the entries always order from oldest to newest so we should revers
utils.ReverseSlice(entries)
}
return entries
}

102
agent/pkg/database/size_enforcer.go
View File

@ -1,102 +0,0 @@
package database
import (
"mizuserver/pkg/config"
"os"
"time"
"github.com/fsnotify/fsnotify"
"github.com/up9inc/mizu/shared/debounce"
"github.com/up9inc/mizu/shared/logger"
"github.com/up9inc/mizu/shared/units"
tapApi "github.com/up9inc/mizu/tap/api"
)
const percentageOfMaxSizeBytesToPrune = 15
func StartEnforcingDatabaseSize() {
watcher, err := fsnotify.NewWatcher()
if err != nil {
logger.Log.Fatalf("Error creating filesystem watcher for db size enforcement: %v\n", err)
return
}
checkFileSizeDebouncer := debounce.NewDebouncer(5*time.Second, func() {
checkFileSize(config.Config.MaxDBSizeBytes)
})
go func() {
for {
select {
case event, ok := <-watcher.Events:
if !ok {
return // closed channel
}
if event.Op == fsnotify.Write {
checkFileSizeDebouncer.SetOn()
}
case err, ok := <-watcher.Errors:
if !ok {
return // closed channel
}
logger.Log.Errorf("filesystem watcher encountered error:%v", err)
}
}
}()
err = watcher.Add(DBPath)
if err != nil {
logger.Log.Fatalf("Error adding %s to filesystem watcher for db size enforcement: %v\n", DBPath, err)
}
}
func checkFileSize(maxSizeBytes int64) {
fileStat, err := os.Stat(DBPath)
if err != nil {
logger.Log.Errorf("Error checking %s file size: %v", DBPath, err)
} else {
if fileStat.Size() > maxSizeBytes {
pruneOldEntries(fileStat.Size())
}
}
}
func pruneOldEntries(currentFileSize int64) {
// sqlite locks the database while delete or VACUUM are running and sqlite is terrible at handling its own db lock while a lot of inserts are attempted, we prevent a significant bottleneck by handling the db lock ourselves here
IsDBLocked = true
defer func() { IsDBLocked = false }()
amountOfBytesToTrim := currentFileSize / (100 / percentageOfMaxSizeBytesToPrune)
rows, err := GetEntriesTable().Limit(10000).Order("id").Rows()
if err != nil {
logger.Log.Errorf("Error getting 10000 first db rows: %v", err)
return
}
entryIdsToRemove := make([]uint, 0)
bytesToBeRemoved := int64(0)
for rows.Next() {
if bytesToBeRemoved >= amountOfBytesToTrim {
break
}
var entry tapApi.MizuEntry
err = DB.ScanRows(rows, &entry)
if err != nil {
logger.Log.Errorf("Error scanning db row: %v", err)
continue
}
entryIdsToRemove = append(entryIdsToRemove, entry.ID)
bytesToBeRemoved += int64(entry.EstimatedSizeBytes)
}
if len(entryIdsToRemove) > 0 {
GetEntriesTable().Where(entryIdsToRemove).Delete(tapApi.MizuEntry{})
// VACUUM causes sqlite to shrink the db file after rows have been deleted, the db file will not shrink without this
DB.Exec("VACUUM")
logger.Log.Errorf("Removed %d rows and cleared %s", len(entryIdsToRemove), units.BytesToHumanReadable(bytesToBeRemoved))
} else {
logger.Log.Error("Found no rows to remove when pruning")
}
}

65
agent/pkg/models/models.go
View File

@ -2,12 +2,12 @@ package models
import (
"encoding/json"
"mizuserver/pkg/rules"
tapApi "github.com/up9inc/mizu/tap/api"
"mizuserver/pkg/rules"
"github.com/google/martian/har"
basenine "github.com/up9inc/basenine/client/go"
"github.com/up9inc/mizu/shared"
"github.com/up9inc/mizu/tap"
)
@ -16,15 +16,9 @@ func GetEntry(r *tapApi.MizuEntry, v tapApi.DataUnmarshaler) error {
return v.UnmarshalData(r)
}
type EntriesFilter struct {
Limit int `form:"limit" validate:"required,min=1,max=200"`
Operator string `form:"operator" validate:"required,oneof='lt' 'gt'"`
Timestamp int64 `form:"timestamp" validate:"required,min=1"`
}
type WebSocketEntryMessage struct {
*shared.WebSocketMessageMetadata
Data *tapApi.BaseEntryDetails `json:"data,omitempty"`
Data map[string]interface{} `json:"data,omitempty"`
}
type WebSocketTappedEntryMessage struct {
@ -42,7 +36,28 @@ type AuthStatus struct {
Model string `json:"model"`
}
func CreateBaseEntryWebSocketMessage(base *tapApi.BaseEntryDetails) ([]byte, error) {
type ToastMessage struct {
Type string `json:"type"`
AutoClose uint `json:"autoClose"`
Text string `json:"text"`
}
type WebSocketToastMessage struct {
*shared.WebSocketMessageMetadata
Data *ToastMessage `json:"data,omitempty"`
}
type WebSocketQueryMetadataMessage struct {
*shared.WebSocketMessageMetadata
Data *basenine.Metadata `json:"data,omitempty"`
}
type WebSocketStartTimeMessage struct {
*shared.WebSocketMessageMetadata
Data int64 `json:"data"`
}
func CreateBaseEntryWebSocketMessage(base map[string]interface{}) ([]byte, error) {
message := &WebSocketEntryMessage{
WebSocketMessageMetadata: &shared.WebSocketMessageMetadata{
MessageType: shared.WebSocketMessageTypeEntry,
@ -72,6 +87,36 @@ func CreateWebsocketOutboundLinkMessage(base *tap.OutboundLink) ([]byte, error)
return json.Marshal(message)
}
func CreateWebsocketToastMessage(base *ToastMessage) ([]byte, error) {
message := &WebSocketToastMessage{
WebSocketMessageMetadata: &shared.WebSocketMessageMetadata{
MessageType: shared.WebSocketMessageTypeToast,
},
Data: base,
}
return json.Marshal(message)
}
func CreateWebsocketQueryMetadataMessage(base *basenine.Metadata) ([]byte, error) {
message := &WebSocketQueryMetadataMessage{
WebSocketMessageMetadata: &shared.WebSocketMessageMetadata{
MessageType: shared.WebSocketMessageTypeQueryMetadata,
},
Data: base,
}
return json.Marshal(message)
}
func CreateWebsocketStartTimeMessage(base int64) ([]byte, error) {
message := &WebSocketStartTimeMessage{
WebSocketMessageMetadata: &shared.WebSocketMessageMetadata{
MessageType: shared.WebSocketMessageTypeStartTime,
},
Data: base,
}
return json.Marshal(message)
}
// ExtendedHAR is the top level object of a HAR log.
type ExtendedHAR struct {
Log *ExtendedLog `json:"log"`

6
agent/pkg/providers/stats_provider_test.go
View File

@ -2,16 +2,10 @@ package providers_test
import (
"fmt"
"mizuserver/pkg/config"
"mizuserver/pkg/database"
"mizuserver/pkg/providers"
"testing"
)
func init() {
database.InitDataBase(config.DefaultDatabasePath)
}
func TestNoEntryAddedCount(t *testing.T) {
entriesStats := providers.GetGeneralStats()

6
agent/pkg/routes/entries_routes.go
View File

@ -1,14 +1,14 @@
package routes
import (
"github.com/gin-gonic/gin"
"mizuserver/pkg/controllers"
"github.com/gin-gonic/gin"
)
// EntriesRoutes defines the group of har entries routes.
func EntriesRoutes(ginApp *gin.Engine) {
routeGroup := ginApp.Group("/entries")
routeGroup.GET("/", controllers.GetEntries) // get entries (base/thin entries)
routeGroup.GET("/:entryId", controllers.GetEntry) // get single (full) entry
routeGroup.GET("/:id", controllers.GetEntry) // get single (full) entry
}

13
agent/pkg/routes/query_routes.go Normal file
View File

@ -0,0 +1,13 @@
package routes
import (
"mizuserver/pkg/controllers"
"github.com/gin-gonic/gin"
)
func QueryRoutes(ginApp *gin.Engine) {
routeGroup := ginApp.Group("/query")
routeGroup.POST("/validate", controllers.PostValidate)
}

10
agent/pkg/sensitiveDataFiltering/consts.go
View File

@ -1,10 +0,0 @@
package sensitiveDataFiltering
const maskedFieldPlaceholderValue = "[REDACTED]"
//these values MUST be all lower case and contain no `-` or `_` characters
var personallyIdentifiableDataFields = []string{"token", "authorization", "authentication", "cookie", "userid", "password",
"username", "user", "key", "passcode", "pass", "auth", "authtoken", "jwt",
"bearer", "clientid", "clientsecret", "redirecturi", "phonenumber",
"zip", "zipcode", "address", "country", "firstname", "lastname",
"middlename", "fname", "lname", "birthdate"}

117
agent/pkg/up9/main.go
View File

@ -7,15 +7,16 @@ import (
"encoding/json"
"fmt"
"io/ioutil"
"mizuserver/pkg/database"
"mizuserver/pkg/utils"
"net/http"
"net/url"
"regexp"
"strings"
"sync"
"time"
"github.com/google/martian/har"
basenine "github.com/up9inc/basenine/client/go"
"github.com/up9inc/mizu/shared"
"github.com/up9inc/mizu/shared/logger"
tapApi "github.com/up9inc/mizu/tap/api"
@ -23,6 +24,7 @@ import (
const (
AnalyzeCheckSleepTime = 5 * time.Second
SentCountLogInterval = 100
)
type GuestToken struct {
@ -204,44 +206,62 @@ func syncEntriesImpl(token string, model string, envPrefix string, uploadInterva
analyzeInformation.AnalyzeDestination = envPrefix
analyzeInformation.SentCount = 0
sleepTime := time.Second * time.Duration(uploadIntervalSec)
// "http or grpc" filter indicates that we're only interested in HTTP and gRPC entries
query := "http or grpc"
var timeFrom time.Time
protocolFilter := "http"
logger.Log.Infof("Getting entries from the database\n")
for {
timeTo := time.Now()
logger.Log.Infof("Getting entries from %v, to %v\n", timeFrom.Format(time.RFC3339Nano), timeTo.Format(time.RFC3339Nano))
entriesArray := database.GetEntriesFromDb(timeFrom, timeTo, &protocolFilter)
var connection *basenine.Connection
var err error
connection, err = basenine.NewConnection(shared.BasenineHost, shared.BaseninePort)
if err != nil {
panic(err)
}
if len(entriesArray) > 0 {
result := make([]har.Entry, 0)
for _, data := range entriesArray {
var pair tapApi.RequestResponsePair
if err := json.Unmarshal([]byte(data.Entry), &pair); err != nil {
continue
}
harEntry, err := utils.NewEntry(&pair)
if err != nil {
continue
}
if data.ResolvedSource != "" {
harEntry.Request.Headers = append(harEntry.Request.Headers, har.Header{Name: "x-mizu-source", Value: data.ResolvedSource})
}
if data.ResolvedDestination != "" {
harEntry.Request.Headers = append(harEntry.Request.Headers, har.Header{Name: "x-mizu-destination", Value: data.ResolvedDestination})
harEntry.Request.URL = utils.SetHostname(harEntry.Request.URL, data.ResolvedDestination)
}
data := make(chan []byte)
meta := make(chan []byte)
// go's default marshal behavior is to encode []byte fields to base64, python's default unmarshal behavior is to not decode []byte fields from base64
if harEntry.Response.Content.Text, err = base64.StdEncoding.DecodeString(string(harEntry.Response.Content.Text)); err != nil {
continue
}
defer func() {
data <- []byte(basenine.CloseChannel)
meta <- []byte(basenine.CloseChannel)
connection.Close()
}()
result = append(result, *harEntry)
handleDataChannel := func(wg *sync.WaitGroup, connection *basenine.Connection, data chan []byte) {
defer wg.Done()
for {
dataBytes := <-data
if string(dataBytes) == basenine.CloseChannel {
return
}
logger.Log.Infof("About to upload %v entries\n", len(result))
var dataMap map[string]interface{}
err = json.Unmarshal(dataBytes, &dataMap)
result := make([]har.Entry, 0)
var entry tapApi.MizuEntry
if err := json.Unmarshal([]byte(dataBytes), &entry); err != nil {
continue
}
harEntry, err := utils.NewEntry(entry.Request, entry.Response, entry.StartTime, entry.ElapsedTime)
if err != nil {
continue
}
if entry.ResolvedSource != "" {
harEntry.Request.Headers = append(harEntry.Request.Headers, har.Header{Name: "x-mizu-source", Value: entry.ResolvedSource})
}
if entry.ResolvedDestination != "" {
harEntry.Request.Headers = append(harEntry.Request.Headers, har.Header{Name: "x-mizu-destination", Value: entry.ResolvedDestination})
harEntry.Request.URL = utils.SetHostname(harEntry.Request.URL, entry.ResolvedDestination)
}
// go's default marshal behavior is to encode []byte fields to base64, python's default unmarshal behavior is to not decode []byte fields from base64
if harEntry.Response.Content.Text, err = base64.StdEncoding.DecodeString(string(harEntry.Response.Content.Text)); err != nil {
continue
}
result = append(result, *harEntry)
body, jMarshalErr := json.Marshal(result)
if jMarshalErr != nil {
@ -273,18 +293,33 @@ func syncEntriesImpl(token string, model string, envPrefix string, uploadInterva
logger.Log.Info("Stopping sync entries")
logger.Log.Fatal(postErr)
}
analyzeInformation.SentCount += len(entriesArray)
logger.Log.Infof("Finish uploading %v entries to %s\n", len(entriesArray), GetTrafficDumpUrl(envPrefix, model))
analyzeInformation.SentCount += 1
logger.Log.Infof("Uploaded %v entries until now", analyzeInformation.SentCount)
} else {
logger.Log.Infof("Nothing to upload")
if analyzeInformation.SentCount%SentCountLogInterval == 0 {
logger.Log.Infof("Uploaded %v entries until now", analyzeInformation.SentCount)
}
}
logger.Log.Infof("Sleeping for %v...\n", sleepTime)
time.Sleep(sleepTime)
timeFrom = timeTo
}
handleMetaChannel := func(wg *sync.WaitGroup, connection *basenine.Connection, meta chan []byte) {
defer wg.Done()
for {
metaBytes := <-meta
if string(metaBytes) == basenine.CloseChannel {
return
}
}
}
var wg sync.WaitGroup
go handleDataChannel(&wg, connection, data)
go handleMetaChannel(&wg, connection, meta)
wg.Add(2)
connection.Query(query, data, meta)
wg.Wait()
}
func UpdateAnalyzeStatus(callback func(data []byte)) {

56
agent/pkg/utils/har.go
View File

@ -10,7 +10,6 @@ import (
"github.com/google/martian/har"
"github.com/up9inc/mizu/shared/logger"
"github.com/up9inc/mizu/tap/api"
)
// Keep it because we might want cookies in the future
@ -120,13 +119,11 @@ func BuildPostParams(rawParams []interface{}) []har.Param {
return params
}
func NewRequest(request *api.GenericMessage) (harRequest *har.Request, err error) {
reqDetails := request.Payload.(map[string]interface{})["details"].(map[string]interface{})
func NewRequest(request map[string]interface{}) (harRequest *har.Request, err error) {
headers, host, scheme, authority, path, _ := BuildHeaders(request["_headers"].([]interface{}))
cookies := make([]har.Cookie, 0) // BuildCookies(request["_cookies"].([]interface{}))
headers, host, scheme, authority, path, _ := BuildHeaders(reqDetails["headers"].([]interface{}))
cookies := make([]har.Cookie, 0) // BuildCookies(reqDetails["cookies"].([]interface{}))
postData, _ := reqDetails["postData"].(map[string]interface{})
postData, _ := request["postData"].(map[string]interface{})
mimeType, _ := postData["mimeType"]
if mimeType == nil || len(mimeType.(string)) == 0 {
mimeType = "text/html"
@ -138,7 +135,7 @@ func NewRequest(request *api.GenericMessage) (harRequest *har.Request, err error
}
queryString := make([]har.QueryString, 0)
for _, _qs := range reqDetails["queryString"].([]interface{}) {
for _, _qs := range request["_queryString"].([]interface{}) {
qs := _qs.(map[string]interface{})
queryString = append(queryString, har.QueryString{
Name: qs["name"].(string),
@ -146,7 +143,7 @@ func NewRequest(request *api.GenericMessage) (harRequest *har.Request, err error
})
}
url := fmt.Sprintf("http://%s%s", host, reqDetails["url"].(string))
url := fmt.Sprintf("http://%s%s", host, request["url"].(string))
if strings.HasPrefix(mimeType.(string), "application/grpc") {
url = fmt.Sprintf("%s://%s%s", scheme, authority, path)
}
@ -157,9 +154,9 @@ func NewRequest(request *api.GenericMessage) (harRequest *har.Request, err error
}
harRequest = &har.Request{
Method: reqDetails["method"].(string),
Method: request["method"].(string),
URL: url,
HTTPVersion: reqDetails["httpVersion"].(string),
HTTPVersion: request["httpVersion"].(string),
HeadersSize: -1,
BodySize: int64(bytes.NewBufferString(postDataText).Len()),
QueryString: queryString,
@ -175,13 +172,11 @@ func NewRequest(request *api.GenericMessage) (harRequest *har.Request, err error
return
}
func NewResponse(response *api.GenericMessage) (harResponse *har.Response, err error) {
resDetails := response.Payload.(map[string]interface{})["details"].(map[string]interface{})
func NewResponse(response map[string]interface{}) (harResponse *har.Response, err error) {
headers, _, _, _, _, _status := BuildHeaders(response["_headers"].([]interface{}))
cookies := make([]har.Cookie, 0) // BuildCookies(response["_cookies"].([]interface{}))
headers, _, _, _, _, _status := BuildHeaders(resDetails["headers"].([]interface{}))
cookies := make([]har.Cookie, 0) // BuildCookies(resDetails["cookies"].([]interface{}))
content, _ := resDetails["content"].(map[string]interface{})
content, _ := response["content"].(map[string]interface{})
mimeType, _ := content["mimeType"]
if mimeType == nil || len(mimeType.(string)) == 0 {
mimeType = "text/html"
@ -200,9 +195,11 @@ func NewResponse(response *api.GenericMessage) (harResponse *har.Response, err e
Size: int64(len(bodyText)),
}
status := int(resDetails["status"].(float64))
status := int(response["status"].(float64))
if strings.HasPrefix(mimeType.(string), "application/grpc") {
status, err = strconv.Atoi(_status)
if _status != "" {
status, err = strconv.Atoi(_status)
}
if err != nil {
logger.Log.Errorf("Failed converting status to int %s (%v,%+v)", err, err, err)
return nil, errors.New("failed converting response status to int for HAR")
@ -210,9 +207,9 @@ func NewResponse(response *api.GenericMessage) (harResponse *har.Response, err e
}
harResponse = &har.Response{
HTTPVersion: resDetails["httpVersion"].(string),
HTTPVersion: response["httpVersion"].(string),
Status: status,
StatusText: resDetails["statusText"].(string),
StatusText: response["statusText"].(string),
HeadersSize: -1,
BodySize: int64(bytes.NewBufferString(bodyText).Len()),
Headers: headers,
@ -222,34 +219,33 @@ func NewResponse(response *api.GenericMessage) (harResponse *har.Response, err e
return
}
func NewEntry(pair *api.RequestResponsePair) (*har.Entry, error) {
harRequest, err := NewRequest(&pair.Request)
func NewEntry(request map[string]interface{}, response map[string]interface{}, startTime time.Time, elapsedTime int64) (*har.Entry, error) {
harRequest, err := NewRequest(request)
if err != nil {
logger.Log.Errorf("Failed converting request to HAR %s (%v,%+v)", err, err, err)
return nil, errors.New("failed converting request to HAR")
}
harResponse, err := NewResponse(&pair.Response)
harResponse, err := NewResponse(response)
if err != nil {
logger.Log.Errorf("Failed converting response to HAR %s (%v,%+v)", err, err, err)
return nil, errors.New("failed converting response to HAR")
}
totalTime := pair.Response.CaptureTime.Sub(pair.Request.CaptureTime).Round(time.Millisecond).Milliseconds()
if totalTime < 1 {
totalTime = 1
if elapsedTime < 1 {
elapsedTime = 1
}
harEntry := har.Entry{
StartedDateTime: pair.Request.CaptureTime,
Time: totalTime,
StartedDateTime: startTime,
Time: elapsedTime,
Request: harRequest,
Response: harResponse,
Cache: &har.Cache{},
Timings: &har.Timings{
Send: -1,
Wait: -1,
Receive: totalTime,
Receive: elapsedTime,
},
}

60
agent/pkg/utils/truncating_logger.go
View File

@ -1,60 +0,0 @@
package utils
import (
"context"
"fmt"
"time"
loggerShared "github.com/up9inc/mizu/shared/logger"
"gorm.io/gorm/logger"
"gorm.io/gorm/utils"
)
// TruncatingLogger implements the gorm logger.Interface interface. Its purpose is to act as gorm's logger while truncating logs to a max of 50 characters to minimise the performance impact
type TruncatingLogger struct {
LogLevel logger.LogLevel
SlowThreshold time.Duration
}
func (truncatingLogger *TruncatingLogger) LogMode(logLevel logger.LogLevel) logger.Interface {
truncatingLogger.LogLevel = logLevel
return truncatingLogger
}
func (truncatingLogger *TruncatingLogger) Info(_ context.Context, message string, __ ...interface{}) {
if truncatingLogger.LogLevel < logger.Info {
return
}
loggerShared.Log.Errorf("gorm info: %.150s", message)
}
func (truncatingLogger *TruncatingLogger) Warn(_ context.Context, message string, __ ...interface{}) {
if truncatingLogger.LogLevel < logger.Warn {
return
}
loggerShared.Log.Errorf("gorm warning: %.150s", message)
}
func (truncatingLogger *TruncatingLogger) Error(_ context.Context, message string, __ ...interface{}) {
if truncatingLogger.LogLevel < logger.Error {
return
}
loggerShared.Log.Errorf("gorm error: %.150s", message)
}
func (truncatingLogger *TruncatingLogger) Trace(ctx context.Context, begin time.Time, fc func() (string, int64), err error) {
if truncatingLogger.LogLevel == logger.Silent {
return
}
elapsed := time.Since(begin)
if err != nil {
sql, rows := fc() // copied into every condition as this is a potentially heavy operation best done only when necessary
truncatingLogger.Error(ctx, fmt.Sprintf("Error in %s: %v - elapsed: %fs affected rows: %d, sql: %s", utils.FileWithLineNum(), err, elapsed.Seconds(), rows, sql))
} else if truncatingLogger.LogLevel >= logger.Warn && elapsed > truncatingLogger.SlowThreshold {
sql, rows := fc()
truncatingLogger.Warn(ctx, fmt.Sprintf("Slow sql query - elapse: %fs rows: %d, sql: %s", elapsed.Seconds(), rows, sql))
} else if truncatingLogger.LogLevel >= logger.Info {
sql, rows := fc()
truncatingLogger.Info(ctx, fmt.Sprintf("Sql query - elapse: %fs rows: %d, sql: %s", elapsed.Seconds(), rows, sql))
}
}

7
cli/config/config.go
View File

@ -3,14 +3,15 @@ package config
import (
"errors"
"fmt"
"github.com/up9inc/mizu/tap/api"
"io/ioutil"
"k8s.io/apimachinery/pkg/util/json"
"os"
"reflect"
"strconv"
"strings"
"github.com/up9inc/mizu/tap/api"
"k8s.io/apimachinery/pkg/util/json"
"github.com/up9inc/mizu/shared"
"github.com/up9inc/mizu/shared/logger"
@ -396,7 +397,7 @@ func getMizuAgentConfig(targetNamespaces []string, mizuApiFilteringOptions *api.
TapperResources: Config.Tap.TapperResources,
MizuResourcesNamespace: Config.MizuResourcesNamespace,
MizuApiFilteringOptions: *mizuApiFilteringOptions,
AgentDatabasePath: fmt.Sprintf("%s%s", shared.DataDirPath, "entries.db"),
AgentDatabasePath: shared.DataDirPath,
}
return &config, nil
}

12
debug.Dockerfile
View File

@ -12,7 +12,7 @@ FROM golang:1.16-alpine AS builder
# Set necessary environment variables needed for our image.
ENV CGO_ENABLED=1 GOOS=linux GOARCH=amd64
RUN apk add libpcap-dev gcc g++ make bash
RUN apk add libpcap-dev gcc g++ make bash perl-utils
# Move to agent working directory (/agent-build).
WORKDIR /app/agent-build
@ -23,7 +23,7 @@ COPY tap/go.mod tap/go.mod ../tap/
COPY tap/api/go.* ../tap/api/
RUN go mod download
# cheap trick to make the build faster (As long as go.mod wasn't changes)
RUN go list -f '{{.Path}}@{{.Version}}' -m all | sed 1d | grep -e 'go-cache' -e 'sqlite' | xargs go get
RUN go list -f '{{.Path}}@{{.Version}}' -m all | sed 1d | grep -e 'go-cache' | xargs go get
ARG COMMIT_HASH
ARG GIT_BRANCH
@ -36,6 +36,12 @@ COPY tap ../tap
COPY agent .
RUN go build -gcflags="all=-N -l" -o mizuagent .
# Download Basenine executable, verify the sha1sum and move it to a directory in $PATH
ADD https://github.com/up9inc/basenine/releases/download/v0.2.6/basenine_linux_amd64 ./basenine_linux_amd64
ADD https://github.com/up9inc/basenine/releases/download/v0.2.6/basenine_linux_amd64.sha256 ./basenine_linux_amd64.sha256
RUN shasum -a 256 -c basenine_linux_amd64.sha256
RUN chmod +x ./basenine_linux_amd64
COPY devops/build_extensions_debug.sh ..
RUN cd .. && /bin/bash build_extensions_debug.sh
@ -43,10 +49,12 @@ RUN cd .. && /bin/bash build_extensions_debug.sh
FROM golang:1.16-alpine
RUN apk add bash libpcap-dev tcpdump
WORKDIR /app
# Copy binary and config files from /build to root folder of scratch container.
COPY --from=builder ["/app/agent-build/mizuagent", "."]
COPY --from=builder ["/app/agent-build/basenine_linux_amd64", "/usr/local/bin/basenine"]
COPY --from=builder ["/app/agent/build/extensions", "extensions"]
COPY --from=site-build ["/app/ui-build/build", "site"]

2
shared/consts.go
View File

@ -14,4 +14,6 @@ const (
GoGCEnvVar = "GOGC"
DefaultApiServerPort = 8899
DebugModeEnvVar = "MIZU_DEBUG"
BasenineHost = "localhost"
BaseninePort = "9099"
)

3
shared/models.go
View File

@ -17,6 +17,9 @@ const (
WebSocketMessageTypeUpdateStatus WebSocketMessageType = "status"
WebSocketMessageTypeAnalyzeStatus WebSocketMessageType = "analyzeStatus"
WebsocketMessageTypeOutboundLink WebSocketMessageType = "outboundLink"
WebSocketMessageTypeToast WebSocketMessageType = "toast"
WebSocketMessageTypeQueryMetadata WebSocketMessageType = "queryMetadata"
WebSocketMessageTypeStartTime WebSocketMessageType = "startTime"
)
type Resources struct {

108
tap/api/api.go
View File

@ -18,7 +18,8 @@ import (
type Protocol struct {
Name string `json:"name"`
LongName string `json:"longName"`
Abbreviation string `json:"abbreviation"`
Abbreviation string `json:"abbr"`
Macro string `json:"macro"`
Version string `json:"version"`
BackgroundColor string `json:"backgroundColor"`
ForegroundColor string `json:"foregroundColor"`
@ -28,6 +29,12 @@ type Protocol struct {
Priority uint8 `json:"priority"`
}
type TCP struct {
IP string `json:"ip"`
Port string `json:"port"`
Name string `json:"name"`
}
type Extension struct {
Protocol *Protocol
Path string
@ -74,6 +81,7 @@ type OutputChannelItem struct {
Timestamp int64
ConnectionInfo *ConnectionInfo
Pair *RequestResponsePair
Summary *BaseEntryDetails
}
type SuperTimer struct {
@ -89,9 +97,10 @@ type Dissector interface {
Register(*Extension)
Ping()
Dissect(b *bufio.Reader, isClient bool, tcpID *TcpID, counterPair *CounterPair, superTimer *SuperTimer, superIdentifier *SuperIdentifier, emitter Emitter, options *TrafficFilteringOptions) error
Analyze(item *OutputChannelItem, entryId string, resolvedSource string, resolvedDestination string) *MizuEntry
Analyze(item *OutputChannelItem, resolvedSource string, resolvedDestination string) *MizuEntry
Summarize(entry *MizuEntry) *BaseEntryDetails
Represent(entry *MizuEntry) (protocol Protocol, object []byte, bodySize int64, err error)
Represent(pIn Protocol, request map[string]interface{}, response map[string]interface{}) (pOut Protocol, object []byte, bodySize int64, err error)
Macros() map[string]string
}
type Emitting struct {
@ -109,39 +118,36 @@ func (e *Emitting) Emit(item *OutputChannelItem) {
}
type MizuEntry struct {
ID uint `gorm:"primarykey"`
CreatedAt time.Time
UpdatedAt time.Time
ProtocolName string `json:"protocolName" gorm:"column:protocolName"`
ProtocolLongName string `json:"protocolLongName" gorm:"column:protocolLongName"`
ProtocolAbbreviation string `json:"protocolAbbreviation" gorm:"column:protocolAbbreviation"`
ProtocolVersion string `json:"protocolVersion" gorm:"column:protocolVersion"`
ProtocolBackgroundColor string `json:"protocolBackgroundColor" gorm:"column:protocolBackgroundColor"`
ProtocolForegroundColor string `json:"protocolForegroundColor" gorm:"column:protocolForegroundColor"`
ProtocolFontSize int8 `json:"protocolFontSize" gorm:"column:protocolFontSize"`
ProtocolReferenceLink string `json:"protocolReferenceLink" gorm:"column:protocolReferenceLink"`
Entry string `json:"entry,omitempty" gorm:"column:entry"`
EntryId string `json:"entryId" gorm:"column:entryId"`
Url string `json:"url" gorm:"column:url"`
Method string `json:"method" gorm:"column:method"`
Status int `json:"status" gorm:"column:status"`
RequestSenderIp string `json:"requestSenderIp" gorm:"column:requestSenderIp"`
Service string `json:"service" gorm:"column:service"`
Timestamp int64 `json:"timestamp" gorm:"column:timestamp"`
ElapsedTime int64 `json:"elapsedTime" gorm:"column:elapsedTime"`
Path string `json:"path" gorm:"column:path"`
ResolvedSource string `json:"resolvedSource,omitempty" gorm:"column:resolvedSource"`
ResolvedDestination string `json:"resolvedDestination,omitempty" gorm:"column:resolvedDestination"`
SourceIp string `json:"sourceIp,omitempty" gorm:"column:sourceIp"`
DestinationIp string `json:"destinationIp,omitempty" gorm:"column:destinationIp"`
SourcePort string `json:"sourcePort,omitempty" gorm:"column:sourcePort"`
DestinationPort string `json:"destinationPort,omitempty" gorm:"column:destinationPort"`
IsOutgoing bool `json:"isOutgoing,omitempty" gorm:"column:isOutgoing"`
ContractStatus ContractStatus `json:"contractStatus,omitempty" gorm:"column:contractStatus"`
ContractRequestReason string `json:"contractRequestReason,omitempty" gorm:"column:contractRequestReason"`
ContractResponseReason string `json:"contractResponseReason,omitempty" gorm:"column:contractResponseReason"`
ContractContent string `json:"contractContent,omitempty" gorm:"column:contractContent"`
EstimatedSizeBytes int `json:"-" gorm:"column:estimatedSizeBytes"`
Id uint `json:"id"`
Protocol Protocol `json:"proto"`
Source *TCP `json:"src"`
Destination *TCP `json:"dst"`
Outgoing bool `json:"outgoing"`
Timestamp int64 `json:"timestamp"`
StartTime time.Time `json:"startTime"`
Request map[string]interface{} `json:"request"`
Response map[string]interface{} `json:"response"`
Base *BaseEntryDetails `json:"base"`
Summary string `json:"summary"`
Url string `json:"url"`
Method string `json:"method"`
Status int `json:"status"`
RequestSenderIp string `json:"requestSenderIp"`
Service string `json:"service"`
ElapsedTime int64 `json:"elapsedTime"`
Path string `json:"path"`
ResolvedSource string `json:"resolvedSource,omitempty"`
ResolvedDestination string `json:"resolvedDestination,omitempty"`
SourceIp string `json:"sourceIp,omitempty"`
DestinationIp string `json:"destinationIp,omitempty"`
SourcePort string `json:"sourcePort,omitempty"`
DestinationPort string `json:"destinationPort,omitempty"`
IsOutgoing bool `json:"isOutgoing,omitempty"`
ContractStatus ContractStatus `json:"contractStatus,omitempty"`
ContractRequestReason string `json:"contractRequestReason,omitempty"`
ContractResponseReason string `json:"contractResponseReason,omitempty"`
ContractContent string `json:"contractContent,omitempty"`
HTTPPair string `json:"httpPair,omitempty"`
}
type MizuEntryWrapper struct {
@ -154,7 +160,7 @@ type MizuEntryWrapper struct {
}
type BaseEntryDetails struct {
Id string `json:"id,omitempty"`
Id uint `json:"id"`
Protocol Protocol `json:"protocol,omitempty"`
Url string `json:"url,omitempty"`
RequestSenderIp string `json:"requestSenderIp,omitempty"`
@ -194,17 +200,8 @@ type DataUnmarshaler interface {
}
func (bed *BaseEntryDetails) UnmarshalData(entry *MizuEntry) error {
bed.Protocol = Protocol{
Name: entry.ProtocolName,
LongName: entry.ProtocolLongName,
Abbreviation: entry.ProtocolAbbreviation,
Version: entry.ProtocolVersion,
BackgroundColor: entry.ProtocolBackgroundColor,
ForegroundColor: entry.ProtocolForegroundColor,
FontSize: entry.ProtocolFontSize,
ReferenceLink: entry.ProtocolReferenceLink,
}
bed.Id = entry.EntryId
bed.Protocol = entry.Protocol
bed.Id = entry.Id
bed.Url = entry.Url
bed.RequestSenderIp = entry.RequestSenderIp
bed.Service = entry.Service
@ -228,6 +225,21 @@ const (
BODY string = "body"
)
type SectionData struct {
Type string `json:"type"`
Title string `json:"title"`
Data string `json:"data"`
Encoding string `json:"encoding,omitempty"`
MimeType string `json:"mimeType,omitempty"`
Selector string `json:"selector,omitempty"`
}
type TableData struct {
Name string `json:"name"`
Value interface{} `json:"value"`
Selector string `json:"selector"`
}
const (
TypeHttpRequest = iota
TypeHttpResponse

591
tap/extensions/amqp/helpers.go
View File

@ -131,97 +131,109 @@ func representProperties(properties map[string]interface{}, rep []interface{}) (
userId := ""
appId := ""
if properties["ContentType"] != nil {
contentType = properties["ContentType"].(string)
if properties["contentType"] != nil {
contentType = properties["contentType"].(string)
}
if properties["ContentEncoding"] != nil {
contentEncoding = properties["ContentEncoding"].(string)
if properties["contentEncoding"] != nil {
contentEncoding = properties["contentEncoding"].(string)
}
if properties["Delivery Mode"] != nil {
deliveryMode = fmt.Sprintf("%g", properties["DeliveryMode"].(float64))
if properties["deliveryMode"] != nil {
deliveryMode = fmt.Sprintf("%g", properties["deliveryMode"].(float64))
}
if properties["Priority"] != nil {
priority = fmt.Sprintf("%g", properties["Priority"].(float64))
if properties["priority"] != nil {
priority = fmt.Sprintf("%g", properties["priority"].(float64))
}
if properties["CorrelationId"] != nil {
correlationId = properties["CorrelationId"].(string)
if properties["correlationId"] != nil {
correlationId = properties["correlationId"].(string)
}
if properties["ReplyTo"] != nil {
replyTo = properties["ReplyTo"].(string)
if properties["replyTo"] != nil {
replyTo = properties["replyTo"].(string)
}
if properties["Expiration"] != nil {
expiration = properties["Expiration"].(string)
if properties["expiration"] != nil {
expiration = properties["expiration"].(string)
}
if properties["MessageId"] != nil {
messageId = properties["MessageId"].(string)
if properties["messageId"] != nil {
messageId = properties["messageId"].(string)
}
if properties["Timestamp"] != nil {
timestamp = properties["Timestamp"].(string)
if properties["timestamp"] != nil {
timestamp = properties["timestamp"].(string)
}
if properties["Type"] != nil {
_type = properties["Type"].(string)
if properties["type"] != nil {
_type = properties["type"].(string)
}
if properties["UserId"] != nil {
userId = properties["UserId"].(string)
if properties["userId"] != nil {
userId = properties["userId"].(string)
}
if properties["AppId"] != nil {
appId = properties["AppId"].(string)
if properties["appId"] != nil {
appId = properties["appId"].(string)
}
props, _ := json.Marshal([]map[string]string{
props, _ := json.Marshal([]api.TableData{
{
"name": "Content Type",
"value": contentType,
Name: "Content Type",
Value: contentType,
Selector: `request.properties.contentType`,
},
{
"name": "Content Encoding",
"value": contentEncoding,
Name: "Content Encoding",
Value: contentEncoding,
Selector: `request.properties.contentEncoding`,
},
{
"name": "Delivery Mode",
"value": deliveryMode,
Name: "Delivery Mode",
Value: deliveryMode,
Selector: `request.properties.deliveryMode`,
},
{
"name": "Priority",
"value": priority,
Name: "Priority",
Value: priority,
Selector: `request.properties.priority`,
},
{
"name": "Correlation ID",
"value": correlationId,
Name: "Correlation ID",
Value: correlationId,
Selector: `request.properties.correlationId`,
},
{
"name": "Reply To",
"value": replyTo,
Name: "Reply To",
Value: replyTo,
Selector: `request.properties.replyTo`,
},
{
"name": "Expiration",
"value": expiration,
Name: "Expiration",
Value: expiration,
Selector: `request.properties.expiration`,
},
{
"name": "Message ID",
"value": messageId,
Name: "Message ID",
Value: messageId,
Selector: `request.properties.messageId`,
},
{
"name": "Timestamp",
"value": timestamp,
Name: "Timestamp",
Value: timestamp,
Selector: `request.properties.timestamp`,
},
{
"name": "Type",
"value": _type,
Name: "Type",
Value: _type,
Selector: `request.properties.type`,
},
{
"name": "User ID",
"value": userId,
Name: "User ID",
Value: userId,
Selector: `request.properties.userId`,
},
{
"name": "App ID",
"value": appId,
Name: "App ID",
Value: appId,
Selector: `request.properties.appId`,
},
})
rep = append(rep, map[string]string{
"type": api.TABLE,
"title": "Properties",
"data": string(props),
rep = append(rep, api.SectionData{
Type: api.TABLE,
Title: "Properties",
Data: string(props),
})
return rep, contentType, contentEncoding
@ -230,56 +242,62 @@ func representProperties(properties map[string]interface{}, rep []interface{}) (
func representBasicPublish(event map[string]interface{}) []interface{} {
rep := make([]interface{}, 0)
details, _ := json.Marshal([]map[string]string{
details, _ := json.Marshal([]api.TableData{
{
"name": "Exchange",
"value": event["Exchange"].(string),
Name: "Exchange",
Value: event["exchange"].(string),
Selector: `request.exchange`,
},
{
"name": "Routing Key",
"value": event["RoutingKey"].(string),
Name: "Routing Key",
Value: event["routingKey"].(string),
Selector: `request.routingKey`,
},
{
"name": "Mandatory",
"value": strconv.FormatBool(event["Mandatory"].(bool)),
Name: "Mandatory",
Value: strconv.FormatBool(event["mandatory"].(bool)),
Selector: `request.mandatory`,
},
{
"name": "Immediate",
"value": strconv.FormatBool(event["Immediate"].(bool)),
Name: "Immediate",
Value: strconv.FormatBool(event["immediate"].(bool)),
Selector: `request.immediate`,
},
})
rep = append(rep, map[string]string{
"type": api.TABLE,
"title": "Details",
"data": string(details),
rep = append(rep, api.SectionData{
Type: api.TABLE,
Title: "Details",
Data: string(details),
})
properties := event["Properties"].(map[string]interface{})
properties := event["properties"].(map[string]interface{})
rep, contentType, _ := representProperties(properties, rep)
if properties["Headers"] != nil {
headers := make([]map[string]string, 0)
for name, value := range properties["Headers"].(map[string]interface{}) {
headers = append(headers, map[string]string{
"name": name,
"value": value.(string),
if properties["headers"] != nil {
headers := make([]api.TableData, 0)
for name, value := range properties["headers"].(map[string]interface{}) {
headers = append(headers, api.TableData{
Name: name,
Value: value.(string),
Selector: fmt.Sprintf(`request.properties.headers["%s"]`, name),
})
}
headersMarshaled, _ := json.Marshal(headers)
rep = append(rep, map[string]string{
"type": api.TABLE,
"title": "Headers",
"data": string(headersMarshaled),
rep = append(rep, api.SectionData{
Type: api.TABLE,
Title: "Headers",
Data: string(headersMarshaled),
})
}
if event["Body"] != nil {
rep = append(rep, map[string]string{
"type": api.BODY,
"title": "Body",
"encoding": "base64",
"mime_type": contentType,
"data": event["Body"].(string),
if event["body"] != nil {
rep = append(rep, api.SectionData{
Type: api.BODY,
Title: "Body",
Encoding: "base64",
MimeType: contentType,
Data: event["body"].(string),
Selector: `request.body`,
})
}
@ -293,70 +311,77 @@ func representBasicDeliver(event map[string]interface{}) []interface{} {
deliveryTag := ""
redelivered := ""
if event["ConsumerTag"] != nil {
consumerTag = event["ConsumerTag"].(string)
if event["consumerTag"] != nil {
consumerTag = event["consumerTag"].(string)
}
if event["DeliveryTag"] != nil {
deliveryTag = fmt.Sprintf("%g", event["DeliveryTag"].(float64))
if event["deliveryTag"] != nil {
deliveryTag = fmt.Sprintf("%g", event["deliveryTag"].(float64))
}
if event["Redelivered"] != nil {
redelivered = strconv.FormatBool(event["Redelivered"].(bool))
if event["redelivered"] != nil {
redelivered = strconv.FormatBool(event["redelivered"].(bool))
}
details, _ := json.Marshal([]map[string]string{
details, _ := json.Marshal([]api.TableData{
{
"name": "Consumer Tag",
"value": consumerTag,
Name: "Consumer Tag",
Value: consumerTag,
Selector: `request.consumerTag`,
},
{
"name": "Delivery Tag",
"value": deliveryTag,
Name: "Delivery Tag",
Value: deliveryTag,
Selector: `request.deliveryTag`,
},
{
"name": "Redelivered",
"value": redelivered,
Name: "Redelivered",
Value: redelivered,
Selector: `request.redelivered`,
},
{
"name": "Exchange",
"value": event["Exchange"].(string),
Name: "Exchange",
Value: event["exchange"].(string),
Selector: `request.exchange`,
},
{
"name": "Routing Key",
"value": event["RoutingKey"].(string),
Name: "Routing Key",
Value: event["routingKey"].(string),
Selector: `request.routingKey`,
},
})
rep = append(rep, map[string]string{
"type": api.TABLE,
"title": "Details",
"data": string(details),
rep = append(rep, api.SectionData{
Type: api.TABLE,
Title: "Details",
Data: string(details),
})
properties := event["Properties"].(map[string]interface{})
properties := event["properties"].(map[string]interface{})
rep, contentType, _ := representProperties(properties, rep)
if properties["Headers"] != nil {
headers := make([]map[string]string, 0)
for name, value := range properties["Headers"].(map[string]interface{}) {
headers = append(headers, map[string]string{
"name": name,
"value": value.(string),
if properties["headers"] != nil {
headers := make([]api.TableData, 0)
for name, value := range properties["headers"].(map[string]interface{}) {
headers = append(headers, api.TableData{
Name: name,
Value: value.(string),
Selector: fmt.Sprintf(`request.properties.headers["%s"]`, name),
})
}
headersMarshaled, _ := json.Marshal(headers)
rep = append(rep, map[string]string{
"type": api.TABLE,
"title": "Headers",
"data": string(headersMarshaled),
rep = append(rep, api.SectionData{
Type: api.TABLE,
Title: "Headers",
Data: string(headersMarshaled),
})
}
if event["Body"] != nil {
rep = append(rep, map[string]string{
"type": api.BODY,
"title": "Body",
"encoding": "base64",
"mime_type": contentType,
"data": event["Body"].(string),
if event["body"] != nil {
rep = append(rep, api.SectionData{
Type: api.BODY,
Title: "Body",
Encoding: "base64",
MimeType: contentType,
Data: event["body"].(string),
Selector: `request.body`,
})
}
@ -366,51 +391,58 @@ func representBasicDeliver(event map[string]interface{}) []interface{} {
func representQueueDeclare(event map[string]interface{}) []interface{} {
rep := make([]interface{}, 0)
details, _ := json.Marshal([]map[string]string{
details, _ := json.Marshal([]api.TableData{
{
"name": "Queue",
"value": event["Queue"].(string),
Name: "Queue",
Value: event["queue"].(string),
Selector: `request.queue`,
},
{
"name": "Passive",
"value": strconv.FormatBool(event["Passive"].(bool)),
Name: "Passive",
Value: strconv.FormatBool(event["passive"].(bool)),
Selector: `request.queue`,
},
{
"name": "Durable",
"value": strconv.FormatBool(event["Durable"].(bool)),
Name: "Durable",
Value: strconv.FormatBool(event["durable"].(bool)),
Selector: `request.durable`,
},
{
"name": "Exclusive",
"value": strconv.FormatBool(event["Exclusive"].(bool)),
Name: "Exclusive",
Value: strconv.FormatBool(event["exclusive"].(bool)),
Selector: `request.exclusive`,
},
{
"name": "Auto Delete",
"value": strconv.FormatBool(event["AutoDelete"].(bool)),
Name: "Auto Delete",
Value: strconv.FormatBool(event["autoDelete"].(bool)),
Selector: `request.autoDelete`,
},
{
"name": "NoWait",
"value": strconv.FormatBool(event["NoWait"].(bool)),
Name: "NoWait",
Value: strconv.FormatBool(event["noWait"].(bool)),
Selector: `request.noWait`,
},
})
rep = append(rep, map[string]string{
"type": api.TABLE,
"title": "Details",
"data": string(details),
rep = append(rep, api.SectionData{
Type: api.TABLE,
Title: "Details",
Data: string(details),
})
if event["Arguments"] != nil {
headers := make([]map[string]string, 0)
for name, value := range event["Arguments"].(map[string]interface{}) {
headers = append(headers, map[string]string{
"name": name,
"value": value.(string),
if event["arguments"] != nil {
headers := make([]api.TableData, 0)
for name, value := range event["arguments"].(map[string]interface{}) {
headers = append(headers, api.TableData{
Name: name,
Value: value.(string),
Selector: fmt.Sprintf(`request.arguments["%s"]`, name),
})
}
headersMarshaled, _ := json.Marshal(headers)
rep = append(rep, map[string]string{
"type": api.TABLE,
"title": "Arguments",
"data": string(headersMarshaled),
rep = append(rep, api.SectionData{
Type: api.TABLE,
Title: "Arguments",
Data: string(headersMarshaled),
})
}
@ -420,55 +452,63 @@ func representQueueDeclare(event map[string]interface{}) []interface{} {
func representExchangeDeclare(event map[string]interface{}) []interface{} {
rep := make([]interface{}, 0)
details, _ := json.Marshal([]map[string]string{
details, _ := json.Marshal([]api.TableData{
{
"name": "Exchange",
"value": event["Exchange"].(string),
Name: "Exchange",
Value: event["exchange"].(string),
Selector: `request.exchange`,
},
{
"name": "Type",
"value": event["Type"].(string),
Name: "Type",
Value: event["type"].(string),
Selector: `request.type`,
},
{
"name": "Passive",
"value": strconv.FormatBool(event["Passive"].(bool)),
Name: "Passive",
Value: strconv.FormatBool(event["passive"].(bool)),
Selector: `request.passive`,
},
{
"name": "Durable",
"value": strconv.FormatBool(event["Durable"].(bool)),
Name: "Durable",
Value: strconv.FormatBool(event["durable"].(bool)),
Selector: `request.durable`,
},
{
"name": "Auto Delete",
"value": strconv.FormatBool(event["AutoDelete"].(bool)),
Name: "Auto Delete",
Value: strconv.FormatBool(event["autoDelete"].(bool)),
Selector: `request.autoDelete`,
},
{
"name": "Internal",
"value": strconv.FormatBool(event["Internal"].(bool)),
Name: "Internal",
Value: strconv.FormatBool(event["internal"].(bool)),
Selector: `request.internal`,
},
{
"name": "NoWait",
"value": strconv.FormatBool(event["NoWait"].(bool)),
Name: "NoWait",
Value: strconv.FormatBool(event["noWait"].(bool)),
Selector: `request.noWait`,
},
})
rep = append(rep, map[string]string{
"type": api.TABLE,
"title": "Details",
"data": string(details),
rep = append(rep, api.SectionData{
Type: api.TABLE,
Title: "Details",
Data: string(details),
})
if event["Arguments"] != nil {
headers := make([]map[string]string, 0)
for name, value := range event["Arguments"].(map[string]interface{}) {
headers = append(headers, map[string]string{
"name": name,
"value": value.(string),
if event["arguments"] != nil {
headers := make([]api.TableData, 0)
for name, value := range event["arguments"].(map[string]interface{}) {
headers = append(headers, api.TableData{
Name: name,
Value: value.(string),
Selector: fmt.Sprintf(`request.arguments["%s"]`, name),
})
}
headersMarshaled, _ := json.Marshal(headers)
rep = append(rep, map[string]string{
"type": api.TABLE,
"title": "Arguments",
"data": string(headersMarshaled),
rep = append(rep, api.SectionData{
Type: api.TABLE,
Title: "Arguments",
Data: string(headersMarshaled),
})
}
@ -478,33 +518,37 @@ func representExchangeDeclare(event map[string]interface{}) []interface{} {
func representConnectionStart(event map[string]interface{}) []interface{} {
rep := make([]interface{}, 0)
details, _ := json.Marshal([]map[string]string{
details, _ := json.Marshal([]api.TableData{
{
"name": "Version Major",
"value": fmt.Sprintf("%g", event["VersionMajor"].(float64)),
Name: "Version Major",
Value: fmt.Sprintf("%g", event["versionMajor"].(float64)),
Selector: `request.versionMajor`,
},
{
"name": "Version Minor",
"value": fmt.Sprintf("%g", event["VersionMinor"].(float64)),
Name: "Version Minor",
Value: fmt.Sprintf("%g", event["versionMinor"].(float64)),
Selector: `request.versionMinor`,
},
{
"name": "Mechanisms",
"value": event["Mechanisms"].(string),
Name: "Mechanisms",
Value: event["mechanisms"].(string),
Selector: `request.mechanisms`,
},
{
"name": "Locales",
"value": event["Locales"].(string),
Name: "Locales",
Value: event["locales"].(string),
Selector: `request.locales`,
},
})
rep = append(rep, map[string]string{
"type": api.TABLE,
"title": "Details",
"data": string(details),
rep = append(rep, api.SectionData{
Type: api.TABLE,
Title: "Details",
Data: string(details),
})
if event["ServerProperties"] != nil {
headers := make([]map[string]string, 0)
for name, value := range event["ServerProperties"].(map[string]interface{}) {
if event["serverProperties"] != nil {
headers := make([]api.TableData, 0)
for name, value := range event["serverProperties"].(map[string]interface{}) {
var outcome string
switch value.(type) {
case string:
@ -517,16 +561,17 @@ func representConnectionStart(event map[string]interface{}) []interface{} {
default:
panic("Unknown data type for the server property!")
}
headers = append(headers, map[string]string{
"name": name,
"value": outcome,
headers = append(headers, api.TableData{
Name: name,
Value: outcome,
Selector: fmt.Sprintf(`request.serverProperties["%s"]`, name),
})
}
headersMarshaled, _ := json.Marshal(headers)
rep = append(rep, map[string]string{
"type": api.TABLE,
"title": "Server Properties",
"data": string(headersMarshaled),
rep = append(rep, api.SectionData{
Type: api.TABLE,
Title: "Server Properties",
Data: string(headersMarshaled),
})
}
@ -536,28 +581,32 @@ func representConnectionStart(event map[string]interface{}) []interface{} {
func representConnectionClose(event map[string]interface{}) []interface{} {
rep := make([]interface{}, 0)
details, _ := json.Marshal([]map[string]string{
details, _ := json.Marshal([]api.TableData{
{
"name": "Reply Code",
"value": fmt.Sprintf("%g", event["ReplyCode"].(float64)),
Name: "Reply Code",
Value: fmt.Sprintf("%g", event["replyCode"].(float64)),
Selector: `request.replyCode`,
},
{
"name": "Reply Text",
"value": event["ReplyText"].(string),
Name: "Reply Text",
Value: event["replyText"].(string),
Selector: `request.replyText`,
},
{
"name": "Class ID",
"value": fmt.Sprintf("%g", event["ClassId"].(float64)),
Name: "Class ID",
Value: fmt.Sprintf("%g", event["classId"].(float64)),
Selector: `request.classId`,
},
{
"name": "Method ID",
"value": fmt.Sprintf("%g", event["MethodId"].(float64)),
Name: "Method ID",
Value: fmt.Sprintf("%g", event["methodId"].(float64)),
Selector: `request.methodId`,
},
})
rep = append(rep, map[string]string{
"type": api.TABLE,
"title": "Details",
"data": string(details),
rep = append(rep, api.SectionData{
Type: api.TABLE,
Title: "Details",
Data: string(details),
})
return rep
@ -566,43 +615,48 @@ func representConnectionClose(event map[string]interface{}) []interface{} {
func representQueueBind(event map[string]interface{}) []interface{} {
rep := make([]interface{}, 0)
details, _ := json.Marshal([]map[string]string{
details, _ := json.Marshal([]api.TableData{
{
"name": "Queue",
"value": event["Queue"].(string),
Name: "Queue",
Value: event["queue"].(string),
Selector: `request.queue`,
},
{
"name": "Exchange",
"value": event["Exchange"].(string),
Name: "Exchange",
Value: event["exchange"].(string),
Selector: `request.exchange`,
},
{
"name": "RoutingKey",
"value": event["RoutingKey"].(string),
Name: "RoutingKey",
Value: event["routingKey"].(string),
Selector: `request.routingKey`,
},
{
"name": "NoWait",
"value": strconv.FormatBool(event["NoWait"].(bool)),
Name: "NoWait",
Value: strconv.FormatBool(event["noWait"].(bool)),
Selector: `request.noWait`,
},
})
rep = append(rep, map[string]string{
"type": api.TABLE,
"title": "Details",
"data": string(details),
rep = append(rep, api.SectionData{
Type: api.TABLE,
Title: "Details",
Data: string(details),
})
if event["Arguments"] != nil {
headers := make([]map[string]string, 0)
for name, value := range event["Arguments"].(map[string]interface{}) {
headers = append(headers, map[string]string{
"name": name,
"value": value.(string),
if event["arguments"] != nil {
headers := make([]api.TableData, 0)
for name, value := range event["arguments"].(map[string]interface{}) {
headers = append(headers, api.TableData{
Name: name,
Value: value.(string),
Selector: fmt.Sprintf(`request.arguments["%s"]`, name),
})
}
headersMarshaled, _ := json.Marshal(headers)
rep = append(rep, map[string]string{
"type": api.TABLE,
"title": "Arguments",
"data": string(headersMarshaled),
rep = append(rep, api.SectionData{
Type: api.TABLE,
Title: "Arguments",
Data: string(headersMarshaled),
})
}
@ -612,51 +666,58 @@ func representQueueBind(event map[string]interface{}) []interface{} {
func representBasicConsume(event map[string]interface{}) []interface{} {
rep := make([]interface{}, 0)
details, _ := json.Marshal([]map[string]string{
details, _ := json.Marshal([]api.TableData{
{
"name": "Queue",
"value": event["Queue"].(string),
Name: "Queue",
Value: event["queue"].(string),
Selector: `request.queue`,
},
{
"name": "Consumer Tag",
"value": event["ConsumerTag"].(string),
Name: "Consumer Tag",
Value: event["consumerTag"].(string),
Selector: `request.consumerTag`,
},
{
"name": "No Local",
"value": strconv.FormatBool(event["NoLocal"].(bool)),
Name: "No Local",
Value: strconv.FormatBool(event["noLocal"].(bool)),
Selector: `request.noLocal`,
},
{
"name": "No Ack",
"value": strconv.FormatBool(event["NoAck"].(bool)),
Name: "No Ack",
Value: strconv.FormatBool(event["noAck"].(bool)),
Selector: `request.noAck`,
},
{
"name": "Exclusive",
"value": strconv.FormatBool(event["Exclusive"].(bool)),
Name: "Exclusive",
Value: strconv.FormatBool(event["exclusive"].(bool)),
Selector: `request.exclusive`,
},
{
"name": "NoWait",
"value": strconv.FormatBool(event["NoWait"].(bool)),
Name: "NoWait",
Value: strconv.FormatBool(event["noWait"].(bool)),
Selector: `request.noWait`,
},
})
rep = append(rep, map[string]string{
"type": api.TABLE,
"title": "Details",
"data": string(details),
rep = append(rep, api.SectionData{
Type: api.TABLE,
Title: "Details",
Data: string(details),
})
if event["Arguments"] != nil {
headers := make([]map[string]string, 0)
for name, value := range event["Arguments"].(map[string]interface{}) {
headers = append(headers, map[string]string{
"name": name,
"value": value.(string),
if event["arguments"] != nil {
headers := make([]api.TableData, 0)
for name, value := range event["arguments"].(map[string]interface{}) {
headers = append(headers, api.TableData{
Name: name,
Value: value.(string),
Selector: fmt.Sprintf(`request.arguments["%s"]`, name),
})
}
headersMarshaled, _ := json.Marshal(headers)
rep = append(rep, map[string]string{
"type": api.TABLE,
"title": "Arguments",
"data": string(headersMarshaled),
rep = append(rep, api.SectionData{
Type: api.TABLE,
Title: "Arguments",
Data: string(headersMarshaled),
})
}

111
tap/extensions/amqp/main.go
View File

@ -16,6 +16,7 @@ var protocol api.Protocol = api.Protocol{
Name: "amqp",
LongName: "Advanced Message Queuing Protocol 0-9-1",
Abbreviation: "AMQP",
Macro: "amqp",
Version: "0-9-1",
BackgroundColor: "#ff6600",
ForegroundColor: "#ffffff",
@ -222,7 +223,7 @@ func (d dissecting) Dissect(b *bufio.Reader, isClient bool, tcpID *api.TcpID, co
}
}
func (d dissecting) Analyze(item *api.OutputChannelItem, entryId string, resolvedSource string, resolvedDestination string) *api.MizuEntry {
func (d dissecting) Analyze(item *api.OutputChannelItem, resolvedSource string, resolvedDestination string) *api.MizuEntry {
request := item.Pair.Request.Payload.(map[string]interface{})
reqDetails := request["details"].(map[string]interface{})
service := "amqp"
@ -235,75 +236,79 @@ func (d dissecting) Analyze(item *api.OutputChannelItem, entryId string, resolve
summary := ""
switch request["method"] {
case basicMethodMap[40]:
summary = reqDetails["Exchange"].(string)
summary = reqDetails["exchange"].(string)
break
case basicMethodMap[60]:
summary = reqDetails["Exchange"].(string)
summary = reqDetails["exchange"].(string)
break
case exchangeMethodMap[10]:
summary = reqDetails["Exchange"].(string)
summary = reqDetails["exchange"].(string)
break
case queueMethodMap[10]:
summary = reqDetails["Queue"].(string)
summary = reqDetails["queue"].(string)
break
case connectionMethodMap[10]:
summary = fmt.Sprintf(
"%s.%s",
strconv.Itoa(int(reqDetails["VersionMajor"].(float64))),
strconv.Itoa(int(reqDetails["VersionMinor"].(float64))),
strconv.Itoa(int(reqDetails["versionMajor"].(float64))),
strconv.Itoa(int(reqDetails["versionMinor"].(float64))),
)
break
case connectionMethodMap[50]:
summary = reqDetails["ReplyText"].(string)
summary = reqDetails["replyText"].(string)
break
case queueMethodMap[20]:
summary = reqDetails["Queue"].(string)
summary = reqDetails["queue"].(string)
break
case basicMethodMap[20]:
summary = reqDetails["Queue"].(string)
summary = reqDetails["queue"].(string)
break
}
request["url"] = summary
entryBytes, _ := json.Marshal(item.Pair)
reqDetails["method"] = request["method"]
return &api.MizuEntry{
ProtocolName: protocol.Name,
ProtocolLongName: protocol.LongName,
ProtocolAbbreviation: protocol.Abbreviation,
ProtocolVersion: protocol.Version,
ProtocolBackgroundColor: protocol.BackgroundColor,
ProtocolForegroundColor: protocol.ForegroundColor,
ProtocolFontSize: protocol.FontSize,
ProtocolReferenceLink: protocol.ReferenceLink,
EntryId: entryId,
Entry: string(entryBytes),
Url: fmt.Sprintf("%s%s", service, summary),
Method: request["method"].(string),
Status: 0,
RequestSenderIp: item.ConnectionInfo.ClientIP,
Service: service,
Timestamp: item.Timestamp,
ElapsedTime: 0,
Path: summary,
ResolvedSource: resolvedSource,
ResolvedDestination: resolvedDestination,
SourceIp: item.ConnectionInfo.ClientIP,
DestinationIp: item.ConnectionInfo.ServerIP,
SourcePort: item.ConnectionInfo.ClientPort,
DestinationPort: item.ConnectionInfo.ServerPort,
IsOutgoing: item.ConnectionInfo.IsOutgoing,
Protocol: protocol,
Source: &api.TCP{
Name: resolvedSource,
IP: item.ConnectionInfo.ClientIP,
Port: item.ConnectionInfo.ClientPort,
},
Destination: &api.TCP{
Name: resolvedDestination,
IP: item.ConnectionInfo.ServerIP,
Port: item.ConnectionInfo.ServerPort,
},
Outgoing: item.ConnectionInfo.IsOutgoing,
Request: reqDetails,
Url: fmt.Sprintf("%s%s", service, summary),
Method: request["method"].(string),
Status: 0,
RequestSenderIp: item.ConnectionInfo.ClientIP,
Service: service,
Timestamp: item.Timestamp,
StartTime: item.Pair.Request.CaptureTime,
ElapsedTime: 0,
Summary: summary,
ResolvedSource: resolvedSource,
ResolvedDestination: resolvedDestination,
SourceIp: item.ConnectionInfo.ClientIP,
DestinationIp: item.ConnectionInfo.ServerIP,
SourcePort: item.ConnectionInfo.ClientPort,
DestinationPort: item.ConnectionInfo.ServerPort,
IsOutgoing: item.ConnectionInfo.IsOutgoing,
}
}
func (d dissecting) Summarize(entry *api.MizuEntry) *api.BaseEntryDetails {
return &api.BaseEntryDetails{
Id: entry.EntryId,
Id: entry.Id,
Protocol: protocol,
Url: entry.Url,
RequestSenderIp: entry.RequestSenderIp,
Service: entry.Service,
Summary: entry.Path,
Summary: entry.Summary,
StatusCode: entry.Status,
Method: entry.Method,
Timestamp: entry.Timestamp,
@ -320,39 +325,35 @@ func (d dissecting) Summarize(entry *api.MizuEntry) *api.BaseEntryDetails {
}
}
func (d dissecting) Represent(entry *api.MizuEntry) (p api.Protocol, object []byte, bodySize int64, err error) {
p = protocol
func (d dissecting) Represent(protoIn api.Protocol, request map[string]interface{}, response map[string]interface{}) (protoOut api.Protocol, object []byte, bodySize int64, err error) {
protoOut = protocol
bodySize = 0
var root map[string]interface{}
json.Unmarshal([]byte(entry.Entry), &root)
representation := make(map[string]interface{}, 0)
request := root["request"].(map[string]interface{})["payload"].(map[string]interface{})
var repRequest []interface{}
details := request["details"].(map[string]interface{})
switch request["method"].(string) {
case basicMethodMap[40]:
repRequest = representBasicPublish(details)
repRequest = representBasicPublish(request)
break
case basicMethodMap[60]:
repRequest = representBasicDeliver(details)
repRequest = representBasicDeliver(request)
break
case queueMethodMap[10]:
repRequest = representQueueDeclare(details)
repRequest = representQueueDeclare(request)
break
case exchangeMethodMap[10]:
repRequest = representExchangeDeclare(details)
repRequest = representExchangeDeclare(request)
break
case connectionMethodMap[10]:
repRequest = representConnectionStart(details)
repRequest = representConnectionStart(request)
break
case connectionMethodMap[50]:
repRequest = representConnectionClose(details)
repRequest = representConnectionClose(request)
break
case queueMethodMap[20]:
repRequest = representQueueBind(details)
repRequest = representQueueBind(request)
break
case basicMethodMap[20]:
repRequest = representBasicConsume(details)
repRequest = representBasicConsume(request)
break
}
representation["request"] = repRequest
@ -360,4 +361,10 @@ func (d dissecting) Represent(entry *api.MizuEntry) (p api.Protocol, object []by
return
}
func (d dissecting) Macros() map[string]string {
return map[string]string{
`amqp`: fmt.Sprintf(`proto.abbr == "%s"`, protocol.Abbreviation),
}
}
var Dissector dissecting

98
tap/extensions/amqp/spec091.go
View File

@ -71,11 +71,11 @@ func isSoftExceptionCode(code int) bool {
}
type ConnectionStart struct {
VersionMajor byte
VersionMinor byte
ServerProperties Table
Mechanisms string
Locales string
VersionMajor byte `json:"versionMajor"`
VersionMinor byte `json:"versionMinor"`
ServerProperties Table `json:"serverProperties"`
Mechanisms string `json:"mechanisms"`
Locales string `json:"locales"`
}
func (msg *ConnectionStart) id() (uint16, uint16) {
@ -429,10 +429,10 @@ func (msg *connectionOpenOk) read(r io.Reader) (err error) {
}
type ConnectionClose struct {
ReplyCode uint16
ReplyText string
ClassId uint16
MethodId uint16
ReplyCode uint16 `json:"relyCode"`
ReplyText string `json:"replyText"`
ClassId uint16 `json:"classId"`
MethodId uint16 `json:"methodId"`
}
func (msg *ConnectionClose) id() (uint16, uint16) {
@ -767,14 +767,14 @@ func (msg *channelCloseOk) read(r io.Reader) (err error) {
type ExchangeDeclare struct {
reserved1 uint16
Exchange string
Type string
Passive bool
Durable bool
AutoDelete bool
Internal bool
NoWait bool
Arguments Table
Exchange string `json:"exchange"`
Type string `json:"type"`
Passive bool `json:"passive"`
Durable bool `json:"durable"`
AutoDelete bool `json:"autoDelete"`
Internal bool `json:"internal"`
NoWait bool `json:"noWait"`
Arguments Table `json:"arguments"`
}
func (msg *ExchangeDeclare) id() (uint16, uint16) {
@ -1163,13 +1163,13 @@ func (msg *exchangeUnbindOk) read(r io.Reader) (err error) {
type QueueDeclare struct {
reserved1 uint16
Queue string
Passive bool
Durable bool
Exclusive bool
AutoDelete bool
NoWait bool
Arguments Table
Queue string `json:"queue"`
Passive bool `json:"passive"`
Durable bool `json:"durable"`
Exclusive bool `json:"exclusive"`
AutoDelete bool `json:"autoDelete"`
NoWait bool `json:"noWait"`
Arguments Table `json:"arguments"`
}
func (msg *QueueDeclare) id() (uint16, uint16) {
@ -1297,11 +1297,11 @@ func (msg *QueueDeclareOk) read(r io.Reader) (err error) {
type QueueBind struct {
reserved1 uint16
Queue string
Exchange string
RoutingKey string
NoWait bool
Arguments Table
Queue string `json:"queue"`
Exchange string `json:"exchange"`
RoutingKey string `json:"routingKey"`
NoWait bool `json:"noWait"`
Arguments Table `json:"arguments"`
}
func (msg *QueueBind) id() (uint16, uint16) {
@ -1737,13 +1737,13 @@ func (msg *basicQosOk) read(r io.Reader) (err error) {
type BasicConsume struct {
reserved1 uint16
Queue string
ConsumerTag string
NoLocal bool
NoAck bool
Exclusive bool
NoWait bool
Arguments Table
Queue string `json:"queue"`
ConsumerTag string `json:"consumerTag"`
NoLocal bool `json:"noLocal"`
NoAck bool `json:"noAck"`
Exclusive bool `json:"exclusive"`
NoWait bool `json:"noWait"`
Arguments Table `json:"arguments"`
}
func (msg *BasicConsume) id() (uint16, uint16) {
@ -1932,12 +1932,12 @@ func (msg *basicCancelOk) read(r io.Reader) (err error) {
type BasicPublish struct {
reserved1 uint16
Exchange string
RoutingKey string
Mandatory bool
Immediate bool
Properties Properties
Body []byte
Exchange string `json:"exchange"`
RoutingKey string `json:"routingKey"`
Mandatory bool `json:"mandatory"`
Immediate bool `json:"immediate"`
Properties Properties `json:"properties"`
Body []byte `json:"body"`
}
func (msg *BasicPublish) id() (uint16, uint16) {
@ -2072,13 +2072,13 @@ func (msg *basicReturn) read(r io.Reader) (err error) {
}
type BasicDeliver struct {
ConsumerTag string
DeliveryTag uint64
Redelivered bool
Exchange string
RoutingKey string
Properties Properties
Body []byte
ConsumerTag string `json:"consumerTag"`
DeliveryTag uint64 `json:"deliveryTag"`
Redelivered bool `json:"redelivered"`
Exchange string `json:"exchange"`
RoutingKey string `json:"routingKey"`
Properties Properties `json:"properties"`
Body []byte `json:"body"`
}
func (msg *BasicDeliver) id() (uint16, uint16) {

26
tap/extensions/amqp/types.go
View File

@ -93,19 +93,19 @@ func (e Error) Error() string {
// Used by header frames to capture routing and header information
type Properties struct {
ContentType string // MIME content type
ContentEncoding string // MIME content encoding
Headers Table // Application or header exchange table
DeliveryMode uint8 // queue implementation use - Transient (1) or Persistent (2)
Priority uint8 // queue implementation use - 0 to 9
CorrelationId string // application use - correlation identifier
ReplyTo string // application use - address to to reply to (ex: RPC)
Expiration string // implementation use - message expiration spec
MessageId string // application use - message identifier
Timestamp time.Time // application use - message timestamp
Type string // application use - message type name
UserId string // application use - creating user id
AppId string // application use - creating application
ContentType string `json:"contentType"` // MIME content type
ContentEncoding string `json:"contentEncoding"` // MIME content encoding
Headers Table `json:"headers"` // Application or header exchange table
DeliveryMode uint8 `json:"deliveryMode"` // queue implementation use - Transient (1) or Persistent (2)
Priority uint8 `json:"priority"` // queue implementation use - 0 to 9
CorrelationId string `json:"correlationId"` // application use - correlation identifier
ReplyTo string `json:"replyTo"` // application use - address to to reply to (ex: RPC)
Expiration string `json:"expiration"` // implementation use - message expiration spec
MessageId string `json:"messageId"` // application use - message identifier
Timestamp time.Time `json:"timestamp"` // application use - message timestamp
Type string `json:"type"` // application use - message type name
UserId string `json:"userId"` // application use - creating user id
AppId string `json:"appId"` // application use - creating application
reserved1 string // was cluster-id - process for buffer consumption
}

35
tap/extensions/http/helpers.go Normal file
View File

@ -0,0 +1,35 @@
package main
import (
"encoding/json"
"fmt"
"github.com/up9inc/mizu/tap/api"
)
func mapSliceRebuildAsMap(mapSlice []interface{}) (newMap map[string]interface{}) {
newMap = make(map[string]interface{})
for _, header := range mapSlice {
h := header.(map[string]interface{})
newMap[h["name"].(string)] = h["value"]
}
return
}
func representMapSliceAsTable(mapSlice []interface{}, selectorPrefix string) (representation string) {
var table []api.TableData
for _, header := range mapSlice {
h := header.(map[string]interface{})
selector := fmt.Sprintf("%s[\"%s\"]", selectorPrefix, h["name"].(string))
table = append(table, api.TableData{
Name: h["name"].(string),
Value: h["value"],
Selector: selector,
})
}
obj, _ := json.Marshal(table)
representation = string(obj)
return
}

263
tap/extensions/http/main.go
View File

@ -17,6 +17,7 @@ var protocol api.Protocol = api.Protocol{
Name: "http",
LongName: "Hypertext Transfer Protocol -- HTTP/1.1",
Abbreviation: "HTTP",
Macro: "http",
Version: "1.1",
BackgroundColor: "#205cf5",
ForegroundColor: "#ffffff",
@ -30,6 +31,7 @@ var http2Protocol api.Protocol = api.Protocol{
Name: "http",
LongName: "Hypertext Transfer Protocol Version 2 (HTTP/2) (gRPC)",
Abbreviation: "HTTP/2",
Macro: "grpc",
Version: "2.0",
BackgroundColor: "#244c5a",
ForegroundColor: "#ffffff",
@ -117,7 +119,7 @@ func SetHostname(address, newHostname string) string {
return replacedUrl.String()
}
func (d dissecting) Analyze(item *api.OutputChannelItem, entryId string, resolvedSource string, resolvedDestination string) *api.MizuEntry {
func (d dissecting) Analyze(item *api.OutputChannelItem, resolvedSource string, resolvedDestination string) *api.MizuEntry {
var host, scheme, authority, path, service string
request := item.Pair.Request.Payload.(map[string]interface{})
@ -145,10 +147,32 @@ func (d dissecting) Analyze(item *api.OutputChannelItem, entryId string, resolve
service = fmt.Sprintf("%s://%s", scheme, authority)
} else {
service = fmt.Sprintf("http://%s", host)
path = reqDetails["url"].(string)
u, err := url.Parse(reqDetails["url"].(string))
if err != nil {
path = reqDetails["url"].(string)
} else {
path = u.Path
}
}
request["url"] = path
request["url"] = reqDetails["url"].(string)
reqDetails["path"] = path
reqDetails["summary"] = path
// Rearrange the maps for the querying
reqDetails["_headers"] = reqDetails["headers"]
reqDetails["headers"] = mapSliceRebuildAsMap(reqDetails["_headers"].([]interface{}))
resDetails["_headers"] = resDetails["headers"]
resDetails["headers"] = mapSliceRebuildAsMap(resDetails["_headers"].([]interface{}))
reqDetails["_cookies"] = reqDetails["cookies"]
reqDetails["cookies"] = mapSliceRebuildAsMap(reqDetails["_cookies"].([]interface{}))
resDetails["_cookies"] = resDetails["cookies"]
resDetails["cookies"] = mapSliceRebuildAsMap(resDetails["_cookies"].([]interface{}))
reqDetails["_queryString"] = reqDetails["queryString"]
reqDetails["queryString"] = mapSliceRebuildAsMap(reqDetails["_queryString"].([]interface{}))
if resolvedDestination != "" {
service = SetHostname(service, resolvedDestination)
} else if resolvedSource != "" {
@ -156,51 +180,59 @@ func (d dissecting) Analyze(item *api.OutputChannelItem, entryId string, resolve
}
elapsedTime := item.Pair.Response.CaptureTime.Sub(item.Pair.Request.CaptureTime).Round(time.Millisecond).Milliseconds()
entryBytes, _ := json.Marshal(item.Pair)
httpPair, _ := json.Marshal(item.Pair)
_protocol := protocol
_protocol.Version = item.Protocol.Version
return &api.MizuEntry{
ProtocolName: protocol.Name,
ProtocolLongName: protocol.LongName,
ProtocolAbbreviation: protocol.Abbreviation,
ProtocolVersion: item.Protocol.Version,
ProtocolBackgroundColor: protocol.BackgroundColor,
ProtocolForegroundColor: protocol.ForegroundColor,
ProtocolFontSize: protocol.FontSize,
ProtocolReferenceLink: protocol.ReferenceLink,
EntryId: entryId,
Entry: string(entryBytes),
Url: fmt.Sprintf("%s%s", service, path),
Method: reqDetails["method"].(string),
Status: int(resDetails["status"].(float64)),
RequestSenderIp: item.ConnectionInfo.ClientIP,
Service: service,
Timestamp: item.Timestamp,
ElapsedTime: elapsedTime,
Path: path,
ResolvedSource: resolvedSource,
ResolvedDestination: resolvedDestination,
SourceIp: item.ConnectionInfo.ClientIP,
DestinationIp: item.ConnectionInfo.ServerIP,
SourcePort: item.ConnectionInfo.ClientPort,
DestinationPort: item.ConnectionInfo.ServerPort,
IsOutgoing: item.ConnectionInfo.IsOutgoing,
Protocol: _protocol,
Source: &api.TCP{
Name: resolvedSource,
IP: item.ConnectionInfo.ClientIP,
Port: item.ConnectionInfo.ClientPort,
},
Destination: &api.TCP{
Name: resolvedDestination,
IP: item.ConnectionInfo.ServerIP,
Port: item.ConnectionInfo.ServerPort,
},
Outgoing: item.ConnectionInfo.IsOutgoing,
Request: reqDetails,
Response: resDetails,
Url: fmt.Sprintf("%s%s", service, path),
Method: reqDetails["method"].(string),
Status: int(resDetails["status"].(float64)),
RequestSenderIp: item.ConnectionInfo.ClientIP,
Service: service,
Timestamp: item.Timestamp,
StartTime: item.Pair.Request.CaptureTime,
ElapsedTime: elapsedTime,
Summary: path,
ResolvedSource: resolvedSource,
ResolvedDestination: resolvedDestination,
SourceIp: item.ConnectionInfo.ClientIP,
DestinationIp: item.ConnectionInfo.ServerIP,
SourcePort: item.ConnectionInfo.ClientPort,
DestinationPort: item.ConnectionInfo.ServerPort,
IsOutgoing: item.ConnectionInfo.IsOutgoing,
HTTPPair: string(httpPair),
}
}
func (d dissecting) Summarize(entry *api.MizuEntry) *api.BaseEntryDetails {
var p api.Protocol
if entry.ProtocolVersion == "2.0" {
if entry.Protocol.Version == "2.0" {
p = http2Protocol
} else {
p = protocol
}
return &api.BaseEntryDetails{
Id: entry.EntryId,
Id: entry.Id,
Protocol: p,
Url: entry.Url,
RequestSenderIp: entry.RequestSenderIp,
Service: entry.Service,
Path: entry.Path,
Summary: entry.Path,
Summary: entry.Summary,
StatusCode: entry.Status,
Method: entry.Method,
Timestamp: entry.Timestamp,
@ -218,45 +250,50 @@ func (d dissecting) Summarize(entry *api.MizuEntry) *api.BaseEntryDetails {
}
func representRequest(request map[string]interface{}) (repRequest []interface{}) {
details, _ := json.Marshal([]map[string]string{
details, _ := json.Marshal([]api.TableData{
{
"name": "Method",
"value": request["method"].(string),
Name: "Method",
Value: request["method"].(string),
Selector: `request.method`,
},
{
"name": "URL",
"value": request["url"].(string),
Name: "URL",
Value: request["url"].(string),
Selector: `request.url`,
},
{
"name": "Body Size",
"value": fmt.Sprintf("%g bytes", request["bodySize"].(float64)),
Name: "Path",
Value: request["path"].(string),
Selector: `request.path`,
},
{
Name: "Body Size (bytes)",
Value: int64(request["bodySize"].(float64)),
Selector: `request.bodySize`,
},
})
repRequest = append(repRequest, map[string]string{
"type": api.TABLE,
"title": "Details",
"data": string(details),
repRequest = append(repRequest, api.SectionData{
Type: api.TABLE,
Title: "Details",
Data: string(details),
})
headers, _ := json.Marshal(request["headers"].([]interface{}))
repRequest = append(repRequest, map[string]string{
"type": api.TABLE,
"title": "Headers",
"data": string(headers),
repRequest = append(repRequest, api.SectionData{
Type: api.TABLE,
Title: "Headers",
Data: representMapSliceAsTable(request["_headers"].([]interface{}), `request.headers`),
})
cookies, _ := json.Marshal(request["cookies"].([]interface{}))
repRequest = append(repRequest, map[string]string{
"type": api.TABLE,
"title": "Cookies",
"data": string(cookies),
repRequest = append(repRequest, api.SectionData{
Type: api.TABLE,
Title: "Cookies",
Data: representMapSliceAsTable(request["_cookies"].([]interface{}), `request.cookies`),
})
queryString, _ := json.Marshal(request["queryString"].([]interface{}))
repRequest = append(repRequest, map[string]string{
"type": api.TABLE,
"title": "Query String",
"data": string(queryString),
repRequest = append(repRequest, api.SectionData{
Type: api.TABLE,
Title: "Query String",
Data: representMapSliceAsTable(request["_queryString"].([]interface{}), `request.queryString`),
})
postData, _ := request["postData"].(map[string]interface{})
@ -266,12 +303,12 @@ func representRequest(request map[string]interface{}) (repRequest []interface{})
}
text, _ := postData["text"]
if text != nil {
repRequest = append(repRequest, map[string]string{
"type": api.BODY,
"title": "POST Data (text/plain)",
"encoding": "",
"mime_type": mimeType.(string),
"data": text.(string),
repRequest = append(repRequest, api.SectionData{
Type: api.BODY,
Title: "POST Data (text/plain)",
MimeType: mimeType.(string),
Data: text.(string),
Selector: `request.postData.text`,
})
}
@ -285,16 +322,16 @@ func representRequest(request map[string]interface{}) (repRequest []interface{})
"value": string(params),
},
})
repRequest = append(repRequest, map[string]string{
"type": api.TABLE,
"title": "POST Data (multipart/form-data)",
"data": string(multipart),
repRequest = append(repRequest, api.SectionData{
Type: api.TABLE,
Title: "POST Data (multipart/form-data)",
Data: string(multipart),
})
} else {
repRequest = append(repRequest, map[string]string{
"type": api.TABLE,
"title": "POST Data (application/x-www-form-urlencoded)",
"data": string(params),
repRequest = append(repRequest, api.SectionData{
Type: api.TABLE,
Title: "POST Data (application/x-www-form-urlencoded)",
Data: representMapSliceAsTable(postData["params"].([]interface{}), `request.postData.params`),
})
}
}
@ -308,38 +345,39 @@ func representResponse(response map[string]interface{}) (repResponse []interface
bodySize = int64(response["bodySize"].(float64))
details, _ := json.Marshal([]map[string]string{
details, _ := json.Marshal([]api.TableData{
{
"name": "Status",
"value": fmt.Sprintf("%g", response["status"].(float64)),
Name: "Status",
Value: int64(response["status"].(float64)),
Selector: `response.status`,
},
{
"name": "Status Text",
"value": response["statusText"].(string),
Name: "Status Text",
Value: response["statusText"].(string),
Selector: `response.statusText`,
},
{
"name": "Body Size",
"value": fmt.Sprintf("%d bytes", bodySize),
Name: "Body Size (bytes)",
Value: bodySize,
Selector: `response.bodySize`,
},
})
repResponse = append(repResponse, map[string]string{
"type": api.TABLE,
"title": "Details",
"data": string(details),
repResponse = append(repResponse, api.SectionData{
Type: api.TABLE,
Title: "Details",
Data: string(details),
})
headers, _ := json.Marshal(response["headers"].([]interface{}))
repResponse = append(repResponse, map[string]string{
"type": api.TABLE,
"title": "Headers",
"data": string(headers),
repResponse = append(repResponse, api.SectionData{
Type: api.TABLE,
Title: "Headers",
Data: representMapSliceAsTable(response["_headers"].([]interface{}), `response.headers`),
})
cookies, _ := json.Marshal(response["cookies"].([]interface{}))
repResponse = append(repResponse, map[string]string{
"type": api.TABLE,
"title": "Cookies",
"data": string(cookies),
repResponse = append(repResponse, api.SectionData{
Type: api.TABLE,
Title: "Cookies",
Data: representMapSliceAsTable(response["_cookies"].([]interface{}), `response.cookies`),
})
content, _ := response["content"].(map[string]interface{})
@ -350,37 +388,40 @@ func representResponse(response map[string]interface{}) (repResponse []interface
encoding, _ := content["encoding"]
text, _ := content["text"]
if text != nil {
repResponse = append(repResponse, map[string]string{
"type": api.BODY,
"title": "Body",
"encoding": encoding.(string),
"mime_type": mimeType.(string),
"data": text.(string),
repResponse = append(repResponse, api.SectionData{
Type: api.BODY,
Title: "Body",
Encoding: encoding.(string),
MimeType: mimeType.(string),
Data: text.(string),
Selector: `response.content.text`,
})
}
return
}
func (d dissecting) Represent(entry *api.MizuEntry) (p api.Protocol, object []byte, bodySize int64, err error) {
if entry.ProtocolVersion == "2.0" {
p = http2Protocol
func (d dissecting) Represent(protoIn api.Protocol, request map[string]interface{}, response map[string]interface{}) (protoOut api.Protocol, object []byte, bodySize int64, err error) {
if protoIn.Version == "2.0" {
protoOut = http2Protocol
} else {
p = protocol
protoOut = protocol
}
var root map[string]interface{}
json.Unmarshal([]byte(entry.Entry), &root)
representation := make(map[string]interface{}, 0)
request := root["request"].(map[string]interface{})["payload"].(map[string]interface{})
response := root["response"].(map[string]interface{})["payload"].(map[string]interface{})
reqDetails := request["details"].(map[string]interface{})
resDetails := response["details"].(map[string]interface{})
repRequest := representRequest(reqDetails)
repResponse, bodySize := representResponse(resDetails)
repRequest := representRequest(request)
repResponse, bodySize := representResponse(response)
representation["request"] = repRequest
representation["response"] = repResponse
object, err = json.Marshal(representation)
return
}
func (d dissecting) Macros() map[string]string {
return map[string]string{
`http`: fmt.Sprintf(`proto.abbr == "%s"`, protocol.Abbreviation),
`grpc`: fmt.Sprintf(`proto.abbr == "%s" and proto.version == "%s"`, protocol.Abbreviation, http2Protocol.Version),
`http2`: fmt.Sprintf(`proto.abbr == "%s" and proto.version == "%s"`, protocol.Abbreviation, http2Protocol.Version),
}
}
var Dissector dissecting

617
tap/extensions/kafka/helpers.go
View File

@ -27,48 +27,54 @@ type KafkaWrapper struct {
}
func representRequestHeader(data map[string]interface{}, rep []interface{}) []interface{} {
requestHeader, _ := json.Marshal([]map[string]string{
requestHeader, _ := json.Marshal([]api.TableData{
{
"name": "ApiKey",
"value": apiNames[int(data["ApiKey"].(float64))],
Name: "ApiKey",
Value: apiNames[int(data["apiKey"].(float64))],
Selector: `request.apiKey`,
},
{
"name": "ApiVersion",
"value": fmt.Sprintf("%d", int(data["ApiVersion"].(float64))),
Name: "ApiVersion",
Value: fmt.Sprintf("%d", int(data["apiVersion"].(float64))),
Selector: `request.apiVersion`,
},
{
"name": "Client ID",
"value": data["ClientID"].(string),
Name: "Client ID",
Value: data["clientID"].(string),
Selector: `request.clientID`,
},
{
"name": "Correlation ID",
"value": fmt.Sprintf("%d", int(data["CorrelationID"].(float64))),
Name: "Correlation ID",
Value: fmt.Sprintf("%d", int(data["correlationID"].(float64))),
Selector: `request.correlationID`,
},
{
"name": "Size",
"value": fmt.Sprintf("%d", int(data["Size"].(float64))),
Name: "Size",
Value: fmt.Sprintf("%d", int(data["size"].(float64))),
Selector: `request.size`,
},
})
rep = append(rep, map[string]string{
"type": api.TABLE,
"title": "Request Header",
"data": string(requestHeader),
rep = append(rep, api.SectionData{
Type: api.TABLE,
Title: "Request Header",
Data: string(requestHeader),
})
return rep
}
func representResponseHeader(data map[string]interface{}, rep []interface{}) []interface{} {
requestHeader, _ := json.Marshal([]map[string]string{
requestHeader, _ := json.Marshal([]api.TableData{
{
"name": "Correlation ID",
"value": fmt.Sprintf("%d", int(data["CorrelationID"].(float64))),
Name: "Correlation ID",
Value: fmt.Sprintf("%d", int(data["correlationID"].(float64))),
Selector: `response.correlationID`,
},
})
rep = append(rep, map[string]string{
"type": api.TABLE,
"title": "Response Header",
"data": string(requestHeader),
rep = append(rep, api.SectionData{
Type: api.TABLE,
Title: "Response Header",
Data: string(requestHeader),
})
return rep
@ -79,46 +85,50 @@ func representMetadataRequest(data map[string]interface{}) []interface{} {
rep = representRequestHeader(data, rep)
payload := data["Payload"].(map[string]interface{})
payload := data["payload"].(map[string]interface{})
topics := ""
allowAutoTopicCreation := ""
includeClusterAuthorizedOperations := ""
includeTopicAuthorizedOperations := ""
if payload["Topics"] != nil {
x, _ := json.Marshal(payload["Topics"].([]interface{}))
if payload["topics"] != nil {
x, _ := json.Marshal(payload["topics"].([]interface{}))
topics = string(x)
}
if payload["AllowAutoTopicCreation"] != nil {
allowAutoTopicCreation = strconv.FormatBool(payload["AllowAutoTopicCreation"].(bool))
if payload["allowAutoTopicCreation"] != nil {
allowAutoTopicCreation = strconv.FormatBool(payload["allowAutoTopicCreation"].(bool))
}
if payload["IncludeClusterAuthorizedOperations"] != nil {
includeClusterAuthorizedOperations = strconv.FormatBool(payload["IncludeClusterAuthorizedOperations"].(bool))
if payload["includeClusterAuthorizedOperations"] != nil {
includeClusterAuthorizedOperations = strconv.FormatBool(payload["includeClusterAuthorizedOperations"].(bool))
}
if payload["IncludeTopicAuthorizedOperations"] != nil {
includeTopicAuthorizedOperations = strconv.FormatBool(payload["IncludeTopicAuthorizedOperations"].(bool))
if payload["includeTopicAuthorizedOperations"] != nil {
includeTopicAuthorizedOperations = strconv.FormatBool(payload["includeTopicAuthorizedOperations"].(bool))
}
repPayload, _ := json.Marshal([]map[string]string{
repPayload, _ := json.Marshal([]api.TableData{
{
"name": "Topics",
"value": topics,
Name: "Topics",
Value: topics,
Selector: `request.payload.topics`,
},
{
"name": "Allow Auto Topic Creation",
"value": allowAutoTopicCreation,
Name: "Allow Auto Topic Creation",
Value: allowAutoTopicCreation,
Selector: `request.payload.allowAutoTopicCreation`,
},
{
"name": "Include Cluster Authorized Operations",
"value": includeClusterAuthorizedOperations,
Name: "Include Cluster Authorized Operations",
Value: includeClusterAuthorizedOperations,
Selector: `request.payload.includeClusterAuthorizedOperations`,
},
{
"name": "Include Topic Authorized Operations",
"value": includeTopicAuthorizedOperations,
Name: "Include Topic Authorized Operations",
Value: includeTopicAuthorizedOperations,
Selector: `request.payload.includeTopicAuthorizedOperations`,
},
})
rep = append(rep, map[string]string{
"type": api.TABLE,
"title": "Payload",
"data": string(repPayload),
rep = append(rep, api.SectionData{
Type: api.TABLE,
Title: "Payload",
Data: string(repPayload),
})
return rep
@ -129,63 +139,69 @@ func representMetadataResponse(data map[string]interface{}) []interface{} {
rep = representResponseHeader(data, rep)
payload := data["Payload"].(map[string]interface{})
payload := data["payload"].(map[string]interface{})
topics := ""
if payload["Topics"] != nil {
_topics, _ := json.Marshal(payload["Topics"].([]interface{}))
if payload["topics"] != nil {
_topics, _ := json.Marshal(payload["topics"].([]interface{}))
topics = string(_topics)
}
brokers := ""
if payload["Brokers"] != nil {
_brokers, _ := json.Marshal(payload["Brokers"].([]interface{}))
if payload["brokers"] != nil {
_brokers, _ := json.Marshal(payload["brokers"].([]interface{}))
brokers = string(_brokers)
}
controllerID := ""
clusterID := ""
throttleTimeMs := ""
clusterAuthorizedOperations := ""
if payload["ControllerID"] != nil {
controllerID = fmt.Sprintf("%d", int(payload["ControllerID"].(float64)))
if payload["controllerID"] != nil {
controllerID = fmt.Sprintf("%d", int(payload["controllerID"].(float64)))
}
if payload["ClusterID"] != nil {
clusterID = payload["ClusterID"].(string)
if payload["clusterID"] != nil {
clusterID = payload["clusterID"].(string)
}
if payload["ThrottleTimeMs"] != nil {
throttleTimeMs = fmt.Sprintf("%d", int(payload["ThrottleTimeMs"].(float64)))
if payload["throttleTimeMs"] != nil {
throttleTimeMs = fmt.Sprintf("%d", int(payload["throttleTimeMs"].(float64)))
}
if payload["ClusterAuthorizedOperations"] != nil {
clusterAuthorizedOperations = fmt.Sprintf("%d", int(payload["ClusterAuthorizedOperations"].(float64)))
if payload["clusterAuthorizedOperations"] != nil {
clusterAuthorizedOperations = fmt.Sprintf("%d", int(payload["clusterAuthorizedOperations"].(float64)))
}
repPayload, _ := json.Marshal([]map[string]string{
repPayload, _ := json.Marshal([]api.TableData{
{
"name": "Throttle Time (ms)",
"value": throttleTimeMs,
Name: "Throttle Time (ms)",
Value: throttleTimeMs,
Selector: `response.payload.throttleTimeMs`,
},
{
"name": "Brokers",
"value": brokers,
Name: "Brokers",
Value: brokers,
Selector: `response.payload.brokers`,
},
{
"name": "Cluster ID",
"value": clusterID,
Name: "Cluster ID",
Value: clusterID,
Selector: `response.payload.clusterID`,
},
{
"name": "Controller ID",
"value": controllerID,
Name: "Controller ID",
Value: controllerID,
Selector: `response.payload.controllerID`,
},
{
"name": "Topics",
"value": topics,
Name: "Topics",
Value: topics,
Selector: `response.payload.topics`,
},
{
"name": "Cluster Authorized Operations",
"value": clusterAuthorizedOperations,
Name: "Cluster Authorized Operations",
Value: clusterAuthorizedOperations,
Selector: `response.payload.clusterAuthorizedOperations`,
},
})
rep = append(rep, map[string]string{
"type": api.TABLE,
"title": "Payload",
"data": string(repPayload),
rep = append(rep, api.SectionData{
Type: api.TABLE,
Title: "Payload",
Data: string(repPayload),
})
return rep
@ -196,29 +212,31 @@ func representApiVersionsRequest(data map[string]interface{}) []interface{} {
rep = representRequestHeader(data, rep)
payload := data["Payload"].(map[string]interface{})
payload := data["payload"].(map[string]interface{})
clientSoftwareName := ""
clientSoftwareVersion := ""
if payload["ClientSoftwareName"] != nil {
clientSoftwareName = payload["ClientSoftwareName"].(string)
if payload["clientSoftwareName"] != nil {
clientSoftwareName = payload["clientSoftwareName"].(string)
}
if payload["ClientSoftwareVersion"] != nil {
clientSoftwareVersion = payload["ClientSoftwareVersion"].(string)
if payload["clientSoftwareVersion"] != nil {
clientSoftwareVersion = payload["clientSoftwareVersion"].(string)
}
repPayload, _ := json.Marshal([]map[string]string{
repPayload, _ := json.Marshal([]api.TableData{
{
"name": "Client Software Name",
"value": clientSoftwareName,
Name: "Client Software Name",
Value: clientSoftwareName,
Selector: `request.payload.clientSoftwareName`,
},
{
"name": "Client Software Version",
"value": clientSoftwareVersion,
Name: "Client Software Version",
Value: clientSoftwareVersion,
Selector: `request.payload.clientSoftwareVersion`,
},
})
rep = append(rep, map[string]string{
"type": api.TABLE,
"title": "Payload",
"data": string(repPayload),
rep = append(rep, api.SectionData{
Type: api.TABLE,
Title: "Payload",
Data: string(repPayload),
})
return rep
@ -229,34 +247,37 @@ func representApiVersionsResponse(data map[string]interface{}) []interface{} {
rep = representResponseHeader(data, rep)
payload := data["Payload"].(map[string]interface{})
payload := data["payload"].(map[string]interface{})
apiKeys := ""
if payload["TopicNames"] != nil {
x, _ := json.Marshal(payload["ApiKeys"].([]interface{}))
if payload["apiKeys"] != nil {
x, _ := json.Marshal(payload["apiKeys"].([]interface{}))
apiKeys = string(x)
}
throttleTimeMs := ""
if payload["ThrottleTimeMs"] != nil {
throttleTimeMs = fmt.Sprintf("%d", int(payload["ThrottleTimeMs"].(float64)))
if payload["throttleTimeMs"] != nil {
throttleTimeMs = fmt.Sprintf("%d", int(payload["throttleTimeMs"].(float64)))
}
repPayload, _ := json.Marshal([]map[string]string{
repPayload, _ := json.Marshal([]api.TableData{
{
"name": "Error Code",
"value": fmt.Sprintf("%d", int(payload["ErrorCode"].(float64))),
Name: "Error Code",
Value: fmt.Sprintf("%d", int(payload["errorCode"].(float64))),
Selector: `response.payload.errorCode`,
},
{
"name": "ApiKeys",
"value": apiKeys,
Name: "ApiKeys",
Value: apiKeys,
Selector: `response.payload.apiKeys`,
},
{
"name": "Throttle Time (ms)",
"value": throttleTimeMs,
Name: "Throttle Time (ms)",
Value: throttleTimeMs,
Selector: `response.payload.throttleTimeMs`,
},
})
rep = append(rep, map[string]string{
"type": api.TABLE,
"title": "Payload",
"data": string(repPayload),
rep = append(rep, api.SectionData{
Type: api.TABLE,
Title: "Payload",
Data: string(repPayload),
})
return rep
@ -267,39 +288,43 @@ func representProduceRequest(data map[string]interface{}) []interface{} {
rep = representRequestHeader(data, rep)
payload := data["Payload"].(map[string]interface{})
payload := data["payload"].(map[string]interface{})
topicData := ""
_topicData := payload["TopicData"]
_topicData := payload["topicData"]
if _topicData != nil {
x, _ := json.Marshal(_topicData.([]interface{}))
topicData = string(x)
}
transactionalID := ""
if payload["TransactionalID"] != nil {
transactionalID = payload["TransactionalID"].(string)
if payload["transactionalID"] != nil {
transactionalID = payload["transactionalID"].(string)
}
repPayload, _ := json.Marshal([]map[string]string{
repPayload, _ := json.Marshal([]api.TableData{
{
"name": "Transactional ID",
"value": transactionalID,
Name: "Transactional ID",
Value: transactionalID,
Selector: `request.payload.transactionalID`,
},
{
"name": "Required Acknowledgements",
"value": fmt.Sprintf("%d", int(payload["RequiredAcks"].(float64))),
Name: "Required Acknowledgements",
Value: fmt.Sprintf("%d", int(payload["requiredAcks"].(float64))),
Selector: `request.payload.requiredAcks`,
},
{
"name": "Timeout",
"value": fmt.Sprintf("%d", int(payload["Timeout"].(float64))),
Name: "Timeout",
Value: fmt.Sprintf("%d", int(payload["timeout"].(float64))),
Selector: `request.payload.timeout`,
},
{
"name": "Topic Data",
"value": topicData,
Name: "Topic Data",
Value: topicData,
Selector: `request.payload.topicData`,
},
})
rep = append(rep, map[string]string{
"type": api.TABLE,
"title": "Payload",
"data": string(repPayload),
rep = append(rep, api.SectionData{
Type: api.TABLE,
Title: "Payload",
Data: string(repPayload),
})
return rep
@ -310,30 +335,32 @@ func representProduceResponse(data map[string]interface{}) []interface{} {
rep = representResponseHeader(data, rep)
payload := data["Payload"].(map[string]interface{})
payload := data["payload"].(map[string]interface{})
responses := ""
if payload["Responses"] != nil {
_responses, _ := json.Marshal(payload["Responses"].([]interface{}))
if payload["responses"] != nil {
_responses, _ := json.Marshal(payload["responses"].([]interface{}))
responses = string(_responses)
}
throttleTimeMs := ""
if payload["ThrottleTimeMs"] != nil {
throttleTimeMs = fmt.Sprintf("%d", int(payload["ThrottleTimeMs"].(float64)))
if payload["throttleTimeMs"] != nil {
throttleTimeMs = fmt.Sprintf("%d", int(payload["throttleTimeMs"].(float64)))
}
repPayload, _ := json.Marshal([]map[string]string{
repPayload, _ := json.Marshal([]api.TableData{
{
"name": "Responses",
"value": string(responses),
Name: "Responses",
Value: string(responses),
Selector: `response.payload.responses`,
},
{
"name": "Throttle Time (ms)",
"value": throttleTimeMs,
Name: "Throttle Time (ms)",
Value: throttleTimeMs,
Selector: `response.payload.throttleTimeMs`,
},
})
rep = append(rep, map[string]string{
"type": api.TABLE,
"title": "Payload",
"data": string(repPayload),
rep = append(rep, api.SectionData{
Type: api.TABLE,
Title: "Payload",
Data: string(repPayload),
})
return rep
@ -344,87 +371,97 @@ func representFetchRequest(data map[string]interface{}) []interface{} {
rep = representRequestHeader(data, rep)
payload := data["Payload"].(map[string]interface{})
payload := data["payload"].(map[string]interface{})
topics := ""
if payload["Topics"] != nil {
_topics, _ := json.Marshal(payload["Topics"].([]interface{}))
if payload["topics"] != nil {
_topics, _ := json.Marshal(payload["topics"].([]interface{}))
topics = string(_topics)
}
replicaId := ""
if payload["ReplicaId"] != nil {
replicaId = fmt.Sprintf("%d", int(payload["ReplicaId"].(float64)))
if payload["replicaId"] != nil {
replicaId = fmt.Sprintf("%d", int(payload["replicaId"].(float64)))
}
maxBytes := ""
if payload["MaxBytes"] != nil {
maxBytes = fmt.Sprintf("%d", int(payload["MaxBytes"].(float64)))
if payload["maxBytes"] != nil {
maxBytes = fmt.Sprintf("%d", int(payload["maxBytes"].(float64)))
}
isolationLevel := ""
if payload["IsolationLevel"] != nil {
isolationLevel = fmt.Sprintf("%d", int(payload["IsolationLevel"].(float64)))
if payload["isolationLevel"] != nil {
isolationLevel = fmt.Sprintf("%d", int(payload["isolationLevel"].(float64)))
}
sessionId := ""
if payload["SessionId"] != nil {
sessionId = fmt.Sprintf("%d", int(payload["SessionId"].(float64)))
if payload["sessionId"] != nil {
sessionId = fmt.Sprintf("%d", int(payload["sessionId"].(float64)))
}
sessionEpoch := ""
if payload["SessionEpoch"] != nil {
sessionEpoch = fmt.Sprintf("%d", int(payload["SessionEpoch"].(float64)))
if payload["sessionEpoch"] != nil {
sessionEpoch = fmt.Sprintf("%d", int(payload["sessionEpoch"].(float64)))
}
forgottenTopicsData := ""
if payload["ForgottenTopicsData"] != nil {
x, _ := json.Marshal(payload["ForgottenTopicsData"].(map[string]interface{}))
if payload["forgottenTopicsData"] != nil {
x, _ := json.Marshal(payload["forgottenTopicsData"].(map[string]interface{}))
forgottenTopicsData = string(x)
}
rackId := ""
if payload["RackId"] != nil {
rackId = payload["RackId"].(string)
if payload["rackId"] != nil {
rackId = payload["rackId"].(string)
}
repPayload, _ := json.Marshal([]map[string]string{
repPayload, _ := json.Marshal([]api.TableData{
{
"name": "Replica ID",
"value": replicaId,
Name: "Replica ID",
Value: replicaId,
Selector: `request.payload.replicaId`,
},
{
"name": "Maximum Wait (ms)",
"value": fmt.Sprintf("%d", int(payload["MaxWaitMs"].(float64))),
Name: "Maximum Wait (ms)",
Value: fmt.Sprintf("%d", int(payload["maxWaitMs"].(float64))),
Selector: `request.payload.maxWaitMs`,
},
{
"name": "Minimum Bytes",
"value": fmt.Sprintf("%d", int(payload["MinBytes"].(float64))),
Name: "Minimum Bytes",
Value: fmt.Sprintf("%d", int(payload["minBytes"].(float64))),
Selector: `request.payload.minBytes`,
},
{
"name": "Maximum Bytes",
"value": maxBytes,
Name: "Maximum Bytes",
Value: maxBytes,
Selector: `request.payload.maxBytes`,
},
{
"name": "Isolation Level",
"value": isolationLevel,
Name: "Isolation Level",
Value: isolationLevel,
Selector: `request.payload.isolationLevel`,
},
{
"name": "Session ID",
"value": sessionId,
Name: "Session ID",
Value: sessionId,
Selector: `request.payload.sessionId`,
},
{
"name": "Session Epoch",
"value": sessionEpoch,
Name: "Session Epoch",
Value: sessionEpoch,
Selector: `request.payload.sessionEpoch`,
},
{
"name": "Topics",
"value": topics,
Name: "Topics",
Value: topics,
Selector: `request.payload.topics`,
},
{
"name": "Forgotten Topics Data",
"value": forgottenTopicsData,
Name: "Forgotten Topics Data",
Value: forgottenTopicsData,
Selector: `request.payload.forgottenTopicsData`,
},
{
"name": "Rack ID",
"value": rackId,
Name: "Rack ID",
Value: rackId,
Selector: `request.payload.rackId`,
},
})
rep = append(rep, map[string]string{
"type": api.TABLE,
"title": "Payload",
"data": string(repPayload),
rep = append(rep, api.SectionData{
Type: api.TABLE,
Title: "Payload",
Data: string(repPayload),
})
return rep
@ -435,46 +472,50 @@ func representFetchResponse(data map[string]interface{}) []interface{} {
rep = representResponseHeader(data, rep)
payload := data["Payload"].(map[string]interface{})
payload := data["payload"].(map[string]interface{})
responses := ""
if payload["Responses"] != nil {
_responses, _ := json.Marshal(payload["Responses"].([]interface{}))
if payload["responses"] != nil {
_responses, _ := json.Marshal(payload["responses"].([]interface{}))
responses = string(_responses)
}
throttleTimeMs := ""
if payload["ThrottleTimeMs"] != nil {
throttleTimeMs = fmt.Sprintf("%d", int(payload["ThrottleTimeMs"].(float64)))
if payload["throttleTimeMs"] != nil {
throttleTimeMs = fmt.Sprintf("%d", int(payload["throttleTimeMs"].(float64)))
}
errorCode := ""
if payload["ErrorCode"] != nil {
errorCode = fmt.Sprintf("%d", int(payload["ErrorCode"].(float64)))
if payload["errorCode"] != nil {
errorCode = fmt.Sprintf("%d", int(payload["errorCode"].(float64)))
}
sessionId := ""
if payload["SessionId"] != nil {
sessionId = fmt.Sprintf("%d", int(payload["SessionId"].(float64)))
if payload["sessionId"] != nil {
sessionId = fmt.Sprintf("%d", int(payload["sessionId"].(float64)))
}
repPayload, _ := json.Marshal([]map[string]string{
repPayload, _ := json.Marshal([]api.TableData{
{
"name": "Throttle Time (ms)",
"value": throttleTimeMs,
Name: "Throttle Time (ms)",
Value: throttleTimeMs,
Selector: `response.payload.throttleTimeMs`,
},
{
"name": "Error Code",
"value": errorCode,
Name: "Error Code",
Value: errorCode,
Selector: `response.payload.errorCode`,
},
{
"name": "Session ID",
"value": sessionId,
Name: "Session ID",
Value: sessionId,
Selector: `response.payload.sessionId`,
},
{
"name": "Responses",
"value": responses,
Name: "Responses",
Value: responses,
Selector: `response.payload.responses`,
},
})
rep = append(rep, map[string]string{
"type": api.TABLE,
"title": "Payload",
"data": string(repPayload),
rep = append(rep, api.SectionData{
Type: api.TABLE,
Title: "Payload",
Data: string(repPayload),
})
return rep
@ -485,26 +526,28 @@ func representListOffsetsRequest(data map[string]interface{}) []interface{} {
rep = representRequestHeader(data, rep)
payload := data["Payload"].(map[string]interface{})
payload := data["payload"].(map[string]interface{})
topics := ""
if payload["Topics"] != nil {
_topics, _ := json.Marshal(payload["Topics"].([]interface{}))
if payload["topics"] != nil {
_topics, _ := json.Marshal(payload["topics"].([]interface{}))
topics = string(_topics)
}
repPayload, _ := json.Marshal([]map[string]string{
repPayload, _ := json.Marshal([]api.TableData{
{
"name": "Replica ID",
"value": fmt.Sprintf("%d", int(payload["ReplicaId"].(float64))),
Name: "Replica ID",
Value: fmt.Sprintf("%d", int(payload["replicaId"].(float64))),
Selector: `request.payload.replicaId`,
},
{
"name": "Topics",
"value": topics,
Name: "Topics",
Value: topics,
Selector: `request.payload.topics`,
},
})
rep = append(rep, map[string]string{
"type": api.TABLE,
"title": "Payload",
"data": string(repPayload),
rep = append(rep, api.SectionData{
Type: api.TABLE,
Title: "Payload",
Data: string(repPayload),
})
return rep
@ -515,26 +558,28 @@ func representListOffsetsResponse(data map[string]interface{}) []interface{} {
rep = representResponseHeader(data, rep)
payload := data["Payload"].(map[string]interface{})
topics, _ := json.Marshal(payload["Topics"].([]interface{}))
payload := data["payload"].(map[string]interface{})
topics, _ := json.Marshal(payload["topics"].([]interface{}))
throttleTimeMs := ""
if payload["ThrottleTimeMs"] != nil {
throttleTimeMs = fmt.Sprintf("%d", int(payload["ThrottleTimeMs"].(float64)))
if payload["throttleTimeMs"] != nil {
throttleTimeMs = fmt.Sprintf("%d", int(payload["throttleTimeMs"].(float64)))
}
repPayload, _ := json.Marshal([]map[string]string{
repPayload, _ := json.Marshal([]api.TableData{
{
"name": "Throttle Time (ms)",
"value": throttleTimeMs,
Name: "Throttle Time (ms)",
Value: throttleTimeMs,
Selector: `response.payload.throttleTimeMs`,
},
{
"name": "Topics",
"value": string(topics),
Name: "Topics",
Value: string(topics),
Selector: `response.payload.topics`,
},
})
rep = append(rep, map[string]string{
"type": api.TABLE,
"title": "Payload",
"data": string(repPayload),
rep = append(rep, api.SectionData{
Type: api.TABLE,
Title: "Payload",
Data: string(repPayload),
})
return rep
@ -545,30 +590,33 @@ func representCreateTopicsRequest(data map[string]interface{}) []interface{} {
rep = representRequestHeader(data, rep)
payload := data["Payload"].(map[string]interface{})
topics, _ := json.Marshal(payload["Topics"].([]interface{}))
payload := data["payload"].(map[string]interface{})
topics, _ := json.Marshal(payload["topics"].([]interface{}))
validateOnly := ""
if payload["ValidateOnly"] != nil {
validateOnly = strconv.FormatBool(payload["ValidateOnly"].(bool))
if payload["validateOnly"] != nil {
validateOnly = strconv.FormatBool(payload["validateOnly"].(bool))
}
repPayload, _ := json.Marshal([]map[string]string{
repPayload, _ := json.Marshal([]api.TableData{
{
"name": "Topics",
"value": string(topics),
Name: "Topics",
Value: string(topics),
Selector: `request.payload.topics`,
},
{
"name": "Timeout (ms)",
"value": fmt.Sprintf("%d", int(payload["TimeoutMs"].(float64))),
Name: "Timeout (ms)",
Value: fmt.Sprintf("%d", int(payload["timeoutMs"].(float64))),
Selector: `request.payload.timeoutMs`,
},
{
"name": "Validate Only",
"value": validateOnly,
Name: "Validate Only",
Value: validateOnly,
Selector: `request.payload.validateOnly`,
},
})
rep = append(rep, map[string]string{
"type": api.TABLE,
"title": "Payload",
"data": string(repPayload),
rep = append(rep, api.SectionData{
Type: api.TABLE,
Title: "Payload",
Data: string(repPayload),
})
return rep
@ -579,26 +627,28 @@ func representCreateTopicsResponse(data map[string]interface{}) []interface{} {
rep = representResponseHeader(data, rep)
payload := data["Payload"].(map[string]interface{})
topics, _ := json.Marshal(payload["Topics"].([]interface{}))
payload := data["payload"].(map[string]interface{})
topics, _ := json.Marshal(payload["topics"].([]interface{}))
throttleTimeMs := ""
if payload["ThrottleTimeMs"] != nil {
throttleTimeMs = fmt.Sprintf("%d", int(payload["ThrottleTimeMs"].(float64)))
if payload["throttleTimeMs"] != nil {
throttleTimeMs = fmt.Sprintf("%d", int(payload["throttleTimeMs"].(float64)))
}
repPayload, _ := json.Marshal([]map[string]string{
repPayload, _ := json.Marshal([]api.TableData{
{
"name": "Throttle Time (ms)",
"value": throttleTimeMs,
Name: "Throttle Time (ms)",
Value: throttleTimeMs,
Selector: `response.payload.throttleTimeMs`,
},
{
"name": "Topics",
"value": string(topics),
Name: "Topics",
Value: string(topics),
Selector: `response.payload.topics`,
},
})
rep = append(rep, map[string]string{
"type": api.TABLE,
"title": "Payload",
"data": string(repPayload),
rep = append(rep, api.SectionData{
Type: api.TABLE,
Title: "Payload",
Data: string(repPayload),
})
return rep
@ -609,35 +659,38 @@ func representDeleteTopicsRequest(data map[string]interface{}) []interface{} {
rep = representRequestHeader(data, rep)
payload := data["Payload"].(map[string]interface{})
payload := data["payload"].(map[string]interface{})
topics := ""
if payload["Topics"] != nil {
x, _ := json.Marshal(payload["Topics"].([]interface{}))
if payload["topics"] != nil {
x, _ := json.Marshal(payload["topics"].([]interface{}))
topics = string(x)
}
topicNames := ""
if payload["TopicNames"] != nil {
x, _ := json.Marshal(payload["TopicNames"].([]interface{}))
if payload["topicNames"] != nil {
x, _ := json.Marshal(payload["topicNames"].([]interface{}))
topicNames = string(x)
}
repPayload, _ := json.Marshal([]map[string]string{
repPayload, _ := json.Marshal([]api.TableData{
{
"name": "TopicNames",
"value": string(topicNames),
Name: "TopicNames",
Value: string(topicNames),
Selector: `request.payload.topicNames`,
},
{
"name": "Topics",
"value": string(topics),
Name: "Topics",
Value: string(topics),
Selector: `request.payload.topics`,
},
{
"name": "Timeout (ms)",
"value": fmt.Sprintf("%d", int(payload["TimeoutMs"].(float64))),
Name: "Timeout (ms)",
Value: fmt.Sprintf("%d", int(payload["timeoutMs"].(float64))),
Selector: `request.payload.timeoutMs`,
},
})
rep = append(rep, map[string]string{
"type": api.TABLE,
"title": "Payload",
"data": string(repPayload),
rep = append(rep, api.SectionData{
Type: api.TABLE,
Title: "Payload",
Data: string(repPayload),
})
return rep
@ -648,26 +701,28 @@ func representDeleteTopicsResponse(data map[string]interface{}) []interface{} {
rep = representResponseHeader(data, rep)
payload := data["Payload"].(map[string]interface{})
responses, _ := json.Marshal(payload["Responses"].([]interface{}))
payload := data["payload"].(map[string]interface{})
responses, _ := json.Marshal(payload["responses"].([]interface{}))
throttleTimeMs := ""
if payload["ThrottleTimeMs"] != nil {
throttleTimeMs = fmt.Sprintf("%d", int(payload["ThrottleTimeMs"].(float64)))
if payload["throttleTimeMs"] != nil {
throttleTimeMs = fmt.Sprintf("%d", int(payload["throttleTimeMs"].(float64)))
}
repPayload, _ := json.Marshal([]map[string]string{
repPayload, _ := json.Marshal([]api.TableData{
{
"name": "Throttle Time (ms)",
"value": throttleTimeMs,
Name: "Throttle Time (ms)",
Value: throttleTimeMs,
Selector: `response.payload.throttleTimeMs`,
},
{
"name": "Responses",
"value": string(responses),
Name: "Responses",
Value: string(responses),
Selector: `response.payload.responses`,
},
})
rep = append(rep, map[string]string{
"type": api.TABLE,
"title": "Payload",
"data": string(repPayload),
rep = append(rep, api.SectionData{
Type: api.TABLE,
Title: "Payload",
Data: string(repPayload),
})
return rep

135
tap/extensions/kafka/main.go
View File

@ -15,6 +15,7 @@ var _protocol api.Protocol = api.Protocol{
Name: "kafka",
LongName: "Apache Kafka Protocol",
Abbreviation: "KAFKA",
Macro: "kafka",
Version: "12",
BackgroundColor: "#000000",
ForegroundColor: "#ffffff",
@ -61,7 +62,7 @@ func (d dissecting) Dissect(b *bufio.Reader, isClient bool, tcpID *api.TcpID, co
}
}
func (d dissecting) Analyze(item *api.OutputChannelItem, entryId string, resolvedSource string, resolvedDestination string) *api.MizuEntry {
func (d dissecting) Analyze(item *api.OutputChannelItem, resolvedSource string, resolvedDestination string) *api.MizuEntry {
request := item.Pair.Request.Payload.(map[string]interface{})
reqDetails := request["details"].(map[string]interface{})
service := "kafka"
@ -70,76 +71,76 @@ func (d dissecting) Analyze(item *api.OutputChannelItem, entryId string, resolve
} else if resolvedSource != "" {
service = resolvedSource
}
apiKey := ApiKey(reqDetails["ApiKey"].(float64))
apiKey := ApiKey(reqDetails["apiKey"].(float64))
summary := ""
switch apiKey {
case Metadata:
_topics := reqDetails["Payload"].(map[string]interface{})["Topics"]
_topics := reqDetails["payload"].(map[string]interface{})["topics"]
if _topics == nil {
break
}
topics := _topics.([]interface{})
for _, topic := range topics {
summary += fmt.Sprintf("%s, ", topic.(map[string]interface{})["Name"].(string))
summary += fmt.Sprintf("%s, ", topic.(map[string]interface{})["name"].(string))
}
if len(summary) > 0 {
summary = summary[:len(summary)-2]
}
break
case ApiVersions:
summary = reqDetails["ClientID"].(string)
summary = reqDetails["clientID"].(string)
break
case Produce:
_topics := reqDetails["Payload"].(map[string]interface{})["TopicData"]
_topics := reqDetails["payload"].(map[string]interface{})["topicData"]
if _topics == nil {
break
}
topics := _topics.([]interface{})
for _, topic := range topics {
summary += fmt.Sprintf("%s, ", topic.(map[string]interface{})["Topic"].(string))
summary += fmt.Sprintf("%s, ", topic.(map[string]interface{})["topic"].(string))
}
if len(summary) > 0 {
summary = summary[:len(summary)-2]
}
break
case Fetch:
_topics := reqDetails["Payload"].(map[string]interface{})["Topics"]
_topics := reqDetails["payload"].(map[string]interface{})["topics"]
if _topics == nil {
break
}
topics := _topics.([]interface{})
for _, topic := range topics {
summary += fmt.Sprintf("%s, ", topic.(map[string]interface{})["Topic"].(string))
summary += fmt.Sprintf("%s, ", topic.(map[string]interface{})["topic"].(string))
}
if len(summary) > 0 {
summary = summary[:len(summary)-2]
}
break
case ListOffsets:
_topics := reqDetails["Payload"].(map[string]interface{})["Topics"]
_topics := reqDetails["payload"].(map[string]interface{})["topics"]
if _topics == nil {
break
}
topics := _topics.([]interface{})
for _, topic := range topics {
summary += fmt.Sprintf("%s, ", topic.(map[string]interface{})["Name"].(string))
summary += fmt.Sprintf("%s, ", topic.(map[string]interface{})["name"].(string))
}
if len(summary) > 0 {
summary = summary[:len(summary)-2]
}
break
case CreateTopics:
topics := reqDetails["Payload"].(map[string]interface{})["Topics"].([]interface{})
topics := reqDetails["payload"].(map[string]interface{})["topics"].([]interface{})
for _, topic := range topics {
summary += fmt.Sprintf("%s, ", topic.(map[string]interface{})["Name"].(string))
summary += fmt.Sprintf("%s, ", topic.(map[string]interface{})["name"].(string))
}
if len(summary) > 0 {
summary = summary[:len(summary)-2]
}
break
case DeleteTopics:
topicNames := reqDetails["TopicNames"].([]string)
topicNames := reqDetails["topicNames"].([]string)
for _, name := range topicNames {
summary += fmt.Sprintf("%s, ", name)
}
@ -148,44 +149,48 @@ func (d dissecting) Analyze(item *api.OutputChannelItem, entryId string, resolve
request["url"] = summary
elapsedTime := item.Pair.Response.CaptureTime.Sub(item.Pair.Request.CaptureTime).Round(time.Millisecond).Milliseconds()
entryBytes, _ := json.Marshal(item.Pair)
return &api.MizuEntry{
ProtocolName: _protocol.Name,
ProtocolLongName: _protocol.LongName,
ProtocolAbbreviation: _protocol.Abbreviation,
ProtocolVersion: _protocol.Version,
ProtocolBackgroundColor: _protocol.BackgroundColor,
ProtocolForegroundColor: _protocol.ForegroundColor,
ProtocolFontSize: _protocol.FontSize,
ProtocolReferenceLink: _protocol.ReferenceLink,
EntryId: entryId,
Entry: string(entryBytes),
Url: fmt.Sprintf("%s%s", service, summary),
Method: apiNames[apiKey],
Status: 0,
RequestSenderIp: item.ConnectionInfo.ClientIP,
Service: service,
Timestamp: item.Timestamp,
ElapsedTime: elapsedTime,
Path: summary,
ResolvedSource: resolvedSource,
ResolvedDestination: resolvedDestination,
SourceIp: item.ConnectionInfo.ClientIP,
DestinationIp: item.ConnectionInfo.ServerIP,
SourcePort: item.ConnectionInfo.ClientPort,
DestinationPort: item.ConnectionInfo.ServerPort,
IsOutgoing: item.ConnectionInfo.IsOutgoing,
Protocol: _protocol,
Source: &api.TCP{
Name: resolvedSource,
IP: item.ConnectionInfo.ClientIP,
Port: item.ConnectionInfo.ClientPort,
},
Destination: &api.TCP{
Name: resolvedDestination,
IP: item.ConnectionInfo.ServerIP,
Port: item.ConnectionInfo.ServerPort,
},
Outgoing: item.ConnectionInfo.IsOutgoing,
Request: reqDetails,
Response: item.Pair.Response.Payload.(map[string]interface{})["details"].(map[string]interface{}),
Url: fmt.Sprintf("%s%s", service, summary),
Method: apiNames[apiKey],
Status: 0,
RequestSenderIp: item.ConnectionInfo.ClientIP,
Service: service,
Timestamp: item.Timestamp,
StartTime: item.Pair.Request.CaptureTime,
ElapsedTime: elapsedTime,
Summary: summary,
ResolvedSource: resolvedSource,
ResolvedDestination: resolvedDestination,
SourceIp: item.ConnectionInfo.ClientIP,
DestinationIp: item.ConnectionInfo.ServerIP,
SourcePort: item.ConnectionInfo.ClientPort,
DestinationPort: item.ConnectionInfo.ServerPort,
IsOutgoing: item.ConnectionInfo.IsOutgoing,
}
}
func (d dissecting) Summarize(entry *api.MizuEntry) *api.BaseEntryDetails {
return &api.BaseEntryDetails{
Id: entry.EntryId,
Id: entry.Id,
Protocol: _protocol,
Url: entry.Url,
RequestSenderIp: entry.RequestSenderIp,
Service: entry.Service,
Summary: entry.Path,
Summary: entry.Summary,
StatusCode: entry.Status,
Method: entry.Method,
Timestamp: entry.Timestamp,
@ -202,49 +207,43 @@ func (d dissecting) Summarize(entry *api.MizuEntry) *api.BaseEntryDetails {
}
}
func (d dissecting) Represent(entry *api.MizuEntry) (p api.Protocol, object []byte, bodySize int64, err error) {
p = _protocol
func (d dissecting) Represent(protoIn api.Protocol, request map[string]interface{}, response map[string]interface{}) (protoOut api.Protocol, object []byte, bodySize int64, err error) {
protoOut = _protocol
bodySize = 0
var root map[string]interface{}
json.Unmarshal([]byte(entry.Entry), &root)
representation := make(map[string]interface{}, 0)
request := root["request"].(map[string]interface{})["payload"].(map[string]interface{})
response := root["response"].(map[string]interface{})["payload"].(map[string]interface{})
reqDetails := request["details"].(map[string]interface{})
resDetails := response["details"].(map[string]interface{})
apiKey := ApiKey(reqDetails["ApiKey"].(float64))
apiKey := ApiKey(request["apiKey"].(float64))
var repRequest []interface{}
var repResponse []interface{}
switch apiKey {
case Metadata:
repRequest = representMetadataRequest(reqDetails)
repResponse = representMetadataResponse(resDetails)
repRequest = representMetadataRequest(request)
repResponse = representMetadataResponse(response)
break
case ApiVersions:
repRequest = representApiVersionsRequest(reqDetails)
repResponse = representApiVersionsResponse(resDetails)
repRequest = representApiVersionsRequest(request)
repResponse = representApiVersionsResponse(response)
break
case Produce:
repRequest = representProduceRequest(reqDetails)
repResponse = representProduceResponse(resDetails)
repRequest = representProduceRequest(request)
repResponse = representProduceResponse(response)
break
case Fetch:
repRequest = representFetchRequest(reqDetails)
repResponse = representFetchResponse(resDetails)
repRequest = representFetchRequest(request)
repResponse = representFetchResponse(response)
break
case ListOffsets:
repRequest = representListOffsetsRequest(reqDetails)
repResponse = representListOffsetsResponse(resDetails)
repRequest = representListOffsetsRequest(request)
repResponse = representListOffsetsResponse(response)
break
case CreateTopics:
repRequest = representCreateTopicsRequest(reqDetails)
repResponse = representCreateTopicsResponse(resDetails)
repRequest = representCreateTopicsRequest(request)
repResponse = representCreateTopicsResponse(response)
break
case DeleteTopics:
repRequest = representDeleteTopicsRequest(reqDetails)
repResponse = representDeleteTopicsResponse(resDetails)
repRequest = representDeleteTopicsRequest(request)
repResponse = representDeleteTopicsResponse(response)
break
}
@ -254,4 +253,10 @@ func (d dissecting) Represent(entry *api.MizuEntry) (p api.Protocol, object []by
return
}
func (d dissecting) Macros() map[string]string {
return map[string]string{
`kafka`: fmt.Sprintf(`proto.abbr == "%s"`, _protocol.Abbreviation),
}
}
var Dissector dissecting

14
tap/extensions/kafka/request.go
View File

@ -10,13 +10,13 @@ import (
)
type Request struct {
Size int32
ApiKey ApiKey
ApiVersion int16
CorrelationID int32
ClientID string
Payload interface{}
CaptureTime time.Time
Size int32 `json:"size"`
ApiKey ApiKey `json:"apiKey"`
ApiVersion int16 `json:"apiVersion"`
CorrelationID int32 `json:"correlationID"`
ClientID string `json:"clientID"`
Payload interface{} `json:"payload"`
CaptureTime time.Time `json:"captureTime"`
}
func ReadRequest(r io.Reader, tcpID *api.TcpID, superTimer *api.SuperTimer) (apiKey ApiKey, apiVersion int16, err error) {

8
tap/extensions/kafka/response.go
View File

@ -10,10 +10,10 @@ import (
)
type Response struct {
Size int32
CorrelationID int32
Payload interface{}
CaptureTime time.Time
Size int32 `json:"size"`
CorrelationID int32 `json:"correlationID"`
Payload interface{} `json:"payload"`
CaptureTime time.Time `json:"captureTime"`
}
func ReadResponse(r io.Reader, tcpID *api.TcpID, superTimer *api.SuperTimer, emitter api.Emitter) (err error) {

894
tap/extensions/kafka/structs.go
View File

File diff suppressed because it is too large Load Diff

38
tap/extensions/redis/helpers.go
View File

@ -2,6 +2,7 @@ package main
import (
"encoding/json"
"fmt"
"github.com/up9inc/mizu/tap/api"
)
@ -24,33 +25,38 @@ type RedisWrapper struct {
Details interface{} `json:"details"`
}
func representGeneric(generic map[string]interface{}) (representation []interface{}) {
details, _ := json.Marshal([]map[string]string{
func representGeneric(generic map[string]interface{}, selectorPrefix string) (representation []interface{}) {
details, _ := json.Marshal([]api.TableData{
{
"name": "Type",
"value": generic["type"].(string),
Name: "Type",
Value: generic["type"].(string),
Selector: fmt.Sprintf("%stype", selectorPrefix),
},
{
"name": "Command",
"value": generic["command"].(string),
Name: "Command",
Value: generic["command"].(string),
Selector: fmt.Sprintf("%scommand", selectorPrefix),
},
{
"name": "Key",
"value": generic["key"].(string),
Name: "Key",
Value: generic["key"].(string),
Selector: fmt.Sprintf("%skey", selectorPrefix),
},
{
"name": "Value",
"value": generic["value"].(string),
Name: "Value",
Value: generic["value"].(string),
Selector: fmt.Sprintf("%svalue", selectorPrefix),
},
{
"name": "Keyword",
"value": generic["keyword"].(string),
Name: "Keyword",
Value: generic["keyword"].(string),
Selector: fmt.Sprintf("%skeyword", selectorPrefix),
},
})
representation = append(representation, map[string]string{
"type": api.TABLE,
"title": "Details",
"data": string(details),
representation = append(representation, api.SectionData{
Type: api.TABLE,
Title: "Details",
Data: string(details),
})
return

86
tap/extensions/redis/main.go
View File

@ -13,6 +13,7 @@ var protocol api.Protocol = api.Protocol{
Name: "redis",
LongName: "Redis Serialization Protocol",
Abbreviation: "REDIS",
Macro: "redis",
Version: "3.x",
BackgroundColor: "#a41e11",
ForegroundColor: "#ffffff",
@ -57,9 +58,12 @@ func (d dissecting) Dissect(b *bufio.Reader, isClient bool, tcpID *api.TcpID, co
}
}
func (d dissecting) Analyze(item *api.OutputChannelItem, entryId string, resolvedSource string, resolvedDestination string) *api.MizuEntry {
func (d dissecting) Analyze(item *api.OutputChannelItem, resolvedSource string, resolvedDestination string) *api.MizuEntry {
request := item.Pair.Request.Payload.(map[string]interface{})
response := item.Pair.Response.Payload.(map[string]interface{})
reqDetails := request["details"].(map[string]interface{})
resDetails := response["details"].(map[string]interface{})
service := "redis"
if resolvedDestination != "" {
service = resolvedDestination
@ -78,45 +82,49 @@ func (d dissecting) Analyze(item *api.OutputChannelItem, entryId string, resolve
}
request["url"] = summary
entryBytes, _ := json.Marshal(item.Pair)
return &api.MizuEntry{
ProtocolName: protocol.Name,
ProtocolLongName: protocol.LongName,
ProtocolAbbreviation: protocol.Abbreviation,
ProtocolVersion: protocol.Version,
ProtocolBackgroundColor: protocol.BackgroundColor,
ProtocolForegroundColor: protocol.ForegroundColor,
ProtocolFontSize: protocol.FontSize,
ProtocolReferenceLink: protocol.ReferenceLink,
EntryId: entryId,
Entry: string(entryBytes),
Url: fmt.Sprintf("%s%s", service, summary),
Method: method,
Status: 0,
RequestSenderIp: item.ConnectionInfo.ClientIP,
Service: service,
Timestamp: item.Timestamp,
ElapsedTime: 0,
Path: summary,
ResolvedSource: resolvedSource,
ResolvedDestination: resolvedDestination,
SourceIp: item.ConnectionInfo.ClientIP,
DestinationIp: item.ConnectionInfo.ServerIP,
SourcePort: item.ConnectionInfo.ClientPort,
DestinationPort: item.ConnectionInfo.ServerPort,
IsOutgoing: item.ConnectionInfo.IsOutgoing,
Protocol: protocol,
Source: &api.TCP{
Name: resolvedSource,
IP: item.ConnectionInfo.ClientIP,
Port: item.ConnectionInfo.ClientPort,
},
Destination: &api.TCP{
Name: resolvedDestination,
IP: item.ConnectionInfo.ServerIP,
Port: item.ConnectionInfo.ServerPort,
},
Outgoing: item.ConnectionInfo.IsOutgoing,
Request: reqDetails,
Response: resDetails,
Url: fmt.Sprintf("%s%s", service, summary),
Method: method,
Status: 0,
RequestSenderIp: item.ConnectionInfo.ClientIP,
Service: service,
Timestamp: item.Timestamp,
StartTime: item.Pair.Request.CaptureTime,
ElapsedTime: 0,
Summary: summary,
ResolvedSource: resolvedSource,
ResolvedDestination: resolvedDestination,
SourceIp: item.ConnectionInfo.ClientIP,
DestinationIp: item.ConnectionInfo.ServerIP,
SourcePort: item.ConnectionInfo.ClientPort,
DestinationPort: item.ConnectionInfo.ServerPort,
IsOutgoing: item.ConnectionInfo.IsOutgoing,
}
}
func (d dissecting) Summarize(entry *api.MizuEntry) *api.BaseEntryDetails {
return &api.BaseEntryDetails{
Id: entry.EntryId,
Id: entry.Id,
Protocol: protocol,
Url: entry.Url,
RequestSenderIp: entry.RequestSenderIp,
Service: entry.Service,
Summary: entry.Path,
Summary: entry.Summary,
StatusCode: entry.Status,
Method: entry.Method,
Timestamp: entry.Timestamp,
@ -133,22 +141,22 @@ func (d dissecting) Summarize(entry *api.MizuEntry) *api.BaseEntryDetails {
}
}
func (d dissecting) Represent(entry *api.MizuEntry) (p api.Protocol, object []byte, bodySize int64, err error) {
p = protocol
func (d dissecting) Represent(protoIn api.Protocol, request map[string]interface{}, response map[string]interface{}) (protoOut api.Protocol, object []byte, bodySize int64, err error) {
protoOut = protocol
bodySize = 0
var root map[string]interface{}
json.Unmarshal([]byte(entry.Entry), &root)
representation := make(map[string]interface{}, 0)
request := root["request"].(map[string]interface{})["payload"].(map[string]interface{})
response := root["response"].(map[string]interface{})["payload"].(map[string]interface{})
reqDetails := request["details"].(map[string]interface{})
resDetails := response["details"].(map[string]interface{})
repRequest := representGeneric(reqDetails)
repResponse := representGeneric(resDetails)
repRequest := representGeneric(request, `request.`)
repResponse := representGeneric(response, `response.`)
representation["request"] = repRequest
representation["response"] = repResponse
object, err = json.Marshal(representation)
return
}
func (d dissecting) Macros() map[string]string {
return map[string]string{
`redis`: fmt.Sprintf(`proto.abbr == "%s"`, protocol.Abbreviation),
}
}
var Dissector dissecting

350
ui/package-lock.json generated
View File

@ -1769,6 +1769,33 @@
}
}
},
"@mapbox/rehype-prism": {
"version": "0.7.0",
"resolved": "https://registry.npmjs.org/@mapbox/rehype-prism/-/rehype-prism-0.7.0.tgz",
"integrity": "sha512-zSG46selA6v+3THhCatTyOt9DuTzxTIVTxTbcj15kFmxPDtjzZ5VoFVCLZfjWFouYa9PiXxcbMLLhJoVzCxh9w==",
"requires": {
"hast-util-to-string": "^1.0.4",
"refractor": "^3.4.0",
"unist-util-visit": "^2.0.3"
},
"dependencies": {
"prismjs": {
"version": "1.24.1",
"resolved": "https://registry.npmjs.org/prismjs/-/prismjs-1.24.1.tgz",
"integrity": "sha512-mNPsedLuk90RVJioIky8ANZEwYm5w9LcvCXrxHlwf4fNVSn8jEipMybMkWUyyF0JhnC+C4VcOVSBuHRKs1L5Ow=="
},
"refractor": {
"version": "3.4.0",
"resolved": "https://registry.npmjs.org/refractor/-/refractor-3.4.0.tgz",
"integrity": "sha512-dBeD02lC5eytm9Gld2Mx0cMcnR+zhSnsTfPpWqFaMgUMJfC9A6bcN3Br/NaXrnBJcuxnLFR90k1jrkaSyV8umg==",
"requires": {
"hastscript": "^6.0.0",
"parse-entities": "^2.0.0",
"prismjs": "~1.24.0"
}
}
}
},
"@material-ui/core": {
"version": "4.12.2",
"resolved": "https://registry.npmjs.org/@material-ui/core/-/core-4.12.2.tgz",
@ -1788,6 +1815,14 @@
"react-transition-group": "^4.4.0"
}
},
"@material-ui/icons": {
"version": "4.11.2",
"resolved": "https://registry.npmjs.org/@material-ui/icons/-/icons-4.11.2.tgz",
"integrity": "sha512-fQNsKX2TxBmqIGJCSi3tGTO/gZ+eJgWmMJkgDiOfyNaunNaxcklJQFaFogYcFl0qFuaEz1qaXYXboa/bUXVSOQ==",
"requires": {
"@babel/runtime": "^7.4.4"
}
},
"@material-ui/lab": {
"version": "4.0.0-alpha.60",
"resolved": "https://registry.npmjs.org/@material-ui/lab/-/lab-4.0.0-alpha.60.tgz",
@ -2406,6 +2441,11 @@
"resolved": "https://registry.npmjs.org/@types/parse-json/-/parse-json-4.0.0.tgz",
"integrity": "sha512-//oorEZjL6sbPcKUaCdIGlIUeH26mgzimjBB77G6XRgnDl/L5wOnpyBGRe/Mmf5CVW3PwEBE1NjiMZ/ssFh4wA=="
},
"@types/parse5": {
"version": "6.0.1",
"resolved": "https://registry.npmjs.org/@types/parse5/-/parse5-6.0.1.tgz",
"integrity": "sha512-ARATsLdrGPUnaBvxLhUlnltcMgn7pQG312S8ccdYlnyijabrX9RN/KN/iGj9Am96CoW8e/K9628BA7Bv4XHdrA=="
},
"@types/prettier": {
"version": "2.2.3",
"resolved": "https://registry.npmjs.org/@types/prettier/-/prettier-2.2.3.tgz",
@ -2629,6 +2669,26 @@
"eslint-visitor-keys": "^2.0.0"
}
},
"@uiw/react-textarea-code-editor": {
"version": "1.4.12",
"resolved": "https://registry.npmjs.org/@uiw/react-textarea-code-editor/-/react-textarea-code-editor-1.4.12.tgz",
"integrity": "sha512-op0aIRxX8hLi8OLwm/23dQ2X4o2xHUoK2pLr1DWBFgNbIh4L+RM5zByNFjV8mHfi/NvE6mQLB6LtMd1qaor5MQ==",
"requires": {
"@babel/runtime": "7.15.4",
"@mapbox/rehype-prism": "0.7.0",
"rehype": "12.0.0"
},
"dependencies": {
"@babel/runtime": {
"version": "7.15.4",
"resolved": "https://registry.npmjs.org/@babel/runtime/-/runtime-7.15.4.tgz",
"integrity": "sha512-99catp6bHCaxr4sJ/DbTGgHS4+Rs2RVd2g7iOap6SLGPDknRK9ztKNsE/Fg6QhSeh1FGE5f6gHGQmvvn3I3xhw==",
"requires": {
"regenerator-runtime": "^0.13.4"
}
}
}
},
"@webassemblyjs/ast": {
"version": "1.9.0",
"resolved": "https://registry.npmjs.org/@webassemblyjs/ast/-/ast-1.9.0.tgz",
@ -3689,6 +3749,11 @@
"resolved": "https://registry.npmjs.org/babylon/-/babylon-6.18.0.tgz",
"integrity": "sha512-q/UEjfGJ2Cm3oKV71DJz9d25TPnq5rhBVL2Q4fA5wcC3jcrdn7+SssEybFIxwAvvP+YCsCYNKughoF33GxgycQ=="
},
"bail": {
"version": "2.0.1",
"resolved": "https://registry.npmjs.org/bail/-/bail-2.0.1.tgz",
"integrity": "sha512-d5FoTAr2S5DSUPKl85WNm2yUwsINN8eidIdIwsOge2t33DaOfOdSmmsI11jMN3GmALCXaw+Y6HMVHDzePshFAA=="
},
"balanced-match": {
"version": "1.0.2",
"resolved": "https://registry.npmjs.org/balanced-match/-/balanced-match-1.0.2.tgz",
@ -4179,6 +4244,11 @@
"resolved": "https://registry.npmjs.org/caseless/-/caseless-0.12.0.tgz",
"integrity": "sha1-G2gcIf+EAzyCZUMJBolCDRhxUdw="
},
"ccount": {
"version": "2.0.0",
"resolved": "https://registry.npmjs.org/ccount/-/ccount-2.0.0.tgz",
"integrity": "sha512-VOR0NWFYX65n9gELQdcpqsie5L5ihBXuZGAgaPEp/U7IOSjnPMEH6geE+2f6lcekaNEfWzAHS45mPvSo5bqsUA=="
},
"chalk": {
"version": "2.4.2",
"resolved": "https://registry.npmjs.org/chalk/-/chalk-2.4.2.tgz",
@ -4199,6 +4269,11 @@
"resolved": "https://registry.npmjs.org/character-entities/-/character-entities-1.2.4.tgz",
"integrity": "sha512-iBMyeEHxfVnIakwOuDXpVkc54HijNgCyQB2w0VfGQThle6NXn50zU6V/u+LDhxHcDUPojn6Kpga3PTAD8W1bQw=="
},
"character-entities-html4": {
"version": "2.0.0",
"resolved": "https://registry.npmjs.org/character-entities-html4/-/character-entities-html4-2.0.0.tgz",
"integrity": "sha512-dwT2xh5ZhUAjyP96k57ilMKoTQyASaw9IAMR9U5c1lCu2RUni6O6jxfpUEdO2RcPT6TJFvr8pqsbami4Jk+2oA=="
},
"character-entities-legacy": {
"version": "1.1.4",
"resolved": "https://registry.npmjs.org/character-entities-legacy/-/character-entities-legacy-1.1.4.tgz",
@ -7534,11 +7609,121 @@
"minimalistic-assert": "^1.0.1"
}
},
"hast-util-from-parse5": {
"version": "7.1.0",
"resolved": "https://registry.npmjs.org/hast-util-from-parse5/-/hast-util-from-parse5-7.1.0.tgz",
"integrity": "sha512-m8yhANIAccpU4K6+121KpPP55sSl9/samzQSQGpb0mTExcNh2WlvjtMwSWFhg6uqD4Rr6Nfa8N6TMypQM51rzQ==",
"requires": {
"@types/hast": "^2.0.0",
"@types/parse5": "^6.0.0",
"@types/unist": "^2.0.0",
"hastscript": "^7.0.0",
"property-information": "^6.0.0",
"vfile": "^5.0.0",
"vfile-location": "^4.0.0",
"web-namespaces": "^2.0.0"
},
"dependencies": {
"comma-separated-tokens": {
"version": "2.0.2",
"resolved": "https://registry.npmjs.org/comma-separated-tokens/-/comma-separated-tokens-2.0.2.tgz",
"integrity": "sha512-G5yTt3KQN4Yn7Yk4ed73hlZ1evrFKXeUW3086p3PRFNp7m2vIjI6Pg+Kgb+oyzhd9F2qdcoj67+y3SdxL5XWsg=="
},
"hast-util-parse-selector": {
"version": "3.1.0",
"resolved": "https://registry.npmjs.org/hast-util-parse-selector/-/hast-util-parse-selector-3.1.0.tgz",
"integrity": "sha512-AyjlI2pTAZEOeu7GeBPZhROx0RHBnydkQIXlhnFzDi0qfXTmGUWoCYZtomHbrdrheV4VFUlPcfJ6LMF5T6sQzg==",
"requires": {
"@types/hast": "^2.0.0"
}
},
"hastscript": {
"version": "7.0.2",
"resolved": "https://registry.npmjs.org/hastscript/-/hastscript-7.0.2.tgz",
"integrity": "sha512-uA8ooUY4ipaBvKcMuPehTAB/YfFLSSzCwFSwT6ltJbocFUKH/GDHLN+tflq7lSRf9H86uOuxOFkh1KgIy3Gg2g==",
"requires": {
"@types/hast": "^2.0.0",
"comma-separated-tokens": "^2.0.0",
"hast-util-parse-selector": "^3.0.0",
"property-information": "^6.0.0",
"space-separated-tokens": "^2.0.0"
}
},
"property-information": {
"version": "6.0.1",
"resolved": "https://registry.npmjs.org/property-information/-/property-information-6.0.1.tgz",
"integrity": "sha512-F4WUUAF7fMeF4/JUFHNBWDaKDXi2jbvqBW/y6o5wsf3j19wTZ7S60TmtB5HoBhtgw7NKQRMWuz5vk2PR0CygUg=="
},
"space-separated-tokens": {
"version": "2.0.1",
"resolved": "https://registry.npmjs.org/space-separated-tokens/-/space-separated-tokens-2.0.1.tgz",
"integrity": "sha512-ekwEbFp5aqSPKaqeY1PGrlGQxPNaq+Cnx4+bE2D8sciBQrHpbwoBbawqTN2+6jPs9IdWxxiUcN0K2pkczD3zmw=="
}
}
},
"hast-util-is-element": {
"version": "2.1.1",
"resolved": "https://registry.npmjs.org/hast-util-is-element/-/hast-util-is-element-2.1.1.tgz",
"integrity": "sha512-ag0fiZfRWsPiR1udvnSbaazJLGv8qd8E+/e3rW8rUZhbKG4HNJmFL4QkEceN+22BgE+uozXY30z/s+2dL6Z++g==",
"requires": {
"@types/hast": "^2.0.0",
"@types/unist": "^2.0.0"
}
},
"hast-util-parse-selector": {
"version": "2.2.5",
"resolved": "https://registry.npmjs.org/hast-util-parse-selector/-/hast-util-parse-selector-2.2.5.tgz",
"integrity": "sha512-7j6mrk/qqkSehsM92wQjdIgWM2/BW61u/53G6xmC8i1OmEdKLHbk419QKQUjz6LglWsfqoiHmyMRkP1BGjecNQ=="
},
"hast-util-to-html": {
"version": "8.0.2",
"resolved": "https://registry.npmjs.org/hast-util-to-html/-/hast-util-to-html-8.0.2.tgz",
"integrity": "sha512-ipLhUTMyyJi9F/LXaNDG9BrRdshP6obCfmUZYbE/+T639IdzqAOkKN4DyrEyID0gbb+rsC3PKf0XlviZwzomhw==",
"requires": {
"@types/hast": "^2.0.0",
"ccount": "^2.0.0",
"comma-separated-tokens": "^2.0.0",
"hast-util-is-element": "^2.0.0",
"hast-util-whitespace": "^2.0.0",
"html-void-elements": "^2.0.0",
"property-information": "^6.0.0",
"space-separated-tokens": "^2.0.0",
"stringify-entities": "^4.0.0",
"unist-util-is": "^5.0.0"
},
"dependencies": {
"comma-separated-tokens": {
"version": "2.0.2",
"resolved": "https://registry.npmjs.org/comma-separated-tokens/-/comma-separated-tokens-2.0.2.tgz",
"integrity": "sha512-G5yTt3KQN4Yn7Yk4ed73hlZ1evrFKXeUW3086p3PRFNp7m2vIjI6Pg+Kgb+oyzhd9F2qdcoj67+y3SdxL5XWsg=="
},
"property-information": {
"version": "6.0.1",
"resolved": "https://registry.npmjs.org/property-information/-/property-information-6.0.1.tgz",
"integrity": "sha512-F4WUUAF7fMeF4/JUFHNBWDaKDXi2jbvqBW/y6o5wsf3j19wTZ7S60TmtB5HoBhtgw7NKQRMWuz5vk2PR0CygUg=="
},
"space-separated-tokens": {
"version": "2.0.1",
"resolved": "https://registry.npmjs.org/space-separated-tokens/-/space-separated-tokens-2.0.1.tgz",
"integrity": "sha512-ekwEbFp5aqSPKaqeY1PGrlGQxPNaq+Cnx4+bE2D8sciBQrHpbwoBbawqTN2+6jPs9IdWxxiUcN0K2pkczD3zmw=="
},
"unist-util-is": {
"version": "5.1.1",
"resolved": "https://registry.npmjs.org/unist-util-is/-/unist-util-is-5.1.1.tgz",
"integrity": "sha512-F5CZ68eYzuSvJjGhCLPL3cYx45IxkqXSetCcRgUXtbcm50X2L9oOWQlfUfDdAf+6Pd27YDblBfdtmsThXmwpbQ=="
}
}
},
"hast-util-to-string": {
"version": "1.0.4",
"resolved": "https://registry.npmjs.org/hast-util-to-string/-/hast-util-to-string-1.0.4.tgz",
"integrity": "sha512-eK0MxRX47AV2eZ+Lyr18DCpQgodvaS3fAQO2+b9Two9F5HEoRPhiUMNzoXArMJfZi2yieFzUBMRl3HNJ3Jus3w=="
},
"hast-util-whitespace": {
"version": "2.0.0",
"resolved": "https://registry.npmjs.org/hast-util-whitespace/-/hast-util-whitespace-2.0.0.tgz",
"integrity": "sha512-Pkw+xBHuV6xFeJprJe2BBEoDV+AvQySaz3pPDRUs5PNZEMQjpXJJueqrpcHIXxnWTcAGi/UOCgVShlkY6kLoqg=="
},
"hastscript": {
"version": "6.0.0",
"resolved": "https://registry.npmjs.org/hastscript/-/hastscript-6.0.0.tgz",
@ -7671,6 +7856,11 @@
"terser": "^4.6.3"
}
},
"html-void-elements": {
"version": "2.0.0",
"resolved": "https://registry.npmjs.org/html-void-elements/-/html-void-elements-2.0.0.tgz",
"integrity": "sha512-4OYzQQsBt0G9bJ/nM9/DDsjm4+fVdzAaPJJcWk5QwA3GIAPxQEeOR0rsI8HbDHQz5Gta8pVvGnnTNSbZVEVvkQ=="
},
"html-webpack-plugin": {
"version": "4.5.0",
"resolved": "https://registry.npmjs.org/html-webpack-plugin/-/html-webpack-plugin-4.5.0.tgz",
@ -13470,6 +13660,14 @@
"refractor": "^3.2.0"
}
},
"react-toastify": {
"version": "8.0.3",
"resolved": "https://registry.npmjs.org/react-toastify/-/react-toastify-8.0.3.tgz",
"integrity": "sha512-rv3koC7f9lKKSkdpYgo/TGzgWlrB/aaiUInF1DyV7BpiM4kyTs+uhu6/r8XDMtBY2FOIHK+FlK3Iv7OzpA/tCA==",
"requires": {
"clsx": "^1.1.1"
}
},
"react-transition-group": {
"version": "4.4.1",
"resolved": "https://registry.npmjs.org/react-transition-group/-/react-transition-group-4.4.1.tgz",
@ -13693,6 +13891,38 @@
}
}
},
"rehype": {
"version": "12.0.0",
"resolved": "https://registry.npmjs.org/rehype/-/rehype-12.0.0.tgz",
"integrity": "sha512-gZcttmf9R5IYHb8AlI1rlmWqXS1yX0rSB/S5ZGJs8atfYZy2DobvH3Ic/gSzB+HL/+oOHPtBguw1TprfhxXBgQ==",
"requires": {
"@types/hast": "^2.0.0",
"rehype-parse": "^8.0.0",
"rehype-stringify": "^9.0.0",
"unified": "^10.0.0"
}
},
"rehype-parse": {
"version": "8.0.3",
"resolved": "https://registry.npmjs.org/rehype-parse/-/rehype-parse-8.0.3.tgz",
"integrity": "sha512-RGw0CVt+0S6KdvpE8bbP2Db9WXclQcIX7A0ufM3QFqAhTo/ddJMQrrI2j3cijlRPZlGK8R3pRgC8U5HyV76IDw==",
"requires": {
"@types/hast": "^2.0.0",
"hast-util-from-parse5": "^7.0.0",
"parse5": "^6.0.0",
"unified": "^10.0.0"
}
},
"rehype-stringify": {
"version": "9.0.2",
"resolved": "https://registry.npmjs.org/rehype-stringify/-/rehype-stringify-9.0.2.tgz",
"integrity": "sha512-BuVA6lAEYtOpXO2xuHLohAzz8UNoQAxAqYRqh4QEEtU39Co+P1JBZhw6wXA9hMWp+JLcmrxWH8+UKcNSr443Fw==",
"requires": {
"@types/hast": "^2.0.0",
"hast-util-to-html": "^8.0.0",
"unified": "^10.0.0"
}
},
"relateurl": {
"version": "0.2.7",
"resolved": "https://registry.npmjs.org/relateurl/-/relateurl-0.2.7.tgz",
@ -15342,6 +15572,22 @@
}
}
},
"stringify-entities": {
"version": "4.0.1",
"resolved": "https://registry.npmjs.org/stringify-entities/-/stringify-entities-4.0.1.tgz",
"integrity": "sha512-gmMQxKXPWIO3NXNSPyWNhlYcBNGpPA/487D+9dLPnU4xBnIrnHdr8cv5rGJOS/1BRxEXRb7uKwg7BA36IWV7xg==",
"requires": {
"character-entities-html4": "^2.0.0",
"character-entities-legacy": "^2.0.0"
},
"dependencies": {
"character-entities-legacy": {
"version": "2.0.0",
"resolved": "https://registry.npmjs.org/character-entities-legacy/-/character-entities-legacy-2.0.0.tgz",
"integrity": "sha512-YwaEtEvWLpFa6Wh3uVLrvirA/ahr9fki/NUd/Bd4OR6EdJ8D22hovYQEOUCBfQfcqnC4IAMGMsHXY1eXgL4ZZA=="
}
}
},
"stringify-object": {
"version": "3.3.0",
"resolved": "https://registry.npmjs.org/stringify-object/-/stringify-object-3.3.0.tgz",
@ -15877,6 +16123,11 @@
"resolved": "https://registry.npmjs.org/trim-newlines/-/trim-newlines-1.0.0.tgz",
"integrity": "sha1-WIeWa7WCpFA6QetST301ARgVphM="
},
"trough": {
"version": "2.0.2",
"resolved": "https://registry.npmjs.org/trough/-/trough-2.0.2.tgz",
"integrity": "sha512-FnHq5sTMxC0sk957wHDzRnemFnNBvt/gSY99HzK8F7UP5WAbvP70yX5bd7CjEQkN+TjdxwI7g7lJ6podqrG2/w=="
},
"true-case-path": {
"version": "1.0.3",
"resolved": "https://registry.npmjs.org/true-case-path/-/true-case-path-1.0.3.tgz",
@ -16037,6 +16288,32 @@
"resolved": "https://registry.npmjs.org/unicode-property-aliases-ecmascript/-/unicode-property-aliases-ecmascript-1.1.0.tgz",
"integrity": "sha512-PqSoPh/pWetQ2phoj5RLiaqIk4kCNwoV3CI+LfGmWLKI3rE3kl1h59XpX2BjgDrmbxD9ARtQobPGU1SguCYuQg=="
},
"unified": {
"version": "10.1.0",
"resolved": "https://registry.npmjs.org/unified/-/unified-10.1.0.tgz",
"integrity": "sha512-4U3ru/BRXYYhKbwXV6lU6bufLikoAavTwev89H5UxY8enDFaAT2VXmIXYNm6hb5oHPng/EXr77PVyDFcptbk5g==",
"requires": {
"@types/unist": "^2.0.0",
"bail": "^2.0.0",
"extend": "^3.0.0",
"is-buffer": "^2.0.0",
"is-plain-obj": "^4.0.0",
"trough": "^2.0.0",
"vfile": "^5.0.0"
},
"dependencies": {
"is-buffer": {
"version": "2.0.5",
"resolved": "https://registry.npmjs.org/is-buffer/-/is-buffer-2.0.5.tgz",
"integrity": "sha512-i2R6zNFDwgEHJyQUtJEk0XFi1i0dPFn/oqjK3/vPCcDeJvW5NQ83V8QbicfF1SupOaB0h8ntgBC2YiE7dfyctQ=="
},
"is-plain-obj": {
"version": "4.0.0",
"resolved": "https://registry.npmjs.org/is-plain-obj/-/is-plain-obj-4.0.0.tgz",
"integrity": "sha512-NXRbBtUdBioI73y/HmOhogw/U5msYPC9DAtGkJXeFcFWSFZw0mCUsPxk/snTuJHzNKA8kLBK4rH97RMB1BfCXw=="
}
}
},
"union-value": {
"version": "1.0.1",
"resolved": "https://registry.npmjs.org/union-value/-/union-value-1.0.1.tgz",
@ -16082,6 +16359,38 @@
"crypto-random-string": "^1.0.0"
}
},
"unist-util-is": {
"version": "4.1.0",
"resolved": "https://registry.npmjs.org/unist-util-is/-/unist-util-is-4.1.0.tgz",
"integrity": "sha512-ZOQSsnce92GrxSqlnEEseX0gi7GH9zTJZ0p9dtu87WRb/37mMPO2Ilx1s/t9vBHrFhbgweUwb+t7cIn5dxPhZg=="
},
"unist-util-stringify-position": {
"version": "3.0.0",
"resolved": "https://registry.npmjs.org/unist-util-stringify-position/-/unist-util-stringify-position-3.0.0.tgz",
"integrity": "sha512-SdfAl8fsDclywZpfMDTVDxA2V7LjtRDTOFd44wUJamgl6OlVngsqWjxvermMYf60elWHbxhuRCZml7AnuXCaSA==",
"requires": {
"@types/unist": "^2.0.0"
}
},
"unist-util-visit": {
"version": "2.0.3",
"resolved": "https://registry.npmjs.org/unist-util-visit/-/unist-util-visit-2.0.3.tgz",
"integrity": "sha512-iJ4/RczbJMkD0712mGktuGpm/U4By4FfDonL7N/9tATGIF4imikjOuagyMY53tnZq3NP6BcmlrHhEKAfGWjh7Q==",
"requires": {
"@types/unist": "^2.0.0",
"unist-util-is": "^4.0.0",
"unist-util-visit-parents": "^3.0.0"
}
},
"unist-util-visit-parents": {
"version": "3.1.1",
"resolved": "https://registry.npmjs.org/unist-util-visit-parents/-/unist-util-visit-parents-3.1.1.tgz",
"integrity": "sha512-1KROIZWo6bcMrZEwiH2UrXDyalAa0uqzWCxCJj6lPOvTve2WkfgCytoDTPaMnodXh1WrXOq0haVYHj99ynJlsg==",
"requires": {
"@types/unist": "^2.0.0",
"unist-util-is": "^4.0.0"
}
},
"universalify": {
"version": "2.0.0",
"resolved": "https://registry.npmjs.org/universalify/-/universalify-2.0.0.tgz",
@ -16306,6 +16615,42 @@
"extsprintf": "^1.2.0"
}
},
"vfile": {
"version": "5.1.0",
"resolved": "https://registry.npmjs.org/vfile/-/vfile-5.1.0.tgz",
"integrity": "sha512-4o7/DJjEaFPYSh0ckv5kcYkJTHQgCKdL8ozMM1jLAxO9ox95IzveDPXCZp08HamdWq8JXTkClDvfAKaeLQeKtg==",
"requires": {
"@types/unist": "^2.0.0",
"is-buffer": "^2.0.0",
"unist-util-stringify-position": "^3.0.0",
"vfile-message": "^3.0.0"
},
"dependencies": {
"is-buffer": {
"version": "2.0.5",
"resolved": "https://registry.npmjs.org/is-buffer/-/is-buffer-2.0.5.tgz",
"integrity": "sha512-i2R6zNFDwgEHJyQUtJEk0XFi1i0dPFn/oqjK3/vPCcDeJvW5NQ83V8QbicfF1SupOaB0h8ntgBC2YiE7dfyctQ=="
}
}
},
"vfile-location": {
"version": "4.0.1",
"resolved": "https://registry.npmjs.org/vfile-location/-/vfile-location-4.0.1.tgz",
"integrity": "sha512-JDxPlTbZrZCQXogGheBHjbRWjESSPEak770XwWPfw5mTc1v1nWGLB/apzZxsx8a0SJVfF8HK8ql8RD308vXRUw==",
"requires": {
"@types/unist": "^2.0.0",
"vfile": "^5.0.0"
}
},
"vfile-message": {
"version": "3.0.2",
"resolved": "https://registry.npmjs.org/vfile-message/-/vfile-message-3.0.2.tgz",
"integrity": "sha512-UUjZYIOg9lDRwwiBAuezLIsu9KlXntdxwG+nXnjuQAHvBpcX3x0eN8h+I7TkY5nkCXj+cWVp4ZqebtGBvok8ww==",
"requires": {
"@types/unist": "^2.0.0",
"unist-util-stringify-position": "^3.0.0"
}
},
"vm-browserify": {
"version": "1.1.2",
"resolved": "https://registry.npmjs.org/vm-browserify/-/vm-browserify-1.1.2.tgz",
@ -16590,6 +16935,11 @@
"minimalistic-assert": "^1.0.0"
}
},
"web-namespaces": {
"version": "2.0.0",
"resolved": "https://registry.npmjs.org/web-namespaces/-/web-namespaces-2.0.0.tgz",
"integrity": "sha512-dE7ELZRVWh0ceQsRgkjLgsAvwTuv3kcjSY/hLjqL0llleUlQBDjE9JkB9FCBY5F2mnFEwiyJoowl8+NVGHe8dw=="
},
"web-vitals": {
"version": "1.1.1",
"resolved": "https://registry.npmjs.org/web-vitals/-/web-vitals-1.1.1.tgz",

3
ui/package.json
View File

@ -4,6 +4,7 @@
"private": true,
"dependencies": {
"@material-ui/core": "^4.11.3",
"@material-ui/icons": "^4.11.2",
"@material-ui/lab": "^4.0.0-alpha.60",
"@testing-library/jest-dom": "^5.11.10",
"@testing-library/react": "^11.2.6",
@ -12,6 +13,7 @@
"@types/node": "^12.20.10",
"@types/react": "^17.0.3",
"@types/react-dom": "^17.0.3",
"@uiw/react-textarea-code-editor": "^1.4.12",
"axios": "^0.21.1",
"jsonpath": "^1.1.1",
"node-sass": "^5.0.0",
@ -23,6 +25,7 @@
"react-scripts": "4.0.3",
"react-scrollable-feed-virtualized": "^1.4.3",
"react-syntax-highlighter": "^15.4.3",
"react-toastify": "^8.0.3",
"typescript": "^4.2.4",
"web-vitals": "^1.1.1"
},

121
ui/src/components/EntriesList.tsx
View File

@ -1,10 +1,7 @@
import {EntryItem} from "./EntryListItem/EntryListItem";
import React, {useCallback, useEffect, useMemo, useRef, useState} from "react";
import React, {useRef} from "react";
import styles from './style/EntriesList.module.sass';
import spinner from './assets/spinner.svg';
import ScrollableFeedVirtualized from "react-scrollable-feed-virtualized";
import {StatusType} from "./Filters";
import Api from "../helpers/api";
import down from "./assets/downImg.svg";
interface EntriesListProps {
@ -13,114 +10,36 @@ interface EntriesListProps {
focusedEntryId: string;
setFocusedEntryId: (id: string) => void;
connectionOpen: boolean;
noMoreDataTop: boolean;
setNoMoreDataTop: (flag: boolean) => void;
noMoreDataBottom: boolean;
setNoMoreDataBottom: (flag: boolean) => void;
methodsFilter: Array<string>;
statusFilter: Array<string>;
pathFilter: string;
serviceFilter: string;
listEntryREF: any;
onScrollEvent: (isAtBottom:boolean) => void;
scrollableList: boolean;
ws: any
openWebSocket: any;
query: string;
updateQuery: any;
queriedCurrent: number;
queriedTotal: number;
startTime: number;
}
enum FetchOperator {
LT = "lt",
GT = "gt"
}
const api = new Api();
export const EntriesList: React.FC<EntriesListProps> = ({entries, setEntries, focusedEntryId, setFocusedEntryId, connectionOpen, noMoreDataTop, setNoMoreDataTop, noMoreDataBottom, setNoMoreDataBottom, methodsFilter, statusFilter, pathFilter, serviceFilter, listEntryREF, onScrollEvent, scrollableList}) => {
const [loadMoreTop, setLoadMoreTop] = useState(false);
const [isLoadingTop, setIsLoadingTop] = useState(false);
export const EntriesList: React.FC<EntriesListProps> = ({entries, setEntries, focusedEntryId, setFocusedEntryId, connectionOpen, listEntryREF, onScrollEvent, scrollableList, ws, openWebSocket, query, updateQuery, queriedCurrent, queriedTotal, startTime}) => {
const scrollableRef = useRef(null);
useEffect(() => {
const list = document.getElementById('list').firstElementChild;
list.addEventListener('scroll', (e) => {
const el: any = e.target;
if(el.scrollTop === 0) {
setLoadMoreTop(true);
} else {
setLoadMoreTop(false);
}
});
}, []);
const filterEntries = useCallback((entry) => {
if(methodsFilter.length > 0 && !methodsFilter.includes(entry.method.toLowerCase())) return;
if(pathFilter && entry.path?.toLowerCase()?.indexOf(pathFilter) === -1) return;
if(serviceFilter && entry.service?.toLowerCase()?.indexOf(serviceFilter) === -1) return;
if(statusFilter.includes(StatusType.SUCCESS) && entry.statusCode >= 400) return;
if(statusFilter.includes(StatusType.ERROR) && entry.statusCode < 400) return;
return entry;
},[methodsFilter, pathFilter, statusFilter, serviceFilter])
const filteredEntries = useMemo(() => {
return entries.filter(filterEntries);
},[entries, filterEntries])
const getOldEntries = useCallback(async () => {
setIsLoadingTop(true);
const data = await api.fetchEntries(FetchOperator.LT, entries[0].timestamp);
setLoadMoreTop(false);
let scrollTo;
if(data.length === 0) {
setNoMoreDataTop(true);
scrollTo = document.getElementById("noMoreDataTop");
} else {
scrollTo = document.getElementById(filteredEntries?.[0]?.id);
}
setIsLoadingTop(false);
const newEntries = [...data, ...entries];
setEntries(newEntries);
if(scrollTo) {
scrollTo.scrollIntoView();
}
},[setLoadMoreTop, setIsLoadingTop, entries, setEntries, filteredEntries, setNoMoreDataTop])
useEffect(() => {
if(!loadMoreTop || connectionOpen || noMoreDataTop) return;
getOldEntries();
}, [loadMoreTop, connectionOpen, noMoreDataTop, getOldEntries]);
const getNewEntries = async () => {
const data = await api.fetchEntries(FetchOperator.GT, entries[entries.length - 1].timestamp);
let scrollTo;
if(data.length === 0) {
setNoMoreDataBottom(true);
}
scrollTo = document.getElementById(filteredEntries?.[filteredEntries.length -1]?.id);
let newEntries = [...entries, ...data];
setEntries(newEntries);
if(scrollTo) {
scrollTo.scrollIntoView({behavior: "smooth"});
}
}
return <>
<div className={styles.list}>
<div id="list" ref={listEntryREF} className={styles.list}>
{isLoadingTop && <div className={styles.spinnerContainer}>
<img alt="spinner" src={spinner} style={{height: 25}}/>
</div>}
<ScrollableFeedVirtualized ref={scrollableRef} itemHeight={48} marginTop={10} onScroll={(isAtBottom) => onScrollEvent(isAtBottom)}>
{noMoreDataTop && !connectionOpen && <div id="noMoreDataTop" className={styles.noMoreDataAvailable}>No more data available</div>}
{filteredEntries.map(entry => <EntryItem key={entry.id}
{false /* TODO: why there is a need for something here (not necessarily false)? */}
{entries.map(entry => <EntryItem key={entry.id}
entry={entry}
setFocusedEntryId={setFocusedEntryId}
isSelected={focusedEntryId === entry.id}
style={{}}/>)}
isSelected={focusedEntryId === entry.id.toString()}
style={{}}
updateQuery={updateQuery}/>)}
</ScrollableFeedVirtualized>
{!connectionOpen && !noMoreDataBottom && <div className={styles.fetchButtonContainer}>
<div className={styles.styledButton} onClick={() => getNewEntries()}>Fetch more entries</div>
{!connectionOpen && <div className={styles.fetchButtonContainer}>
<div className={styles.styledButton} onClick={() => {ws.close(); openWebSocket(query);}}>Reconnect</div>
</div>}
<button type="button"
className={`${styles.btnLive} ${scrollableList ? styles.showButton : styles.hideButton}`}
@ -129,10 +48,10 @@ export const EntriesList: React.FC<EntriesListProps> = ({entries, setEntries, fo
</button>
</div>
{entries?.length > 0 && <div className={styles.footer}>
<div><b>{filteredEntries?.length !== entries.length && `${filteredEntries?.length} / `} {entries?.length}</b> requests</div>
<div>Started listening at <span style={{marginRight: 5, fontWeight: 600, fontSize: 13}}>{new Date(+entries[0].timestamp)?.toLocaleString()}</span></div>
</div>}
<div className={styles.footer}>
<div>Displaying <b>{entries?.length}</b> results (queried <b>{queriedCurrent}</b>/<b>{queriedTotal}</b>)</div>
{startTime !== 0 && <div>Started listening at <span style={{marginRight: 5, fontWeight: 600, fontSize: 13}}>{new Date(startTime).toLocaleString()}</span></div>}
</div>
</div>
</>;
};

57
ui/src/components/EntryDetailed.tsx
View File

@ -3,7 +3,7 @@ import EntryViewer from "./EntryDetailed/EntryViewer";
import {makeStyles} from "@material-ui/core";
import Protocol from "./UI/Protocol"
import StatusCode from "./UI/StatusCode";
import {EndpointPath} from "./UI/EndpointPath";
import {Summary} from "./UI/Summary";
const useStyles = makeStyles(() => ({
entryTitle: {
@ -28,50 +28,79 @@ const useStyles = makeStyles(() => ({
interface EntryDetailedProps {
entryData: any
updateQuery: any
}
export const formatSize = (n: number) => n > 1000 ? `${Math.round(n / 1000)}KB` : `${n} B`;
const EntryTitle: React.FC<any> = ({protocol, data, bodySize, elapsedTime}) => {
const EntryTitle: React.FC<any> = ({protocol, data, bodySize, elapsedTime, updateQuery}) => {
const classes = useStyles();
const {response} = JSON.parse(data.entry);
const response = data.response;
return <div className={classes.entryTitle}>
<Protocol protocol={protocol} horizontal={true}/>
<Protocol protocol={protocol} horizontal={true} updateQuery={null}/>
<div style={{right: "30px", position: "absolute", display: "flex"}}>
{response.payload && <div style={{margin: "0 18px", opacity: 0.5}}>{formatSize(bodySize)}</div>}
{response.payload && <div style={{marginRight: 18, opacity: 0.5}}>{Math.round(elapsedTime)}ms</div>}
{response && <div
className="queryable"
style={{margin: "0 18px", opacity: 0.5}}
onClick={() => {
updateQuery(`response.bodySize == ${bodySize}`)
}}
>
{formatSize(bodySize)}
</div>}
{response && <div
className="queryable"
style={{marginRight: 18, opacity: 0.5}}
onClick={() => {
updateQuery(`elapsedTime >= ${elapsedTime}`)
}}
>
{Math.round(elapsedTime)}ms
</div>}
</div>
</div>;
};
const EntrySummary: React.FC<any> = ({data}) => {
const EntrySummary: React.FC<any> = ({data, updateQuery}) => {
const classes = useStyles();
const {response, request} = JSON.parse(data.entry);
const response = data.response;
return <div className={classes.entrySummary}>
{response?.payload && response.payload?.details && "status" in response.payload.details && <div style={{marginRight: 8}}>
<StatusCode statusCode={response.payload.details.status}/>
{response && "status" in response && <div style={{marginRight: 8}}>
<StatusCode statusCode={response.status} updateQuery={updateQuery}/>
</div>}
<div style={{flexGrow: 1, overflow: 'hidden'}}>
<EndpointPath method={request?.payload.method} path={request?.payload.url}/>
<Summary method={data.method} summary={data.summary} updateQuery={updateQuery}/>
</div>
</div>;
};
export const EntryDetailed: React.FC<EntryDetailedProps> = ({entryData}) => {
export const EntryDetailed: React.FC<EntryDetailedProps> = ({entryData, updateQuery}) => {
return <>
<EntryTitle
protocol={entryData.protocol}
data={entryData.data}
bodySize={entryData.bodySize}
elapsedTime={entryData.data.elapsedTime}
updateQuery={updateQuery}
/>
{entryData.data && <EntrySummary data={entryData.data}/>}
{entryData.data && <EntrySummary data={entryData.data} updateQuery={updateQuery}/>}
<>
{entryData.data && <EntryViewer representation={entryData.representation} isRulesEnabled={entryData.isRulesEnabled} rulesMatched={entryData.rulesMatched} contractStatus={entryData.data.contractStatus} requestReason={entryData.data.contractRequestReason} responseReason={entryData.data.contractResponseReason} contractContent={entryData.data.contractContent} elapsedTime={entryData.data.elapsedTime} color={entryData.protocol.backgroundColor}/>}
{entryData.data && <EntryViewer
representation={entryData.representation}
isRulesEnabled={entryData.isRulesEnabled}
rulesMatched={entryData.rulesMatched}
contractStatus={entryData.data.contractStatus}
requestReason={entryData.data.contractRequestReason}
responseReason={entryData.data.contractResponseReason}
contractContent={entryData.data.contractContent}
elapsedTime={entryData.data.elapsedTime}
color={entryData.protocol.backgroundColor}
updateQuery={updateQuery}
/>}
</>
</>
};

42
ui/src/components/EntryDetailed/EntrySections.tsx
View File

@ -9,11 +9,29 @@ import ProtobufDecoder from "protobuf-decoder";
interface EntryViewLineProps {
label: string;
value: number | string;
updateQuery: any;
selector: string;
overrideQueryValue?: string;
}
const EntryViewLine: React.FC<EntryViewLineProps> = ({label, value}) => {
const EntryViewLine: React.FC<EntryViewLineProps> = ({label, value, updateQuery, selector, overrideQueryValue}) => {
return (label && value && <tr className={styles.dataLine}>
<td className={styles.dataKey}>{label}</td>
<td
className={`queryable ${styles.dataKey}`}
onClick={() => {
if (!selector) {
return
} else if (overrideQueryValue) {
updateQuery(`${selector} == ${overrideQueryValue}`)
} else if (typeof(value) === "string") {
updateQuery(`${selector} == "${JSON.stringify(value).slice(1, -1)}"`)
} else {
updateQuery(`${selector} == ${value}`)
}
}}
>
{label}
</td>
<td>
<FancyTextDisplay
className={styles.dataValue}
@ -62,15 +80,19 @@ export const EntrySectionContainer: React.FC<EntrySectionContainerProps> = ({tit
interface EntryBodySectionProps {
content: any,
color: string,
updateQuery: any,
encoding?: string,
contentType?: string,
selector?: string,
}
export const EntryBodySection: React.FC<EntryBodySectionProps> = ({
color,
updateQuery,
content,
encoding,
contentType,
selector,
}) => {
const MAXIMUM_BYTES_TO_HIGHLIGHT = 10000; // The maximum of chars to highlight in body, in case the response can be megabytes
const supportedLanguages = [['html', 'html'], ['json', 'json'], ['application/grpc', 'json']]; // [[indicator, languageToUse],...]
@ -107,8 +129,8 @@ export const EntryBodySection: React.FC<EntryBodySectionProps> = ({
{content && content?.length > 0 && <EntrySectionContainer title='Body' color={color}>
<table>
<tbody>
<EntryViewLine label={'Mime type'} value={contentType}/>
<EntryViewLine label={'Encoding'} value={encoding}/>
<EntryViewLine label={'Mime type'} value={contentType} updateQuery={updateQuery} selector={selector} overrideQueryValue={`r".*"`}/>
<EntryViewLine label={'Encoding'} value={encoding} updateQuery={updateQuery} selector={selector} overrideQueryValue={`r".*"`}/>
</tbody>
</table>
@ -132,17 +154,23 @@ interface EntrySectionProps {
title: string,
color: string,
arrayToIterate: any[],
updateQuery: any,
}
export const EntryTableSection: React.FC<EntrySectionProps> = ({title, color, arrayToIterate}) => {
export const EntryTableSection: React.FC<EntrySectionProps> = ({title, color, arrayToIterate, updateQuery}) => {
return <React.Fragment>
{
arrayToIterate && arrayToIterate.length > 0 ?
<EntrySectionContainer title={title} color={color}>
<table>
<tbody>
{arrayToIterate.map(({name, value}, index) => <EntryViewLine key={index} label={name}
value={value}/>)}
{arrayToIterate.map(({name, value, selector}, index) => <EntryViewLine
key={index}
label={name}
value={value}
updateQuery={updateQuery}
selector={selector}
/>)}
</tbody>
</table>
</EntrySectionContainer> : <span/>

16
ui/src/components/EntryDetailed/EntryViewer.tsx
View File

@ -8,7 +8,7 @@ enum SectionTypes {
SectionBody = "body",
}
const SectionsRepresentation: React.FC<any> = ({data, color}) => {
const SectionsRepresentation: React.FC<any> = ({data, color, updateQuery}) => {
const sections = []
if (data) {
@ -16,12 +16,12 @@ const SectionsRepresentation: React.FC<any> = ({data, color}) => {
switch (row.type) {
case SectionTypes.SectionTable:
sections.push(
<EntryTableSection key={i} title={row.title} color={color} arrayToIterate={JSON.parse(row.data)}/>
<EntryTableSection key={i} title={row.title} color={color} arrayToIterate={JSON.parse(row.data)} updateQuery={updateQuery}/>
)
break;
case SectionTypes.SectionBody:
sections.push(
<EntryBodySection key={i} color={color} content={row.data} encoding={row.encoding} contentType={row.mime_type}/>
<EntryBodySection key={i} color={color} content={row.data} updateQuery={updateQuery} encoding={row.encoding} contentType={row.mimeType} selector={row.selector}/>
)
break;
default:
@ -33,7 +33,7 @@ const SectionsRepresentation: React.FC<any> = ({data, color}) => {
return <>{sections}</>;
}
const AutoRepresentation: React.FC<any> = ({representation, isRulesEnabled, rulesMatched, contractStatus, requestReason, responseReason, contractContent, elapsedTime, color}) => {
const AutoRepresentation: React.FC<any> = ({representation, isRulesEnabled, rulesMatched, contractStatus, requestReason, responseReason, contractContent, elapsedTime, color, updateQuery}) => {
var TABS = [
{
tab: 'Request'
@ -85,10 +85,10 @@ const AutoRepresentation: React.FC<any> = ({representation, isRulesEnabled, rule
<Tabs tabs={TABS} currentTab={currentTab} color={color} onChange={setCurrentTab} leftAligned/>
</div>
{currentTab === TABS[0].tab && <React.Fragment>
<SectionsRepresentation data={request} color={color}/>
<SectionsRepresentation data={request} color={color} updateQuery={updateQuery}/>
</React.Fragment>}
{response && currentTab === TABS[responseTabIndex].tab && <React.Fragment>
<SectionsRepresentation data={response} color={color}/>
<SectionsRepresentation data={response} color={color} updateQuery={updateQuery}/>
</React.Fragment>}
{isRulesEnabled && currentTab === TABS[rulesTabIndex].tab && <React.Fragment>
<EntryTablePolicySection title={'Rule'} color={color} latency={elapsedTime} arrayToIterate={rulesMatched ? rulesMatched : []}/>
@ -110,9 +110,10 @@ interface Props {
contractContent: string;
color: string;
elapsedTime: number;
updateQuery: any;
}
const EntryViewer: React.FC<Props> = ({representation, isRulesEnabled, rulesMatched, contractStatus, requestReason, responseReason, contractContent, elapsedTime, color}) => {
const EntryViewer: React.FC<Props> = ({representation, isRulesEnabled, rulesMatched, contractStatus, requestReason, responseReason, contractContent, elapsedTime, color, updateQuery}) => {
return <AutoRepresentation
representation={representation}
isRulesEnabled={isRulesEnabled}
@ -123,6 +124,7 @@ const EntryViewer: React.FC<Props> = ({representation, isRulesEnabled, rulesMatc
contractContent={contractContent}
elapsedTime={elapsedTime}
color={color}
updateQuery={updateQuery}
/>
};

79
ui/src/components/EntryListItem/EntryListItem.tsx
View File

@ -2,7 +2,7 @@ import React from "react";
import styles from './EntryListItem.module.sass';
import StatusCode, {getClassification, StatusCodeClassification} from "../UI/StatusCode";
import Protocol, {ProtocolInterface} from "../UI/Protocol"
import {EndpointPath} from "../UI/EndpointPath";
import {Summary} from "../UI/Summary";
import ingoingIconSuccess from "../assets/ingoing-traffic-success.svg"
import ingoingIconFailure from "../assets/ingoing-traffic-failure.svg"
import ingoingIconNeutral from "../assets/ingoing-traffic-neutral.svg"
@ -15,7 +15,7 @@ interface Entry {
method?: string,
summary: string,
service: string,
id: string,
id: number,
statusCode?: number;
url?: string;
timestamp: Date;
@ -40,9 +40,10 @@ interface EntryProps {
setFocusedEntryId: (id: string) => void;
isSelected?: boolean;
style: object;
updateQuery: any;
}
export const EntryItem: React.FC<EntryProps> = ({entry, setFocusedEntryId, isSelected, style}) => {
export const EntryItem: React.FC<EntryProps> = ({entry, setFocusedEntryId, isSelected, style, updateQuery}) => {
const classification = getClassification(entry.statusCode)
const numberOfRules = entry.rules.numberOfRules
let ingoingIcon;
@ -115,10 +116,10 @@ export const EntryItem: React.FC<EntryProps> = ({entry, setFocusedEntryId, isSel
return <>
<div
id={entry.id}
id={entry.id.toString()}
className={`${styles.row}
${isSelected && !rule && !contractEnabled ? styles.rowSelected : additionalRulesProperties}`}
onClick={() => setFocusedEntryId(entry.id)}
onClick={() => setFocusedEntryId(entry.id.toString())}
style={{
border: isSelected ? `1px ${entry.protocol.backgroundColor} solid` : "1px transparent solid",
position: "absolute",
@ -127,14 +128,26 @@ export const EntryItem: React.FC<EntryProps> = ({entry, setFocusedEntryId, isSel
width: "calc(100% - 25px)",
}}
>
<Protocol protocol={entry.protocol} horizontal={false}/>
<Protocol
protocol={entry.protocol}
horizontal={false}
updateQuery={updateQuery}
/>
{((entry.protocol.name === "http" && "statusCode" in entry) || entry.statusCode !== 0) && <div>
<StatusCode statusCode={entry.statusCode}/>
<StatusCode statusCode={entry.statusCode} updateQuery={updateQuery}/>
</div>}
<div className={styles.endpointServiceContainer}>
<EndpointPath method={entry.method} path={entry.summary}/>
<Summary method={entry.method} summary={entry.summary} updateQuery={updateQuery}/>
<div className={styles.service}>
<span title="Service Name">{entry.service}</span>
<span
title="Service Name"
className="queryable"
onClick={() => {
updateQuery(`service == "${entry.service}"`)
}}
>
{entry.service}
</span>
</div>
</div>
{
@ -152,17 +165,53 @@ export const EntryItem: React.FC<EntryProps> = ({entry, setFocusedEntryId, isSel
: ""
}
<div className={styles.separatorRight}>
<span className={styles.port} title="Source Port">{entry.sourcePort}</span>
<span
className={`queryable ${styles.port}`}
title="Source Port"
onClick={() => {
updateQuery(`src.port == "${entry.sourcePort}"`)
}}
>
{entry.sourcePort}
</span>
{entry.isOutgoing ?
<img src={outgoingIcon} alt="Ingoing traffic" title="Ingoing"/>
<img
src={outgoingIcon}
alt="Ingoing traffic"
title="Ingoing"
onClick={() => {
updateQuery(`outgoing == true`)
}}
/>
:
<img src={ingoingIcon} alt="Outgoing traffic" title="Outgoing"/>
<img
src={ingoingIcon}
alt="Outgoing traffic"
title="Outgoing"
onClick={() => {
updateQuery(`outgoing == false`)
}}
/>
}
<span className={styles.port} title="Destination Port">{entry.destinationPort}</span>
<span
className={`queryable ${styles.port}`}
title="Destination Port"
onClick={() => {
updateQuery(`dst.port == "${entry.destinationPort}"`)
}}
>
{entry.destinationPort}
</span>
</div>
<div className={styles.timestamp}>
<span title="Timestamp">
{new Date(+entry.timestamp)?.toLocaleString()}
<span
title="Timestamp"
className="queryable"
onClick={() => {
updateQuery(`timestamp >= datetime("${new Date(+entry.timestamp)?.toLocaleString("en-US", {timeZone: 'UTC' })}")`)
}}
>
{new Date(+entry.timestamp)?.toLocaleString("en-US")}
</span>
</div>
</div>

398
ui/src/components/Filters.tsx
View File

@ -1,137 +1,303 @@
import React from "react";
import React, {useRef, useState} from "react";
import styles from './style/Filters.module.sass';
import {FilterSelect} from "./UI/FilterSelect";
import {TextField} from "@material-ui/core";
import {ALL_KEY} from "./UI/Select";
import {Button, Grid, Modal, Box, Typography, Backdrop, Fade, Divider} from "@material-ui/core";
import CodeEditor from '@uiw/react-textarea-code-editor';
import MenuBookIcon from '@material-ui/icons/MenuBook';
import {SyntaxHighlighter} from "./UI/SyntaxHighlighter/index";
import filterUIExample1 from "./assets/filter-ui-example-1.png"
import filterUIExample2 from "./assets/filter-ui-example-2.png"
interface FiltersProps {
methodsFilter: Array<string>;
setMethodsFilter: (methods: Array<string>) => void;
statusFilter: Array<string>;
setStatusFilter: (methods: Array<string>) => void;
pathFilter: string
setPathFilter: (val: string) => void;
serviceFilter: string
setServiceFilter: (val: string) => void;
query: string
setQuery: any
backgroundColor: string
ws: any
openWebSocket: (query: string) => void;
}
export const Filters: React.FC<FiltersProps> = ({methodsFilter, setMethodsFilter, statusFilter, setStatusFilter, pathFilter, setPathFilter, serviceFilter, setServiceFilter}) => {
export const Filters: React.FC<FiltersProps> = ({query, setQuery, backgroundColor, ws, openWebSocket}) => {
return <div className={styles.container}>
<MethodFilter methodsFilter={methodsFilter} setMethodsFilter={setMethodsFilter}/>
<StatusTypesFilter statusFilter={statusFilter} setStatusFilter={setStatusFilter}/>
<ServiceFilter serviceFilter={serviceFilter} setServiceFilter={setServiceFilter}/>
<PathFilter pathFilter={pathFilter} setPathFilter={setPathFilter}/>
<QueryForm
query={query}
setQuery={setQuery}
backgroundColor={backgroundColor}
ws={ws}
openWebSocket={openWebSocket}
/>
</div>;
};
const _toUpperCase = v => v.toUpperCase();
interface QueryFormProps {
query: string
setQuery: any
backgroundColor: string
ws: any
openWebSocket: (query: string) => void;
}
const FilterContainer: React.FC = ({children}) => {
return <div className={styles.filterContainer}>
{children}
</div>;
const style = {
position: 'absolute',
top: '50%',
left: '50%',
transform: 'translate(-50%, -50%)',
width: '80vw',
bgcolor: 'background.paper',
borderRadius: '5px',
boxShadow: 24,
p: 4,
color: '#000',
};
enum HTTPMethod {
GET = "get",
PUT = "put",
POST = "post",
DELETE = "delete",
OPTIONS="options",
PATCH = "patch"
}
export const QueryForm: React.FC<QueryFormProps> = ({query, setQuery, backgroundColor, ws, openWebSocket}) => {
interface MethodFilterProps {
methodsFilter: Array<string>;
setMethodsFilter: (methods: Array<string>) => void;
}
const formRef = useRef<HTMLFormElement>(null);
const MethodFilter: React.FC<MethodFilterProps> = ({methodsFilter, setMethodsFilter}) => {
const [openModal, setOpenModal] = useState(false);
const methodClicked = (val) => {
if(val === ALL_KEY) {
setMethodsFilter([]);
return;
}
if(methodsFilter.includes(val)) {
setMethodsFilter(methodsFilter.filter(method => method !== val))
} else {
setMethodsFilter([...methodsFilter, val]);
}
const handleOpenModal = () => setOpenModal(true);
const handleCloseModal = () => setOpenModal(false);
const handleChange = async (e) => {
setQuery(e.target.value);
}
return <FilterContainer>
<FilterSelect
items={Object.values(HTTPMethod)}
allowMultiple={true}
value={methodsFilter}
onChange={(val) => methodClicked(val)}
transformDisplay={_toUpperCase}
label={"Methods"}
/>
</FilterContainer>;
};
export enum StatusType {
SUCCESS = "success",
ERROR = "error"
}
interface StatusTypesFilterProps {
statusFilter: Array<string>;
setStatusFilter: (methods: Array<string>) => void;
}
const StatusTypesFilter: React.FC<StatusTypesFilterProps> = ({statusFilter, setStatusFilter}) => {
const statusClicked = (val) => {
if(val === ALL_KEY) {
setStatusFilter([]);
return;
}
setStatusFilter([val]);
const handleSubmit = (e) => {
ws.close()
openWebSocket(query)
e.preventDefault();
}
return <FilterContainer>
<FilterSelect
items={Object.values(StatusType)}
allowMultiple={true}
value={statusFilter}
onChange={(val) => statusClicked(val)}
transformDisplay={_toUpperCase}
label="Status"
/>
</FilterContainer>;
};
return <>
<form
ref={formRef}
onSubmit={handleSubmit}
style={{
width: '100%',
}}
>
<Grid container spacing={2}>
<Grid
item
xs={8}
style={{
maxHeight: '25vh',
overflowY: 'auto',
}}
>
<label>
<CodeEditor
value={query}
language="py"
placeholder="Mizu Filter Syntax"
onChange={handleChange}
padding={8}
style={{
fontSize: 14,
backgroundColor: `${backgroundColor}`,
fontFamily: 'ui-monospace,SFMono-Regular,SF Mono,Consolas,Liberation Mono,Menlo,monospace',
}}
/>
</label>
</Grid>
<Grid item xs={4}>
<Button type="submit" variant="contained" style={{margin: "2px 0px 0px 0px"}}>Apply</Button>
<Button
title="Open Filtering Guide (Cheatsheet)"
variant="contained"
style={{margin: "2px 0px 0px 10px", minWidth: "26px"}}
onClick={handleOpenModal}
>
<MenuBookIcon fontSize="inherit"></MenuBookIcon>
</Button>
</Grid>
</Grid>
</form>
interface PathFilterProps {
pathFilter: string;
setPathFilter: (val: string) => void;
<Modal
aria-labelledby="transition-modal-title"
aria-describedby="transition-modal-description"
open={openModal}
onClose={handleCloseModal}
closeAfterTransition
BackdropComponent={Backdrop}
BackdropProps={{
timeout: 500,
}}
style={{overflow: 'auto'}}
>
<Fade in={openModal}>
<Box sx={style}>
<Typography id="modal-modal-title" variant="h5" component="h2" style={{textAlign: 'center'}}>
Filtering Guide (Cheatsheet)
</Typography>
<Typography id="modal-modal-description">
<p>Mizu has a rich filtering syntax that let's you query the results both flexibly and efficiently.</p>
<p>Here are some examples that you can try;</p>
</Typography>
<Grid container>
<Grid item xs style={{margin: "10px"}}>
<Typography id="modal-modal-description">
This is a simple query that matches to HTTP packets with request path "/catalogue":
</Typography>
<SyntaxHighlighter
isWrapped={false}
showLineNumbers={false}
code={`http and request.path == "/catalogue"`}
language="python"
/>
<Typography id="modal-modal-description">
The same query can be negated for HTTP path and written like this:
</Typography>
<SyntaxHighlighter
isWrapped={false}
showLineNumbers={false}
code={`http and request.path != "/catalogue"`}
language="python"
/>
<Typography id="modal-modal-description">
The syntax supports regular expressions. Here is a query that matches the HTTP requests that send JSON to a server:
</Typography>
<SyntaxHighlighter
isWrapped={false}
showLineNumbers={false}
code={`http and request.headers["Accept"] == r"application/json.*"`}
language="python"
/>
<Typography id="modal-modal-description">
Here is another query that matches HTTP responses with status code 4xx:
</Typography>
<SyntaxHighlighter
isWrapped={false}
showLineNumbers={false}
code={`http and response.status == r"4.*"`}
language="python"
/>
<Typography id="modal-modal-description">
The same exact query can be as integer comparison:
</Typography>
<SyntaxHighlighter
isWrapped={false}
showLineNumbers={false}
code={`http and response.status >= 400`}
language="python"
/>
<Typography id="modal-modal-description">
The results can be queried based on their timestamps:
</Typography>
<SyntaxHighlighter
isWrapped={false}
showLineNumbers={false}
code={`timestamp < datetime("10/28/2021, 9:13:02 PM")`}
language="python"
/>
</Grid>
<Divider orientation="vertical" flexItem />
<Grid item xs style={{margin: "10px"}}>
<Typography id="modal-modal-description">
Since Mizu supports various protocols like gRPC, AMQP, Kafka and Redis. It's possible to write complex queries that match multiple protocols like this:
</Typography>
<SyntaxHighlighter
isWrapped={false}
showLineNumbers={false}
code={`(http and request.method == "PUT") or (amqp and request.queue.startsWith("test"))\n or (kafka and response.payload.errorCode == 2) or (redis and request.key == "example")\n or (grpc and request.headers[":path"] == r".*foo.*")`}
language="python"
/>
<Typography id="modal-modal-description">
By clicking the UI elements in both left-pane and right-pane, you can automatically select a field and update the query:
</Typography>
<img
src={filterUIExample1}
width={600}
alt="Clicking to UI elements (left-pane)"
title="Clicking to UI elements (left-pane)"
/>
<Typography id="modal-modal-description">
Such that; clicking this in left-pane, would append the query below:
</Typography>
<SyntaxHighlighter
isWrapped={false}
showLineNumbers={false}
code={`and service == "http://carts.sock-shop"`}
language="python"
/>
<Typography id="modal-modal-description">
Another queriable UI element example, this time from the right-pane:
</Typography>
<img
src={filterUIExample2}
width={300}
alt="Clicking to UI elements (right-pane)"
title="Clicking to UI elements (right-pane)"
/>
<Typography id="modal-modal-description">
A query that compares one selector to another is also a valid query:
</Typography>
<SyntaxHighlighter
isWrapped={false}
showLineNumbers={false}
code={`http and (request.query["x"] == response.headers["y"]\n or response.content.text.contains(request.query["x"]))`}
language="python"
/>
</Grid>
<Divider orientation="vertical" flexItem />
<Grid item xs style={{margin: "10px"}}>
<Typography id="modal-modal-description">
There are a few helper methods included the in the filter language* to help building queries more easily.
</Typography>
<br></br>
<Typography id="modal-modal-description">
true if the given selector's value starts with the string:
</Typography>
<SyntaxHighlighter
isWrapped={false}
showLineNumbers={false}
code={`request.path.startsWith("something")`}
language="python"
/>
<Typography id="modal-modal-description">
true if the given selector's value ends with the string:
</Typography>
<SyntaxHighlighter
isWrapped={false}
showLineNumbers={false}
code={`request.path.endsWith("something")`}
language="python"
/>
<Typography id="modal-modal-description">
true if the given selector's value contains the string:
</Typography>
<SyntaxHighlighter
isWrapped={false}
showLineNumbers={false}
code={`request.path.contains("something")`}
language="python"
/>
<Typography id="modal-modal-description">
returns the UNIX timestamp which is the equivalent of the time that's provided by the string. Invalid input evaluates to false:
</Typography>
<SyntaxHighlighter
isWrapped={false}
showLineNumbers={false}
code={`timestamp >= datetime("10/19/2021, 6:29:02 PM")`}
language="python"
/>
<Typography id="modal-modal-description">
limits the number of records that are streamed back as a result of a query. Always evaluates to true:
</Typography>
<SyntaxHighlighter
isWrapped={false}
showLineNumbers={false}
code={`and limit(100)`}
language="python"
/>
</Grid>
</Grid>
<br></br>
<Typography id="modal-modal-description" style={{fontSize: 12, fontStyle: 'italic'}}>
*The filtering functionality is provided through <b>Basenine</b> database server. Please refer to <a href="https://github.com/up9inc/basenine/wiki/BFL-Syntax-Reference"><b>BFL Syntax Reference</b></a> for more information.
</Typography>
</Box>
</Fade>
</Modal>
</>
}
const PathFilter: React.FC<PathFilterProps> = ({pathFilter, setPathFilter}) => {
return <FilterContainer>
<div className={styles.filterLabel}>Path</div>
<div>
<TextField value={pathFilter} variant="outlined" className={styles.filterText} style={{minWidth: '150px'}} onChange={(e: any) => setPathFilter(e.target.value)}/>
</div>
</FilterContainer>;
};
interface ServiceFilterProps {
serviceFilter: string;
setServiceFilter: (val: string) => void;
}
const ServiceFilter: React.FC<ServiceFilterProps> = ({serviceFilter, setServiceFilter}) => {
return <FilterContainer>
<div className={styles.filterLabel}>Service</div>
<div>
<TextField value={serviceFilter} variant="outlined" className={styles.filterText} style={{minWidth: '150px'}} onChange={(e: any) => setServiceFilter(e.target.value)}/>
</div>
</FilterContainer>;
};

153
ui/src/components/TrafficPage.tsx
View File

@ -10,6 +10,8 @@ import pauseIcon from './assets/pause.svg';
import variables from '../variables.module.scss';
import {StatusBar} from "./UI/StatusBar";
import Api, {MizuWebsocketURL} from "../helpers/api";
import { ToastContainer, toast } from 'react-toastify';
import 'react-toastify/dist/ReactToastify.css';
const useLayoutStyles = makeStyles(() => ({
details: {
@ -34,7 +36,6 @@ const useLayoutStyles = makeStyles(() => ({
enum ConnectionStatus {
Closed,
Connected,
Paused
}
interface TrafficPageProps {
@ -52,25 +53,52 @@ export const TrafficPage: React.FC<TrafficPageProps> = ({setAnalyzeStatus, onTLS
const [focusedEntryId, setFocusedEntryId] = useState(null);
const [selectedEntryData, setSelectedEntryData] = useState(null);
const [connection, setConnection] = useState(ConnectionStatus.Closed);
const [noMoreDataTop, setNoMoreDataTop] = useState(false);
const [noMoreDataBottom, setNoMoreDataBottom] = useState(false);
const [methodsFilter, setMethodsFilter] = useState([]);
const [statusFilter, setStatusFilter] = useState([]);
const [pathFilter, setPathFilter] = useState("");
const [serviceFilter, setServiceFilter] = useState("");
const [tappingStatus, setTappingStatus] = useState(null);
const [disableScrollList, setDisableScrollList] = useState(false);
const [query, setQueryDefault] = useState("");
const [queryBackgroundColor, setQueryBackgroundColor] = useState("#f5f5f5");
const [queriedCurrent, setQueriedCurrent] = useState(0);
const [queriedTotal, setQueriedTotal] = useState(0);
const [startTime, setStartTime] = useState(0);
const setQuery = async (query) => {
if (!query) {
setQueryBackgroundColor("#f5f5f5")
} else {
const data = await api.validateQuery(query);
if (data.valid) {
setQueryBackgroundColor("#d2fad2")
} else {
setQueryBackgroundColor("#fad6dc")
}
}
setQueryDefault(query)
}
const updateQuery = (addition) => {
if (query) {
setQuery(`${query} and ${addition}`)
} else {
setQuery(addition)
}
}
const ws = useRef(null);
const listEntry = useRef(null);
const openWebSocket = () => {
const openWebSocket = (query) => {
setEntries([])
ws.current = new WebSocket(MizuWebsocketURL);
ws.current.onopen = () => setConnection(ConnectionStatus.Connected);
ws.current.onopen = () => {
ws.current.send(query)
setConnection(ConnectionStatus.Connected);
}
ws.current.onclose = () => setConnection(ConnectionStatus.Closed);
}
@ -81,11 +109,7 @@ export const TrafficPage: React.FC<TrafficPageProps> = ({setAnalyzeStatus, onTLS
switch (message.messageType) {
case "entry":
const entry = message.data
if (connection === ConnectionStatus.Paused) {
setNoMoreDataBottom(false)
return;
}
if (!focusedEntryId) setFocusedEntryId(entry.id)
if (!focusedEntryId) setFocusedEntryId(entry.id.toString())
let newEntries = [...entries];
setEntries([...newEntries, entry])
if(listEntry.current) {
@ -103,6 +127,25 @@ export const TrafficPage: React.FC<TrafficPageProps> = ({setAnalyzeStatus, onTLS
case "outboundLink":
onTLSDetected(message.Data.DstIP);
break;
case "toast":
toast[message.data.type](message.data.text, {
position: "bottom-right",
theme: "colored",
autoClose: message.data.autoClose,
hideProgressBar: false,
closeOnClick: true,
pauseOnHover: true,
draggable: true,
progress: undefined,
});
break;
case "queryMetadata":
setQueriedCurrent(message.data.current)
setQueriedTotal(message.data.total)
break;
case "startTime":
setStartTime(message.data);
break;
default:
console.error(`unsupported websocket message type, Got: ${message.messageType}`)
}
@ -111,7 +154,7 @@ export const TrafficPage: React.FC<TrafficPageProps> = ({setAnalyzeStatus, onTLS
useEffect(() => {
(async () => {
openWebSocket();
openWebSocket("rlimit(100)");
try{
const tapStatusResponse = await api.tapStatus();
setTappingStatus(tapStatusResponse);
@ -139,14 +182,17 @@ export const TrafficPage: React.FC<TrafficPageProps> = ({setAnalyzeStatus, onTLS
}, [focusedEntryId])
const toggleConnection = () => {
setConnection(connection === ConnectionStatus.Connected ? ConnectionStatus.Paused : ConnectionStatus.Connected);
if (connection === ConnectionStatus.Connected) {
ws.current.close();
} else {
openWebSocket(query);
setConnection(ConnectionStatus.Connected);
}
}
const getConnectionStatusClass = (isContainer) => {
const container = isContainer ? "Container" : "";
switch (connection) {
case ConnectionStatus.Paused:
return "orangeIndicator" + container;
case ConnectionStatus.Connected:
return "greenIndicator" + container;
default:
@ -156,8 +202,6 @@ export const TrafficPage: React.FC<TrafficPageProps> = ({setAnalyzeStatus, onTLS
const getConnectionTitle = () => {
switch (connection) {
case ConnectionStatus.Paused:
return "traffic paused";
case ConnectionStatus.Connected:
return "connected, waiting for traffic"
default:
@ -176,8 +220,10 @@ export const TrafficPage: React.FC<TrafficPageProps> = ({setAnalyzeStatus, onTLS
return (
<div className="TrafficPage">
<div className="TrafficPageHeader">
{connection !== ConnectionStatus.Closed && <img style={{cursor: "pointer", marginRight: 15, height: 30}} alt="pause"
src={connection === ConnectionStatus.Connected ? pauseIcon : playIcon} onClick={toggleConnection}/>}
<img className="playPauseIcon" style={{visibility: connection === ConnectionStatus.Connected ? "visible" : "hidden"}} alt="pause"
src={pauseIcon} onClick={toggleConnection}/>
<img className="playPauseIcon" style={{position: "absolute", visibility: connection === ConnectionStatus.Connected ? "hidden" : "visible"}} alt="play"
src={playIcon} onClick={toggleConnection}/>
<div className="connectionText">
{getConnectionTitle()}
<div className={"indicatorContainer " + getConnectionStatusClass(true)}>
@ -185,42 +231,51 @@ export const TrafficPage: React.FC<TrafficPageProps> = ({setAnalyzeStatus, onTLS
</div>
</div>
</div>
{entries.length > 0 && <div className="TrafficPage-Container">
{<div className="TrafficPage-Container">
<div className="TrafficPage-ListContainer">
<Filters methodsFilter={methodsFilter}
setMethodsFilter={setMethodsFilter}
statusFilter={statusFilter}
setStatusFilter={setStatusFilter}
pathFilter={pathFilter}
setPathFilter={setPathFilter}
serviceFilter={serviceFilter}
setServiceFilter={setServiceFilter}
<Filters
query={query}
setQuery={setQuery}
backgroundColor={queryBackgroundColor}
ws={ws.current}
openWebSocket={openWebSocket}
/>
<div className={styles.container}>
<EntriesList entries={entries}
setEntries={setEntries}
focusedEntryId={focusedEntryId}
setFocusedEntryId={setFocusedEntryId}
connectionOpen={connection === ConnectionStatus.Connected}
noMoreDataBottom={noMoreDataBottom}
setNoMoreDataBottom={setNoMoreDataBottom}
noMoreDataTop={noMoreDataTop}
setNoMoreDataTop={setNoMoreDataTop}
methodsFilter={methodsFilter}
statusFilter={statusFilter}
pathFilter={pathFilter}
serviceFilter={serviceFilter}
listEntryREF={listEntry}
onScrollEvent={onScrollEvent}
scrollableList={disableScrollList}
<EntriesList
entries={entries}
setEntries={setEntries}
focusedEntryId={focusedEntryId}
setFocusedEntryId={setFocusedEntryId}
connectionOpen={connection === ConnectionStatus.Connected}
listEntryREF={listEntry}
onScrollEvent={onScrollEvent}
scrollableList={disableScrollList}
ws={ws.current}
openWebSocket={openWebSocket}
query={query}
updateQuery={updateQuery}
queriedCurrent={queriedCurrent}
queriedTotal={queriedTotal}
startTime={startTime}
/>
</div>
</div>
<div className={classes.details}>
{selectedEntryData && <EntryDetailed entryData={selectedEntryData}/>}
{selectedEntryData && <EntryDetailed entryData={selectedEntryData} updateQuery={updateQuery}/>}
</div>
</div>}
{tappingStatus?.pods != null && <StatusBar tappingStatus={tappingStatus}/>}
<ToastContainer
position="bottom-right"
autoClose={5000}
hideProgressBar={false}
newestOnTop={false}
closeOnClick
rtl={false}
pauseOnFocusLoss
draggable
pauseOnHover
/>
</div>
)
};

15
ui/src/components/UI/EndpointPath.tsx
View File

@ -1,15 +0,0 @@
import miscStyles from "./style/misc.module.sass";
import React from "react";
import styles from './style/EndpointPath.module.sass';
interface EndpointPathProps {
method: string,
path: string
}
export const EndpointPath: React.FC<EndpointPathProps> = ({method, path}) => {
return <div className={styles.container}>
{method && <span title="Method" className={`${miscStyles.protocol} ${miscStyles.method}`}>{method}</span>}
{path && <div title="Summary" className={styles.path}>{path}</div>}
</div>
};

28
ui/src/components/UI/FilterSelect.tsx
View File

@ -1,28 +0,0 @@
import React from "react";
import { MenuItem } from '@material-ui/core';
import style from './style/FilterSelect.module.sass';
import { Select, SelectProps } from "./Select";
interface FilterSelectProps extends SelectProps {
items: string[];
value: string | string[];
onChange: (string) => void;
label?: string;
allowMultiple?: boolean;
transformDisplay?: (string) => string;
}
export const FilterSelect: React.FC<FilterSelectProps> = ({items, value, onChange, label, allowMultiple= false, transformDisplay}) => {
return <Select
value={value}
multiple={allowMultiple}
label={label}
onChange={onChange}
transformDisplay={transformDisplay}
labelOnTop={true}
labelClassName={style.SelectLabel}
trimItemsWhenMultiple={true}
>
{items?.map(item => <MenuItem key={item} value={item}><span className='uppercase'>{item}</span></MenuItem>)}
</Select>
};

35
ui/src/components/UI/Protocol.tsx
View File

@ -4,7 +4,8 @@ import styles from './style/Protocol.module.sass';
export interface ProtocolInterface {
name: string
longName: string
abbreviation: string
abbr: string
macro: string
backgroundColor: string
foregroundColor: string
fontSize: number
@ -16,9 +17,10 @@ export interface ProtocolInterface {
interface ProtocolProps {
protocol: ProtocolInterface
horizontal: boolean
updateQuery: any
}
const Protocol: React.FC<ProtocolProps> = ({protocol, horizontal}) => {
const Protocol: React.FC<ProtocolProps> = ({protocol, horizontal, updateQuery}) => {
if (horizontal) {
return <a target="_blank" rel="noopener noreferrer" href={protocol.referenceLink}>
<span
@ -28,25 +30,26 @@ const Protocol: React.FC<ProtocolProps> = ({protocol, horizontal}) => {
color: protocol.foregroundColor,
fontSize: 13,
}}
title={protocol.abbreviation}
title={protocol.abbr}
>
{protocol.longName}
</span>
</a>
} else {
return <a target="_blank" rel="noopener noreferrer" href={protocol.referenceLink}>
<span
className={`${styles.base} ${styles.vertical}`}
style={{
backgroundColor: protocol.backgroundColor,
color: protocol.foregroundColor,
fontSize: protocol.fontSize,
}}
title={protocol.longName}
>
{protocol.abbreviation}
</span>
</a>
return <span
className={`${styles.base} ${styles.vertical}`}
style={{
backgroundColor: protocol.backgroundColor,
color: protocol.foregroundColor,
fontSize: protocol.fontSize,
}}
title={protocol.longName}
onClick={() => {
updateQuery(protocol.macro)
}}
>
{protocol.abbr}
</span>
}
};

11
ui/src/components/UI/StatusCode.tsx
View File

@ -9,16 +9,21 @@ export enum StatusCodeClassification {
interface EntryProps {
statusCode: number
updateQuery: any
}
const StatusCode: React.FC<EntryProps> = ({statusCode}) => {
const StatusCode: React.FC<EntryProps> = ({statusCode, updateQuery}) => {
const classification = getClassification(statusCode)
return <span
title="Status Code"
className={`${styles[classification]} ${styles.base}`}>
{statusCode}
className={`queryable ${styles[classification]} ${styles.base}`}
onClick={() => {
updateQuery(`response.status == ${statusCode}`)
}}
>
{statusCode}
</span>
};

32
ui/src/components/UI/Summary.tsx Normal file
View File

@ -0,0 +1,32 @@
import miscStyles from "./style/misc.module.sass";
import React from "react";
import styles from './style/Summary.module.sass';
interface SummaryProps {
method: string
summary: string
updateQuery: any
}
export const Summary: React.FC<SummaryProps> = ({method, summary, updateQuery}) => {
return <div className={styles.container}>
{method && <span
title="Method"
className={`queryable ${miscStyles.protocol} ${miscStyles.method}`}
onClick={() => {
updateQuery(`method == "${method}"`)
}}
>
{method}
</span>}
{summary && <div
title="Summary"
className={`queryable ${styles.summary}`}
onClick={() => {
updateQuery(`summary == "${summary}"`)
}}
>
{summary}
</div>}
</div>
};

2
ui/src/components/UI/SyntaxHighlighter/highlighterStyle.ts
View File

@ -112,7 +112,7 @@ export const highlighterStyle = {
"color": "#C6C5FE"
},
"operator": {
"color": "#EDEDED"
"color": "#A1A1A1"
},
"entity": {
"color": "#fdab2b",

1
ui/src/components/UI/style/Protocol.module.sass
View File

@ -6,7 +6,6 @@
background-color: #000
color: #fff
margin-left: -8px
margin-bottom: -4px
.vertical
line-height: 22px

4
ui/src/components/UI/style/EndpointPath.module.sass → ui/src/components/UI/style/Summary.module.sass
View File

@ -2,7 +2,7 @@
display: flex
align-items: center
.path
.summary
text-overflow: ellipsis
overflow: hidden
white-space: nowrap
white-space: nowrap

BIN
ui/src/components/assets/filter-ui-example-1.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 40 KiB

BIN
ui/src/components/assets/filter-ui-example-2.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 16 KiB

9
ui/src/components/style/EntriesList.module.sass
View File

@ -38,15 +38,6 @@
border: 1px solid #627ef7
background-color: rgba(255, 255, 255, 0.06)
.spinnerContainer
display: flex
justify-content: center
margin-bottom: 10px
.noMoreDataAvailable
text-align: center
font-weight: 600
color: $secondary-font-color
.fetchButtonContainer
width: 100%
display: flex

7
ui/src/components/style/Filters.module.sass
View File

@ -4,9 +4,6 @@
display: flex
flex-direction: row
align-items: center
min-height: 3rem
overflow-y: hidden
overflow-x: auto
padding: .5rem 0
border-bottom: 1px solid #BCC6DD
margin-right: 20px
@ -29,8 +26,8 @@
input
padding: 4px 12px
background: $main-background-color
border-radius: 12px
font-size: 12px
border-radius: 4px
font-size: 14px
border: 1px solid #BCC6DD
fieldset
border: none

5
ui/src/components/style/TrafficPage.sass
View File

@ -110,3 +110,8 @@
align-items: center
height: 17px
font-size: 16px
.playPauseIcon
cursor: pointer
margin-right: 15px
height: 30px

16
ui/src/helpers/api.js
View File

@ -29,13 +29,8 @@ export default class Api {
return response.data;
}
getEntry = async (entryId) => {
const response = await this.client.get(`/entries/${entryId}`);
return response.data;
}
fetchEntries = async (operator, timestamp) => {
const response = await this.client.get(`/entries?limit=50&operator=${operator}&timestamp=${timestamp}`);
getEntry = async (id) => {
const response = await this.client.get(`/entries/${id}`);
return response.data;
}
@ -48,4 +43,11 @@ export default class Api {
const response = await this.client.get("/status/auth");
return response.data;
}
validateQuery = async (query) => {
const form = new FormData();
form.append('query', query)
const response = await this.client.post(`/query/validate`, form);
return response.data;
}
}

13
ui/src/index.sass
View File

@ -22,6 +22,11 @@ code
.uppercase
text-transform: uppercase
.queryable
cursor: pointer
&:hover
text-decoration: underline
/****
* Button
***/
@ -31,11 +36,13 @@ button
&:not(.MuiFab-root)
&.MuiButtonBase-root
box-sizing: border-box
font-weight: 500
font-weight: 600
line-height: 1
border-radius: 20px
border-radius: 4px
letter-spacing: 0.02857em
text-transform: uppercase
background-color: $blue-color
color: #fff
text-transform: none
img:not(.custom)
max-width: 13px
max-height: 13px