Migrate from SQLite to Basenine and introduce a new filtering syntax (#279)

* Fix the OOMKilled error by calling `debug.FreeOSMemory` periodically

* Remove `MAX_NUMBER_OF_GOROUTINES` environment variable

* Change the line

* Increase the default value of `TCP_STREAM_CHANNEL_TIMEOUT_MS` to `10000`

* Write the client and integrate to the new real-time database

* Refactor the WebSocket implementaiton for `/ws`

* Adapt the UI to the new filtering system

* Fix the rest of the issues in the UI

* Increase the buffer of the scanner

* Implement accessing single records

* Increase the buffer of another scanner

* Populate `Request` and `Response` fields of `MizuEntry`

* Add syntax highlighting for the query

* Add database to `Dockerfile`

* Fix some issues

* Update the `realtime_dbms` Git module commit hash

* Upgrade Gin version and print the query string

* Revert "Upgrade Gin version and print the query string"

This reverts commit aa09f904ee.

* Use WebSocket's itself to query instead of the query string

* Fix some errors related to conversion to HAR

* Fix the issues caused by the latest merge

* Fix the build error

* Fix PR validation GitHub workflow

* Replace the git submodule with latest Basenine version `0.1.0`

Remove `realtime_client.go` and use the official client library `github.com/up9inc/basenine/client/go` instead.

* Move Basenine host and port constants to `shared` module

* Reliably execute and wait for Basenine to become available

* Upgrade Basenine version

* Properly close WebSocket and data channel

* Fix the issues caused by the recent merge commit

* Clean up the TypeScript code

* Update `.gitignore`

* Limit the database size

* Add `Macros` method signature to `Dissector` interface and set the macros provided by the protocol extensions

* Run `go mod tidy` on `agent`

* Upgrade `github.com/up9inc/basenine/client/go` version

* Implement a mechanism to update the query using click events in the UI and use it for protocol macros

* Update the query on click to timestamps

* Fix some issues in the WebSocket and channel handling

* Update the query on clicks to status code

* Update the query on clicks to method, path and service

* Update the query on clicks to is outgoing, source and destination ports

* Add an API endpoint to validate the query against syntax errors

* Move the query background color state into `TrafficPage`

* Fix the logic in `setQuery`

* Display a toast message in case of a syntax error in the query

* Remove a call to `fmt.Printf`

* Upgrade Basenine version to `0.1.3`

* Fix an issue related to getting `MAX_ENTRIES_DB_BYTES` environment variable

* Have the `path` key in request details, in HTTP

* Rearrange the HTTP headers for the querying

* Do the same thing for `cookies` and `queryString`

* Update the query on click to table elements

Add the selectors for `TABLE` type representations in HTTP extension.

* Update the query on click to `bodySize` and `elapsedTime` in `EntryTitle`

* Add the selectors for `TABLE` type representations in AMQP extension

* Add the selectors for `TABLE` type representations in Kafka extension

* Add the selectors for `TABLE` type representations in Redis extension

* Define a struct in `tap/api.go` for the section representation data

* Add the selectors for `BODY` type representations

* Add `request.path` to the HTTP request details

* Change the summary string's field name from `path` to `summary`

* Introduce `queryable` CSS class for queryable UI elements and underline them on hover

* Instead of `N requests` at the bottom, make it `Displaying N results (queried X/Y)` and live update the values

Upgrade Basenine version to `0.2.0`.

* Verify the sha256sum of Basenine executable inside `Dockerfile`

* Pass the start time to web UI through WebSocket and always show the `EntriesList` footer

* Pipe the `stderr` of Basenine as well

* Fix the layout issues related to `CodeEditor` in the UI

* Use the correct `shasum` command in `Dockerfile`

* Upgrade Basenine version to `0.2.1`

* Limit the height of `CodeEditor` container

* Remove `Paused` enum `ConnectionStatus` in UI

* Fix the issue caused by the recent merge

* Add the filtering guide (cheatsheet)

* Update open cheatsheet button's title

* Update cheatsheet content

* Remove the old SQLite code, adapt the `--analyze` related code to Basenine

* Change the method signature of `NewEntry`

* Change the method signature of `Represent`

* Introduce `HTTPPair` field in `MizuEntry` specific to HTTP

* Remove `Entry`, `EntryId` and `EstimatedSizeBytes` fields from `MizuEntry`

Also remove the `getEstimatedEntrySizeBytes` method.

* Remove `gorm.io/gorm` dependency

* Remove unused `sensitiveDataFiltering` folder

* Increase the left margin of open cheatsheet button

* Add `overflow: auto` to the cheatsheet `Modal`

* Fix `GetEntry` method

* Fix the macro for gRPC

* Fix an interface conversion in case of AMQP

* Fix two more interface conversion errors in AMQP

* Make the `syncEntriesImpl` method blocking

* Fix a grammar mistake in the cheatsheet

* Adapt to the changes in the recent merge commit

* Improve the cheatsheet text

* Always display the timestamp in `en-US`

* Upgrade Basenine version to `0.2.2`

* Fix the order of closing Basenine connections and channels

* Don't close the Basenine channels at all

* Upgrade Basenine version to `0.2.3`

* Set the initial filter to `rlimit(100)`

* Make Basenine persistent

* Upgrade Basenine version to `0.2.4`

* Update `debug.Dockerfile`

* Fix a failing test

* Upgrade Basenine version to `0.2.5`

* Revert "Do not show play icon when disconnected (#428)"

This reverts commit 8af2e562f8.

* Upgrade Basenine version to `0.2.6`

* Make all non-informative things informative

* Make `100` a constant

* Use `===` in JavaScript no matter what

* Remove a forgotten `console.log`

* Add a comment and update the `query` in `syncEntriesImpl`

* Don't call `panic` in `GetEntry`

* Replace `panic` calls in `startBasenineServer` with `logger.Log.Panicf`

* Remove unnecessary `\n` characters in the logs

.gitignore

@@ -29,3 +29,6 @@ build
 
 # pprof
 pprof/*
+
+# Database Files
+*.bin

Dockerfile

@@ -13,7 +13,7 @@ FROM golang:1.16-alpine AS builder
 # Set necessary environment variables needed for our image.
 ENV CGO_ENABLED=1 GOOS=linux GOARCH=amd64
 
-RUN apk add libpcap-dev gcc g++ make bash
+RUN apk add libpcap-dev gcc g++ make bash perl-utils
 
 # Move to agent working directory (/agent-build).
 WORKDIR /app/agent-build
@@ -24,7 +24,7 @@ COPY tap/go.mod tap/go.mod ../tap/
 COPY tap/api/go.* ../tap/api/
 RUN go mod download
 # cheap trick to make the build faster (As long as go.mod wasn't changes)
-RUN go list -f '{{.Path}}@{{.Version}}' -m all | sed 1d | grep -e 'go-cache' -e 'sqlite' | xargs go get
+RUN go list -f '{{.Path}}@{{.Version}}' -m all | sed 1d | grep -e 'go-cache' | xargs go get
 
 ARG COMMIT_HASH
 ARG GIT_BRANCH
@@ -41,16 +41,24 @@ RUN go build -ldflags="-s -w \
      -X 'mizuserver/pkg/version.BuildTimestamp=${BUILD_TIMESTAMP}' \
      -X 'mizuserver/pkg/version.SemVer=${SEM_VER}'" -o mizuagent .
 
+# Download Basenine executable, verify the sha1sum and move it to a directory in $PATH
+ADD https://github.com/up9inc/basenine/releases/download/v0.2.6/basenine_linux_amd64 ./basenine_linux_amd64
+ADD https://github.com/up9inc/basenine/releases/download/v0.2.6/basenine_linux_amd64.sha256 ./basenine_linux_amd64.sha256
+RUN shasum -a 256 -c basenine_linux_amd64.sha256
+RUN chmod +x ./basenine_linux_amd64
+
 COPY devops/build_extensions.sh ..
 RUN cd .. && /bin/bash build_extensions.sh
 
 FROM alpine:3.14
 
 RUN apk add bash libpcap-dev tcpdump
+
 WORKDIR /app
 
 # Copy binary and config files from /build to root folder of scratch container.
 COPY --from=builder ["/app/agent-build/mizuagent", "."]
+COPY --from=builder ["/app/agent-build/basenine_linux_amd64", "/usr/local/bin/basenine"]
 COPY --from=builder ["/app/agent/build/extensions", "extensions"]
 COPY --from=site-build ["/app/ui-build/build", "site"]
 RUN mkdir /app/data/

acceptanceTests/tap_test.go

@@ -472,7 +472,7 @@ func TestTapRedact(t *testing.T) {
 		entryPayload := entryRequest["payload"].(map[string]interface{})
 		entryDetails := entryPayload["details"].(map[string]interface{})
 
-		headers := entryDetails["headers"].([]interface{})
+		headers := entryDetails["_headers"].([]interface{})
 		for _, headerInterface := range headers {
 			header := headerInterface.(map[string]interface{})
 			if header["name"].(string) != "User-Agent" {
@@ -587,7 +587,7 @@ func TestTapNoRedact(t *testing.T) {
 		entryPayload := entryRequest["payload"].(map[string]interface{})
 		entryDetails := entryPayload["details"].(map[string]interface{})
 
-		headers := entryDetails["headers"].([]interface{})
+		headers := entryDetails["_headers"].([]interface{})
 		for _, headerInterface := range headers {
 			header := headerInterface.(map[string]interface{})
 			if header["name"].(string) != "User-Agent" {
@@ -808,7 +808,7 @@ func TestTapIgnoredUserAgents(t *testing.T) {
 			entryPayload := entryRequest["payload"].(map[string]interface{})
 			entryDetails := entryPayload["details"].(map[string]interface{})
 
-			entryHeaders := entryDetails["headers"].([]interface{})
+			entryHeaders := entryDetails["_headers"].([]interface{})
 			for _, headerInterface := range entryHeaders {
 				header := headerInterface.(map[string]interface{})
 				if header["name"].(string) != ignoredUserAgentCustomHeader {

agent/go.mod

@@ -3,8 +3,8 @@ module mizuserver
 go 1.16
 
 require (
+	github.com/antelman107/net-wait-go v0.0.0-20210623112055-cf684aebda7b
 	github.com/djherbis/atime v1.0.0
-	github.com/fsnotify/fsnotify v1.4.9
 	github.com/getkin/kin-openapi v0.76.0
 	github.com/gin-contrib/static v0.0.1
 	github.com/gin-gonic/gin v1.7.2
@@ -16,13 +16,12 @@ require (
 	github.com/op/go-logging v0.0.0-20160315200505-970db520ece7
 	github.com/orcaman/concurrent-map v0.0.0-20210106121528-16402b402231
 	github.com/patrickmn/go-cache v2.1.0+incompatible
+	github.com/up9inc/basenine/client/go v0.0.0-20211106180626-0193408db715
 	github.com/up9inc/mizu/shared v0.0.0
 	github.com/up9inc/mizu/tap v0.0.0
 	github.com/up9inc/mizu/tap/api v0.0.0
 	github.com/yalp/jsonpath v0.0.0-20180802001716-5cc68e5049a0
-	go.mongodb.org/mongo-driver v1.7.1
+	golang.org/x/text v0.3.5 // indirect
-	gorm.io/driver/sqlite v1.1.4
-	gorm.io/gorm v1.21.8
 	k8s.io/api v0.21.2
 	k8s.io/apimachinery v0.21.2
 	k8s.io/client-go v0.21.2

agent/go.sum

@@ -52,6 +52,8 @@ github.com/alecthomas/template v0.0.0-20190718012654-fb15b899a751/go.mod h1:LOuy
 github.com/alecthomas/units v0.0.0-20151022065526-2efee857e7cf/go.mod h1:ybxpYRFXyAe+OPACYpWeL0wqObRcbAqCMya13uyzqw0=
 github.com/alecthomas/units v0.0.0-20190717042225-c3de453c63f4/go.mod h1:ybxpYRFXyAe+OPACYpWeL0wqObRcbAqCMya13uyzqw0=
 github.com/andreyvit/diff v0.0.0-20170406064948-c7f18ee00883/go.mod h1:rCTlJbsFo29Kk6CurOXKm700vrz8f0KW0JNfpkRJY/8=
+github.com/antelman107/net-wait-go v0.0.0-20210623112055-cf684aebda7b h1:8m+eVxVVDDyJFidv7Ck1OwqnDaQR6pTSRGlCC2Dnw0A=
+github.com/antelman107/net-wait-go v0.0.0-20210623112055-cf684aebda7b/go.mod h1:+tQQjzrp2501Nd6JXrb9s/XsNvFK3ZbxOnCdQl/vDRo=
 github.com/armon/circbuf v0.0.0-20150827004946-bbbad097214e/go.mod h1:3U/XgcO3hCbHZ8TKRvWD2dDTCfh9M9ya+I9JpbB7O8o=
 github.com/armon/consul-api v0.0.0-20180202201655-eb2c6b5be1b6/go.mod h1:grANhF5doyWs3UAsr3K4I6qtAmlQcZDesFNEHPZAzj8=
 github.com/armon/go-metrics v0.0.0-20180917152333-f0300d1749da/go.mod h1:Q73ZrmVTwzkszR9V5SSuryQ31EELlFMUz1kKyl939pY=
@@ -94,7 +96,6 @@ github.com/djherbis/atime v1.0.0 h1:ySLvBAM0EvOGaX7TI4dAM5lWj+RdJUCKtGSEHN8SGBg=
 github.com/djherbis/atime v1.0.0/go.mod h1:5W+KBIuTwVGcqjIfaTwt+KSYX1o6uep8dtevevQP/f8=
 github.com/docker/distribution v2.7.1+incompatible/go.mod h1:J2gT2udsDAN96Uj4KfcMRqY0/ypR+oyYUYmja8H+y+w=
 github.com/docker/go-units v0.3.3/go.mod h1:fgPhTUdO+D/Jk86RDLlptpiXQzgHJF7gydDDbaIK4Dk=
-github.com/docker/go-units v0.4.0 h1:3uh0PgVws3nIA0Q+MwDC8yjEPf9zjRfZZWXZYDct3Tw=
 github.com/docker/go-units v0.4.0/go.mod h1:fgPhTUdO+D/Jk86RDLlptpiXQzgHJF7gydDDbaIK4Dk=
 github.com/docopt/docopt-go v0.0.0-20180111231733-ee0de3bc6815/go.mod h1:WwZ+bS3ebgob9U8Nd0kOddGdZWjyMGR8Wziv+TBNwSE=
 github.com/elazarl/goproxy v0.0.0-20180725130230-947c36da3153/go.mod h1:/Zj4wYkgs4iZTTu3o/KG3Itv/qCCa8VVMlb3i9OVuzc=
@@ -111,7 +112,6 @@ github.com/fatih/color v1.7.0/go.mod h1:Zm6kSWBoL9eyXnKyktHP6abPY2pDugNf5Kwzbycv
 github.com/form3tech-oss/jwt-go v3.2.2+incompatible h1:TcekIExNqud5crz4xD2pavyTgWiPvpYe4Xau31I0PRk=
 github.com/form3tech-oss/jwt-go v3.2.2+incompatible/go.mod h1:pbq4aXjuKjdthFRnoDwaVPLA+WlJuPGy+QneDUgJi2k=
 github.com/fsnotify/fsnotify v1.4.7/go.mod h1:jwhsz4b93w/PPRr/qN1Yymfu8t87LnFCMoQvtojpjFo=
-github.com/fsnotify/fsnotify v1.4.9 h1:hsms1Qyu0jgnwNXIxa+/V/PDsU6CfLf6CNO8H7IWoS4=
 github.com/fsnotify/fsnotify v1.4.9/go.mod h1:znqG4EE+3YCdAaPaxE2ZRY/06pZUdp0tY4IgpuI1SZQ=
 github.com/fvbommel/sortorder v1.0.1/go.mod h1:uk88iVf1ovNn1iLfgUVU2F9o5eO30ui720w+kxuqRs0=
 github.com/getkin/kin-openapi v0.76.0 h1:j77zg3Ec+k+r+GA3d8hBoXpAc6KX9TbBPrwQGBIy2sY=
@@ -195,31 +195,7 @@ github.com/go-playground/validator/v10 v10.4.1/go.mod h1:nlOn6nFhuKACm19sB/8EGNn
 github.com/go-playground/validator/v10 v10.5.0 h1:X9rflw/KmpACwT8zdrm1upefpvdy6ur8d1kWyq6sg3E=
 github.com/go-playground/validator/v10 v10.5.0/go.mod h1:xm76BBt941f7yWdGnI2DVPFFg1UK3YY04qifoXU3lOk=
 github.com/go-stack/stack v1.8.0/go.mod h1:v0f6uXyyMGvRgIKkXu+yp6POWl0qKG85gN/melR3HDY=
-github.com/gobuffalo/attrs v0.0.0-20190224210810-a9411de4debd/go.mod h1:4duuawTqi2wkkpB4ePgWMaai6/Kc6WEz83bhFwpHzj0=
-github.com/gobuffalo/depgen v0.0.0-20190329151759-d478694a28d3/go.mod h1:3STtPUQYuzV0gBVOY3vy6CfMm/ljR4pABfrTeHNLHUY=
-github.com/gobuffalo/depgen v0.1.0/go.mod h1:+ifsuy7fhi15RWncXQQKjWS9JPkdah5sZvtHc2RXGlg=
-github.com/gobuffalo/envy v1.6.15/go.mod h1:n7DRkBerg/aorDM8kbduw5dN3oXGswK5liaSCx4T5NI=
-github.com/gobuffalo/envy v1.7.0/go.mod h1:n7DRkBerg/aorDM8kbduw5dN3oXGswK5liaSCx4T5NI=
-github.com/gobuffalo/flect v0.1.0/go.mod h1:d2ehjJqGOH/Kjqcoz+F7jHTBbmDb38yXA598Hb50EGs=
-github.com/gobuffalo/flect v0.1.1/go.mod h1:8JCgGVbRjJhVgD6399mQr4fx5rRfGKVzFjbj6RE/9UI=
-github.com/gobuffalo/flect v0.1.3/go.mod h1:8JCgGVbRjJhVgD6399mQr4fx5rRfGKVzFjbj6RE/9UI=
-github.com/gobuffalo/genny v0.0.0-20190329151137-27723ad26ef9/go.mod h1:rWs4Z12d1Zbf19rlsn0nurr75KqhYp52EAGGxTbBhNk=
-github.com/gobuffalo/genny v0.0.0-20190403191548-3ca520ef0d9e/go.mod h1:80lIj3kVJWwOrXWWMRzzdhW3DsrdjILVil/SFKBzF28=
-github.com/gobuffalo/genny v0.1.0/go.mod h1:XidbUqzak3lHdS//TPu2OgiFB+51Ur5f7CSnXZ/JDvo=
-github.com/gobuffalo/genny v0.1.1/go.mod h1:5TExbEyY48pfunL4QSXxlDOmdsD44RRq4mVZ0Ex28Xk=
-github.com/gobuffalo/gitgen v0.0.0-20190315122116-cc086187d211/go.mod h1:vEHJk/E9DmhejeLeNt7UVvlSGv3ziL+djtTr3yyzcOw=
-github.com/gobuffalo/gogen v0.0.0-20190315121717-8f38393713f5/go.mod h1:V9QVDIxsgKNZs6L2IYiGR8datgMhB577vzTDqypH360=
-github.com/gobuffalo/gogen v0.1.0/go.mod h1:8NTelM5qd8RZ15VjQTFkAW6qOMx5wBbW4dSCS3BY8gg=
-github.com/gobuffalo/gogen v0.1.1/go.mod h1:y8iBtmHmGc4qa3urIyo1shvOD8JftTtfcKi+71xfDNE=
 github.com/gobuffalo/here v0.6.0/go.mod h1:wAG085dHOYqUpf+Ap+WOdrPTp5IYcDAs/x7PLa8Y5fM=
-github.com/gobuffalo/logger v0.0.0-20190315122211-86e12af44bc2/go.mod h1:QdxcLw541hSGtBnhUc4gaNIXRjiDppFGaDqzbrBd3v8=
-github.com/gobuffalo/mapi v1.0.1/go.mod h1:4VAGh89y6rVOvm5A8fKFxYG+wIW6LO1FMTG9hnKStFc=
-github.com/gobuffalo/mapi v1.0.2/go.mod h1:4VAGh89y6rVOvm5A8fKFxYG+wIW6LO1FMTG9hnKStFc=
-github.com/gobuffalo/packd v0.0.0-20190315124812-a385830c7fc0/go.mod h1:M2Juc+hhDXf/PnmBANFCqx4DM3wRbgDvnVWeG2RIxq4=
-github.com/gobuffalo/packd v0.1.0/go.mod h1:M2Juc+hhDXf/PnmBANFCqx4DM3wRbgDvnVWeG2RIxq4=
-github.com/gobuffalo/packr/v2 v2.0.9/go.mod h1:emmyGweYTm6Kdper+iywB6YK5YzuKchGtJQZ0Odn4pQ=
-github.com/gobuffalo/packr/v2 v2.2.0/go.mod h1:CaAwI0GPIAv+5wKLtv8Afwl+Cm78K/I/VCm/3ptBN+0=
-github.com/gobuffalo/syncx v0.0.0-20190224160051-33c29581e754/go.mod h1:HhnNqWY95UYwwW3uSASeV7vtgYkT2t16hJgV3AEPUpw=
 github.com/gogo/protobuf v1.1.1/go.mod h1:r8qH/GZQm5c6nD/R0oafs1akxWv10x8SbQlK7atdtwQ=
 github.com/gogo/protobuf v1.2.1/go.mod h1:hp+jE20tsWTFYpLwKvXlhS1hjn+gTNwPg2I6zVXpSg4=
 github.com/gogo/protobuf v1.3.2 h1:Ov1cvc58UF3b5XjBnZv7+opcTcQFZebYjWzi34vdm4Q=
@@ -251,7 +227,6 @@ github.com/golang/protobuf v1.4.1/go.mod h1:U8fpvMrcmy5pZrNK1lt4xCsGvpyWQ/VVv6QD
 github.com/golang/protobuf v1.4.2/go.mod h1:oDoupMAO8OvCJWAcko0GGGIgR6R6ocIYbsSw735rRwI=
 github.com/golang/protobuf v1.4.3 h1:JjCZWpVbqXDqFVmTfYWEVTMIYrL/NPdPSCHPJ0T/raM=
 github.com/golang/protobuf v1.4.3/go.mod h1:oDoupMAO8OvCJWAcko0GGGIgR6R6ocIYbsSw735rRwI=
-github.com/golang/snappy v0.0.1/go.mod h1:/XxbfmMg8lxefKM7IXC3fBNl/7bRcc72aCRzEWrmP2Q=
 github.com/golangplus/testing v0.0.0-20180327235837-af21d9c3145e/go.mod h1:0AA//k/eakGydO4jKRoRL2j92ZKSzTgj9tclaCrvXHk=
 github.com/google/btree v0.0.0-20180813153112-4030bb1f1f0c/go.mod h1:lNA+9X1NB3Zf8V7Ke586lFgjr2dZNuvo3lPJSGZ5JPQ=
 github.com/google/btree v1.0.0/go.mod h1:lNA+9X1NB3Zf8V7Ke586lFgjr2dZNuvo3lPJSGZ5JPQ=
@@ -321,12 +296,6 @@ github.com/ianlancetaylor/demangle v0.0.0-20181102032728-5e5cf60278f6/go.mod h1:
 github.com/imdario/mergo v0.3.5 h1:JboBksRwiiAJWvIYJVo46AfV+IAIKZpfrSzVKj42R4Q=
 github.com/imdario/mergo v0.3.5/go.mod h1:2EnlNZ0deacrJVfApfmtdGgDfMuh/nq6Ok1EcJh5FfA=
 github.com/inconshreveable/mousetrap v1.0.0/go.mod h1:PxqpIevigyE2G7u3NXJIT2ANytuPF1OarO4DADm73n8=
-github.com/jinzhu/inflection v1.0.0 h1:K317FqzuhWc8YvSVlFMCCUb36O/S9MCKRDI7QkRKD/E=
-github.com/jinzhu/inflection v1.0.0/go.mod h1:h+uFLlag+Qp1Va5pdKtLDYj+kHp5pxUVkryuEj+Srlc=
-github.com/jinzhu/now v1.1.1/go.mod h1:d3SSVoowX0Lcu0IBviAWJpolVfI5UJVZZ7cO71lE/z8=
-github.com/jinzhu/now v1.1.2 h1:eVKgfIdy9b6zbWBMgFpfDPoAMifwSZagU9HmEU6zgiI=
-github.com/jinzhu/now v1.1.2/go.mod h1:d3SSVoowX0Lcu0IBviAWJpolVfI5UJVZZ7cO71lE/z8=
-github.com/joho/godotenv v1.3.0/go.mod h1:7hK45KPybAkOC6peb+G5yklZfMxEjkZhHbwpqxOKXbg=
 github.com/jonboulle/clockwork v0.1.0/go.mod h1:Ii8DK3G1RaLaWxj9trq07+26W01tbo22gdxWY5EU2bo=
 github.com/json-iterator/go v1.1.6/go.mod h1:+SdeFBvtyEkXs7REEP0seUULqWtbJapLOCVDaaPEHmU=
 github.com/json-iterator/go v1.1.9/go.mod h1:KdQUCv79m/52Kvf8AW2vK1V8akMuk1QjK/uOdHXbAo4=
@@ -336,14 +305,10 @@ github.com/jstemmer/go-junit-report v0.0.0-20190106144839-af01ea7f8024/go.mod h1
 github.com/jstemmer/go-junit-report v0.9.1/go.mod h1:Brl9GWCQeLvo8nXZwPNNblvFj/XSXhF0NWZEnDohbsk=
 github.com/jtolds/gls v4.20.0+incompatible/go.mod h1:QJZ7F/aHp+rZTRtaJ1ow/lLfFfVYBRgL+9YlvaHOwJU=
 github.com/julienschmidt/httprouter v1.2.0/go.mod h1:SYymIcj16QtmaHHD7aYtjjsJG7VTCxuUUipMqKk8s4w=
-github.com/karrick/godirwalk v1.8.0/go.mod h1:H5KPZjojv4lE+QYImBI8xVtrBRgYrIVsaRPx4tDPEn4=
-github.com/karrick/godirwalk v1.10.3/go.mod h1:RoGL9dQei4vP9ilrpETWE8CLOZ1kiN0LhBygSwrAsHA=
 github.com/kisielk/errcheck v1.1.0/go.mod h1:EZBBE59ingxPouuu3KfxchcWSUPOHkagtvWXihfKN4Q=
 github.com/kisielk/errcheck v1.5.0/go.mod h1:pFxgyoBC7bSaBwPgfKdkLd5X25qrDl4LWUI2bnpBCr8=
 github.com/kisielk/gotool v1.0.0/go.mod h1:XhKaO+MFFWcvkIS/tQcRk01m1F5IRFswLeQ+oQHNcck=
-github.com/klauspost/compress v1.9.5/go.mod h1:RyIbtBH6LamlWaDj8nUwkbUhJ87Yi3uG0guNDohfE1A=
 github.com/konsorten/go-windows-terminal-sequences v1.0.1/go.mod h1:T0+1ngSBFLxvqU3pZ+m/2kptfBszLMUkC4ZK/EgS/cQ=
-github.com/konsorten/go-windows-terminal-sequences v1.0.2/go.mod h1:T0+1ngSBFLxvqU3pZ+m/2kptfBszLMUkC4ZK/EgS/cQ=
 github.com/kr/logfmt v0.0.0-20140226030751-b84e30acd515/go.mod h1:+0opPa2QZZtGFBFZlji/RkVcI2GknAs/DXo4wKdlNEc=
 github.com/kr/pretty v0.1.0/go.mod h1:dAy3ld7l9f0ibDNOQOHHMYYIIbhfbHSm3C4ZsoJORNo=
 github.com/kr/pretty v0.2.0/go.mod h1:ipq/a2n7PKx3OHsz4KJII5eveXtPO4qwEXGdVfWzfnI=
@@ -364,16 +329,12 @@ github.com/mailru/easyjson v0.0.0-20190614124828-94de47d64c63/go.mod h1:C1wdFJiN
 github.com/mailru/easyjson v0.0.0-20190626092158-b2ccc519800e/go.mod h1:C1wdFJiN94OJF2b5HbByQZoLdCWB1Yqtg26g4irojpc=
 github.com/mailru/easyjson v0.7.0 h1:aizVhC/NAAcKWb+5QsU1iNOZb4Yws5UO2I+aIprQITM=
 github.com/mailru/easyjson v0.7.0/go.mod h1:KAzv3t3aY1NaHWoQz1+4F1ccyAH66Jk7yos7ldAVICs=
-github.com/markbates/oncer v0.0.0-20181203154359-bf2de49a0be2/go.mod h1:Ld9puTsIW75CHf65OeIOkyKbteujpZVXDpWK6YGZbxE=
 github.com/markbates/pkger v0.17.1/go.mod h1:0JoVlrol20BSywW79rN3kdFFsE5xYM+rSCQDXbLhiuI=
-github.com/markbates/safe v1.0.1/go.mod h1:nAqgmRi7cY2nqMc92/bSEeQA+R4OheNU2T1kNSCBdG0=
 github.com/mattn/go-colorable v0.0.9/go.mod h1:9vuHe8Xs5qXnSaW/c/ABM9alt+Vo+STaOChaDxuIBZU=
 github.com/mattn/go-isatty v0.0.3/go.mod h1:M+lRXTBqGeGNdLjl/ufCoiOlB5xdOkqRJdNxMWT7Zi4=
 github.com/mattn/go-isatty v0.0.12 h1:wuysRhFDzyxgEmMf5xjvJ2M9dZoWAXNNr5LSBS7uHXY=
 github.com/mattn/go-isatty v0.0.12/go.mod h1:cbi8OIDigv2wuxKPP5vlRcQ1OAZbq2CE4Kysco4FUpU=
 github.com/mattn/go-runewidth v0.0.7/go.mod h1:H031xJmbD/WCDINGzjvQ9THkh0rPKHF+m2gUSrubnMI=
-github.com/mattn/go-sqlite3 v1.14.5 h1:1IdxlwTNazvbKJQSxoJ5/9ECbEeaTTyeU7sEAZ5KKTQ=
-github.com/mattn/go-sqlite3 v1.14.5/go.mod h1:WVKg1VTActs4Qso6iwGbiFih2UIHo0ENGwNd0Lj+XmI=
 github.com/matttproud/golang_protobuf_extensions v1.0.1/go.mod h1:D8He9yQNgCq6Z5Ld7szi9bcBfOoFv/3dc6xSMkL2PC0=
 github.com/matttproud/golang_protobuf_extensions v1.0.2-0.20181231171920-c182affec369/go.mod h1:BSXmuO+STAnVfrANrmjBb36TMTDstsz7MSK+HVaYKv4=
 github.com/miekg/dns v1.0.14/go.mod h1:W1PPwlIAgtquWBMBEV9nkV9Cazfe8ScdGz/Lj7v3Nrg=
@@ -395,7 +356,6 @@ github.com/modern-go/reflect2 v0.0.0-20180701023420-4b7aa43c6742/go.mod h1:bx2lN
 github.com/modern-go/reflect2 v1.0.1 h1:9f412s+6RmYXLWZSEzVVgPGK7C2PphHj5RJrvfx9AWI=
 github.com/modern-go/reflect2 v1.0.1/go.mod h1:bx2lNnkwVCuqBIxFjflWJWanXIb3RllmbCylyMrvgv0=
 github.com/monochromegane/go-gitignore v0.0.0-20200626010858-205db1a8cc00/go.mod h1:Pm3mSP3c5uWn86xMLZ5Sa7JB9GsEZySvHYXCTK4E9q4=
-github.com/montanaflynn/stats v0.0.0-20171201202039-1bf9dbcd8cbe/go.mod h1:wL8QJuTMNUDYhXwkmfOly8iTdp5TEcJFWZD2D7SIkUc=
 github.com/munnerz/goautoneg v0.0.0-20120707110453-a547fc61f48d/go.mod h1:+n7T8mK8HuQTcFwEeznm/DIxMOiR9yIdICNftLE1DvQ=
 github.com/mwitkow/go-conntrack v0.0.0-20161129095857-cc309e4a2223/go.mod h1:qRWi+5nqEBWmkhHvq77mSJWrCKwh8bxhgT7d/eI7P4U=
 github.com/mxk/go-flowrate v0.0.0-20140419014527-cca7078d478f h1:y5//uYreIhSUg3J1GEMiLbxo1LJaP8RfCpH6pymGZus=
@@ -419,7 +379,6 @@ github.com/patrickmn/go-cache v2.1.0+incompatible h1:HRMgzkcYKYpi3C8ajMPV8OFXaaR
 github.com/patrickmn/go-cache v2.1.0+incompatible/go.mod h1:3Qf8kWWT7OJRJbdiICTKqZju1ZixQ/KpMGzzAfe6+WQ=
 github.com/pborman/uuid v1.2.0/go.mod h1:X/NO0urCmaxf9VXbdlT7C2Yzkj2IKimNn4k+gtPdI/k=
 github.com/pelletier/go-toml v1.2.0/go.mod h1:5z9KED0ma1S8pY6P1sdut58dfprrGBbd/94hg7ilaic=
-github.com/pelletier/go-toml v1.7.0/go.mod h1:vwGMzjaWMwyfHwgIBhI2YUM4fB6nL6lVAvS1LBMMhTE=
 github.com/peterbourgon/diskv v2.0.1+incompatible/go.mod h1:uqqh8zWWbv1HBMNONnaR/tNboyR3/BZd58JJSHlUSCU=
 github.com/pkg/errors v0.8.0/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0=
 github.com/pkg/errors v0.8.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0=
@@ -447,8 +406,6 @@ github.com/prometheus/procfs v0.1.3/go.mod h1:lV6e/gmhEcM9IjHGsFOCxxuZ+z1YqCvr4O
 github.com/prometheus/procfs v0.2.0/go.mod h1:lV6e/gmhEcM9IjHGsFOCxxuZ+z1YqCvr4OA4YeYWdaU=
 github.com/prometheus/tsdb v0.7.1/go.mod h1:qhTCs0VvXwvX/y3TZrWD7rabWM+ijKTux40TwIPHuXU=
 github.com/rogpeppe/fastuuid v0.0.0-20150106093220-6724a57986af/go.mod h1:XWv6SoW27p1b0cqNHllgS5HIMJraePCO15w5zCzIWYg=
-github.com/rogpeppe/go-internal v1.1.0/go.mod h1:M8bDsm7K2OlrFYOpmOWEs/qY81heoFRclV5y23lUDJ4=
-github.com/rogpeppe/go-internal v1.2.2/go.mod h1:M8bDsm7K2OlrFYOpmOWEs/qY81heoFRclV5y23lUDJ4=
 github.com/rogpeppe/go-internal v1.3.0/go.mod h1:M8bDsm7K2OlrFYOpmOWEs/qY81heoFRclV5y23lUDJ4=
 github.com/russross/blackfriday v1.5.2/go.mod h1:JO/DiYxRf+HjHt06OyowR9PTA263kcR/rfWxYHBV53g=
 github.com/russross/blackfriday/v2 v2.0.1/go.mod h1:+Rmxgy9KzJVeS9/2gXHxylqXiyQDYRxCVz55jmeOWTM=
@@ -458,8 +415,6 @@ github.com/sergi/go-diff v1.0.0/go.mod h1:0CfEIISq7TuYL3j771MWULgwwjU+GofnZX9QAm
 github.com/sergi/go-diff v1.1.0/go.mod h1:STckp+ISIX8hZLjrqAeVduY0gWCT9IjLuqbuNXdaHfM=
 github.com/shurcooL/sanitized_anchor_name v1.0.0/go.mod h1:1NzhyTcUVG4SuEtjjoZeVRXNmyL/1OwPU0+IJeTBvfc=
 github.com/sirupsen/logrus v1.2.0/go.mod h1:LxeOpSwHxABJmUn/MG1IvRgCAasNZTLOkJPxbbu5VWo=
-github.com/sirupsen/logrus v1.4.0/go.mod h1:LxeOpSwHxABJmUn/MG1IvRgCAasNZTLOkJPxbbu5VWo=
-github.com/sirupsen/logrus v1.4.1/go.mod h1:ni0Sbl8bgC9z8RoU9G6nDWqqs/fq4eDPysMBDgk/93Q=
 github.com/sirupsen/logrus v1.4.2/go.mod h1:tLMulIdttU9McNUspp0xgXVQah82FyeX6MwdIuYE2rE=
 github.com/smartystreets/assertions v0.0.0-20180927180507-b2de0cb4f26d/go.mod h1:OnSkiWE9lh6wB0YB77sQom3nweQdgAjqCqsofrRNTgc=
 github.com/smartystreets/goconvey v1.6.4/go.mod h1:syvi0/a8iFYH4r/RixwvyeAJjdLS9QV7WQ/tjFTllLA=
@@ -468,7 +423,6 @@ github.com/spaolacci/murmur3 v0.0.0-20180118202830-f09979ecbc72/go.mod h1:JwIasO
 github.com/spf13/afero v1.1.2/go.mod h1:j4pytiNVoe2o6bmDsKpLACNPDBIoEAkihy7loJ1B0CQ=
 github.com/spf13/afero v1.2.2/go.mod h1:9ZxEEn6pIJ8Rxe320qSDBk6AsU0r9pR7Q4OcevTdifk=
 github.com/spf13/cast v1.3.0/go.mod h1:Qx5cxh0v+4UWYiBimWS+eyWzqEqokIECu5etghLkUJE=
-github.com/spf13/cobra v0.0.3/go.mod h1:1l0Ry5zgKvJasoi3XT1TypsSe7PqH0Sj9dhYf7v3XqQ=
 github.com/spf13/cobra v1.0.0/go.mod h1:/6GTrnGXV9HjY+aR4k0oJ5tcvakLuG6EuKReYlHNrgE=
 github.com/spf13/cobra v1.1.1/go.mod h1:WnodtKOvamDL/PwE2M4iKs8aMDBZ5Q5klgD3qfVJQMI=
 github.com/spf13/jwalterweatherman v1.0.0/go.mod h1:cQK4TGJAtQXfYWX+Ddv3mKDzgVb68N+wFjFa4jdeBTo=
@@ -485,8 +439,9 @@ github.com/stretchr/testify v1.2.2/go.mod h1:a8OnRcib4nhh0OaRAV+Yts87kKdq0PP7pXf
 github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI=
 github.com/stretchr/testify v1.4.0/go.mod h1:j7eGeouHqKxXV5pUuKE4zz7dFj8WfuZ+81PSLYec5m4=
 github.com/stretchr/testify v1.5.1/go.mod h1:5W2xD1RspED5o8YsWQXVCued0rvSQ+mT+I5cxcmMvtA=
-github.com/stretchr/testify v1.6.1 h1:hDPOHmpOpP40lSULcqw7IrRb/u7w6RpDC9399XyoNd0=
 github.com/stretchr/testify v1.6.1/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
+github.com/stretchr/testify v1.7.0 h1:nwc3DEeHmmLAfoZucVR881uASk0Mfjw8xYJ99tb5CcY=
+github.com/stretchr/testify v1.7.0/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
 github.com/subosito/gotenv v1.2.0/go.mod h1:N0PQaV/YGNqwC0u51sEeR/aUtSLEXKX9iv69rRypqCw=
 github.com/tidwall/pretty v1.0.0/go.mod h1:XNkn88O1ChpSDQmQeStsy+sBenx6DDtFZJxhVysOjyk=
 github.com/tmc/grpc-websocket-proxy v0.0.0-20190109142713-0ad062ec5ee5/go.mod h1:ncp9v5uamzpCO7NfCPTXjqaC+bZgJeR0sMTm6dMHP7U=
@@ -495,26 +450,22 @@ github.com/ugorji/go v1.1.7 h1:/68gy2h+1mWMrwZFeD1kQialdSzAb432dtpeJ42ovdo=
 github.com/ugorji/go v1.1.7/go.mod h1:kZn38zHttfInRq0xu/PH0az30d+z6vm202qpg1oXVMw=
 github.com/ugorji/go/codec v1.1.7 h1:2SvQaVZ1ouYrrKKwoSk2pzd4A9evlKJb9oTL+OaLUSs=
 github.com/ugorji/go/codec v1.1.7/go.mod h1:Ax+UKWsSmolVDwsd+7N3ZtXu+yMGCf907BLYF3GoBXY=
+github.com/up9inc/basenine/client/go v0.0.0-20211106180626-0193408db715 h1:3RNTMQZO/4g5gRn4R98cPwCjCrsMklmcOS0g+QwCh5c=
+github.com/up9inc/basenine/client/go v0.0.0-20211106180626-0193408db715/go.mod h1:SvJGPoa/6erhUQV7kvHBwM/0x5LyO6XaG2lUaCaKiUI=
 github.com/vektah/gqlparser v1.1.2/go.mod h1:1ycwN7Ij5njmMkPPAOaRFY4rET2Enx7IkVv3vaXspKw=
 github.com/vishvananda/netns v0.0.0-20210104183010-2eb08e3e575f h1:p4VB7kIXpOQvVn1ZaTIVp+3vuYAXFe3OJEvjbUYJLaA=
 github.com/vishvananda/netns v0.0.0-20210104183010-2eb08e3e575f/go.mod h1:DD4vA1DwXk04H54A1oHXtwZmA0grkVMdPxx/VGLCah0=
-github.com/xdg-go/pbkdf2 v1.0.0/go.mod h1:jrpuAogTd400dnrH08LKmI/xc1MbPOebTwRqcT5RDeI=
-github.com/xdg-go/scram v1.0.2/go.mod h1:1WAq6h33pAW+iRreB34OORO2Nf7qel3VV3fjBj+hCSs=
-github.com/xdg-go/stringprep v1.0.2/go.mod h1:8F9zXuvzgwmyT5DUm4GUfZGDdT3W+LCvS6+da4O5kxM=
 github.com/xiang90/probing v0.0.0-20190116061207-43a291ad63a2/go.mod h1:UETIi67q53MR2AWcXfiuqkDkRtnGDLqkBTpCHuJHxtU=
 github.com/xlab/treeprint v0.0.0-20181112141820-a009c3971eca/go.mod h1:ce1O1j6UtZfjr22oyGxGLbauSBp2YVXpARAosm7dHBg=
 github.com/xordataexchange/crypt v0.0.3-0.20170626215501-b2862e3d0a77/go.mod h1:aYKd//L2LvnjZzWKhF00oedf4jCCReLcmhLdhm1A27Q=
 github.com/yalp/jsonpath v0.0.0-20180802001716-5cc68e5049a0 h1:6fRhSjgLCkTD3JnJxvaJ4Sj+TYblw757bqYgZaOq5ZY=
 github.com/yalp/jsonpath v0.0.0-20180802001716-5cc68e5049a0/go.mod h1:/LWChgwKmvncFJFHJ7Gvn9wZArjbV5/FppcK2fKk/tI=
-github.com/youmark/pkcs8 v0.0.0-20181117223130-1be2e3e5546d/go.mod h1:rHwXgn7JulP+udvsHwJoVG1YGAP6VLg4y9I5dyZdqmA=
 github.com/yuin/goldmark v1.1.27/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74=
 github.com/yuin/goldmark v1.2.1/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74=
 go.etcd.io/bbolt v1.3.2/go.mod h1:IbVyRI1SCnLcuJnV2u8VeU0CEYM7e686BmAb1XKL+uU=
 go.mongodb.org/mongo-driver v1.0.3/go.mod h1:u7ryQJ+DOzQmeO7zB6MHyr8jkEQvC8vH7qLUO4lqsUM=
 go.mongodb.org/mongo-driver v1.1.1/go.mod h1:u7ryQJ+DOzQmeO7zB6MHyr8jkEQvC8vH7qLUO4lqsUM=
 go.mongodb.org/mongo-driver v1.1.2/go.mod h1:u7ryQJ+DOzQmeO7zB6MHyr8jkEQvC8vH7qLUO4lqsUM=
-go.mongodb.org/mongo-driver v1.7.1 h1:jwqTeEM3x6L9xDXrCxN0Hbg7vdGfPBOTIkr0+/LYZDA=
-go.mongodb.org/mongo-driver v1.7.1/go.mod h1:Q4oFMbo1+MSNqICAdYMlC/zSTrwCogR4R8NzkI+yfU8=
 go.opencensus.io v0.21.0/go.mod h1:mSImk1erAIZhrmZN+AvHh14ztQfjbGwt4TtuofqLduU=
 go.opencensus.io v0.22.0/go.mod h1:+kGneAE2xo2IficOXnaByMWTGM9T73dGwxeWcUqIpI8=
 go.opencensus.io v0.22.2/go.mod h1:yxeiOL68Rb0Xd1ddK5vPZ/oVn4vY4Ynel7k9FzqtOIw=
@@ -527,13 +478,11 @@ golang.org/x/crypto v0.0.0-20180904163835-0709b304e793/go.mod h1:6SG95UA2DQfeDnf
 golang.org/x/crypto v0.0.0-20181029021203-45a5f77698d3/go.mod h1:6SG95UA2DQfeDnfUPMdvaQW0Q7yPrPDi9nlGo2tz2b4=
 golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w=
 golang.org/x/crypto v0.0.0-20190320223903-b7391e95e576/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w=
-golang.org/x/crypto v0.0.0-20190422162423-af44ce270edf/go.mod h1:WFFai1msRO1wXaEeE5yQxYXgSfI8pQAWXbQop6sCtWE=
 golang.org/x/crypto v0.0.0-20190510104115-cbcb75029529/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI=
 golang.org/x/crypto v0.0.0-20190605123033-f99c8df09eb5/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI=
 golang.org/x/crypto v0.0.0-20190611184440-5c40567a22f8/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI=
 golang.org/x/crypto v0.0.0-20190617133340-57b3e21c3d56/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI=
 golang.org/x/crypto v0.0.0-20191011191535-87dc89f01550/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI=
-golang.org/x/crypto v0.0.0-20200302210943-78000ba7a073/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto=
 golang.org/x/crypto v0.0.0-20200622213623-75b288015ac9/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto=
 golang.org/x/crypto v0.0.0-20201002170205-7f63de1d35b0/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto=
 golang.org/x/crypto v0.0.0-20210220033148-5ea612d1eb83 h1:/ZScEX8SfEmUGRHs0gxpqteO5nfNW6axyZbBdw9A12g=
@@ -611,7 +560,6 @@ golang.org/x/sync v0.0.0-20180314180146-1d60e4601c6f/go.mod h1:RxMgew5VJxzue5/jJ
 golang.org/x/sync v0.0.0-20181108010431-42b317875d0f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
 golang.org/x/sync v0.0.0-20181221193216-37e7f081c4d4/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
 golang.org/x/sync v0.0.0-20190227155943-e225da77a7e6/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
-golang.org/x/sync v0.0.0-20190412183630-56d357773e84/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
 golang.org/x/sync v0.0.0-20190423024810-112230192c58/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
 golang.org/x/sync v0.0.0-20190911185100-cd5d95a43a6e/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
 golang.org/x/sync v0.0.0-20201020160332-67f06af15bc9/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
@@ -625,13 +573,10 @@ golang.org/x/sys v0.0.0-20181116152217-5ac8a444bdc5/go.mod h1:STP8DvDyc/dI5b8T5h
 golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
 golang.org/x/sys v0.0.0-20190312061237-fead79001313/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
 golang.org/x/sys v0.0.0-20190321052220-f7bb7a8bee54/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
-golang.org/x/sys v0.0.0-20190403152447-81d4e9dc473e/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
 golang.org/x/sys v0.0.0-20190412213103-97732733099d/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
-golang.org/x/sys v0.0.0-20190419153524-e8e3143a4f4a/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
 golang.org/x/sys v0.0.0-20190422165155-953cdadca894/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
 golang.org/x/sys v0.0.0-20190502145724-3ef323f4f1fd/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
 golang.org/x/sys v0.0.0-20190507160741-ecd444e8653b/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
-golang.org/x/sys v0.0.0-20190531175056-4c3a928424d2/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
 golang.org/x/sys v0.0.0-20190606165138-5da285871e9c/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
 golang.org/x/sys v0.0.0-20190616124812-15dcb6c0061f/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
 golang.org/x/sys v0.0.0-20190624142023-c5567b49c5d0/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
@@ -685,13 +630,9 @@ golang.org/x/tools v0.0.0-20190311212946-11955173bddd/go.mod h1:LCzVGOaR6xXOjkQ3
 golang.org/x/tools v0.0.0-20190312151545-0bb0c0a6e846/go.mod h1:LCzVGOaR6xXOjkQ3onu1FJEFr0SW1gC7cKk1uF8kGRs=
 golang.org/x/tools v0.0.0-20190312170243-e65039ee4138/go.mod h1:LCzVGOaR6xXOjkQ3onu1FJEFr0SW1gC7cKk1uF8kGRs=
 golang.org/x/tools v0.0.0-20190328211700-ab21143f2384/go.mod h1:LCzVGOaR6xXOjkQ3onu1FJEFr0SW1gC7cKk1uF8kGRs=
-golang.org/x/tools v0.0.0-20190329151228-23e29df326fe/go.mod h1:LCzVGOaR6xXOjkQ3onu1FJEFr0SW1gC7cKk1uF8kGRs=
-golang.org/x/tools v0.0.0-20190416151739-9c9e1878f421/go.mod h1:LCzVGOaR6xXOjkQ3onu1FJEFr0SW1gC7cKk1uF8kGRs=
-golang.org/x/tools v0.0.0-20190420181800-aa740d480789/go.mod h1:LCzVGOaR6xXOjkQ3onu1FJEFr0SW1gC7cKk1uF8kGRs=
 golang.org/x/tools v0.0.0-20190425150028-36563e24a262/go.mod h1:RgjU9mgBXZiqYHBnxXauZ1Gv1EHHAz9KjViQ78xBX0Q=
 golang.org/x/tools v0.0.0-20190506145303-2d16b83fe98c/go.mod h1:RgjU9mgBXZiqYHBnxXauZ1Gv1EHHAz9KjViQ78xBX0Q=
 golang.org/x/tools v0.0.0-20190524140312-2c0ae7006135/go.mod h1:RgjU9mgBXZiqYHBnxXauZ1Gv1EHHAz9KjViQ78xBX0Q=
-golang.org/x/tools v0.0.0-20190531172133-b3315ee88b7d/go.mod h1:/rFqwRUd4F7ZHNgwSSTFct+R/Kf4OFW1sUzUTQQTgfc=
 golang.org/x/tools v0.0.0-20190606124116-d0a3d012864b/go.mod h1:/rFqwRUd4F7ZHNgwSSTFct+R/Kf4OFW1sUzUTQQTgfc=
 golang.org/x/tools v0.0.0-20190614205625-5aca471b1d59/go.mod h1:/rFqwRUd4F7ZHNgwSSTFct+R/Kf4OFW1sUzUTQQTgfc=
 golang.org/x/tools v0.0.0-20190617190820-da514acc4774/go.mod h1:/rFqwRUd4F7ZHNgwSSTFct+R/Kf4OFW1sUzUTQQTgfc=
@@ -805,11 +746,6 @@ gopkg.in/yaml.v2 v2.4.0/go.mod h1:RDklbk79AGWmwhnvt/jBztapEOGDOx6ZbXqjP6csGnQ=
 gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
 gopkg.in/yaml.v3 v3.0.0-20210107192922-496545a6307b h1:h8qDotaEPuJATrMmW04NCwg7v22aHH28wwpauUhK9Oo=
 gopkg.in/yaml.v3 v3.0.0-20210107192922-496545a6307b/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
-gorm.io/driver/sqlite v1.1.4 h1:PDzwYE+sI6De2+mxAneV9Xs11+ZyKV6oxD3wDGkaNvM=
-gorm.io/driver/sqlite v1.1.4/go.mod h1:mJCeTFr7+crvS+TRnWc5Z3UvwxUN1BGBLMrf5LA9DYw=
-gorm.io/gorm v1.20.7/go.mod h1:0HFTzE/SqkGTzK6TlDPPQbAYCluiVvhzoA1+aVyzenw=
-gorm.io/gorm v1.21.8 h1:2CEwZSzogdhsKPlJ9OvBKTdlWIpELXb6HbfLfMNhSYI=
-gorm.io/gorm v1.21.8/go.mod h1:F+OptMscr0P2F2qU97WT1WimdH9GaQPoDW7AYd5i2Y0=
 gotest.tools/v3 v3.0.2/go.mod h1:3SzNCllyD9/Y+b5r9JIKQ474KzkZyqLqEfYqMsX94Bk=
 gotest.tools/v3 v3.0.3/go.mod h1:Z7Lb0S5l+klDB31fvDQX8ss/FlKDxtlFlw3Oa8Ymbl8=
 honnef.co/go/tools v0.0.0-20190102054323-c2f93a96b099/go.mod h1:rf3lG4BRIbNafJWhAfAdb/ePZxsR/4RtNHQocxwk9r4=

agent/main.go

@@ -6,13 +6,10 @@ import (
 	"errors"
 	"flag"
 	"fmt"
-	"github.com/up9inc/mizu/shared/kubernetes"
 	"io/ioutil"
-	v1 "k8s.io/api/core/v1"
 	"mizuserver/pkg/api"
 	"mizuserver/pkg/config"
 	"mizuserver/pkg/controllers"
-	"mizuserver/pkg/database"
 	"mizuserver/pkg/models"
 	"mizuserver/pkg/providers"
 	"mizuserver/pkg/routes"
@@ -20,6 +17,7 @@ import (
 	"mizuserver/pkg/utils"
 	"net/http"
 	"os"
+	"os/exec"
 	"os/signal"
 	"path"
 	"path/filepath"
@@ -28,10 +26,15 @@ import (
 	"syscall"
 	"time"
 
+	"github.com/up9inc/mizu/shared/kubernetes"
+	v1 "k8s.io/api/core/v1"
+
+	"github.com/antelman107/net-wait-go/wait"
 	"github.com/gin-contrib/static"
 	"github.com/gin-gonic/gin"
 	"github.com/gorilla/websocket"
 	"github.com/op/go-logging"
+	basenine "github.com/up9inc/basenine/client/go"
 	"github.com/up9inc/mizu/shared"
 	"github.com/up9inc/mizu/shared/logger"
 	"github.com/up9inc/mizu/tap"
@@ -49,10 +52,12 @@ var harsDir = flag.String("hars-dir", "", "Directory to read hars from")
 var extensions []*tapApi.Extension             // global
 var extensionsMap map[string]*tapApi.Extension // global
 
+var startTime int64
+
 const (
-	socketConnectionRetries = 10
+	socketConnectionRetries    = 10
 	socketConnectionRetryDelay = time.Second * 2
-	socketHandshakeTimeout = time.Second * 2
+	socketHandshakeTimeout     = time.Second * 2
 )
 
 func main() {
@@ -109,7 +114,8 @@ func main() {
 
 		go pipeTapChannelToSocket(socketConnection, filteredOutputItemsChannel)
 	} else if *apiServerMode {
-		database.InitDataBase(config.Config.AgentDatabasePath)
+		startBasenineServer(shared.BasenineHost, shared.BaseninePort)
+		startTime = time.Now().UnixNano() / int64(time.Millisecond)
 		api.StartResolving(*namespace)
 
 		outputItemsChannel := make(chan *tapApi.OutputChannelItem)
@@ -142,6 +148,53 @@ func main() {
 	logger.Log.Info("Exiting")
 }
 
+func startBasenineServer(host string, port string) {
+	cmd := exec.Command("basenine", "-addr", host, "-port", port, "-persistent")
+	cmd.Dir = config.Config.AgentDatabasePath
+	cmd.Stdout = os.Stdout
+	cmd.Stderr = os.Stderr
+	err := cmd.Start()
+	if err != nil {
+		logger.Log.Panicf("Failed starting Basenine: %v", err)
+	}
+
+	if !wait.New(
+		wait.WithProto("tcp"),
+		wait.WithWait(200*time.Millisecond),
+		wait.WithBreak(50*time.Millisecond),
+		wait.WithDeadline(5*time.Second),
+		wait.WithDebug(true),
+	).Do([]string{fmt.Sprintf("%s:%s", host, port)}) {
+		logger.Log.Panicf("Basenine is not available: %v", err)
+	}
+
+	// Make a channel to gracefully exit Basenine.
+	channel := make(chan os.Signal)
+	signal.Notify(channel, os.Interrupt, syscall.SIGTERM)
+
+	// Handle the channel.
+	go func() {
+		<-channel
+		cmd.Process.Signal(syscall.SIGTERM)
+	}()
+
+	// Limit the database size to default 200MB
+	err = basenine.Limit(host, port, config.Config.MaxDBSizeBytes)
+	if err != nil {
+		logger.Log.Panicf("Error while limiting database size: %v", err)
+	}
+
+	for _, extension := range extensions {
+		macros := extension.Dissector.Macros()
+		for macro, expanded := range macros {
+			err = basenine.Macro(host, port, macro, expanded)
+			if err != nil {
+				logger.Log.Panicf("Error while adding a macro: %v", err)
+			}
+		}
+	}
+}
+
 func loadExtensions() {
 	dir, _ := filepath.Abs(filepath.Dir(os.Args[0]))
 	extensionsDir := path.Join(dir, "./extensions/")
@@ -200,7 +253,8 @@ func hostApi(socketHarOutputChannel chan<- *tapApi.OutputChannelItem) {
 	app.Use(static.ServeRoot("/", "./site"))
 	app.Use(CORSMiddleware()) // This has to be called after the static middleware, does not work if its called before
 
-	api.WebSocketRoutes(app, &eventHandlers)
+	api.WebSocketRoutes(app, &eventHandlers, startTime)
+	routes.QueryRoutes(app)
 	routes.EntriesRoutes(app)
 	routes.MetadataRoutes(app)
 	routes.StatusRoutes(app)
@@ -361,7 +415,7 @@ func dialSocketWithRetry(socketAddress string, retryAmount int, retryDelay time.
 		socketConnection, _, err := dialer.Dial(socketAddress, nil)
 		if err != nil {
 			if i < retryAmount {
-				logger.Log.Infof("socket connection to %s failed: %v, retrying %d out of %d in %d seconds...", socketAddress, err, i, retryAmount, retryDelay / time.Second)
+				logger.Log.Infof("socket connection to %s failed: %v, retrying %d out of %d in %d seconds...", socketAddress, err, i, retryAmount, retryDelay/time.Second)
 				time.Sleep(retryDelay)
 			}
 		} else {
@@ -371,8 +425,7 @@ func dialSocketWithRetry(socketAddress string, retryAmount int, retryDelay time.
 	return nil, lastErr
 }
 
-
+func startMizuTapperSyncer(ctx context.Context) (*kubernetes.MizuTapperSyncer, error) {
-func startMizuTapperSyncer(ctx context.Context) (*kubernetes.MizuTapperSyncer, error){
 	provider, err := kubernetes.NewProviderInCluster()
 	if err != nil {
 		return nil, err

agent/pkg/api/main.go

@@ -5,7 +5,6 @@ import (
 	"context"
 	"encoding/json"
 	"fmt"
-	"mizuserver/pkg/database"
 	"mizuserver/pkg/holder"
 	"mizuserver/pkg/providers"
 	"os"
@@ -14,15 +13,16 @@ import (
 	"strings"
 	"time"
 
-	"go.mongodb.org/mongo-driver/bson/primitive"
-
 	"github.com/google/martian/har"
+	"github.com/up9inc/mizu/shared"
 	"github.com/up9inc/mizu/shared/logger"
 	tapApi "github.com/up9inc/mizu/tap/api"
 
 	"mizuserver/pkg/models"
 	"mizuserver/pkg/resolver"
 	"mizuserver/pkg/utils"
+
+	basenine "github.com/up9inc/basenine/client/go"
 )
 
 var k8sResolver *resolver.Resolver
@@ -99,6 +99,12 @@ func startReadingChannel(outputItems <-chan *tapApi.OutputChannelItem, extension
 		panic("Channel of captured messages is nil")
 	}
 
+	connection, err := basenine.NewConnection(shared.BasenineHost, shared.BaseninePort)
+	if err != nil {
+		panic(err)
+	}
+	connection.InsertMode()
+
 	disableOASValidation := false
 	ctx := context.Background()
 	doc, contractContent, router, err := loadOAS(ctx)
@@ -112,13 +118,13 @@ func startReadingChannel(outputItems <-chan *tapApi.OutputChannelItem, extension
 
 		extension := extensionsMap[item.Protocol.Name]
 		resolvedSource, resolvedDestionation := resolveIP(item.ConnectionInfo)
-		mizuEntry := extension.Dissector.Analyze(item, primitive.NewObjectID().Hex(), resolvedSource, resolvedDestionation)
+		mizuEntry := extension.Dissector.Analyze(item, resolvedSource, resolvedDestionation)
 		baseEntry := extension.Dissector.Summarize(mizuEntry)
-		mizuEntry.EstimatedSizeBytes = getEstimatedEntrySizeBytes(mizuEntry)
+		mizuEntry.Base = baseEntry
 		if extension.Protocol.Name == "http" {
 			if !disableOASValidation {
 				var httpPair tapApi.HTTPRequestResponsePair
-				json.Unmarshal([]byte(mizuEntry.Entry), &httpPair)
+				json.Unmarshal([]byte(mizuEntry.HTTPPair), &httpPair)
 
 				contract := handleOAS(ctx, doc, router, httpPair.Request.Payload.RawRequest, httpPair.Response.Payload.RawResponse, contractContent)
 				baseEntry.ContractStatus = contract.Status
@@ -128,18 +134,18 @@ func startReadingChannel(outputItems <-chan *tapApi.OutputChannelItem, extension
 				mizuEntry.ContractContent = contract.Content
 			}
 
-			var pair tapApi.RequestResponsePair
+			harEntry, err := utils.NewEntry(mizuEntry.Request, mizuEntry.Response, mizuEntry.StartTime, mizuEntry.ElapsedTime)
-			json.Unmarshal([]byte(mizuEntry.Entry), &pair)
-			harEntry, err := utils.NewEntry(&pair)
 			if err == nil {
 				rules, _, _ := models.RunValidationRulesState(*harEntry, mizuEntry.Service)
 				baseEntry.Rules = rules
 			}
 		}
-		database.CreateEntry(mizuEntry)
 
-		baseEntryBytes, _ := models.CreateBaseEntryWebSocketMessage(baseEntry)
+		data, err := json.Marshal(mizuEntry)
-		BroadcastToBrowserClients(baseEntryBytes)
+		if err != nil {
+			panic(err)
+		}
+		connection.SendText(string(data))
 	}
 }
 
@@ -171,21 +177,3 @@ func CheckIsServiceIP(address string) bool {
 	}
 	return k8sResolver.CheckIsServiceIP(address)
 }
-
-// gives a rough estimate of the size this will take up in the db, good enough for maintaining db size limit accurately
-func getEstimatedEntrySizeBytes(mizuEntry *tapApi.MizuEntry) int {
-	sizeBytes := len(mizuEntry.Entry)
-	sizeBytes += len(mizuEntry.EntryId)
-	sizeBytes += len(mizuEntry.Service)
-	sizeBytes += len(mizuEntry.Url)
-	sizeBytes += len(mizuEntry.Method)
-	sizeBytes += len(mizuEntry.RequestSenderIp)
-	sizeBytes += len(mizuEntry.ResolvedDestination)
-	sizeBytes += len(mizuEntry.ResolvedSource)
-	sizeBytes += 8 // Status bytes (sqlite integer is always 8 bytes)
-	sizeBytes += 8 // Timestamp bytes
-	sizeBytes += 8 // SizeBytes bytes
-	sizeBytes += 1 // IsOutgoing bytes
-
-	return sizeBytes
-}

agent/pkg/api/socket_routes.go

@@ -1,13 +1,18 @@
 package api
 
 import (
+	"encoding/json"
 	"errors"
+	"fmt"
+	"mizuserver/pkg/models"
 	"net/http"
 	"sync"
 	"time"
 
 	"github.com/gin-gonic/gin"
 	"github.com/gorilla/websocket"
+	basenine "github.com/up9inc/basenine/client/go"
+	"github.com/up9inc/mizu/shared"
 	"github.com/up9inc/mizu/shared/debounce"
 	"github.com/up9inc/mizu/shared/logger"
 )
@@ -39,17 +44,17 @@ func init() {
 	connectedWebsockets = make(map[int]*SocketConnection, 0)
 }
 
-func WebSocketRoutes(app *gin.Engine, eventHandlers EventHandlers) {
+func WebSocketRoutes(app *gin.Engine, eventHandlers EventHandlers, startTime int64) {
 	app.GET("/ws", func(c *gin.Context) {
-		websocketHandler(c.Writer, c.Request, eventHandlers, false)
+		websocketHandler(c.Writer, c.Request, eventHandlers, false, startTime)
 	})
 	app.GET("/wsTapper", func(c *gin.Context) {
-		websocketHandler(c.Writer, c.Request, eventHandlers, true)
+		websocketHandler(c.Writer, c.Request, eventHandlers, true, startTime)
 	})
 }
 
-func websocketHandler(w http.ResponseWriter, r *http.Request, eventHandlers EventHandlers, isTapper bool) {
+func websocketHandler(w http.ResponseWriter, r *http.Request, eventHandlers EventHandlers, isTapper bool, startTime int64) {
-	conn, err := websocketUpgrader.Upgrade(w, r, nil)
+	ws, err := websocketUpgrader.Upgrade(w, r, nil)
 	if err != nil {
 		logger.Log.Errorf("Failed to set websocket upgrade: %v", err)
 		return
@@ -59,23 +64,103 @@ func websocketHandler(w http.ResponseWriter, r *http.Request, eventHandlers Even
 
 	connectedWebsocketIdCounter++
 	socketId := connectedWebsocketIdCounter
-	connectedWebsockets[socketId] = &SocketConnection{connection: conn, lock: &sync.Mutex{}, eventHandlers: eventHandlers, isTapper: isTapper}
+	connectedWebsockets[socketId] = &SocketConnection{connection: ws, lock: &sync.Mutex{}, eventHandlers: eventHandlers, isTapper: isTapper}
 
 	websocketIdsLock.Unlock()
 
+	var connection *basenine.Connection
+	var isQuerySet bool
+
+	// `!isTapper` means it's a connection from the web UI
+	if !isTapper {
+		connection, err = basenine.NewConnection(shared.BasenineHost, shared.BaseninePort)
+		if err != nil {
+			panic(err)
+		}
+	}
+
+	data := make(chan []byte)
+	meta := make(chan []byte)
+
 	defer func() {
+		data <- []byte(basenine.CloseChannel)
+		meta <- []byte(basenine.CloseChannel)
+		connection.Close()
 		socketCleanup(socketId, connectedWebsockets[socketId])
 	}()
 
 	eventHandlers.WebSocketConnect(socketId, isTapper)
 
+	startTimeBytes, _ := models.CreateWebsocketStartTimeMessage(startTime)
+	BroadcastToBrowserClients(startTimeBytes)
+
 	for {
-		_, msg, err := conn.ReadMessage()
+		_, msg, err := ws.ReadMessage()
 		if err != nil {
 			logger.Log.Errorf("Error reading message, socket id: %d, error: %v", socketId, err)
 			break
 		}
-		eventHandlers.WebSocketMessage(socketId, msg)
+
+		if !isTapper && !isQuerySet {
+			query := string(msg)
+			err = basenine.Validate(shared.BasenineHost, shared.BaseninePort, query)
+			if err != nil {
+				toastBytes, _ := models.CreateWebsocketToastMessage(&models.ToastMessage{
+					Type:      "error",
+					AutoClose: 5000,
+					Text:      fmt.Sprintf("Syntax error: %s", err.Error()),
+				})
+				BroadcastToBrowserClients(toastBytes)
+				break
+			}
+
+			isQuerySet = true
+
+			handleDataChannel := func(c *basenine.Connection, data chan []byte) {
+				for {
+					bytes := <-data
+
+					if string(bytes) == basenine.CloseChannel {
+						return
+					}
+
+					var dataMap map[string]interface{}
+					err = json.Unmarshal(bytes, &dataMap)
+
+					base := dataMap["base"].(map[string]interface{})
+					base["id"] = uint(dataMap["id"].(float64))
+
+					baseEntryBytes, _ := models.CreateBaseEntryWebSocketMessage(base)
+					BroadcastToBrowserClients(baseEntryBytes)
+				}
+			}
+
+			handleMetaChannel := func(c *basenine.Connection, meta chan []byte) {
+				for {
+					bytes := <-meta
+
+					if string(bytes) == basenine.CloseChannel {
+						return
+					}
+
+					var metadata *basenine.Metadata
+					err = json.Unmarshal(bytes, &metadata)
+					if err != nil {
+						logger.Log.Debugf("Error recieving metadata: %v\n", err.Error())
+					}
+
+					metadataBytes, _ := models.CreateWebsocketQueryMetadataMessage(metadata)
+					BroadcastToBrowserClients(metadataBytes)
+				}
+			}
+
+			go handleDataChannel(connection, data)
+			go handleMetaChannel(connection, meta)
+
+			connection.Query(query, data, meta)
+		} else {
+			eventHandlers.WebSocketMessage(socketId, msg)
+		}
 	}
 }
 

agent/pkg/controllers/entries_controller.go

@@ -2,14 +2,17 @@ package controllers
 
 import (
 	"encoding/json"
-	"fmt"
-	"github.com/gin-gonic/gin"
-	tapApi "github.com/up9inc/mizu/tap/api"
-	"mizuserver/pkg/database"
 	"mizuserver/pkg/models"
 	"mizuserver/pkg/utils"
-	"mizuserver/pkg/validation"
 	"net/http"
+	"strconv"
+
+	"github.com/gin-gonic/gin"
+
+	basenine "github.com/up9inc/basenine/client/go"
+	"github.com/up9inc/mizu/shared"
+	"github.com/up9inc/mizu/shared/logger"
+	tapApi "github.com/up9inc/mizu/tap/api"
 )
 
 var extensionsMap map[string]*tapApi.Extension // global
@@ -18,68 +21,36 @@ func InitExtensionsMap(ref map[string]*tapApi.Extension) {
 	extensionsMap = ref
 }
 
-func GetEntries(c *gin.Context) {
+func Error(c *gin.Context, err error) bool {
-	entriesFilter := &models.EntriesFilter{}
-
-	if err := c.BindQuery(entriesFilter); err != nil {
-		c.JSON(http.StatusBadRequest, err)
-	}
-	err := validation.Validate(entriesFilter)
 	if err != nil {
-		c.JSON(http.StatusBadRequest, err)
+		logger.Log.Errorf("Error getting entry: %v", err)
+		c.Error(err)
+		c.AbortWithStatusJSON(http.StatusInternalServerError, gin.H{"error": true, "msg": err.Error()})
+		return true // signal that there was an error and the caller should return
 	}
-
+	return false // no error, can continue
-	order := database.OperatorToOrderMapping[entriesFilter.Operator]
-	operatorSymbol := database.OperatorToSymbolMapping[entriesFilter.Operator]
-	var entries []tapApi.MizuEntry
-	database.GetEntriesTable().
-		Order(fmt.Sprintf("timestamp %s", order)).
-		Where(fmt.Sprintf("timestamp %s %v", operatorSymbol, entriesFilter.Timestamp)).
-		Limit(entriesFilter.Limit).
-		Find(&entries)
-
-	if len(entries) > 0 && order == database.OrderDesc {
-		// the entries always order from oldest to newest - we should reverse
-		utils.ReverseSlice(entries)
-	}
-
-	baseEntries := make([]tapApi.BaseEntryDetails, 0)
-	for _, entry := range entries {
-		baseEntryDetails := tapApi.BaseEntryDetails{}
-		if err := models.GetEntry(&entry, &baseEntryDetails); err != nil {
-			continue
-		}
-
-		var pair tapApi.RequestResponsePair
-		json.Unmarshal([]byte(entry.Entry), &pair)
-		harEntry, err := utils.NewEntry(&pair)
-		if err == nil {
-			rules, _, _ := models.RunValidationRulesState(*harEntry, entry.Service)
-			baseEntryDetails.Rules = rules
-		}
-
-		baseEntries = append(baseEntries, baseEntryDetails)
-	}
-
-	c.JSON(http.StatusOK, baseEntries)
 }
 
 func GetEntry(c *gin.Context) {
-	var entryData tapApi.MizuEntry
+	id, _ := strconv.Atoi(c.Param("id"))
-	database.GetEntriesTable().
+	var entry tapApi.MizuEntry
-		Where(map[string]string{"entryId": c.Param("entryId")}).
+	bytes, err := basenine.Single(shared.BasenineHost, shared.BaseninePort, id)
-		First(&entryData)
+	if Error(c, err) {
+		return // exit
+	}
+	err = json.Unmarshal(bytes, &entry)
+	if Error(c, err) {
+		return // exit
+	}
 
-	extension := extensionsMap[entryData.ProtocolName]
+	extension := extensionsMap[entry.Protocol.Name]
-	protocol, representation, bodySize, _ := extension.Dissector.Represent(&entryData)
+	protocol, representation, bodySize, _ := extension.Dissector.Represent(entry.Protocol, entry.Request, entry.Response)
 
 	var rules []map[string]interface{}
 	var isRulesEnabled bool
-	if entryData.ProtocolName == "http" {
+	if entry.Protocol.Name == "http" {
-		var pair tapApi.RequestResponsePair
+		harEntry, _ := utils.NewEntry(entry.Request, entry.Response, entry.StartTime, entry.ElapsedTime)
-		json.Unmarshal([]byte(entryData.Entry), &pair)
+		_, rulesMatched, _isRulesEnabled := models.RunValidationRulesState(*harEntry, entry.Service)
-		harEntry, _ := utils.NewEntry(&pair)
-		_, rulesMatched, _isRulesEnabled := models.RunValidationRulesState(*harEntry, entryData.Service)
 		isRulesEnabled = _isRulesEnabled
 		inrec, _ := json.Marshal(rulesMatched)
 		json.Unmarshal(inrec, &rules)
@@ -89,7 +60,7 @@ func GetEntry(c *gin.Context) {
 		Protocol:       protocol,
 		Representation: string(representation),
 		BodySize:       bodySize,
-		Data:           entryData,
+		Data:           entry,
 		Rules:          rules,
 		IsRulesEnabled: isRulesEnabled,
 	})

agent/pkg/controllers/query_controller.go

@@ -0,0 +1,31 @@
+package controllers
+
+import (
+	"net/http"
+
+	"github.com/gin-gonic/gin"
+	basenine "github.com/up9inc/basenine/client/go"
+	"github.com/up9inc/mizu/shared"
+)
+
+type ValidateResponse struct {
+	Valid   bool   `json:"valid"`
+	Message string `json:"message"`
+}
+
+func PostValidate(c *gin.Context) {
+	query := c.PostForm("query")
+	valid := true
+	message := ""
+
+	err := basenine.Validate(shared.BasenineHost, shared.BaseninePort, query)
+	if err != nil {
+		valid = false
+		message = err.Error()
+	}
+
+	c.JSON(http.StatusOK, ValidateResponse{
+		Valid:   valid,
+		Message: message,
+	})
+}

agent/pkg/database/main.go

@@ -1,78 +0,0 @@
-package database
-
-import (
-	"fmt"
-	"mizuserver/pkg/utils"
-	"time"
-
-	"gorm.io/driver/sqlite"
-	"gorm.io/gorm"
-	"gorm.io/gorm/logger"
-
-	tapApi "github.com/up9inc/mizu/tap/api"
-)
-
-const (
-	OrderDesc  = "desc"
-	OrderAsc   = "asc"
-	LT         = "lt"
-	GT         = "gt"
-	TimeFormat = "2006-01-02 15:04:05.000000000"
-)
-
-var (
-	DB                      *gorm.DB
-	IsDBLocked              = false
-	OperatorToSymbolMapping = map[string]string{
-		LT: "<",
-		GT: ">",
-	}
-	OperatorToOrderMapping = map[string]string{
-		LT: OrderDesc,
-		GT: OrderAsc,
-	}
-)
-
-var DBPath string
-
-func GetEntriesTable() *gorm.DB {
-	return DB.Table("mizu_entries")
-}
-
-func CreateEntry(entry *tapApi.MizuEntry) {
-	if IsDBLocked {
-		return
-	}
-	GetEntriesTable().Create(entry)
-}
-
-func InitDataBase(databasePath string) *gorm.DB {
-	DBPath = databasePath
-	DB, _ = gorm.Open(sqlite.Open(databasePath), &gorm.Config{
-		Logger: &utils.TruncatingLogger{LogLevel: logger.Warn, SlowThreshold: 500 * time.Millisecond},
-	})
-	_ = DB.AutoMigrate(&tapApi.MizuEntry{}) // this will ensure table is created
-	go StartEnforcingDatabaseSize()
-	return DB
-}
-
-func GetEntriesFromDb(timeFrom time.Time, timeTo time.Time, protocolName *string) []tapApi.MizuEntry {
-	order := OrderDesc
-	protocolNameCondition := "1 = 1"
-	if protocolName != nil {
-		protocolNameCondition = fmt.Sprintf("protocolName = '%s'", *protocolName)
-	}
-
-	var entries []tapApi.MizuEntry
-	GetEntriesTable().
-		Where(protocolNameCondition).
-		Where(fmt.Sprintf("created_at BETWEEN '%s' AND '%s'", timeFrom.Format(TimeFormat), timeTo.Format(TimeFormat))).
-		Order(fmt.Sprintf("timestamp %s", order)).
-		Find(&entries)
-
-	if len(entries) > 0 {
-		// the entries always order from oldest to newest so we should revers
-		utils.ReverseSlice(entries)
-	}
-	return entries
-}

agent/pkg/database/size_enforcer.go

@@ -1,102 +0,0 @@
-package database
-
-import (
-	"mizuserver/pkg/config"
-	"os"
-	"time"
-
-	"github.com/fsnotify/fsnotify"
-	"github.com/up9inc/mizu/shared/debounce"
-	"github.com/up9inc/mizu/shared/logger"
-	"github.com/up9inc/mizu/shared/units"
-	tapApi "github.com/up9inc/mizu/tap/api"
-)
-
-const percentageOfMaxSizeBytesToPrune = 15
-
-func StartEnforcingDatabaseSize() {
-	watcher, err := fsnotify.NewWatcher()
-	if err != nil {
-		logger.Log.Fatalf("Error creating filesystem watcher for db size enforcement: %v\n", err)
-		return
-	}
-
-	checkFileSizeDebouncer := debounce.NewDebouncer(5*time.Second, func() {
-		checkFileSize(config.Config.MaxDBSizeBytes)
-	})
-
-	go func() {
-		for {
-			select {
-			case event, ok := <-watcher.Events:
-				if !ok {
-					return // closed channel
-				}
-				if event.Op == fsnotify.Write {
-					checkFileSizeDebouncer.SetOn()
-				}
-			case err, ok := <-watcher.Errors:
-				if !ok {
-					return // closed channel
-				}
-				logger.Log.Errorf("filesystem watcher encountered error:%v", err)
-			}
-		}
-	}()
-
-	err = watcher.Add(DBPath)
-	if err != nil {
-		logger.Log.Fatalf("Error adding %s to filesystem watcher for db size enforcement: %v\n", DBPath, err)
-	}
-}
-
-func checkFileSize(maxSizeBytes int64) {
-	fileStat, err := os.Stat(DBPath)
-	if err != nil {
-		logger.Log.Errorf("Error checking %s file size: %v", DBPath, err)
-	} else {
-		if fileStat.Size() > maxSizeBytes {
-			pruneOldEntries(fileStat.Size())
-		}
-	}
-}
-
-func pruneOldEntries(currentFileSize int64) {
-	// sqlite locks the database while delete or VACUUM are running and sqlite is terrible at handling its own db lock while a lot of inserts are attempted, we prevent a significant bottleneck by handling the db lock ourselves here
-	IsDBLocked = true
-	defer func() { IsDBLocked = false }()
-
-	amountOfBytesToTrim := currentFileSize / (100 / percentageOfMaxSizeBytesToPrune)
-
-	rows, err := GetEntriesTable().Limit(10000).Order("id").Rows()
-	if err != nil {
-		logger.Log.Errorf("Error getting 10000 first db rows: %v", err)
-		return
-	}
-
-	entryIdsToRemove := make([]uint, 0)
-	bytesToBeRemoved := int64(0)
-	for rows.Next() {
-		if bytesToBeRemoved >= amountOfBytesToTrim {
-			break
-		}
-		var entry tapApi.MizuEntry
-		err = DB.ScanRows(rows, &entry)
-		if err != nil {
-			logger.Log.Errorf("Error scanning db row: %v", err)
-			continue
-		}
-
-		entryIdsToRemove = append(entryIdsToRemove, entry.ID)
-		bytesToBeRemoved += int64(entry.EstimatedSizeBytes)
-	}
-
-	if len(entryIdsToRemove) > 0 {
-		GetEntriesTable().Where(entryIdsToRemove).Delete(tapApi.MizuEntry{})
-		// VACUUM causes sqlite to shrink the db file after rows have been deleted, the db file will not shrink without this
-		DB.Exec("VACUUM")
-		logger.Log.Errorf("Removed %d rows and cleared %s", len(entryIdsToRemove), units.BytesToHumanReadable(bytesToBeRemoved))
-	} else {
-		logger.Log.Error("Found no rows to remove when pruning")
-	}
-}

agent/pkg/models/models.go

@@ -2,12 +2,12 @@ package models
 
 import (
 	"encoding/json"
+	"mizuserver/pkg/rules"
 
 	tapApi "github.com/up9inc/mizu/tap/api"
 
-	"mizuserver/pkg/rules"
-
 	"github.com/google/martian/har"
+	basenine "github.com/up9inc/basenine/client/go"
 	"github.com/up9inc/mizu/shared"
 	"github.com/up9inc/mizu/tap"
 )
@@ -16,15 +16,9 @@ func GetEntry(r *tapApi.MizuEntry, v tapApi.DataUnmarshaler) error {
 	return v.UnmarshalData(r)
 }
 
-type EntriesFilter struct {
-	Limit     int    `form:"limit" validate:"required,min=1,max=200"`
-	Operator  string `form:"operator" validate:"required,oneof='lt' 'gt'"`
-	Timestamp int64  `form:"timestamp" validate:"required,min=1"`
-}
-
 type WebSocketEntryMessage struct {
 	*shared.WebSocketMessageMetadata
-	Data *tapApi.BaseEntryDetails `json:"data,omitempty"`
+	Data map[string]interface{} `json:"data,omitempty"`
 }
 
 type WebSocketTappedEntryMessage struct {
@@ -42,7 +36,28 @@ type AuthStatus struct {
 	Model string `json:"model"`
 }
 
-func CreateBaseEntryWebSocketMessage(base *tapApi.BaseEntryDetails) ([]byte, error) {
+type ToastMessage struct {
+	Type      string `json:"type"`
+	AutoClose uint   `json:"autoClose"`
+	Text      string `json:"text"`
+}
+
+type WebSocketToastMessage struct {
+	*shared.WebSocketMessageMetadata
+	Data *ToastMessage `json:"data,omitempty"`
+}
+
+type WebSocketQueryMetadataMessage struct {
+	*shared.WebSocketMessageMetadata
+	Data *basenine.Metadata `json:"data,omitempty"`
+}
+
+type WebSocketStartTimeMessage struct {
+	*shared.WebSocketMessageMetadata
+	Data int64 `json:"data"`
+}
+
+func CreateBaseEntryWebSocketMessage(base map[string]interface{}) ([]byte, error) {
 	message := &WebSocketEntryMessage{
 		WebSocketMessageMetadata: &shared.WebSocketMessageMetadata{
 			MessageType: shared.WebSocketMessageTypeEntry,
@@ -72,6 +87,36 @@ func CreateWebsocketOutboundLinkMessage(base *tap.OutboundLink) ([]byte, error)
 	return json.Marshal(message)
 }
 
+func CreateWebsocketToastMessage(base *ToastMessage) ([]byte, error) {
+	message := &WebSocketToastMessage{
+		WebSocketMessageMetadata: &shared.WebSocketMessageMetadata{
+			MessageType: shared.WebSocketMessageTypeToast,
+		},
+		Data: base,
+	}
+	return json.Marshal(message)
+}
+
+func CreateWebsocketQueryMetadataMessage(base *basenine.Metadata) ([]byte, error) {
+	message := &WebSocketQueryMetadataMessage{
+		WebSocketMessageMetadata: &shared.WebSocketMessageMetadata{
+			MessageType: shared.WebSocketMessageTypeQueryMetadata,
+		},
+		Data: base,
+	}
+	return json.Marshal(message)
+}
+
+func CreateWebsocketStartTimeMessage(base int64) ([]byte, error) {
+	message := &WebSocketStartTimeMessage{
+		WebSocketMessageMetadata: &shared.WebSocketMessageMetadata{
+			MessageType: shared.WebSocketMessageTypeStartTime,
+		},
+		Data: base,
+	}
+	return json.Marshal(message)
+}
+
 // ExtendedHAR is the top level object of a HAR log.
 type ExtendedHAR struct {
 	Log *ExtendedLog `json:"log"`

agent/pkg/providers/stats_provider_test.go

@@ -2,16 +2,10 @@ package providers_test
 
 import (
 	"fmt"
-	"mizuserver/pkg/config"
-	"mizuserver/pkg/database"
 	"mizuserver/pkg/providers"
 	"testing"
 )
 
-func init() {
-	database.InitDataBase(config.DefaultDatabasePath)
-}
-
 func TestNoEntryAddedCount(t *testing.T) {
 	entriesStats := providers.GetGeneralStats()
 

agent/pkg/routes/entries_routes.go

@@ -1,14 +1,14 @@
 package routes
 
 import (
-	"github.com/gin-gonic/gin"
 	"mizuserver/pkg/controllers"
+
+	"github.com/gin-gonic/gin"
 )
 
 // EntriesRoutes defines the group of har entries routes.
 func EntriesRoutes(ginApp *gin.Engine) {
 	routeGroup := ginApp.Group("/entries")
 
-	routeGroup.GET("/", controllers.GetEntries)        // get entries (base/thin entries)
+	routeGroup.GET("/:id", controllers.GetEntry) // get single (full) entry
-	routeGroup.GET("/:entryId", controllers.GetEntry) // get single (full) entry
 }

agent/pkg/routes/query_routes.go

@@ -0,0 +1,13 @@
+package routes
+
+import (
+	"mizuserver/pkg/controllers"
+
+	"github.com/gin-gonic/gin"
+)
+
+func QueryRoutes(ginApp *gin.Engine) {
+	routeGroup := ginApp.Group("/query")
+
+	routeGroup.POST("/validate", controllers.PostValidate)
+}

agent/pkg/sensitiveDataFiltering/consts.go

@@ -1,10 +0,0 @@
-package sensitiveDataFiltering
-
-const maskedFieldPlaceholderValue = "[REDACTED]"
-
-//these values MUST be all lower case and contain no `-` or `_` characters
-var personallyIdentifiableDataFields = []string{"token", "authorization", "authentication", "cookie", "userid", "password",
-	"username", "user", "key", "passcode", "pass", "auth", "authtoken", "jwt",
-	"bearer", "clientid", "clientsecret", "redirecturi", "phonenumber",
-	"zip", "zipcode", "address", "country", "firstname", "lastname",
-	"middlename", "fname", "lname", "birthdate"}

agent/pkg/up9/main.go

@@ -7,15 +7,16 @@ import (
 	"encoding/json"
 	"fmt"
 	"io/ioutil"
-	"mizuserver/pkg/database"
 	"mizuserver/pkg/utils"
 	"net/http"
 	"net/url"
 	"regexp"
 	"strings"
+	"sync"
 	"time"
 
 	"github.com/google/martian/har"
+	basenine "github.com/up9inc/basenine/client/go"
 	"github.com/up9inc/mizu/shared"
 	"github.com/up9inc/mizu/shared/logger"
 	tapApi "github.com/up9inc/mizu/tap/api"
@@ -23,6 +24,7 @@ import (
 
 const (
 	AnalyzeCheckSleepTime = 5 * time.Second
+	SentCountLogInterval  = 100
 )
 
 type GuestToken struct {
@@ -204,44 +206,62 @@ func syncEntriesImpl(token string, model string, envPrefix string, uploadInterva
 	analyzeInformation.AnalyzeDestination = envPrefix
 	analyzeInformation.SentCount = 0
 
-	sleepTime := time.Second * time.Duration(uploadIntervalSec)
+	// "http or grpc" filter indicates that we're only interested in HTTP and gRPC entries
+	query := "http or grpc"
 
-	var timeFrom time.Time
+	logger.Log.Infof("Getting entries from the database\n")
-	protocolFilter := "http"
 
-	for {
+	var connection *basenine.Connection
-		timeTo := time.Now()
+	var err error
-		logger.Log.Infof("Getting entries from %v, to %v\n", timeFrom.Format(time.RFC3339Nano), timeTo.Format(time.RFC3339Nano))
+	connection, err = basenine.NewConnection(shared.BasenineHost, shared.BaseninePort)
-		entriesArray := database.GetEntriesFromDb(timeFrom, timeTo, &protocolFilter)
+	if err != nil {
+		panic(err)
+	}
 
-		if len(entriesArray) > 0 {
+	data := make(chan []byte)
-			result := make([]har.Entry, 0)
+	meta := make(chan []byte)
-			for _, data := range entriesArray {
-				var pair tapApi.RequestResponsePair
-				if err := json.Unmarshal([]byte(data.Entry), &pair); err != nil {
-					continue
-				}
-				harEntry, err := utils.NewEntry(&pair)
-				if err != nil {
-					continue
-				}
-				if data.ResolvedSource != "" {
-					harEntry.Request.Headers = append(harEntry.Request.Headers, har.Header{Name: "x-mizu-source", Value: data.ResolvedSource})
-				}
-				if data.ResolvedDestination != "" {
-					harEntry.Request.Headers = append(harEntry.Request.Headers, har.Header{Name: "x-mizu-destination", Value: data.ResolvedDestination})
-					harEntry.Request.URL = utils.SetHostname(harEntry.Request.URL, data.ResolvedDestination)
-				}
 
-				// go's default marshal behavior is to encode []byte fields to base64, python's default unmarshal behavior is to not decode []byte fields from base64
+	defer func() {
-				if harEntry.Response.Content.Text, err = base64.StdEncoding.DecodeString(string(harEntry.Response.Content.Text)); err != nil {
+		data <- []byte(basenine.CloseChannel)
-					continue
+		meta <- []byte(basenine.CloseChannel)
-				}
+		connection.Close()
+	}()
 
-				result = append(result, *harEntry)
+	handleDataChannel := func(wg *sync.WaitGroup, connection *basenine.Connection, data chan []byte) {
+		defer wg.Done()
+		for {
+			dataBytes := <-data
+
+			if string(dataBytes) == basenine.CloseChannel {
+				return
 			}
 
-			logger.Log.Infof("About to upload %v entries\n", len(result))
+			var dataMap map[string]interface{}
+			err = json.Unmarshal(dataBytes, &dataMap)
+
+			result := make([]har.Entry, 0)
+			var entry tapApi.MizuEntry
+			if err := json.Unmarshal([]byte(dataBytes), &entry); err != nil {
+				continue
+			}
+			harEntry, err := utils.NewEntry(entry.Request, entry.Response, entry.StartTime, entry.ElapsedTime)
+			if err != nil {
+				continue
+			}
+			if entry.ResolvedSource != "" {
+				harEntry.Request.Headers = append(harEntry.Request.Headers, har.Header{Name: "x-mizu-source", Value: entry.ResolvedSource})
+			}
+			if entry.ResolvedDestination != "" {
+				harEntry.Request.Headers = append(harEntry.Request.Headers, har.Header{Name: "x-mizu-destination", Value: entry.ResolvedDestination})
+				harEntry.Request.URL = utils.SetHostname(harEntry.Request.URL, entry.ResolvedDestination)
+			}
+
+			// go's default marshal behavior is to encode []byte fields to base64, python's default unmarshal behavior is to not decode []byte fields from base64
+			if harEntry.Response.Content.Text, err = base64.StdEncoding.DecodeString(string(harEntry.Response.Content.Text)); err != nil {
+				continue
+			}
+
+			result = append(result, *harEntry)
 
 			body, jMarshalErr := json.Marshal(result)
 			if jMarshalErr != nil {
@@ -273,18 +293,33 @@ func syncEntriesImpl(token string, model string, envPrefix string, uploadInterva
 				logger.Log.Info("Stopping sync entries")
 				logger.Log.Fatal(postErr)
 			}
-			analyzeInformation.SentCount += len(entriesArray)
+			analyzeInformation.SentCount += 1
-			logger.Log.Infof("Finish uploading %v entries to %s\n", len(entriesArray), GetTrafficDumpUrl(envPrefix, model))
 
-			logger.Log.Infof("Uploaded %v entries until now", analyzeInformation.SentCount)
+			if analyzeInformation.SentCount%SentCountLogInterval == 0 {
-		} else {
+				logger.Log.Infof("Uploaded %v entries until now", analyzeInformation.SentCount)
-			logger.Log.Infof("Nothing to upload")
+			}
 		}
-
-		logger.Log.Infof("Sleeping for %v...\n", sleepTime)
-		time.Sleep(sleepTime)
-		timeFrom = timeTo
 	}
+
+	handleMetaChannel := func(wg *sync.WaitGroup, connection *basenine.Connection, meta chan []byte) {
+		defer wg.Done()
+		for {
+			metaBytes := <-meta
+
+			if string(metaBytes) == basenine.CloseChannel {
+				return
+			}
+		}
+	}
+
+	var wg sync.WaitGroup
+	go handleDataChannel(&wg, connection, data)
+	go handleMetaChannel(&wg, connection, meta)
+	wg.Add(2)
+
+	connection.Query(query, data, meta)
+
+	wg.Wait()
 }
 
 func UpdateAnalyzeStatus(callback func(data []byte)) {

agent/pkg/utils/har.go

@@ -10,7 +10,6 @@ import (
 
 	"github.com/google/martian/har"
 	"github.com/up9inc/mizu/shared/logger"
-	"github.com/up9inc/mizu/tap/api"
 )
 
 // Keep it because we might want cookies in the future
@@ -120,13 +119,11 @@ func BuildPostParams(rawParams []interface{}) []har.Param {
 	return params
 }
 
-func NewRequest(request *api.GenericMessage) (harRequest *har.Request, err error) {
+func NewRequest(request map[string]interface{}) (harRequest *har.Request, err error) {
-	reqDetails := request.Payload.(map[string]interface{})["details"].(map[string]interface{})
+	headers, host, scheme, authority, path, _ := BuildHeaders(request["_headers"].([]interface{}))
+	cookies := make([]har.Cookie, 0) // BuildCookies(request["_cookies"].([]interface{}))
 
-	headers, host, scheme, authority, path, _ := BuildHeaders(reqDetails["headers"].([]interface{}))
+	postData, _ := request["postData"].(map[string]interface{})
-	cookies := make([]har.Cookie, 0) // BuildCookies(reqDetails["cookies"].([]interface{}))
-
-	postData, _ := reqDetails["postData"].(map[string]interface{})
 	mimeType, _ := postData["mimeType"]
 	if mimeType == nil || len(mimeType.(string)) == 0 {
 		mimeType = "text/html"
@@ -138,7 +135,7 @@ func NewRequest(request *api.GenericMessage) (harRequest *har.Request, err error
 	}
 
 	queryString := make([]har.QueryString, 0)
-	for _, _qs := range reqDetails["queryString"].([]interface{}) {
+	for _, _qs := range request["_queryString"].([]interface{}) {
 		qs := _qs.(map[string]interface{})
 		queryString = append(queryString, har.QueryString{
 			Name:  qs["name"].(string),
@@ -146,7 +143,7 @@ func NewRequest(request *api.GenericMessage) (harRequest *har.Request, err error
 		})
 	}
 
-	url := fmt.Sprintf("http://%s%s", host, reqDetails["url"].(string))
+	url := fmt.Sprintf("http://%s%s", host, request["url"].(string))
 	if strings.HasPrefix(mimeType.(string), "application/grpc") {
 		url = fmt.Sprintf("%s://%s%s", scheme, authority, path)
 	}
@@ -157,9 +154,9 @@ func NewRequest(request *api.GenericMessage) (harRequest *har.Request, err error
 	}
 
 	harRequest = &har.Request{
-		Method:      reqDetails["method"].(string),
+		Method:      request["method"].(string),
 		URL:         url,
-		HTTPVersion: reqDetails["httpVersion"].(string),
+		HTTPVersion: request["httpVersion"].(string),
 		HeadersSize: -1,
 		BodySize:    int64(bytes.NewBufferString(postDataText).Len()),
 		QueryString: queryString,
@@ -175,13 +172,11 @@ func NewRequest(request *api.GenericMessage) (harRequest *har.Request, err error
 	return
 }
 
-func NewResponse(response *api.GenericMessage) (harResponse *har.Response, err error) {
+func NewResponse(response map[string]interface{}) (harResponse *har.Response, err error) {
-	resDetails := response.Payload.(map[string]interface{})["details"].(map[string]interface{})
+	headers, _, _, _, _, _status := BuildHeaders(response["_headers"].([]interface{}))
+	cookies := make([]har.Cookie, 0) // BuildCookies(response["_cookies"].([]interface{}))
 
-	headers, _, _, _, _, _status := BuildHeaders(resDetails["headers"].([]interface{}))
+	content, _ := response["content"].(map[string]interface{})
-	cookies := make([]har.Cookie, 0) // BuildCookies(resDetails["cookies"].([]interface{}))
-
-	content, _ := resDetails["content"].(map[string]interface{})
 	mimeType, _ := content["mimeType"]
 	if mimeType == nil || len(mimeType.(string)) == 0 {
 		mimeType = "text/html"
@@ -200,9 +195,11 @@ func NewResponse(response *api.GenericMessage) (harResponse *har.Response, err e
 		Size:     int64(len(bodyText)),
 	}
 
-	status := int(resDetails["status"].(float64))
+	status := int(response["status"].(float64))
 	if strings.HasPrefix(mimeType.(string), "application/grpc") {
-		status, err = strconv.Atoi(_status)
+		if _status != "" {
+			status, err = strconv.Atoi(_status)
+		}
 		if err != nil {
 			logger.Log.Errorf("Failed converting status to int %s (%v,%+v)", err, err, err)
 			return nil, errors.New("failed converting response status to int for HAR")
@@ -210,9 +207,9 @@ func NewResponse(response *api.GenericMessage) (harResponse *har.Response, err e
 	}
 
 	harResponse = &har.Response{
-		HTTPVersion: resDetails["httpVersion"].(string),
+		HTTPVersion: response["httpVersion"].(string),
 		Status:      status,
-		StatusText:  resDetails["statusText"].(string),
+		StatusText:  response["statusText"].(string),
 		HeadersSize: -1,
 		BodySize:    int64(bytes.NewBufferString(bodyText).Len()),
 		Headers:     headers,
@@ -222,34 +219,33 @@ func NewResponse(response *api.GenericMessage) (harResponse *har.Response, err e
 	return
 }
 
-func NewEntry(pair *api.RequestResponsePair) (*har.Entry, error) {
+func NewEntry(request map[string]interface{}, response map[string]interface{}, startTime time.Time, elapsedTime int64) (*har.Entry, error) {
-	harRequest, err := NewRequest(&pair.Request)
+	harRequest, err := NewRequest(request)
 	if err != nil {
 		logger.Log.Errorf("Failed converting request to HAR %s (%v,%+v)", err, err, err)
 		return nil, errors.New("failed converting request to HAR")
 	}
 
-	harResponse, err := NewResponse(&pair.Response)
+	harResponse, err := NewResponse(response)
 	if err != nil {
 		logger.Log.Errorf("Failed converting response to HAR %s (%v,%+v)", err, err, err)
 		return nil, errors.New("failed converting response to HAR")
 	}
 
-	totalTime := pair.Response.CaptureTime.Sub(pair.Request.CaptureTime).Round(time.Millisecond).Milliseconds()
+	if elapsedTime < 1 {
-	if totalTime < 1 {
+		elapsedTime = 1
-		totalTime = 1
 	}
 
 	harEntry := har.Entry{
-		StartedDateTime: pair.Request.CaptureTime,
+		StartedDateTime: startTime,
-		Time:            totalTime,
+		Time:            elapsedTime,
 		Request:         harRequest,
 		Response:        harResponse,
 		Cache:           &har.Cache{},
 		Timings: &har.Timings{
 			Send:    -1,
 			Wait:    -1,
-			Receive: totalTime,
+			Receive: elapsedTime,
 		},
 	}
 

agent/pkg/utils/truncating_logger.go

@@ -1,60 +0,0 @@
-package utils
-
-import (
-	"context"
-	"fmt"
-	"time"
-
-	loggerShared "github.com/up9inc/mizu/shared/logger"
-	"gorm.io/gorm/logger"
-	"gorm.io/gorm/utils"
-)
-
-// TruncatingLogger implements the gorm logger.Interface interface. Its purpose is to act as gorm's logger while truncating logs to a max of 50 characters to minimise the performance impact
-type TruncatingLogger struct {
-	LogLevel      logger.LogLevel
-	SlowThreshold time.Duration
-}
-
-func (truncatingLogger *TruncatingLogger) LogMode(logLevel logger.LogLevel) logger.Interface {
-	truncatingLogger.LogLevel = logLevel
-	return truncatingLogger
-}
-
-func (truncatingLogger *TruncatingLogger) Info(_ context.Context, message string, __ ...interface{}) {
-	if truncatingLogger.LogLevel < logger.Info {
-		return
-	}
-	loggerShared.Log.Errorf("gorm info: %.150s", message)
-}
-
-func (truncatingLogger *TruncatingLogger) Warn(_ context.Context, message string, __ ...interface{}) {
-	if truncatingLogger.LogLevel < logger.Warn {
-		return
-	}
-	loggerShared.Log.Errorf("gorm warning: %.150s", message)
-}
-
-func (truncatingLogger *TruncatingLogger) Error(_ context.Context, message string, __ ...interface{}) {
-	if truncatingLogger.LogLevel < logger.Error {
-		return
-	}
-	loggerShared.Log.Errorf("gorm error: %.150s", message)
-}
-
-func (truncatingLogger *TruncatingLogger) Trace(ctx context.Context, begin time.Time, fc func() (string, int64), err error) {
-	if truncatingLogger.LogLevel == logger.Silent {
-		return
-	}
-	elapsed := time.Since(begin)
-	if err != nil {
-		sql, rows := fc() // copied into every condition as this is a potentially heavy operation best done only when necessary
-		truncatingLogger.Error(ctx, fmt.Sprintf("Error in %s: %v - elapsed: %fs affected rows: %d, sql: %s", utils.FileWithLineNum(), err, elapsed.Seconds(), rows, sql))
-	} else if truncatingLogger.LogLevel >= logger.Warn && elapsed > truncatingLogger.SlowThreshold {
-		sql, rows := fc()
-		truncatingLogger.Warn(ctx, fmt.Sprintf("Slow sql query - elapse: %fs rows: %d, sql: %s", elapsed.Seconds(), rows, sql))
-	} else if truncatingLogger.LogLevel >= logger.Info {
-		sql, rows := fc()
-		truncatingLogger.Info(ctx, fmt.Sprintf("Sql query - elapse: %fs rows: %d, sql: %s", elapsed.Seconds(), rows, sql))
-	}
-}

cli/config/config.go

@@ -3,14 +3,15 @@ package config
 import (
 	"errors"
 	"fmt"
-	"github.com/up9inc/mizu/tap/api"
 	"io/ioutil"
-	"k8s.io/apimachinery/pkg/util/json"
 	"os"
 	"reflect"
 	"strconv"
 	"strings"
 
+	"github.com/up9inc/mizu/tap/api"
+	"k8s.io/apimachinery/pkg/util/json"
+
 	"github.com/up9inc/mizu/shared"
 	"github.com/up9inc/mizu/shared/logger"
 
@@ -396,7 +397,7 @@ func getMizuAgentConfig(targetNamespaces []string, mizuApiFilteringOptions *api.
 		TapperResources:         Config.Tap.TapperResources,
 		MizuResourcesNamespace:  Config.MizuResourcesNamespace,
 		MizuApiFilteringOptions: *mizuApiFilteringOptions,
-		AgentDatabasePath:       fmt.Sprintf("%s%s", shared.DataDirPath, "entries.db"),
+		AgentDatabasePath:       shared.DataDirPath,
 	}
 	return &config, nil
 }

debug.Dockerfile

@@ -12,7 +12,7 @@ FROM golang:1.16-alpine AS builder
 # Set necessary environment variables needed for our image.
 ENV CGO_ENABLED=1 GOOS=linux GOARCH=amd64
 
-RUN apk add libpcap-dev gcc g++ make bash
+RUN apk add libpcap-dev gcc g++ make bash perl-utils
 
 # Move to agent working directory (/agent-build).
 WORKDIR /app/agent-build
@@ -23,7 +23,7 @@ COPY tap/go.mod tap/go.mod ../tap/
 COPY tap/api/go.* ../tap/api/
 RUN go mod download
 # cheap trick to make the build faster (As long as go.mod wasn't changes)
-RUN go list -f '{{.Path}}@{{.Version}}' -m all | sed 1d | grep -e 'go-cache' -e 'sqlite' | xargs go get
+RUN go list -f '{{.Path}}@{{.Version}}' -m all | sed 1d | grep -e 'go-cache' | xargs go get
 
 ARG COMMIT_HASH
 ARG GIT_BRANCH
@@ -36,6 +36,12 @@ COPY tap ../tap
 COPY agent .
 RUN go build -gcflags="all=-N -l" -o mizuagent .
 
+# Download Basenine executable, verify the sha1sum and move it to a directory in $PATH
+ADD https://github.com/up9inc/basenine/releases/download/v0.2.6/basenine_linux_amd64 ./basenine_linux_amd64
+ADD https://github.com/up9inc/basenine/releases/download/v0.2.6/basenine_linux_amd64.sha256 ./basenine_linux_amd64.sha256
+RUN shasum -a 256 -c basenine_linux_amd64.sha256
+RUN chmod +x ./basenine_linux_amd64
+
 COPY devops/build_extensions_debug.sh ..
 RUN cd .. && /bin/bash build_extensions_debug.sh
 
@@ -43,10 +49,12 @@ RUN cd .. && /bin/bash build_extensions_debug.sh
 FROM golang:1.16-alpine
 
 RUN apk add bash libpcap-dev tcpdump
+
 WORKDIR /app
 
 # Copy binary and config files from /build to root folder of scratch container.
 COPY --from=builder ["/app/agent-build/mizuagent", "."]
+COPY --from=builder ["/app/agent-build/basenine_linux_amd64", "/usr/local/bin/basenine"]
 COPY --from=builder ["/app/agent/build/extensions", "extensions"]
 COPY --from=site-build ["/app/ui-build/build", "site"]
 

shared/consts.go

@@ -14,4 +14,6 @@ const (
 	GoGCEnvVar                       = "GOGC"
 	DefaultApiServerPort             = 8899
 	DebugModeEnvVar                  = "MIZU_DEBUG"
+	BasenineHost                     = "localhost"
+	BaseninePort                     = "9099"
 )

shared/models.go

@@ -17,6 +17,9 @@ const (
 	WebSocketMessageTypeUpdateStatus  WebSocketMessageType = "status"
 	WebSocketMessageTypeAnalyzeStatus WebSocketMessageType = "analyzeStatus"
 	WebsocketMessageTypeOutboundLink  WebSocketMessageType = "outboundLink"
+	WebSocketMessageTypeToast         WebSocketMessageType = "toast"
+	WebSocketMessageTypeQueryMetadata WebSocketMessageType = "queryMetadata"
+	WebSocketMessageTypeStartTime     WebSocketMessageType = "startTime"
 )
 
 type Resources struct {

tap/api/api.go

@@ -18,7 +18,8 @@ import (
 type Protocol struct {
 	Name            string   `json:"name"`
 	LongName        string   `json:"longName"`
-	Abbreviation    string   `json:"abbreviation"`
+	Abbreviation    string   `json:"abbr"`
+	Macro           string   `json:"macro"`
 	Version         string   `json:"version"`
 	BackgroundColor string   `json:"backgroundColor"`
 	ForegroundColor string   `json:"foregroundColor"`
@@ -28,6 +29,12 @@ type Protocol struct {
 	Priority        uint8    `json:"priority"`
 }
 
+type TCP struct {
+	IP   string `json:"ip"`
+	Port string `json:"port"`
+	Name string `json:"name"`
+}
+
 type Extension struct {
 	Protocol   *Protocol
 	Path       string
@@ -74,6 +81,7 @@ type OutputChannelItem struct {
 	Timestamp      int64
 	ConnectionInfo *ConnectionInfo
 	Pair           *RequestResponsePair
+	Summary        *BaseEntryDetails
 }
 
 type SuperTimer struct {
@@ -89,9 +97,10 @@ type Dissector interface {
 	Register(*Extension)
 	Ping()
 	Dissect(b *bufio.Reader, isClient bool, tcpID *TcpID, counterPair *CounterPair, superTimer *SuperTimer, superIdentifier *SuperIdentifier, emitter Emitter, options *TrafficFilteringOptions) error
-	Analyze(item *OutputChannelItem, entryId string, resolvedSource string, resolvedDestination string) *MizuEntry
+	Analyze(item *OutputChannelItem, resolvedSource string, resolvedDestination string) *MizuEntry
 	Summarize(entry *MizuEntry) *BaseEntryDetails
-	Represent(entry *MizuEntry) (protocol Protocol, object []byte, bodySize int64, err error)
+	Represent(pIn Protocol, request map[string]interface{}, response map[string]interface{}) (pOut Protocol, object []byte, bodySize int64, err error)
+	Macros() map[string]string
 }
 
 type Emitting struct {
@@ -109,39 +118,36 @@ func (e *Emitting) Emit(item *OutputChannelItem) {
 }
 
 type MizuEntry struct {
-	ID                      uint `gorm:"primarykey"`
+	Id                     uint                   `json:"id"`
-	CreatedAt               time.Time
+	Protocol               Protocol               `json:"proto"`
-	UpdatedAt               time.Time
+	Source                 *TCP                   `json:"src"`
-	ProtocolName            string         `json:"protocolName" gorm:"column:protocolName"`
+	Destination            *TCP                   `json:"dst"`
-	ProtocolLongName        string         `json:"protocolLongName" gorm:"column:protocolLongName"`
+	Outgoing               bool                   `json:"outgoing"`
-	ProtocolAbbreviation    string         `json:"protocolAbbreviation" gorm:"column:protocolAbbreviation"`
+	Timestamp              int64                  `json:"timestamp"`
-	ProtocolVersion         string         `json:"protocolVersion" gorm:"column:protocolVersion"`
+	StartTime              time.Time              `json:"startTime"`
-	ProtocolBackgroundColor string         `json:"protocolBackgroundColor" gorm:"column:protocolBackgroundColor"`
+	Request                map[string]interface{} `json:"request"`
-	ProtocolForegroundColor string         `json:"protocolForegroundColor" gorm:"column:protocolForegroundColor"`
+	Response               map[string]interface{} `json:"response"`
-	ProtocolFontSize        int8           `json:"protocolFontSize" gorm:"column:protocolFontSize"`
+	Base                   *BaseEntryDetails      `json:"base"`
-	ProtocolReferenceLink   string         `json:"protocolReferenceLink" gorm:"column:protocolReferenceLink"`
+	Summary                string                 `json:"summary"`
-	Entry                   string         `json:"entry,omitempty" gorm:"column:entry"`
+	Url                    string                 `json:"url"`
-	EntryId                 string         `json:"entryId" gorm:"column:entryId"`
+	Method                 string                 `json:"method"`
-	Url                     string         `json:"url" gorm:"column:url"`
+	Status                 int                    `json:"status"`
-	Method                  string         `json:"method" gorm:"column:method"`
+	RequestSenderIp        string                 `json:"requestSenderIp"`
-	Status                  int            `json:"status" gorm:"column:status"`
+	Service                string                 `json:"service"`
-	RequestSenderIp         string         `json:"requestSenderIp" gorm:"column:requestSenderIp"`
+	ElapsedTime            int64                  `json:"elapsedTime"`
-	Service                 string         `json:"service" gorm:"column:service"`
+	Path                   string                 `json:"path"`
-	Timestamp               int64          `json:"timestamp" gorm:"column:timestamp"`
+	ResolvedSource         string                 `json:"resolvedSource,omitempty"`
-	ElapsedTime             int64          `json:"elapsedTime" gorm:"column:elapsedTime"`
+	ResolvedDestination    string                 `json:"resolvedDestination,omitempty"`
-	Path                    string         `json:"path" gorm:"column:path"`
+	SourceIp               string                 `json:"sourceIp,omitempty"`
-	ResolvedSource          string         `json:"resolvedSource,omitempty" gorm:"column:resolvedSource"`
+	DestinationIp          string                 `json:"destinationIp,omitempty"`
-	ResolvedDestination     string         `json:"resolvedDestination,omitempty" gorm:"column:resolvedDestination"`
+	SourcePort             string                 `json:"sourcePort,omitempty"`
-	SourceIp                string         `json:"sourceIp,omitempty" gorm:"column:sourceIp"`
+	DestinationPort        string                 `json:"destinationPort,omitempty"`
-	DestinationIp           string         `json:"destinationIp,omitempty" gorm:"column:destinationIp"`
+	IsOutgoing             bool                   `json:"isOutgoing,omitempty"`
-	SourcePort              string         `json:"sourcePort,omitempty" gorm:"column:sourcePort"`
+	ContractStatus         ContractStatus         `json:"contractStatus,omitempty"`
-	DestinationPort         string         `json:"destinationPort,omitempty" gorm:"column:destinationPort"`
+	ContractRequestReason  string                 `json:"contractRequestReason,omitempty"`
-	IsOutgoing              bool           `json:"isOutgoing,omitempty" gorm:"column:isOutgoing"`
+	ContractResponseReason string                 `json:"contractResponseReason,omitempty"`
-	ContractStatus          ContractStatus `json:"contractStatus,omitempty" gorm:"column:contractStatus"`
+	ContractContent        string                 `json:"contractContent,omitempty"`
-	ContractRequestReason   string         `json:"contractRequestReason,omitempty" gorm:"column:contractRequestReason"`
+	HTTPPair               string                 `json:"httpPair,omitempty"`
-	ContractResponseReason  string         `json:"contractResponseReason,omitempty" gorm:"column:contractResponseReason"`
-	ContractContent         string         `json:"contractContent,omitempty" gorm:"column:contractContent"`
-	EstimatedSizeBytes      int            `json:"-" gorm:"column:estimatedSizeBytes"`
 }
 
 type MizuEntryWrapper struct {
@@ -154,7 +160,7 @@ type MizuEntryWrapper struct {
 }
 
 type BaseEntryDetails struct {
-	Id              string          `json:"id,omitempty"`
+	Id              uint            `json:"id"`
 	Protocol        Protocol        `json:"protocol,omitempty"`
 	Url             string          `json:"url,omitempty"`
 	RequestSenderIp string          `json:"requestSenderIp,omitempty"`
@@ -194,17 +200,8 @@ type DataUnmarshaler interface {
 }
 
 func (bed *BaseEntryDetails) UnmarshalData(entry *MizuEntry) error {
-	bed.Protocol = Protocol{
+	bed.Protocol = entry.Protocol
-		Name:            entry.ProtocolName,
+	bed.Id = entry.Id
-		LongName:        entry.ProtocolLongName,
-		Abbreviation:    entry.ProtocolAbbreviation,
-		Version:         entry.ProtocolVersion,
-		BackgroundColor: entry.ProtocolBackgroundColor,
-		ForegroundColor: entry.ProtocolForegroundColor,
-		FontSize:        entry.ProtocolFontSize,
-		ReferenceLink:   entry.ProtocolReferenceLink,
-	}
-	bed.Id = entry.EntryId
 	bed.Url = entry.Url
 	bed.RequestSenderIp = entry.RequestSenderIp
 	bed.Service = entry.Service
@@ -228,6 +225,21 @@ const (
 	BODY  string = "body"
 )
 
+type SectionData struct {
+	Type     string `json:"type"`
+	Title    string `json:"title"`
+	Data     string `json:"data"`
+	Encoding string `json:"encoding,omitempty"`
+	MimeType string `json:"mimeType,omitempty"`
+	Selector string `json:"selector,omitempty"`
+}
+
+type TableData struct {
+	Name     string      `json:"name"`
+	Value    interface{} `json:"value"`
+	Selector string      `json:"selector"`
+}
+
 const (
 	TypeHttpRequest = iota
 	TypeHttpResponse

tap/extensions/amqp/helpers.go

@@ -131,97 +131,109 @@ func representProperties(properties map[string]interface{}, rep []interface{}) (
 	userId := ""
 	appId := ""
 
-	if properties["ContentType"] != nil {
+	if properties["contentType"] != nil {
-		contentType = properties["ContentType"].(string)
+		contentType = properties["contentType"].(string)
 	}
-	if properties["ContentEncoding"] != nil {
+	if properties["contentEncoding"] != nil {
-		contentEncoding = properties["ContentEncoding"].(string)
+		contentEncoding = properties["contentEncoding"].(string)
 	}
-	if properties["Delivery Mode"] != nil {
+	if properties["deliveryMode"] != nil {
-		deliveryMode = fmt.Sprintf("%g", properties["DeliveryMode"].(float64))
+		deliveryMode = fmt.Sprintf("%g", properties["deliveryMode"].(float64))
 	}
-	if properties["Priority"] != nil {
+	if properties["priority"] != nil {
-		priority = fmt.Sprintf("%g", properties["Priority"].(float64))
+		priority = fmt.Sprintf("%g", properties["priority"].(float64))
 	}
-	if properties["CorrelationId"] != nil {
+	if properties["correlationId"] != nil {
-		correlationId = properties["CorrelationId"].(string)
+		correlationId = properties["correlationId"].(string)
 	}
-	if properties["ReplyTo"] != nil {
+	if properties["replyTo"] != nil {
-		replyTo = properties["ReplyTo"].(string)
+		replyTo = properties["replyTo"].(string)
 	}
-	if properties["Expiration"] != nil {
+	if properties["expiration"] != nil {
-		expiration = properties["Expiration"].(string)
+		expiration = properties["expiration"].(string)
 	}
-	if properties["MessageId"] != nil {
+	if properties["messageId"] != nil {
-		messageId = properties["MessageId"].(string)
+		messageId = properties["messageId"].(string)
 	}
-	if properties["Timestamp"] != nil {
+	if properties["timestamp"] != nil {
-		timestamp = properties["Timestamp"].(string)
+		timestamp = properties["timestamp"].(string)
 	}
-	if properties["Type"] != nil {
+	if properties["type"] != nil {
-		_type = properties["Type"].(string)
+		_type = properties["type"].(string)
 	}
-	if properties["UserId"] != nil {
+	if properties["userId"] != nil {
-		userId = properties["UserId"].(string)
+		userId = properties["userId"].(string)
 	}
-	if properties["AppId"] != nil {
+	if properties["appId"] != nil {
-		appId = properties["AppId"].(string)
+		appId = properties["appId"].(string)
 	}
 
-	props, _ := json.Marshal([]map[string]string{
+	props, _ := json.Marshal([]api.TableData{
 		{
-			"name":  "Content Type",
+			Name:     "Content Type",
-			"value": contentType,
+			Value:    contentType,
+			Selector: `request.properties.contentType`,
 		},
 		{
-			"name":  "Content Encoding",
+			Name:     "Content Encoding",
-			"value": contentEncoding,
+			Value:    contentEncoding,
+			Selector: `request.properties.contentEncoding`,
 		},
 		{
-			"name":  "Delivery Mode",
+			Name:     "Delivery Mode",
-			"value": deliveryMode,
+			Value:    deliveryMode,
+			Selector: `request.properties.deliveryMode`,
 		},
 		{
-			"name":  "Priority",
+			Name:     "Priority",
-			"value": priority,
+			Value:    priority,
+			Selector: `request.properties.priority`,
 		},
 		{
-			"name":  "Correlation ID",
+			Name:     "Correlation ID",
-			"value": correlationId,
+			Value:    correlationId,
+			Selector: `request.properties.correlationId`,
 		},
 		{
-			"name":  "Reply To",
+			Name:     "Reply To",
-			"value": replyTo,
+			Value:    replyTo,
+			Selector: `request.properties.replyTo`,
 		},
 		{
-			"name":  "Expiration",
+			Name:     "Expiration",
-			"value": expiration,
+			Value:    expiration,
+			Selector: `request.properties.expiration`,
 		},
 		{
-			"name":  "Message ID",
+			Name:     "Message ID",
-			"value": messageId,
+			Value:    messageId,
+			Selector: `request.properties.messageId`,
 		},
 		{
-			"name":  "Timestamp",
+			Name:     "Timestamp",
-			"value": timestamp,
+			Value:    timestamp,
+			Selector: `request.properties.timestamp`,
 		},
 		{
-			"name":  "Type",
+			Name:     "Type",
-			"value": _type,
+			Value:    _type,
+			Selector: `request.properties.type`,
 		},
 		{
-			"name":  "User ID",
+			Name:     "User ID",
-			"value": userId,
+			Value:    userId,
+			Selector: `request.properties.userId`,
 		},
 		{
-			"name":  "App ID",
+			Name:     "App ID",
-			"value": appId,
+			Value:    appId,
+			Selector: `request.properties.appId`,
 		},
 	})
-	rep = append(rep, map[string]string{
+	rep = append(rep, api.SectionData{
-		"type":  api.TABLE,
+		Type:  api.TABLE,
-		"title": "Properties",
+		Title: "Properties",
-		"data":  string(props),
+		Data:  string(props),
 	})
 
 	return rep, contentType, contentEncoding
@@ -230,56 +242,62 @@ func representProperties(properties map[string]interface{}, rep []interface{}) (
 func representBasicPublish(event map[string]interface{}) []interface{} {
 	rep := make([]interface{}, 0)
 
-	details, _ := json.Marshal([]map[string]string{
+	details, _ := json.Marshal([]api.TableData{
 		{
-			"name":  "Exchange",
+			Name:     "Exchange",
-			"value": event["Exchange"].(string),
+			Value:    event["exchange"].(string),
+			Selector: `request.exchange`,
 		},
 		{
-			"name":  "Routing Key",
+			Name:     "Routing Key",
-			"value": event["RoutingKey"].(string),
+			Value:    event["routingKey"].(string),
+			Selector: `request.routingKey`,
 		},
 		{
-			"name":  "Mandatory",
+			Name:     "Mandatory",
-			"value": strconv.FormatBool(event["Mandatory"].(bool)),
+			Value:    strconv.FormatBool(event["mandatory"].(bool)),
+			Selector: `request.mandatory`,
 		},
 		{
-			"name":  "Immediate",
+			Name:     "Immediate",
-			"value": strconv.FormatBool(event["Immediate"].(bool)),
+			Value:    strconv.FormatBool(event["immediate"].(bool)),
+			Selector: `request.immediate`,
 		},
 	})
-	rep = append(rep, map[string]string{
+	rep = append(rep, api.SectionData{
-		"type":  api.TABLE,
+		Type:  api.TABLE,
-		"title": "Details",
+		Title: "Details",
-		"data":  string(details),
+		Data:  string(details),
 	})
 
-	properties := event["Properties"].(map[string]interface{})
+	properties := event["properties"].(map[string]interface{})
 	rep, contentType, _ := representProperties(properties, rep)
 
-	if properties["Headers"] != nil {
+	if properties["headers"] != nil {
-		headers := make([]map[string]string, 0)
+		headers := make([]api.TableData, 0)
-		for name, value := range properties["Headers"].(map[string]interface{}) {
+		for name, value := range properties["headers"].(map[string]interface{}) {
-			headers = append(headers, map[string]string{
+			headers = append(headers, api.TableData{
-				"name":  name,
+				Name:     name,
-				"value": value.(string),
+				Value:    value.(string),
+				Selector: fmt.Sprintf(`request.properties.headers["%s"]`, name),
 			})
 		}
 		headersMarshaled, _ := json.Marshal(headers)
-		rep = append(rep, map[string]string{
+		rep = append(rep, api.SectionData{
-			"type":  api.TABLE,
+			Type:  api.TABLE,
-			"title": "Headers",
+			Title: "Headers",
-			"data":  string(headersMarshaled),
+			Data:  string(headersMarshaled),
 		})
 	}
 
-	if event["Body"] != nil {
+	if event["body"] != nil {
-		rep = append(rep, map[string]string{
+		rep = append(rep, api.SectionData{
-			"type":      api.BODY,
+			Type:     api.BODY,
-			"title":     "Body",
+			Title:    "Body",
-			"encoding":  "base64",
+			Encoding: "base64",
-			"mime_type": contentType,
+			MimeType: contentType,
-			"data":      event["Body"].(string),
+			Data:     event["body"].(string),
+			Selector: `request.body`,
 		})
 	}
 
@@ -293,70 +311,77 @@ func representBasicDeliver(event map[string]interface{}) []interface{} {
 	deliveryTag := ""
 	redelivered := ""
 
-	if event["ConsumerTag"] != nil {
+	if event["consumerTag"] != nil {
-		consumerTag = event["ConsumerTag"].(string)
+		consumerTag = event["consumerTag"].(string)
 	}
-	if event["DeliveryTag"] != nil {
+	if event["deliveryTag"] != nil {
-		deliveryTag = fmt.Sprintf("%g", event["DeliveryTag"].(float64))
+		deliveryTag = fmt.Sprintf("%g", event["deliveryTag"].(float64))
 	}
-	if event["Redelivered"] != nil {
+	if event["redelivered"] != nil {
-		redelivered = strconv.FormatBool(event["Redelivered"].(bool))
+		redelivered = strconv.FormatBool(event["redelivered"].(bool))
 	}
 
-	details, _ := json.Marshal([]map[string]string{
+	details, _ := json.Marshal([]api.TableData{
 		{
-			"name":  "Consumer Tag",
+			Name:     "Consumer Tag",
-			"value": consumerTag,
+			Value:    consumerTag,
+			Selector: `request.consumerTag`,
 		},
 		{
-			"name":  "Delivery Tag",
+			Name:     "Delivery Tag",
-			"value": deliveryTag,
+			Value:    deliveryTag,
+			Selector: `request.deliveryTag`,
 		},
 		{
-			"name":  "Redelivered",
+			Name:     "Redelivered",
-			"value": redelivered,
+			Value:    redelivered,
+			Selector: `request.redelivered`,
 		},
 		{
-			"name":  "Exchange",
+			Name:     "Exchange",
-			"value": event["Exchange"].(string),
+			Value:    event["exchange"].(string),
+			Selector: `request.exchange`,
 		},
 		{
-			"name":  "Routing Key",
+			Name:     "Routing Key",
-			"value": event["RoutingKey"].(string),
+			Value:    event["routingKey"].(string),
+			Selector: `request.routingKey`,
 		},
 	})
-	rep = append(rep, map[string]string{
+	rep = append(rep, api.SectionData{
-		"type":  api.TABLE,
+		Type:  api.TABLE,
-		"title": "Details",
+		Title: "Details",
-		"data":  string(details),
+		Data:  string(details),
 	})
 
-	properties := event["Properties"].(map[string]interface{})
+	properties := event["properties"].(map[string]interface{})
 	rep, contentType, _ := representProperties(properties, rep)
 
-	if properties["Headers"] != nil {
+	if properties["headers"] != nil {
-		headers := make([]map[string]string, 0)
+		headers := make([]api.TableData, 0)
-		for name, value := range properties["Headers"].(map[string]interface{}) {
+		for name, value := range properties["headers"].(map[string]interface{}) {
-			headers = append(headers, map[string]string{
+			headers = append(headers, api.TableData{
-				"name":  name,
+				Name:     name,
-				"value": value.(string),
+				Value:    value.(string),
+				Selector: fmt.Sprintf(`request.properties.headers["%s"]`, name),
 			})
 		}
 		headersMarshaled, _ := json.Marshal(headers)
-		rep = append(rep, map[string]string{
+		rep = append(rep, api.SectionData{
-			"type":  api.TABLE,
+			Type:  api.TABLE,
-			"title": "Headers",
+			Title: "Headers",
-			"data":  string(headersMarshaled),
+			Data:  string(headersMarshaled),
 		})
 	}
 
-	if event["Body"] != nil {
+	if event["body"] != nil {
-		rep = append(rep, map[string]string{
+		rep = append(rep, api.SectionData{
-			"type":      api.BODY,
+			Type:     api.BODY,
-			"title":     "Body",
+			Title:    "Body",
-			"encoding":  "base64",
+			Encoding: "base64",
-			"mime_type": contentType,
+			MimeType: contentType,
-			"data":      event["Body"].(string),
+			Data:     event["body"].(string),
+			Selector: `request.body`,
 		})
 	}
 
@@ -366,51 +391,58 @@ func representBasicDeliver(event map[string]interface{}) []interface{} {
 func representQueueDeclare(event map[string]interface{}) []interface{} {
 	rep := make([]interface{}, 0)
 
-	details, _ := json.Marshal([]map[string]string{
+	details, _ := json.Marshal([]api.TableData{
 		{
-			"name":  "Queue",
+			Name:     "Queue",
-			"value": event["Queue"].(string),
+			Value:    event["queue"].(string),
+			Selector: `request.queue`,
 		},
 		{
-			"name":  "Passive",
+			Name:     "Passive",
-			"value": strconv.FormatBool(event["Passive"].(bool)),
+			Value:    strconv.FormatBool(event["passive"].(bool)),
+			Selector: `request.queue`,
 		},
 		{
-			"name":  "Durable",
+			Name:     "Durable",
-			"value": strconv.FormatBool(event["Durable"].(bool)),
+			Value:    strconv.FormatBool(event["durable"].(bool)),
+			Selector: `request.durable`,
 		},
 		{
-			"name":  "Exclusive",
+			Name:     "Exclusive",
-			"value": strconv.FormatBool(event["Exclusive"].(bool)),
+			Value:    strconv.FormatBool(event["exclusive"].(bool)),
+			Selector: `request.exclusive`,
 		},
 		{
-			"name":  "Auto Delete",
+			Name:     "Auto Delete",
-			"value": strconv.FormatBool(event["AutoDelete"].(bool)),
+			Value:    strconv.FormatBool(event["autoDelete"].(bool)),
+			Selector: `request.autoDelete`,
 		},
 		{
-			"name":  "NoWait",
+			Name:     "NoWait",
-			"value": strconv.FormatBool(event["NoWait"].(bool)),
+			Value:    strconv.FormatBool(event["noWait"].(bool)),
+			Selector: `request.noWait`,
 		},
 	})
-	rep = append(rep, map[string]string{
+	rep = append(rep, api.SectionData{
-		"type":  api.TABLE,
+		Type:  api.TABLE,
-		"title": "Details",
+		Title: "Details",
-		"data":  string(details),
+		Data:  string(details),
 	})
 
-	if event["Arguments"] != nil {
+	if event["arguments"] != nil {
-		headers := make([]map[string]string, 0)
+		headers := make([]api.TableData, 0)
-		for name, value := range event["Arguments"].(map[string]interface{}) {
+		for name, value := range event["arguments"].(map[string]interface{}) {
-			headers = append(headers, map[string]string{
+			headers = append(headers, api.TableData{
-				"name":  name,
+				Name:     name,
-				"value": value.(string),
+				Value:    value.(string),
+				Selector: fmt.Sprintf(`request.arguments["%s"]`, name),
 			})
 		}
 		headersMarshaled, _ := json.Marshal(headers)
-		rep = append(rep, map[string]string{
+		rep = append(rep, api.SectionData{
-			"type":  api.TABLE,
+			Type:  api.TABLE,
-			"title": "Arguments",
+			Title: "Arguments",
-			"data":  string(headersMarshaled),
+			Data:  string(headersMarshaled),
 		})
 	}
 
@@ -420,55 +452,63 @@ func representQueueDeclare(event map[string]interface{}) []interface{} {
 func representExchangeDeclare(event map[string]interface{}) []interface{} {
 	rep := make([]interface{}, 0)
 
-	details, _ := json.Marshal([]map[string]string{
+	details, _ := json.Marshal([]api.TableData{
 		{
-			"name":  "Exchange",
+			Name:     "Exchange",
-			"value": event["Exchange"].(string),
+			Value:    event["exchange"].(string),
+			Selector: `request.exchange`,
 		},
 		{
-			"name":  "Type",
+			Name:     "Type",
-			"value": event["Type"].(string),
+			Value:    event["type"].(string),
+			Selector: `request.type`,
 		},
 		{
-			"name":  "Passive",
+			Name:     "Passive",
-			"value": strconv.FormatBool(event["Passive"].(bool)),
+			Value:    strconv.FormatBool(event["passive"].(bool)),
+			Selector: `request.passive`,
 		},
 		{
-			"name":  "Durable",
+			Name:     "Durable",
-			"value": strconv.FormatBool(event["Durable"].(bool)),
+			Value:    strconv.FormatBool(event["durable"].(bool)),
+			Selector: `request.durable`,
 		},
 		{
-			"name":  "Auto Delete",
+			Name:     "Auto Delete",
-			"value": strconv.FormatBool(event["AutoDelete"].(bool)),
+			Value:    strconv.FormatBool(event["autoDelete"].(bool)),
+			Selector: `request.autoDelete`,
 		},
 		{
-			"name":  "Internal",
+			Name:     "Internal",
-			"value": strconv.FormatBool(event["Internal"].(bool)),
+			Value:    strconv.FormatBool(event["internal"].(bool)),
+			Selector: `request.internal`,
 		},
 		{
-			"name":  "NoWait",
+			Name:     "NoWait",
-			"value": strconv.FormatBool(event["NoWait"].(bool)),
+			Value:    strconv.FormatBool(event["noWait"].(bool)),
+			Selector: `request.noWait`,
 		},
 	})
-	rep = append(rep, map[string]string{
+	rep = append(rep, api.SectionData{
-		"type":  api.TABLE,
+		Type:  api.TABLE,
-		"title": "Details",
+		Title: "Details",
-		"data":  string(details),
+		Data:  string(details),
 	})
 
-	if event["Arguments"] != nil {
+	if event["arguments"] != nil {
-		headers := make([]map[string]string, 0)
+		headers := make([]api.TableData, 0)
-		for name, value := range event["Arguments"].(map[string]interface{}) {
+		for name, value := range event["arguments"].(map[string]interface{}) {
-			headers = append(headers, map[string]string{
+			headers = append(headers, api.TableData{
-				"name":  name,
+				Name:     name,
-				"value": value.(string),
+				Value:    value.(string),
+				Selector: fmt.Sprintf(`request.arguments["%s"]`, name),
 			})
 		}
 		headersMarshaled, _ := json.Marshal(headers)
-		rep = append(rep, map[string]string{
+		rep = append(rep, api.SectionData{
-			"type":  api.TABLE,
+			Type:  api.TABLE,
-			"title": "Arguments",
+			Title: "Arguments",
-			"data":  string(headersMarshaled),
+			Data:  string(headersMarshaled),
 		})
 	}
 
@@ -478,33 +518,37 @@ func representExchangeDeclare(event map[string]interface{}) []interface{} {
 func representConnectionStart(event map[string]interface{}) []interface{} {
 	rep := make([]interface{}, 0)
 
-	details, _ := json.Marshal([]map[string]string{
+	details, _ := json.Marshal([]api.TableData{
 		{
-			"name":  "Version Major",
+			Name:     "Version Major",
-			"value": fmt.Sprintf("%g", event["VersionMajor"].(float64)),
+			Value:    fmt.Sprintf("%g", event["versionMajor"].(float64)),
+			Selector: `request.versionMajor`,
 		},
 		{
-			"name":  "Version Minor",
+			Name:     "Version Minor",
-			"value": fmt.Sprintf("%g", event["VersionMinor"].(float64)),
+			Value:    fmt.Sprintf("%g", event["versionMinor"].(float64)),
+			Selector: `request.versionMinor`,
 		},
 		{
-			"name":  "Mechanisms",
+			Name:     "Mechanisms",
-			"value": event["Mechanisms"].(string),
+			Value:    event["mechanisms"].(string),
+			Selector: `request.mechanisms`,
 		},
 		{
-			"name":  "Locales",
+			Name:     "Locales",
-			"value": event["Locales"].(string),
+			Value:    event["locales"].(string),
+			Selector: `request.locales`,
 		},
 	})
-	rep = append(rep, map[string]string{
+	rep = append(rep, api.SectionData{
-		"type":  api.TABLE,
+		Type:  api.TABLE,
-		"title": "Details",
+		Title: "Details",
-		"data":  string(details),
+		Data:  string(details),
 	})
 
-	if event["ServerProperties"] != nil {
+	if event["serverProperties"] != nil {
-		headers := make([]map[string]string, 0)
+		headers := make([]api.TableData, 0)
-		for name, value := range event["ServerProperties"].(map[string]interface{}) {
+		for name, value := range event["serverProperties"].(map[string]interface{}) {
 			var outcome string
 			switch value.(type) {
 			case string:
@@ -517,16 +561,17 @@ func representConnectionStart(event map[string]interface{}) []interface{} {
 			default:
 				panic("Unknown data type for the server property!")
 			}
-			headers = append(headers, map[string]string{
+			headers = append(headers, api.TableData{
-				"name":  name,
+				Name:     name,
-				"value": outcome,
+				Value:    outcome,
+				Selector: fmt.Sprintf(`request.serverProperties["%s"]`, name),
 			})
 		}
 		headersMarshaled, _ := json.Marshal(headers)
-		rep = append(rep, map[string]string{
+		rep = append(rep, api.SectionData{
-			"type":  api.TABLE,
+			Type:  api.TABLE,
-			"title": "Server Properties",
+			Title: "Server Properties",
-			"data":  string(headersMarshaled),
+			Data:  string(headersMarshaled),
 		})
 	}
 
@@ -536,28 +581,32 @@ func representConnectionStart(event map[string]interface{}) []interface{} {
 func representConnectionClose(event map[string]interface{}) []interface{} {
 	rep := make([]interface{}, 0)
 
-	details, _ := json.Marshal([]map[string]string{
+	details, _ := json.Marshal([]api.TableData{
 		{
-			"name":  "Reply Code",
+			Name:     "Reply Code",
-			"value": fmt.Sprintf("%g", event["ReplyCode"].(float64)),
+			Value:    fmt.Sprintf("%g", event["replyCode"].(float64)),
+			Selector: `request.replyCode`,
 		},
 		{
-			"name":  "Reply Text",
+			Name:     "Reply Text",
-			"value": event["ReplyText"].(string),
+			Value:    event["replyText"].(string),
+			Selector: `request.replyText`,
 		},
 		{
-			"name":  "Class ID",
+			Name:     "Class ID",
-			"value": fmt.Sprintf("%g", event["ClassId"].(float64)),
+			Value:    fmt.Sprintf("%g", event["classId"].(float64)),
+			Selector: `request.classId`,
 		},
 		{
-			"name":  "Method ID",
+			Name:     "Method ID",
-			"value": fmt.Sprintf("%g", event["MethodId"].(float64)),
+			Value:    fmt.Sprintf("%g", event["methodId"].(float64)),
+			Selector: `request.methodId`,
 		},
 	})
-	rep = append(rep, map[string]string{
+	rep = append(rep, api.SectionData{
-		"type":  api.TABLE,
+		Type:  api.TABLE,
-		"title": "Details",
+		Title: "Details",
-		"data":  string(details),
+		Data:  string(details),
 	})
 
 	return rep
@@ -566,43 +615,48 @@ func representConnectionClose(event map[string]interface{}) []interface{} {
 func representQueueBind(event map[string]interface{}) []interface{} {
 	rep := make([]interface{}, 0)
 
-	details, _ := json.Marshal([]map[string]string{
+	details, _ := json.Marshal([]api.TableData{
 		{
-			"name":  "Queue",
+			Name:     "Queue",
-			"value": event["Queue"].(string),
+			Value:    event["queue"].(string),
+			Selector: `request.queue`,
 		},
 		{
-			"name":  "Exchange",
+			Name:     "Exchange",
-			"value": event["Exchange"].(string),
+			Value:    event["exchange"].(string),
+			Selector: `request.exchange`,
 		},
 		{
-			"name":  "RoutingKey",
+			Name:     "RoutingKey",
-			"value": event["RoutingKey"].(string),
+			Value:    event["routingKey"].(string),
+			Selector: `request.routingKey`,
 		},
 		{
-			"name":  "NoWait",
+			Name:     "NoWait",
-			"value": strconv.FormatBool(event["NoWait"].(bool)),
+			Value:    strconv.FormatBool(event["noWait"].(bool)),
+			Selector: `request.noWait`,
 		},
 	})
-	rep = append(rep, map[string]string{
+	rep = append(rep, api.SectionData{
-		"type":  api.TABLE,
+		Type:  api.TABLE,
-		"title": "Details",
+		Title: "Details",
-		"data":  string(details),
+		Data:  string(details),
 	})
 
-	if event["Arguments"] != nil {
+	if event["arguments"] != nil {
-		headers := make([]map[string]string, 0)
+		headers := make([]api.TableData, 0)
-		for name, value := range event["Arguments"].(map[string]interface{}) {
+		for name, value := range event["arguments"].(map[string]interface{}) {
-			headers = append(headers, map[string]string{
+			headers = append(headers, api.TableData{
-				"name":  name,
+				Name:     name,
-				"value": value.(string),
+				Value:    value.(string),
+				Selector: fmt.Sprintf(`request.arguments["%s"]`, name),
 			})
 		}
 		headersMarshaled, _ := json.Marshal(headers)
-		rep = append(rep, map[string]string{
+		rep = append(rep, api.SectionData{
-			"type":  api.TABLE,
+			Type:  api.TABLE,
-			"title": "Arguments",
+			Title: "Arguments",
-			"data":  string(headersMarshaled),
+			Data:  string(headersMarshaled),
 		})
 	}
 
@@ -612,51 +666,58 @@ func representQueueBind(event map[string]interface{}) []interface{} {
 func representBasicConsume(event map[string]interface{}) []interface{} {
 	rep := make([]interface{}, 0)
 
-	details, _ := json.Marshal([]map[string]string{
+	details, _ := json.Marshal([]api.TableData{
 		{
-			"name":  "Queue",
+			Name:     "Queue",
-			"value": event["Queue"].(string),
+			Value:    event["queue"].(string),
+			Selector: `request.queue`,
 		},
 		{
-			"name":  "Consumer Tag",
+			Name:     "Consumer Tag",
-			"value": event["ConsumerTag"].(string),
+			Value:    event["consumerTag"].(string),
+			Selector: `request.consumerTag`,
 		},
 		{
-			"name":  "No Local",
+			Name:     "No Local",
-			"value": strconv.FormatBool(event["NoLocal"].(bool)),
+			Value:    strconv.FormatBool(event["noLocal"].(bool)),
+			Selector: `request.noLocal`,
 		},
 		{
-			"name":  "No Ack",
+			Name:     "No Ack",
-			"value": strconv.FormatBool(event["NoAck"].(bool)),
+			Value:    strconv.FormatBool(event["noAck"].(bool)),
+			Selector: `request.noAck`,
 		},
 		{
-			"name":  "Exclusive",
+			Name:     "Exclusive",
-			"value": strconv.FormatBool(event["Exclusive"].(bool)),
+			Value:    strconv.FormatBool(event["exclusive"].(bool)),
+			Selector: `request.exclusive`,
 		},
 		{
-			"name":  "NoWait",
+			Name:     "NoWait",
-			"value": strconv.FormatBool(event["NoWait"].(bool)),
+			Value:    strconv.FormatBool(event["noWait"].(bool)),
+			Selector: `request.noWait`,
 		},
 	})
-	rep = append(rep, map[string]string{
+	rep = append(rep, api.SectionData{
-		"type":  api.TABLE,
+		Type:  api.TABLE,
-		"title": "Details",
+		Title: "Details",
-		"data":  string(details),
+		Data:  string(details),
 	})
 
-	if event["Arguments"] != nil {
+	if event["arguments"] != nil {
-		headers := make([]map[string]string, 0)
+		headers := make([]api.TableData, 0)
-		for name, value := range event["Arguments"].(map[string]interface{}) {
+		for name, value := range event["arguments"].(map[string]interface{}) {
-			headers = append(headers, map[string]string{
+			headers = append(headers, api.TableData{
-				"name":  name,
+				Name:     name,
-				"value": value.(string),
+				Value:    value.(string),
+				Selector: fmt.Sprintf(`request.arguments["%s"]`, name),
 			})
 		}
 		headersMarshaled, _ := json.Marshal(headers)
-		rep = append(rep, map[string]string{
+		rep = append(rep, api.SectionData{
-			"type":  api.TABLE,
+			Type:  api.TABLE,
-			"title": "Arguments",
+			Title: "Arguments",
-			"data":  string(headersMarshaled),
+			Data:  string(headersMarshaled),
 		})
 	}
 

tap/extensions/amqp/main.go

@@ -16,6 +16,7 @@ var protocol api.Protocol = api.Protocol{
 	Name:            "amqp",
 	LongName:        "Advanced Message Queuing Protocol 0-9-1",
 	Abbreviation:    "AMQP",
+	Macro:           "amqp",
 	Version:         "0-9-1",
 	BackgroundColor: "#ff6600",
 	ForegroundColor: "#ffffff",
@@ -222,7 +223,7 @@ func (d dissecting) Dissect(b *bufio.Reader, isClient bool, tcpID *api.TcpID, co
 	}
 }
 
-func (d dissecting) Analyze(item *api.OutputChannelItem, entryId string, resolvedSource string, resolvedDestination string) *api.MizuEntry {
+func (d dissecting) Analyze(item *api.OutputChannelItem, resolvedSource string, resolvedDestination string) *api.MizuEntry {
 	request := item.Pair.Request.Payload.(map[string]interface{})
 	reqDetails := request["details"].(map[string]interface{})
 	service := "amqp"
@@ -235,75 +236,79 @@ func (d dissecting) Analyze(item *api.OutputChannelItem, entryId string, resolve
 	summary := ""
 	switch request["method"] {
 	case basicMethodMap[40]:
-		summary = reqDetails["Exchange"].(string)
+		summary = reqDetails["exchange"].(string)
 		break
 	case basicMethodMap[60]:
-		summary = reqDetails["Exchange"].(string)
+		summary = reqDetails["exchange"].(string)
 		break
 	case exchangeMethodMap[10]:
-		summary = reqDetails["Exchange"].(string)
+		summary = reqDetails["exchange"].(string)
 		break
 	case queueMethodMap[10]:
-		summary = reqDetails["Queue"].(string)
+		summary = reqDetails["queue"].(string)
 		break
 	case connectionMethodMap[10]:
 		summary = fmt.Sprintf(
 			"%s.%s",
-			strconv.Itoa(int(reqDetails["VersionMajor"].(float64))),
+			strconv.Itoa(int(reqDetails["versionMajor"].(float64))),
-			strconv.Itoa(int(reqDetails["VersionMinor"].(float64))),
+			strconv.Itoa(int(reqDetails["versionMinor"].(float64))),
 		)
 		break
 	case connectionMethodMap[50]:
-		summary = reqDetails["ReplyText"].(string)
+		summary = reqDetails["replyText"].(string)
 		break
 	case queueMethodMap[20]:
-		summary = reqDetails["Queue"].(string)
+		summary = reqDetails["queue"].(string)
 		break
 	case basicMethodMap[20]:
-		summary = reqDetails["Queue"].(string)
+		summary = reqDetails["queue"].(string)
 		break
 	}
 
 	request["url"] = summary
-	entryBytes, _ := json.Marshal(item.Pair)
+	reqDetails["method"] = request["method"]
 	return &api.MizuEntry{
-		ProtocolName:            protocol.Name,
+		Protocol: protocol,
-		ProtocolLongName:        protocol.LongName,
+		Source: &api.TCP{
-		ProtocolAbbreviation:    protocol.Abbreviation,
+			Name: resolvedSource,
-		ProtocolVersion:         protocol.Version,
+			IP:   item.ConnectionInfo.ClientIP,
-		ProtocolBackgroundColor: protocol.BackgroundColor,
+			Port: item.ConnectionInfo.ClientPort,
-		ProtocolForegroundColor: protocol.ForegroundColor,
+		},
-		ProtocolFontSize:        protocol.FontSize,
+		Destination: &api.TCP{
-		ProtocolReferenceLink:   protocol.ReferenceLink,
+			Name: resolvedDestination,
-		EntryId:                 entryId,
+			IP:   item.ConnectionInfo.ServerIP,
-		Entry:                   string(entryBytes),
+			Port: item.ConnectionInfo.ServerPort,
-		Url:                     fmt.Sprintf("%s%s", service, summary),
+		},
-		Method:                  request["method"].(string),
+		Outgoing:            item.ConnectionInfo.IsOutgoing,
-		Status:                  0,
+		Request:             reqDetails,
-		RequestSenderIp:         item.ConnectionInfo.ClientIP,
+		Url:                 fmt.Sprintf("%s%s", service, summary),
-		Service:                 service,
+		Method:              request["method"].(string),
-		Timestamp:               item.Timestamp,
+		Status:              0,
-		ElapsedTime:             0,
+		RequestSenderIp:     item.ConnectionInfo.ClientIP,
-		Path:                    summary,
+		Service:             service,
-		ResolvedSource:          resolvedSource,
+		Timestamp:           item.Timestamp,
-		ResolvedDestination:     resolvedDestination,
+		StartTime:           item.Pair.Request.CaptureTime,
-		SourceIp:                item.ConnectionInfo.ClientIP,
+		ElapsedTime:         0,
-		DestinationIp:           item.ConnectionInfo.ServerIP,
+		Summary:             summary,
-		SourcePort:              item.ConnectionInfo.ClientPort,
+		ResolvedSource:      resolvedSource,
-		DestinationPort:         item.ConnectionInfo.ServerPort,
+		ResolvedDestination: resolvedDestination,
-		IsOutgoing:              item.ConnectionInfo.IsOutgoing,
+		SourceIp:            item.ConnectionInfo.ClientIP,
+		DestinationIp:       item.ConnectionInfo.ServerIP,
+		SourcePort:          item.ConnectionInfo.ClientPort,
+		DestinationPort:     item.ConnectionInfo.ServerPort,
+		IsOutgoing:          item.ConnectionInfo.IsOutgoing,
 	}
 
 }
 
 func (d dissecting) Summarize(entry *api.MizuEntry) *api.BaseEntryDetails {
 	return &api.BaseEntryDetails{
-		Id:              entry.EntryId,
+		Id:              entry.Id,
 		Protocol:        protocol,
 		Url:             entry.Url,
 		RequestSenderIp: entry.RequestSenderIp,
 		Service:         entry.Service,
-		Summary:         entry.Path,
+		Summary:         entry.Summary,
 		StatusCode:      entry.Status,
 		Method:          entry.Method,
 		Timestamp:       entry.Timestamp,
@@ -320,39 +325,35 @@ func (d dissecting) Summarize(entry *api.MizuEntry) *api.BaseEntryDetails {
 	}
 }
 
-func (d dissecting) Represent(entry *api.MizuEntry) (p api.Protocol, object []byte, bodySize int64, err error) {
+func (d dissecting) Represent(protoIn api.Protocol, request map[string]interface{}, response map[string]interface{}) (protoOut api.Protocol, object []byte, bodySize int64, err error) {
-	p = protocol
+	protoOut = protocol
 	bodySize = 0
-	var root map[string]interface{}
-	json.Unmarshal([]byte(entry.Entry), &root)
 	representation := make(map[string]interface{}, 0)
-	request := root["request"].(map[string]interface{})["payload"].(map[string]interface{})
 	var repRequest []interface{}
-	details := request["details"].(map[string]interface{})
 	switch request["method"].(string) {
 	case basicMethodMap[40]:
-		repRequest = representBasicPublish(details)
+		repRequest = representBasicPublish(request)
 		break
 	case basicMethodMap[60]:
-		repRequest = representBasicDeliver(details)
+		repRequest = representBasicDeliver(request)
 		break
 	case queueMethodMap[10]:
-		repRequest = representQueueDeclare(details)
+		repRequest = representQueueDeclare(request)
 		break
 	case exchangeMethodMap[10]:
-		repRequest = representExchangeDeclare(details)
+		repRequest = representExchangeDeclare(request)
 		break
 	case connectionMethodMap[10]:
-		repRequest = representConnectionStart(details)
+		repRequest = representConnectionStart(request)
 		break
 	case connectionMethodMap[50]:
-		repRequest = representConnectionClose(details)
+		repRequest = representConnectionClose(request)
 		break
 	case queueMethodMap[20]:
-		repRequest = representQueueBind(details)
+		repRequest = representQueueBind(request)
 		break
 	case basicMethodMap[20]:
-		repRequest = representBasicConsume(details)
+		repRequest = representBasicConsume(request)
 		break
 	}
 	representation["request"] = repRequest
@@ -360,4 +361,10 @@ func (d dissecting) Represent(entry *api.MizuEntry) (p api.Protocol, object []by
 	return
 }
 
+func (d dissecting) Macros() map[string]string {
+	return map[string]string{
+		`amqp`: fmt.Sprintf(`proto.abbr == "%s"`, protocol.Abbreviation),
+	}
+}
+
 var Dissector dissecting

tap/extensions/amqp/spec091.go

@@ -71,11 +71,11 @@ func isSoftExceptionCode(code int) bool {
 }
 
 type ConnectionStart struct {
-	VersionMajor     byte
+	VersionMajor     byte   `json:"versionMajor"`
-	VersionMinor     byte
+	VersionMinor     byte   `json:"versionMinor"`
-	ServerProperties Table
+	ServerProperties Table  `json:"serverProperties"`
-	Mechanisms       string
+	Mechanisms       string `json:"mechanisms"`
-	Locales          string
+	Locales          string `json:"locales"`
 }
 
 func (msg *ConnectionStart) id() (uint16, uint16) {
@@ -429,10 +429,10 @@ func (msg *connectionOpenOk) read(r io.Reader) (err error) {
 }
 
 type ConnectionClose struct {
-	ReplyCode uint16
+	ReplyCode uint16 `json:"relyCode"`
-	ReplyText string
+	ReplyText string `json:"replyText"`
-	ClassId   uint16
+	ClassId   uint16 `json:"classId"`
-	MethodId  uint16
+	MethodId  uint16 `json:"methodId"`
 }
 
 func (msg *ConnectionClose) id() (uint16, uint16) {
@@ -767,14 +767,14 @@ func (msg *channelCloseOk) read(r io.Reader) (err error) {
 
 type ExchangeDeclare struct {
 	reserved1  uint16
-	Exchange   string
+	Exchange   string `json:"exchange"`
-	Type       string
+	Type       string `json:"type"`
-	Passive    bool
+	Passive    bool   `json:"passive"`
-	Durable    bool
+	Durable    bool   `json:"durable"`
-	AutoDelete bool
+	AutoDelete bool   `json:"autoDelete"`
-	Internal   bool
+	Internal   bool   `json:"internal"`
-	NoWait     bool
+	NoWait     bool   `json:"noWait"`
-	Arguments  Table
+	Arguments  Table  `json:"arguments"`
 }
 
 func (msg *ExchangeDeclare) id() (uint16, uint16) {
@@ -1163,13 +1163,13 @@ func (msg *exchangeUnbindOk) read(r io.Reader) (err error) {
 
 type QueueDeclare struct {
 	reserved1  uint16
-	Queue      string
+	Queue      string `json:"queue"`
-	Passive    bool
+	Passive    bool   `json:"passive"`
-	Durable    bool
+	Durable    bool   `json:"durable"`
-	Exclusive  bool
+	Exclusive  bool   `json:"exclusive"`
-	AutoDelete bool
+	AutoDelete bool   `json:"autoDelete"`
-	NoWait     bool
+	NoWait     bool   `json:"noWait"`
-	Arguments  Table
+	Arguments  Table  `json:"arguments"`
 }
 
 func (msg *QueueDeclare) id() (uint16, uint16) {
@@ -1297,11 +1297,11 @@ func (msg *QueueDeclareOk) read(r io.Reader) (err error) {
 
 type QueueBind struct {
 	reserved1  uint16
-	Queue      string
+	Queue      string `json:"queue"`
-	Exchange   string
+	Exchange   string `json:"exchange"`
-	RoutingKey string
+	RoutingKey string `json:"routingKey"`
-	NoWait     bool
+	NoWait     bool   `json:"noWait"`
-	Arguments  Table
+	Arguments  Table  `json:"arguments"`
 }
 
 func (msg *QueueBind) id() (uint16, uint16) {
@@ -1737,13 +1737,13 @@ func (msg *basicQosOk) read(r io.Reader) (err error) {
 
 type BasicConsume struct {
 	reserved1   uint16
-	Queue       string
+	Queue       string `json:"queue"`
-	ConsumerTag string
+	ConsumerTag string `json:"consumerTag"`
-	NoLocal     bool
+	NoLocal     bool   `json:"noLocal"`
-	NoAck       bool
+	NoAck       bool   `json:"noAck"`
-	Exclusive   bool
+	Exclusive   bool   `json:"exclusive"`
-	NoWait      bool
+	NoWait      bool   `json:"noWait"`
-	Arguments   Table
+	Arguments   Table  `json:"arguments"`
 }
 
 func (msg *BasicConsume) id() (uint16, uint16) {
@@ -1932,12 +1932,12 @@ func (msg *basicCancelOk) read(r io.Reader) (err error) {
 
 type BasicPublish struct {
 	reserved1  uint16
-	Exchange   string
+	Exchange   string     `json:"exchange"`
-	RoutingKey string
+	RoutingKey string     `json:"routingKey"`
-	Mandatory  bool
+	Mandatory  bool       `json:"mandatory"`
-	Immediate  bool
+	Immediate  bool       `json:"immediate"`
-	Properties Properties
+	Properties Properties `json:"properties"`
-	Body       []byte
+	Body       []byte     `json:"body"`
 }
 
 func (msg *BasicPublish) id() (uint16, uint16) {
@@ -2072,13 +2072,13 @@ func (msg *basicReturn) read(r io.Reader) (err error) {
 }
 
 type BasicDeliver struct {
-	ConsumerTag string
+	ConsumerTag string     `json:"consumerTag"`
-	DeliveryTag uint64
+	DeliveryTag uint64     `json:"deliveryTag"`
-	Redelivered bool
+	Redelivered bool       `json:"redelivered"`
-	Exchange    string
+	Exchange    string     `json:"exchange"`
-	RoutingKey  string
+	RoutingKey  string     `json:"routingKey"`
-	Properties  Properties
+	Properties  Properties `json:"properties"`
-	Body        []byte
+	Body        []byte     `json:"body"`
 }
 
 func (msg *BasicDeliver) id() (uint16, uint16) {

tap/extensions/amqp/types.go

@@ -93,19 +93,19 @@ func (e Error) Error() string {
 
 // Used by header frames to capture routing and header information
 type Properties struct {
-	ContentType     string    // MIME content type
+	ContentType     string    `json:"contentType"`     // MIME content type
-	ContentEncoding string    // MIME content encoding
+	ContentEncoding string    `json:"contentEncoding"` // MIME content encoding
-	Headers         Table     // Application or header exchange table
+	Headers         Table     `json:"headers"`         // Application or header exchange table
-	DeliveryMode    uint8     // queue implementation use - Transient (1) or Persistent (2)
+	DeliveryMode    uint8     `json:"deliveryMode"`    // queue implementation use - Transient (1) or Persistent (2)
-	Priority        uint8     // queue implementation use - 0 to 9
+	Priority        uint8     `json:"priority"`        // queue implementation use - 0 to 9
-	CorrelationId   string    // application use - correlation identifier
+	CorrelationId   string    `json:"correlationId"`   // application use - correlation identifier
-	ReplyTo         string    // application use - address to to reply to (ex: RPC)
+	ReplyTo         string    `json:"replyTo"`         // application use - address to to reply to (ex: RPC)
-	Expiration      string    // implementation use - message expiration spec
+	Expiration      string    `json:"expiration"`      // implementation use - message expiration spec
-	MessageId       string    // application use - message identifier
+	MessageId       string    `json:"messageId"`       // application use - message identifier
-	Timestamp       time.Time // application use - message timestamp
+	Timestamp       time.Time `json:"timestamp"`       // application use - message timestamp
-	Type            string    // application use - message type name
+	Type            string    `json:"type"`            // application use - message type name
-	UserId          string    // application use - creating user id
+	UserId          string    `json:"userId"`          // application use - creating user id
-	AppId           string    // application use - creating application
+	AppId           string    `json:"appId"`           // application use - creating application
 	reserved1       string    // was cluster-id - process for buffer consumption
 }
 

tap/extensions/http/helpers.go

@@ -0,0 +1,35 @@
+package main
+
+import (
+	"encoding/json"
+	"fmt"
+
+	"github.com/up9inc/mizu/tap/api"
+)
+
+func mapSliceRebuildAsMap(mapSlice []interface{}) (newMap map[string]interface{}) {
+	newMap = make(map[string]interface{})
+	for _, header := range mapSlice {
+		h := header.(map[string]interface{})
+		newMap[h["name"].(string)] = h["value"]
+	}
+
+	return
+}
+
+func representMapSliceAsTable(mapSlice []interface{}, selectorPrefix string) (representation string) {
+	var table []api.TableData
+	for _, header := range mapSlice {
+		h := header.(map[string]interface{})
+		selector := fmt.Sprintf("%s[\"%s\"]", selectorPrefix, h["name"].(string))
+		table = append(table, api.TableData{
+			Name:     h["name"].(string),
+			Value:    h["value"],
+			Selector: selector,
+		})
+	}
+
+	obj, _ := json.Marshal(table)
+	representation = string(obj)
+	return
+}

tap/extensions/http/main.go

@@ -17,6 +17,7 @@ var protocol api.Protocol = api.Protocol{
 	Name:            "http",
 	LongName:        "Hypertext Transfer Protocol -- HTTP/1.1",
 	Abbreviation:    "HTTP",
+	Macro:           "http",
 	Version:         "1.1",
 	BackgroundColor: "#205cf5",
 	ForegroundColor: "#ffffff",
@@ -30,6 +31,7 @@ var http2Protocol api.Protocol = api.Protocol{
 	Name:            "http",
 	LongName:        "Hypertext Transfer Protocol Version 2 (HTTP/2) (gRPC)",
 	Abbreviation:    "HTTP/2",
+	Macro:           "grpc",
 	Version:         "2.0",
 	BackgroundColor: "#244c5a",
 	ForegroundColor: "#ffffff",
@@ -117,7 +119,7 @@ func SetHostname(address, newHostname string) string {
 	return replacedUrl.String()
 }
 
-func (d dissecting) Analyze(item *api.OutputChannelItem, entryId string, resolvedSource string, resolvedDestination string) *api.MizuEntry {
+func (d dissecting) Analyze(item *api.OutputChannelItem, resolvedSource string, resolvedDestination string) *api.MizuEntry {
 	var host, scheme, authority, path, service string
 
 	request := item.Pair.Request.Payload.(map[string]interface{})
@@ -145,10 +147,32 @@ func (d dissecting) Analyze(item *api.OutputChannelItem, entryId string, resolve
 		service = fmt.Sprintf("%s://%s", scheme, authority)
 	} else {
 		service = fmt.Sprintf("http://%s", host)
-		path = reqDetails["url"].(string)
+		u, err := url.Parse(reqDetails["url"].(string))
+		if err != nil {
+			path = reqDetails["url"].(string)
+		} else {
+			path = u.Path
+		}
 	}
 
-	request["url"] = path
+	request["url"] = reqDetails["url"].(string)
+	reqDetails["path"] = path
+	reqDetails["summary"] = path
+
+	// Rearrange the maps for the querying
+	reqDetails["_headers"] = reqDetails["headers"]
+	reqDetails["headers"] = mapSliceRebuildAsMap(reqDetails["_headers"].([]interface{}))
+	resDetails["_headers"] = resDetails["headers"]
+	resDetails["headers"] = mapSliceRebuildAsMap(resDetails["_headers"].([]interface{}))
+
+	reqDetails["_cookies"] = reqDetails["cookies"]
+	reqDetails["cookies"] = mapSliceRebuildAsMap(reqDetails["_cookies"].([]interface{}))
+	resDetails["_cookies"] = resDetails["cookies"]
+	resDetails["cookies"] = mapSliceRebuildAsMap(resDetails["_cookies"].([]interface{}))
+
+	reqDetails["_queryString"] = reqDetails["queryString"]
+	reqDetails["queryString"] = mapSliceRebuildAsMap(reqDetails["_queryString"].([]interface{}))
+
 	if resolvedDestination != "" {
 		service = SetHostname(service, resolvedDestination)
 	} else if resolvedSource != "" {
@@ -156,51 +180,59 @@ func (d dissecting) Analyze(item *api.OutputChannelItem, entryId string, resolve
 	}
 
 	elapsedTime := item.Pair.Response.CaptureTime.Sub(item.Pair.Request.CaptureTime).Round(time.Millisecond).Milliseconds()
-	entryBytes, _ := json.Marshal(item.Pair)
+	httpPair, _ := json.Marshal(item.Pair)
+	_protocol := protocol
+	_protocol.Version = item.Protocol.Version
 	return &api.MizuEntry{
-		ProtocolName:            protocol.Name,
+		Protocol: _protocol,
-		ProtocolLongName:        protocol.LongName,
+		Source: &api.TCP{
-		ProtocolAbbreviation:    protocol.Abbreviation,
+			Name: resolvedSource,
-		ProtocolVersion:         item.Protocol.Version,
+			IP:   item.ConnectionInfo.ClientIP,
-		ProtocolBackgroundColor: protocol.BackgroundColor,
+			Port: item.ConnectionInfo.ClientPort,
-		ProtocolForegroundColor: protocol.ForegroundColor,
+		},
-		ProtocolFontSize:        protocol.FontSize,
+		Destination: &api.TCP{
-		ProtocolReferenceLink:   protocol.ReferenceLink,
+			Name: resolvedDestination,
-		EntryId:                 entryId,
+			IP:   item.ConnectionInfo.ServerIP,
-		Entry:                   string(entryBytes),
+			Port: item.ConnectionInfo.ServerPort,
-		Url:                     fmt.Sprintf("%s%s", service, path),
+		},
-		Method:                  reqDetails["method"].(string),
+		Outgoing:            item.ConnectionInfo.IsOutgoing,
-		Status:                  int(resDetails["status"].(float64)),
+		Request:             reqDetails,
-		RequestSenderIp:         item.ConnectionInfo.ClientIP,
+		Response:            resDetails,
-		Service:                 service,
+		Url:                 fmt.Sprintf("%s%s", service, path),
-		Timestamp:               item.Timestamp,
+		Method:              reqDetails["method"].(string),
-		ElapsedTime:             elapsedTime,
+		Status:              int(resDetails["status"].(float64)),
-		Path:                    path,
+		RequestSenderIp:     item.ConnectionInfo.ClientIP,
-		ResolvedSource:          resolvedSource,
+		Service:             service,
-		ResolvedDestination:     resolvedDestination,
+		Timestamp:           item.Timestamp,
-		SourceIp:                item.ConnectionInfo.ClientIP,
+		StartTime:           item.Pair.Request.CaptureTime,
-		DestinationIp:           item.ConnectionInfo.ServerIP,
+		ElapsedTime:         elapsedTime,
-		SourcePort:              item.ConnectionInfo.ClientPort,
+		Summary:             path,
-		DestinationPort:         item.ConnectionInfo.ServerPort,
+		ResolvedSource:      resolvedSource,
-		IsOutgoing:              item.ConnectionInfo.IsOutgoing,
+		ResolvedDestination: resolvedDestination,
+		SourceIp:            item.ConnectionInfo.ClientIP,
+		DestinationIp:       item.ConnectionInfo.ServerIP,
+		SourcePort:          item.ConnectionInfo.ClientPort,
+		DestinationPort:     item.ConnectionInfo.ServerPort,
+		IsOutgoing:          item.ConnectionInfo.IsOutgoing,
+		HTTPPair:            string(httpPair),
 	}
 }
 
 func (d dissecting) Summarize(entry *api.MizuEntry) *api.BaseEntryDetails {
 	var p api.Protocol
-	if entry.ProtocolVersion == "2.0" {
+	if entry.Protocol.Version == "2.0" {
 		p = http2Protocol
 	} else {
 		p = protocol
 	}
 	return &api.BaseEntryDetails{
-		Id:              entry.EntryId,
+		Id:              entry.Id,
 		Protocol:        p,
 		Url:             entry.Url,
 		RequestSenderIp: entry.RequestSenderIp,
 		Service:         entry.Service,
 		Path:            entry.Path,
-		Summary:         entry.Path,
+		Summary:         entry.Summary,
 		StatusCode:      entry.Status,
 		Method:          entry.Method,
 		Timestamp:       entry.Timestamp,
@@ -218,45 +250,50 @@ func (d dissecting) Summarize(entry *api.MizuEntry) *api.BaseEntryDetails {
 }
 
 func representRequest(request map[string]interface{}) (repRequest []interface{}) {
-	details, _ := json.Marshal([]map[string]string{
+	details, _ := json.Marshal([]api.TableData{
 		{
-			"name":  "Method",
+			Name:     "Method",
-			"value": request["method"].(string),
+			Value:    request["method"].(string),
+			Selector: `request.method`,
 		},
 		{
-			"name":  "URL",
+			Name:     "URL",
-			"value": request["url"].(string),
+			Value:    request["url"].(string),
+			Selector: `request.url`,
 		},
 		{
-			"name":  "Body Size",
+			Name:     "Path",
-			"value": fmt.Sprintf("%g bytes", request["bodySize"].(float64)),
+			Value:    request["path"].(string),
+			Selector: `request.path`,
+		},
+		{
+			Name:     "Body Size (bytes)",
+			Value:    int64(request["bodySize"].(float64)),
+			Selector: `request.bodySize`,
 		},
 	})
-	repRequest = append(repRequest, map[string]string{
+	repRequest = append(repRequest, api.SectionData{
-		"type":  api.TABLE,
+		Type:  api.TABLE,
-		"title": "Details",
+		Title: "Details",
-		"data":  string(details),
+		Data:  string(details),
 	})
 
-	headers, _ := json.Marshal(request["headers"].([]interface{}))
+	repRequest = append(repRequest, api.SectionData{
-	repRequest = append(repRequest, map[string]string{
+		Type:  api.TABLE,
-		"type":  api.TABLE,
+		Title: "Headers",
-		"title": "Headers",
+		Data:  representMapSliceAsTable(request["_headers"].([]interface{}), `request.headers`),
-		"data":  string(headers),
 	})
 
-	cookies, _ := json.Marshal(request["cookies"].([]interface{}))
+	repRequest = append(repRequest, api.SectionData{
-	repRequest = append(repRequest, map[string]string{
+		Type:  api.TABLE,
-		"type":  api.TABLE,
+		Title: "Cookies",
-		"title": "Cookies",
+		Data:  representMapSliceAsTable(request["_cookies"].([]interface{}), `request.cookies`),
-		"data":  string(cookies),
 	})
 
-	queryString, _ := json.Marshal(request["queryString"].([]interface{}))
+	repRequest = append(repRequest, api.SectionData{
-	repRequest = append(repRequest, map[string]string{
+		Type:  api.TABLE,
-		"type":  api.TABLE,
+		Title: "Query String",
-		"title": "Query String",
+		Data:  representMapSliceAsTable(request["_queryString"].([]interface{}), `request.queryString`),
-		"data":  string(queryString),
 	})
 
 	postData, _ := request["postData"].(map[string]interface{})
@@ -266,12 +303,12 @@ func representRequest(request map[string]interface{}) (repRequest []interface{})
 	}
 	text, _ := postData["text"]
 	if text != nil {
-		repRequest = append(repRequest, map[string]string{
+		repRequest = append(repRequest, api.SectionData{
-			"type":      api.BODY,
+			Type:     api.BODY,
-			"title":     "POST Data (text/plain)",
+			Title:    "POST Data (text/plain)",
-			"encoding":  "",
+			MimeType: mimeType.(string),
-			"mime_type": mimeType.(string),
+			Data:     text.(string),
-			"data":      text.(string),
+			Selector: `request.postData.text`,
 		})
 	}
 
@@ -285,16 +322,16 @@ func representRequest(request map[string]interface{}) (repRequest []interface{})
 						"value": string(params),
 					},
 				})
-				repRequest = append(repRequest, map[string]string{
+				repRequest = append(repRequest, api.SectionData{
-					"type":  api.TABLE,
+					Type:  api.TABLE,
-					"title": "POST Data (multipart/form-data)",
+					Title: "POST Data (multipart/form-data)",
-					"data":  string(multipart),
+					Data:  string(multipart),
 				})
 			} else {
-				repRequest = append(repRequest, map[string]string{
+				repRequest = append(repRequest, api.SectionData{
-					"type":  api.TABLE,
+					Type:  api.TABLE,
-					"title": "POST Data (application/x-www-form-urlencoded)",
+					Title: "POST Data (application/x-www-form-urlencoded)",
-					"data":  string(params),
+					Data:  representMapSliceAsTable(postData["params"].([]interface{}), `request.postData.params`),
 				})
 			}
 		}
@@ -308,38 +345,39 @@ func representResponse(response map[string]interface{}) (repResponse []interface
 
 	bodySize = int64(response["bodySize"].(float64))
 
-	details, _ := json.Marshal([]map[string]string{
+	details, _ := json.Marshal([]api.TableData{
 		{
-			"name":  "Status",
+			Name:     "Status",
-			"value": fmt.Sprintf("%g", response["status"].(float64)),
+			Value:    int64(response["status"].(float64)),
+			Selector: `response.status`,
 		},
 		{
-			"name":  "Status Text",
+			Name:     "Status Text",
-			"value": response["statusText"].(string),
+			Value:    response["statusText"].(string),
+			Selector: `response.statusText`,
 		},
 		{
-			"name":  "Body Size",
+			Name:     "Body Size (bytes)",
-			"value": fmt.Sprintf("%d bytes", bodySize),
+			Value:    bodySize,
+			Selector: `response.bodySize`,
 		},
 	})
-	repResponse = append(repResponse, map[string]string{
+	repResponse = append(repResponse, api.SectionData{
-		"type":  api.TABLE,
+		Type:  api.TABLE,
-		"title": "Details",
+		Title: "Details",
-		"data":  string(details),
+		Data:  string(details),
 	})
 
-	headers, _ := json.Marshal(response["headers"].([]interface{}))
+	repResponse = append(repResponse, api.SectionData{
-	repResponse = append(repResponse, map[string]string{
+		Type:  api.TABLE,
-		"type":  api.TABLE,
+		Title: "Headers",
-		"title": "Headers",
+		Data:  representMapSliceAsTable(response["_headers"].([]interface{}), `response.headers`),
-		"data":  string(headers),
 	})
 
-	cookies, _ := json.Marshal(response["cookies"].([]interface{}))
+	repResponse = append(repResponse, api.SectionData{
-	repResponse = append(repResponse, map[string]string{
+		Type:  api.TABLE,
-		"type":  api.TABLE,
+		Title: "Cookies",
-		"title": "Cookies",
+		Data:  representMapSliceAsTable(response["_cookies"].([]interface{}), `response.cookies`),
-		"data":  string(cookies),
 	})
 
 	content, _ := response["content"].(map[string]interface{})
@@ -350,37 +388,40 @@ func representResponse(response map[string]interface{}) (repResponse []interface
 	encoding, _ := content["encoding"]
 	text, _ := content["text"]
 	if text != nil {
-		repResponse = append(repResponse, map[string]string{
+		repResponse = append(repResponse, api.SectionData{
-			"type":      api.BODY,
+			Type:     api.BODY,
-			"title":     "Body",
+			Title:    "Body",
-			"encoding":  encoding.(string),
+			Encoding: encoding.(string),
-			"mime_type": mimeType.(string),
+			MimeType: mimeType.(string),
-			"data":      text.(string),
+			Data:     text.(string),
+			Selector: `response.content.text`,
 		})
 	}
 
 	return
 }
 
-func (d dissecting) Represent(entry *api.MizuEntry) (p api.Protocol, object []byte, bodySize int64, err error) {
+func (d dissecting) Represent(protoIn api.Protocol, request map[string]interface{}, response map[string]interface{}) (protoOut api.Protocol, object []byte, bodySize int64, err error) {
-	if entry.ProtocolVersion == "2.0" {
+	if protoIn.Version == "2.0" {
-		p = http2Protocol
+		protoOut = http2Protocol
 	} else {
-		p = protocol
+		protoOut = protocol
 	}
-	var root map[string]interface{}
-	json.Unmarshal([]byte(entry.Entry), &root)
 	representation := make(map[string]interface{}, 0)
-	request := root["request"].(map[string]interface{})["payload"].(map[string]interface{})
+	repRequest := representRequest(request)
-	response := root["response"].(map[string]interface{})["payload"].(map[string]interface{})
+	repResponse, bodySize := representResponse(response)
-	reqDetails := request["details"].(map[string]interface{})
-	resDetails := response["details"].(map[string]interface{})
-	repRequest := representRequest(reqDetails)
-	repResponse, bodySize := representResponse(resDetails)
 	representation["request"] = repRequest
 	representation["response"] = repResponse
 	object, err = json.Marshal(representation)
 	return
 }
 
+func (d dissecting) Macros() map[string]string {
+	return map[string]string{
+		`http`:  fmt.Sprintf(`proto.abbr == "%s"`, protocol.Abbreviation),
+		`grpc`:  fmt.Sprintf(`proto.abbr == "%s" and proto.version == "%s"`, protocol.Abbreviation, http2Protocol.Version),
+		`http2`: fmt.Sprintf(`proto.abbr == "%s" and proto.version == "%s"`, protocol.Abbreviation, http2Protocol.Version),
+	}
+}
+
 var Dissector dissecting

tap/extensions/kafka/helpers.go

@@ -27,48 +27,54 @@ type KafkaWrapper struct {
 }
 
 func representRequestHeader(data map[string]interface{}, rep []interface{}) []interface{} {
-	requestHeader, _ := json.Marshal([]map[string]string{
+	requestHeader, _ := json.Marshal([]api.TableData{
 		{
-			"name":  "ApiKey",
+			Name:     "ApiKey",
-			"value": apiNames[int(data["ApiKey"].(float64))],
+			Value:    apiNames[int(data["apiKey"].(float64))],
+			Selector: `request.apiKey`,
 		},
 		{
-			"name":  "ApiVersion",
+			Name:     "ApiVersion",
-			"value": fmt.Sprintf("%d", int(data["ApiVersion"].(float64))),
+			Value:    fmt.Sprintf("%d", int(data["apiVersion"].(float64))),
+			Selector: `request.apiVersion`,
 		},
 		{
-			"name":  "Client ID",
+			Name:     "Client ID",
-			"value": data["ClientID"].(string),
+			Value:    data["clientID"].(string),
+			Selector: `request.clientID`,
 		},
 		{
-			"name":  "Correlation ID",
+			Name:     "Correlation ID",
-			"value": fmt.Sprintf("%d", int(data["CorrelationID"].(float64))),
+			Value:    fmt.Sprintf("%d", int(data["correlationID"].(float64))),
+			Selector: `request.correlationID`,
 		},
 		{
-			"name":  "Size",
+			Name:     "Size",
-			"value": fmt.Sprintf("%d", int(data["Size"].(float64))),
+			Value:    fmt.Sprintf("%d", int(data["size"].(float64))),
+			Selector: `request.size`,
 		},
 	})
-	rep = append(rep, map[string]string{
+	rep = append(rep, api.SectionData{
-		"type":  api.TABLE,
+		Type:  api.TABLE,
-		"title": "Request Header",
+		Title: "Request Header",
-		"data":  string(requestHeader),
+		Data:  string(requestHeader),
 	})
 
 	return rep
 }
 
 func representResponseHeader(data map[string]interface{}, rep []interface{}) []interface{} {
-	requestHeader, _ := json.Marshal([]map[string]string{
+	requestHeader, _ := json.Marshal([]api.TableData{
 		{
-			"name":  "Correlation ID",
+			Name:     "Correlation ID",
-			"value": fmt.Sprintf("%d", int(data["CorrelationID"].(float64))),
+			Value:    fmt.Sprintf("%d", int(data["correlationID"].(float64))),
+			Selector: `response.correlationID`,
 		},
 	})
-	rep = append(rep, map[string]string{
+	rep = append(rep, api.SectionData{
-		"type":  api.TABLE,
+		Type:  api.TABLE,
-		"title": "Response Header",
+		Title: "Response Header",
-		"data":  string(requestHeader),
+		Data:  string(requestHeader),
 	})
 
 	return rep
@@ -79,46 +85,50 @@ func representMetadataRequest(data map[string]interface{}) []interface{} {
 
 	rep = representRequestHeader(data, rep)
 
-	payload := data["Payload"].(map[string]interface{})
+	payload := data["payload"].(map[string]interface{})
 	topics := ""
 	allowAutoTopicCreation := ""
 	includeClusterAuthorizedOperations := ""
 	includeTopicAuthorizedOperations := ""
-	if payload["Topics"] != nil {
+	if payload["topics"] != nil {
-		x, _ := json.Marshal(payload["Topics"].([]interface{}))
+		x, _ := json.Marshal(payload["topics"].([]interface{}))
 		topics = string(x)
 	}
-	if payload["AllowAutoTopicCreation"] != nil {
+	if payload["allowAutoTopicCreation"] != nil {
-		allowAutoTopicCreation = strconv.FormatBool(payload["AllowAutoTopicCreation"].(bool))
+		allowAutoTopicCreation = strconv.FormatBool(payload["allowAutoTopicCreation"].(bool))
 	}
-	if payload["IncludeClusterAuthorizedOperations"] != nil {
+	if payload["includeClusterAuthorizedOperations"] != nil {
-		includeClusterAuthorizedOperations = strconv.FormatBool(payload["IncludeClusterAuthorizedOperations"].(bool))
+		includeClusterAuthorizedOperations = strconv.FormatBool(payload["includeClusterAuthorizedOperations"].(bool))
 	}
-	if payload["IncludeTopicAuthorizedOperations"] != nil {
+	if payload["includeTopicAuthorizedOperations"] != nil {
-		includeTopicAuthorizedOperations = strconv.FormatBool(payload["IncludeTopicAuthorizedOperations"].(bool))
+		includeTopicAuthorizedOperations = strconv.FormatBool(payload["includeTopicAuthorizedOperations"].(bool))
 	}
-	repPayload, _ := json.Marshal([]map[string]string{
+	repPayload, _ := json.Marshal([]api.TableData{
 		{
-			"name":  "Topics",
+			Name:     "Topics",
-			"value": topics,
+			Value:    topics,
+			Selector: `request.payload.topics`,
 		},
 		{
-			"name":  "Allow Auto Topic Creation",
+			Name:     "Allow Auto Topic Creation",
-			"value": allowAutoTopicCreation,
+			Value:    allowAutoTopicCreation,
+			Selector: `request.payload.allowAutoTopicCreation`,
 		},
 		{
-			"name":  "Include Cluster Authorized Operations",
+			Name:     "Include Cluster Authorized Operations",
-			"value": includeClusterAuthorizedOperations,
+			Value:    includeClusterAuthorizedOperations,
+			Selector: `request.payload.includeClusterAuthorizedOperations`,
 		},
 		{
-			"name":  "Include Topic Authorized Operations",
+			Name:     "Include Topic Authorized Operations",
-			"value": includeTopicAuthorizedOperations,
+			Value:    includeTopicAuthorizedOperations,
+			Selector: `request.payload.includeTopicAuthorizedOperations`,
 		},
 	})
-	rep = append(rep, map[string]string{
+	rep = append(rep, api.SectionData{
-		"type":  api.TABLE,
+		Type:  api.TABLE,
-		"title": "Payload",
+		Title: "Payload",
-		"data":  string(repPayload),
+		Data:  string(repPayload),
 	})
 
 	return rep
@@ -129,63 +139,69 @@ func representMetadataResponse(data map[string]interface{}) []interface{} {
 
 	rep = representResponseHeader(data, rep)
 
-	payload := data["Payload"].(map[string]interface{})
+	payload := data["payload"].(map[string]interface{})
 	topics := ""
-	if payload["Topics"] != nil {
+	if payload["topics"] != nil {
-		_topics, _ := json.Marshal(payload["Topics"].([]interface{}))
+		_topics, _ := json.Marshal(payload["topics"].([]interface{}))
 		topics = string(_topics)
 	}
 	brokers := ""
-	if payload["Brokers"] != nil {
+	if payload["brokers"] != nil {
-		_brokers, _ := json.Marshal(payload["Brokers"].([]interface{}))
+		_brokers, _ := json.Marshal(payload["brokers"].([]interface{}))
 		brokers = string(_brokers)
 	}
 	controllerID := ""
 	clusterID := ""
 	throttleTimeMs := ""
 	clusterAuthorizedOperations := ""
-	if payload["ControllerID"] != nil {
+	if payload["controllerID"] != nil {
-		controllerID = fmt.Sprintf("%d", int(payload["ControllerID"].(float64)))
+		controllerID = fmt.Sprintf("%d", int(payload["controllerID"].(float64)))
 	}
-	if payload["ClusterID"] != nil {
+	if payload["clusterID"] != nil {
-		clusterID = payload["ClusterID"].(string)
+		clusterID = payload["clusterID"].(string)
 	}
-	if payload["ThrottleTimeMs"] != nil {
+	if payload["throttleTimeMs"] != nil {
-		throttleTimeMs = fmt.Sprintf("%d", int(payload["ThrottleTimeMs"].(float64)))
+		throttleTimeMs = fmt.Sprintf("%d", int(payload["throttleTimeMs"].(float64)))
 	}
-	if payload["ClusterAuthorizedOperations"] != nil {
+	if payload["clusterAuthorizedOperations"] != nil {
-		clusterAuthorizedOperations = fmt.Sprintf("%d", int(payload["ClusterAuthorizedOperations"].(float64)))
+		clusterAuthorizedOperations = fmt.Sprintf("%d", int(payload["clusterAuthorizedOperations"].(float64)))
 	}
-	repPayload, _ := json.Marshal([]map[string]string{
+	repPayload, _ := json.Marshal([]api.TableData{
 		{
-			"name":  "Throttle Time (ms)",
+			Name:     "Throttle Time (ms)",
-			"value": throttleTimeMs,
+			Value:    throttleTimeMs,
+			Selector: `response.payload.throttleTimeMs`,
 		},
 		{
-			"name":  "Brokers",
+			Name:     "Brokers",
-			"value": brokers,
+			Value:    brokers,
+			Selector: `response.payload.brokers`,
 		},
 		{
-			"name":  "Cluster ID",
+			Name:     "Cluster ID",
-			"value": clusterID,
+			Value:    clusterID,
+			Selector: `response.payload.clusterID`,
 		},
 		{
-			"name":  "Controller ID",
+			Name:     "Controller ID",
-			"value": controllerID,
+			Value:    controllerID,
+			Selector: `response.payload.controllerID`,
 		},
 		{
-			"name":  "Topics",
+			Name:     "Topics",
-			"value": topics,
+			Value:    topics,
+			Selector: `response.payload.topics`,
 		},
 		{
-			"name":  "Cluster Authorized Operations",
+			Name:     "Cluster Authorized Operations",
-			"value": clusterAuthorizedOperations,
+			Value:    clusterAuthorizedOperations,
+			Selector: `response.payload.clusterAuthorizedOperations`,
 		},
 	})
-	rep = append(rep, map[string]string{
+	rep = append(rep, api.SectionData{
-		"type":  api.TABLE,
+		Type:  api.TABLE,
-		"title": "Payload",
+		Title: "Payload",
-		"data":  string(repPayload),
+		Data:  string(repPayload),
 	})
 
 	return rep
@@ -196,29 +212,31 @@ func representApiVersionsRequest(data map[string]interface{}) []interface{} {
 
 	rep = representRequestHeader(data, rep)
 
-	payload := data["Payload"].(map[string]interface{})
+	payload := data["payload"].(map[string]interface{})
 	clientSoftwareName := ""
 	clientSoftwareVersion := ""
-	if payload["ClientSoftwareName"] != nil {
+	if payload["clientSoftwareName"] != nil {
-		clientSoftwareName = payload["ClientSoftwareName"].(string)
+		clientSoftwareName = payload["clientSoftwareName"].(string)
 	}
-	if payload["ClientSoftwareVersion"] != nil {
+	if payload["clientSoftwareVersion"] != nil {
-		clientSoftwareVersion = payload["ClientSoftwareVersion"].(string)
+		clientSoftwareVersion = payload["clientSoftwareVersion"].(string)
 	}
-	repPayload, _ := json.Marshal([]map[string]string{
+	repPayload, _ := json.Marshal([]api.TableData{
 		{
-			"name":  "Client Software Name",
+			Name:     "Client Software Name",
-			"value": clientSoftwareName,
+			Value:    clientSoftwareName,
+			Selector: `request.payload.clientSoftwareName`,
 		},
 		{
-			"name":  "Client Software Version",
+			Name:     "Client Software Version",
-			"value": clientSoftwareVersion,
+			Value:    clientSoftwareVersion,
+			Selector: `request.payload.clientSoftwareVersion`,
 		},
 	})
-	rep = append(rep, map[string]string{
+	rep = append(rep, api.SectionData{
-		"type":  api.TABLE,
+		Type:  api.TABLE,
-		"title": "Payload",
+		Title: "Payload",
-		"data":  string(repPayload),
+		Data:  string(repPayload),
 	})
 
 	return rep
@@ -229,34 +247,37 @@ func representApiVersionsResponse(data map[string]interface{}) []interface{} {
 
 	rep = representResponseHeader(data, rep)
 
-	payload := data["Payload"].(map[string]interface{})
+	payload := data["payload"].(map[string]interface{})
 	apiKeys := ""
-	if payload["TopicNames"] != nil {
+	if payload["apiKeys"] != nil {
-		x, _ := json.Marshal(payload["ApiKeys"].([]interface{}))
+		x, _ := json.Marshal(payload["apiKeys"].([]interface{}))
 		apiKeys = string(x)
 	}
 	throttleTimeMs := ""
-	if payload["ThrottleTimeMs"] != nil {
+	if payload["throttleTimeMs"] != nil {
-		throttleTimeMs = fmt.Sprintf("%d", int(payload["ThrottleTimeMs"].(float64)))
+		throttleTimeMs = fmt.Sprintf("%d", int(payload["throttleTimeMs"].(float64)))
 	}
-	repPayload, _ := json.Marshal([]map[string]string{
+	repPayload, _ := json.Marshal([]api.TableData{
 		{
-			"name":  "Error Code",
+			Name:     "Error Code",
-			"value": fmt.Sprintf("%d", int(payload["ErrorCode"].(float64))),
+			Value:    fmt.Sprintf("%d", int(payload["errorCode"].(float64))),
+			Selector: `response.payload.errorCode`,
 		},
 		{
-			"name":  "ApiKeys",
+			Name:     "ApiKeys",
-			"value": apiKeys,
+			Value:    apiKeys,
+			Selector: `response.payload.apiKeys`,
 		},
 		{
-			"name":  "Throttle Time (ms)",
+			Name:     "Throttle Time (ms)",
-			"value": throttleTimeMs,
+			Value:    throttleTimeMs,
+			Selector: `response.payload.throttleTimeMs`,
 		},
 	})
-	rep = append(rep, map[string]string{
+	rep = append(rep, api.SectionData{
-		"type":  api.TABLE,
+		Type:  api.TABLE,
-		"title": "Payload",
+		Title: "Payload",
-		"data":  string(repPayload),
+		Data:  string(repPayload),
 	})
 
 	return rep
@@ -267,39 +288,43 @@ func representProduceRequest(data map[string]interface{}) []interface{} {
 
 	rep = representRequestHeader(data, rep)
 
-	payload := data["Payload"].(map[string]interface{})
+	payload := data["payload"].(map[string]interface{})
 	topicData := ""
-	_topicData := payload["TopicData"]
+	_topicData := payload["topicData"]
 	if _topicData != nil {
 		x, _ := json.Marshal(_topicData.([]interface{}))
 		topicData = string(x)
 	}
 	transactionalID := ""
-	if payload["TransactionalID"] != nil {
+	if payload["transactionalID"] != nil {
-		transactionalID = payload["TransactionalID"].(string)
+		transactionalID = payload["transactionalID"].(string)
 	}
-	repPayload, _ := json.Marshal([]map[string]string{
+	repPayload, _ := json.Marshal([]api.TableData{
 		{
-			"name":  "Transactional ID",
+			Name:     "Transactional ID",
-			"value": transactionalID,
+			Value:    transactionalID,
+			Selector: `request.payload.transactionalID`,
 		},
 		{
-			"name":  "Required Acknowledgements",
+			Name:     "Required Acknowledgements",
-			"value": fmt.Sprintf("%d", int(payload["RequiredAcks"].(float64))),
+			Value:    fmt.Sprintf("%d", int(payload["requiredAcks"].(float64))),
+			Selector: `request.payload.requiredAcks`,
 		},
 		{
-			"name":  "Timeout",
+			Name:     "Timeout",
-			"value": fmt.Sprintf("%d", int(payload["Timeout"].(float64))),
+			Value:    fmt.Sprintf("%d", int(payload["timeout"].(float64))),
+			Selector: `request.payload.timeout`,
 		},
 		{
-			"name":  "Topic Data",
+			Name:     "Topic Data",
-			"value": topicData,
+			Value:    topicData,
+			Selector: `request.payload.topicData`,
 		},
 	})
-	rep = append(rep, map[string]string{
+	rep = append(rep, api.SectionData{
-		"type":  api.TABLE,
+		Type:  api.TABLE,
-		"title": "Payload",
+		Title: "Payload",
-		"data":  string(repPayload),
+		Data:  string(repPayload),
 	})
 
 	return rep
@@ -310,30 +335,32 @@ func representProduceResponse(data map[string]interface{}) []interface{} {
 
 	rep = representResponseHeader(data, rep)
 
-	payload := data["Payload"].(map[string]interface{})
+	payload := data["payload"].(map[string]interface{})
 	responses := ""
-	if payload["Responses"] != nil {
+	if payload["responses"] != nil {
-		_responses, _ := json.Marshal(payload["Responses"].([]interface{}))
+		_responses, _ := json.Marshal(payload["responses"].([]interface{}))
 		responses = string(_responses)
 	}
 	throttleTimeMs := ""
-	if payload["ThrottleTimeMs"] != nil {
+	if payload["throttleTimeMs"] != nil {
-		throttleTimeMs = fmt.Sprintf("%d", int(payload["ThrottleTimeMs"].(float64)))
+		throttleTimeMs = fmt.Sprintf("%d", int(payload["throttleTimeMs"].(float64)))
 	}
-	repPayload, _ := json.Marshal([]map[string]string{
+	repPayload, _ := json.Marshal([]api.TableData{
 		{
-			"name":  "Responses",
+			Name:     "Responses",
-			"value": string(responses),
+			Value:    string(responses),
+			Selector: `response.payload.responses`,
 		},
 		{
-			"name":  "Throttle Time (ms)",
+			Name:     "Throttle Time (ms)",
-			"value": throttleTimeMs,
+			Value:    throttleTimeMs,
+			Selector: `response.payload.throttleTimeMs`,
 		},
 	})
-	rep = append(rep, map[string]string{
+	rep = append(rep, api.SectionData{
-		"type":  api.TABLE,
+		Type:  api.TABLE,
-		"title": "Payload",
+		Title: "Payload",
-		"data":  string(repPayload),
+		Data:  string(repPayload),
 	})
 
 	return rep
@@ -344,87 +371,97 @@ func representFetchRequest(data map[string]interface{}) []interface{} {
 
 	rep = representRequestHeader(data, rep)
 
-	payload := data["Payload"].(map[string]interface{})
+	payload := data["payload"].(map[string]interface{})
 	topics := ""
-	if payload["Topics"] != nil {
+	if payload["topics"] != nil {
-		_topics, _ := json.Marshal(payload["Topics"].([]interface{}))
+		_topics, _ := json.Marshal(payload["topics"].([]interface{}))
 		topics = string(_topics)
 	}
 	replicaId := ""
-	if payload["ReplicaId"] != nil {
+	if payload["replicaId"] != nil {
-		replicaId = fmt.Sprintf("%d", int(payload["ReplicaId"].(float64)))
+		replicaId = fmt.Sprintf("%d", int(payload["replicaId"].(float64)))
 	}
 	maxBytes := ""
-	if payload["MaxBytes"] != nil {
+	if payload["maxBytes"] != nil {
-		maxBytes = fmt.Sprintf("%d", int(payload["MaxBytes"].(float64)))
+		maxBytes = fmt.Sprintf("%d", int(payload["maxBytes"].(float64)))
 	}
 	isolationLevel := ""
-	if payload["IsolationLevel"] != nil {
+	if payload["isolationLevel"] != nil {
-		isolationLevel = fmt.Sprintf("%d", int(payload["IsolationLevel"].(float64)))
+		isolationLevel = fmt.Sprintf("%d", int(payload["isolationLevel"].(float64)))
 	}
 	sessionId := ""
-	if payload["SessionId"] != nil {
+	if payload["sessionId"] != nil {
-		sessionId = fmt.Sprintf("%d", int(payload["SessionId"].(float64)))
+		sessionId = fmt.Sprintf("%d", int(payload["sessionId"].(float64)))
 	}
 	sessionEpoch := ""
-	if payload["SessionEpoch"] != nil {
+	if payload["sessionEpoch"] != nil {
-		sessionEpoch = fmt.Sprintf("%d", int(payload["SessionEpoch"].(float64)))
+		sessionEpoch = fmt.Sprintf("%d", int(payload["sessionEpoch"].(float64)))
 	}
 	forgottenTopicsData := ""
-	if payload["ForgottenTopicsData"] != nil {
+	if payload["forgottenTopicsData"] != nil {
-		x, _ := json.Marshal(payload["ForgottenTopicsData"].(map[string]interface{}))
+		x, _ := json.Marshal(payload["forgottenTopicsData"].(map[string]interface{}))
 		forgottenTopicsData = string(x)
 	}
 	rackId := ""
-	if payload["RackId"] != nil {
+	if payload["rackId"] != nil {
-		rackId = payload["RackId"].(string)
+		rackId = payload["rackId"].(string)
 	}
-	repPayload, _ := json.Marshal([]map[string]string{
+	repPayload, _ := json.Marshal([]api.TableData{
 		{
-			"name":  "Replica ID",
+			Name:     "Replica ID",
-			"value": replicaId,
+			Value:    replicaId,
+			Selector: `request.payload.replicaId`,
 		},
 		{
-			"name":  "Maximum Wait (ms)",
+			Name:     "Maximum Wait (ms)",
-			"value": fmt.Sprintf("%d", int(payload["MaxWaitMs"].(float64))),
+			Value:    fmt.Sprintf("%d", int(payload["maxWaitMs"].(float64))),
+			Selector: `request.payload.maxWaitMs`,
 		},
 		{
-			"name":  "Minimum Bytes",
+			Name:     "Minimum Bytes",
-			"value": fmt.Sprintf("%d", int(payload["MinBytes"].(float64))),
+			Value:    fmt.Sprintf("%d", int(payload["minBytes"].(float64))),
+			Selector: `request.payload.minBytes`,
 		},
 		{
-			"name":  "Maximum Bytes",
+			Name:     "Maximum Bytes",
-			"value": maxBytes,
+			Value:    maxBytes,
+			Selector: `request.payload.maxBytes`,
 		},
 		{
-			"name":  "Isolation Level",
+			Name:     "Isolation Level",
-			"value": isolationLevel,
+			Value:    isolationLevel,
+			Selector: `request.payload.isolationLevel`,
 		},
 		{
-			"name":  "Session ID",
+			Name:     "Session ID",
-			"value": sessionId,
+			Value:    sessionId,
+			Selector: `request.payload.sessionId`,
 		},
 		{
-			"name":  "Session Epoch",
+			Name:     "Session Epoch",
-			"value": sessionEpoch,
+			Value:    sessionEpoch,
+			Selector: `request.payload.sessionEpoch`,
 		},
 		{
-			"name":  "Topics",
+			Name:     "Topics",
-			"value": topics,
+			Value:    topics,
+			Selector: `request.payload.topics`,
 		},
 		{
-			"name":  "Forgotten Topics Data",
+			Name:     "Forgotten Topics Data",
-			"value": forgottenTopicsData,
+			Value:    forgottenTopicsData,
+			Selector: `request.payload.forgottenTopicsData`,
 		},
 		{
-			"name":  "Rack ID",
+			Name:     "Rack ID",
-			"value": rackId,
+			Value:    rackId,
+			Selector: `request.payload.rackId`,
 		},
 	})
-	rep = append(rep, map[string]string{
+	rep = append(rep, api.SectionData{
-		"type":  api.TABLE,
+		Type:  api.TABLE,
-		"title": "Payload",
+		Title: "Payload",
-		"data":  string(repPayload),
+		Data:  string(repPayload),
 	})
 
 	return rep
@@ -435,46 +472,50 @@ func representFetchResponse(data map[string]interface{}) []interface{} {
 
 	rep = representResponseHeader(data, rep)
 
-	payload := data["Payload"].(map[string]interface{})
+	payload := data["payload"].(map[string]interface{})
 	responses := ""
-	if payload["Responses"] != nil {
+	if payload["responses"] != nil {
-		_responses, _ := json.Marshal(payload["Responses"].([]interface{}))
+		_responses, _ := json.Marshal(payload["responses"].([]interface{}))
 		responses = string(_responses)
 	}
 	throttleTimeMs := ""
-	if payload["ThrottleTimeMs"] != nil {
+	if payload["throttleTimeMs"] != nil {
-		throttleTimeMs = fmt.Sprintf("%d", int(payload["ThrottleTimeMs"].(float64)))
+		throttleTimeMs = fmt.Sprintf("%d", int(payload["throttleTimeMs"].(float64)))
 	}
 	errorCode := ""
-	if payload["ErrorCode"] != nil {
+	if payload["errorCode"] != nil {
-		errorCode = fmt.Sprintf("%d", int(payload["ErrorCode"].(float64)))
+		errorCode = fmt.Sprintf("%d", int(payload["errorCode"].(float64)))
 	}
 	sessionId := ""
-	if payload["SessionId"] != nil {
+	if payload["sessionId"] != nil {
-		sessionId = fmt.Sprintf("%d", int(payload["SessionId"].(float64)))
+		sessionId = fmt.Sprintf("%d", int(payload["sessionId"].(float64)))
 	}
-	repPayload, _ := json.Marshal([]map[string]string{
+	repPayload, _ := json.Marshal([]api.TableData{
 		{
-			"name":  "Throttle Time (ms)",
+			Name:     "Throttle Time (ms)",
-			"value": throttleTimeMs,
+			Value:    throttleTimeMs,
+			Selector: `response.payload.throttleTimeMs`,
 		},
 		{
-			"name":  "Error Code",
+			Name:     "Error Code",
-			"value": errorCode,
+			Value:    errorCode,
+			Selector: `response.payload.errorCode`,
 		},
 		{
-			"name":  "Session ID",
+			Name:     "Session ID",
-			"value": sessionId,
+			Value:    sessionId,
+			Selector: `response.payload.sessionId`,
 		},
 		{
-			"name":  "Responses",
+			Name:     "Responses",
-			"value": responses,
+			Value:    responses,
+			Selector: `response.payload.responses`,
 		},
 	})
-	rep = append(rep, map[string]string{
+	rep = append(rep, api.SectionData{
-		"type":  api.TABLE,
+		Type:  api.TABLE,
-		"title": "Payload",
+		Title: "Payload",
-		"data":  string(repPayload),
+		Data:  string(repPayload),
 	})
 
 	return rep
@@ -485,26 +526,28 @@ func representListOffsetsRequest(data map[string]interface{}) []interface{} {
 
 	rep = representRequestHeader(data, rep)
 
-	payload := data["Payload"].(map[string]interface{})
+	payload := data["payload"].(map[string]interface{})
 	topics := ""
-	if payload["Topics"] != nil {
+	if payload["topics"] != nil {
-		_topics, _ := json.Marshal(payload["Topics"].([]interface{}))
+		_topics, _ := json.Marshal(payload["topics"].([]interface{}))
 		topics = string(_topics)
 	}
-	repPayload, _ := json.Marshal([]map[string]string{
+	repPayload, _ := json.Marshal([]api.TableData{
 		{
-			"name":  "Replica ID",
+			Name:     "Replica ID",
-			"value": fmt.Sprintf("%d", int(payload["ReplicaId"].(float64))),
+			Value:    fmt.Sprintf("%d", int(payload["replicaId"].(float64))),
+			Selector: `request.payload.replicaId`,
 		},
 		{
-			"name":  "Topics",
+			Name:     "Topics",
-			"value": topics,
+			Value:    topics,
+			Selector: `request.payload.topics`,
 		},
 	})
-	rep = append(rep, map[string]string{
+	rep = append(rep, api.SectionData{
-		"type":  api.TABLE,
+		Type:  api.TABLE,
-		"title": "Payload",
+		Title: "Payload",
-		"data":  string(repPayload),
+		Data:  string(repPayload),
 	})
 
 	return rep
@@ -515,26 +558,28 @@ func representListOffsetsResponse(data map[string]interface{}) []interface{} {
 
 	rep = representResponseHeader(data, rep)
 
-	payload := data["Payload"].(map[string]interface{})
+	payload := data["payload"].(map[string]interface{})
-	topics, _ := json.Marshal(payload["Topics"].([]interface{}))
+	topics, _ := json.Marshal(payload["topics"].([]interface{}))
 	throttleTimeMs := ""
-	if payload["ThrottleTimeMs"] != nil {
+	if payload["throttleTimeMs"] != nil {
-		throttleTimeMs = fmt.Sprintf("%d", int(payload["ThrottleTimeMs"].(float64)))
+		throttleTimeMs = fmt.Sprintf("%d", int(payload["throttleTimeMs"].(float64)))
 	}
-	repPayload, _ := json.Marshal([]map[string]string{
+	repPayload, _ := json.Marshal([]api.TableData{
 		{
-			"name":  "Throttle Time (ms)",
+			Name:     "Throttle Time (ms)",
-			"value": throttleTimeMs,
+			Value:    throttleTimeMs,
+			Selector: `response.payload.throttleTimeMs`,
 		},
 		{
-			"name":  "Topics",
+			Name:     "Topics",
-			"value": string(topics),
+			Value:    string(topics),
+			Selector: `response.payload.topics`,
 		},
 	})
-	rep = append(rep, map[string]string{
+	rep = append(rep, api.SectionData{
-		"type":  api.TABLE,
+		Type:  api.TABLE,
-		"title": "Payload",
+		Title: "Payload",
-		"data":  string(repPayload),
+		Data:  string(repPayload),
 	})
 
 	return rep
@@ -545,30 +590,33 @@ func representCreateTopicsRequest(data map[string]interface{}) []interface{} {
 
 	rep = representRequestHeader(data, rep)
 
-	payload := data["Payload"].(map[string]interface{})
+	payload := data["payload"].(map[string]interface{})
-	topics, _ := json.Marshal(payload["Topics"].([]interface{}))
+	topics, _ := json.Marshal(payload["topics"].([]interface{}))
 	validateOnly := ""
-	if payload["ValidateOnly"] != nil {
+	if payload["validateOnly"] != nil {
-		validateOnly = strconv.FormatBool(payload["ValidateOnly"].(bool))
+		validateOnly = strconv.FormatBool(payload["validateOnly"].(bool))
 	}
-	repPayload, _ := json.Marshal([]map[string]string{
+	repPayload, _ := json.Marshal([]api.TableData{
 		{
-			"name":  "Topics",
+			Name:     "Topics",
-			"value": string(topics),
+			Value:    string(topics),
+			Selector: `request.payload.topics`,
 		},
 		{
-			"name":  "Timeout (ms)",
+			Name:     "Timeout (ms)",
-			"value": fmt.Sprintf("%d", int(payload["TimeoutMs"].(float64))),
+			Value:    fmt.Sprintf("%d", int(payload["timeoutMs"].(float64))),
+			Selector: `request.payload.timeoutMs`,
 		},
 		{
-			"name":  "Validate Only",
+			Name:     "Validate Only",
-			"value": validateOnly,
+			Value:    validateOnly,
+			Selector: `request.payload.validateOnly`,
 		},
 	})
-	rep = append(rep, map[string]string{
+	rep = append(rep, api.SectionData{
-		"type":  api.TABLE,
+		Type:  api.TABLE,
-		"title": "Payload",
+		Title: "Payload",
-		"data":  string(repPayload),
+		Data:  string(repPayload),
 	})
 
 	return rep
@@ -579,26 +627,28 @@ func representCreateTopicsResponse(data map[string]interface{}) []interface{} {
 
 	rep = representResponseHeader(data, rep)
 
-	payload := data["Payload"].(map[string]interface{})
+	payload := data["payload"].(map[string]interface{})
-	topics, _ := json.Marshal(payload["Topics"].([]interface{}))
+	topics, _ := json.Marshal(payload["topics"].([]interface{}))
 	throttleTimeMs := ""
-	if payload["ThrottleTimeMs"] != nil {
+	if payload["throttleTimeMs"] != nil {
-		throttleTimeMs = fmt.Sprintf("%d", int(payload["ThrottleTimeMs"].(float64)))
+		throttleTimeMs = fmt.Sprintf("%d", int(payload["throttleTimeMs"].(float64)))
 	}
-	repPayload, _ := json.Marshal([]map[string]string{
+	repPayload, _ := json.Marshal([]api.TableData{
 		{
-			"name":  "Throttle Time (ms)",
+			Name:     "Throttle Time (ms)",
-			"value": throttleTimeMs,
+			Value:    throttleTimeMs,
+			Selector: `response.payload.throttleTimeMs`,
 		},
 		{
-			"name":  "Topics",
+			Name:     "Topics",
-			"value": string(topics),
+			Value:    string(topics),
+			Selector: `response.payload.topics`,
 		},
 	})
-	rep = append(rep, map[string]string{
+	rep = append(rep, api.SectionData{
-		"type":  api.TABLE,
+		Type:  api.TABLE,
-		"title": "Payload",
+		Title: "Payload",
-		"data":  string(repPayload),
+		Data:  string(repPayload),
 	})
 
 	return rep
@@ -609,35 +659,38 @@ func representDeleteTopicsRequest(data map[string]interface{}) []interface{} {
 
 	rep = representRequestHeader(data, rep)
 
-	payload := data["Payload"].(map[string]interface{})
+	payload := data["payload"].(map[string]interface{})
 	topics := ""
-	if payload["Topics"] != nil {
+	if payload["topics"] != nil {
-		x, _ := json.Marshal(payload["Topics"].([]interface{}))
+		x, _ := json.Marshal(payload["topics"].([]interface{}))
 		topics = string(x)
 	}
 	topicNames := ""
-	if payload["TopicNames"] != nil {
+	if payload["topicNames"] != nil {
-		x, _ := json.Marshal(payload["TopicNames"].([]interface{}))
+		x, _ := json.Marshal(payload["topicNames"].([]interface{}))
 		topicNames = string(x)
 	}
-	repPayload, _ := json.Marshal([]map[string]string{
+	repPayload, _ := json.Marshal([]api.TableData{
 		{
-			"name":  "TopicNames",
+			Name:     "TopicNames",
-			"value": string(topicNames),
+			Value:    string(topicNames),
+			Selector: `request.payload.topicNames`,
 		},
 		{
-			"name":  "Topics",
+			Name:     "Topics",
-			"value": string(topics),
+			Value:    string(topics),
+			Selector: `request.payload.topics`,
 		},
 		{
-			"name":  "Timeout (ms)",
+			Name:     "Timeout (ms)",
-			"value": fmt.Sprintf("%d", int(payload["TimeoutMs"].(float64))),
+			Value:    fmt.Sprintf("%d", int(payload["timeoutMs"].(float64))),
+			Selector: `request.payload.timeoutMs`,
 		},
 	})
-	rep = append(rep, map[string]string{
+	rep = append(rep, api.SectionData{
-		"type":  api.TABLE,
+		Type:  api.TABLE,
-		"title": "Payload",
+		Title: "Payload",
-		"data":  string(repPayload),
+		Data:  string(repPayload),
 	})
 
 	return rep
@@ -648,26 +701,28 @@ func representDeleteTopicsResponse(data map[string]interface{}) []interface{} {
 
 	rep = representResponseHeader(data, rep)
 
-	payload := data["Payload"].(map[string]interface{})
+	payload := data["payload"].(map[string]interface{})
-	responses, _ := json.Marshal(payload["Responses"].([]interface{}))
+	responses, _ := json.Marshal(payload["responses"].([]interface{}))
 	throttleTimeMs := ""
-	if payload["ThrottleTimeMs"] != nil {
+	if payload["throttleTimeMs"] != nil {
-		throttleTimeMs = fmt.Sprintf("%d", int(payload["ThrottleTimeMs"].(float64)))
+		throttleTimeMs = fmt.Sprintf("%d", int(payload["throttleTimeMs"].(float64)))
 	}
-	repPayload, _ := json.Marshal([]map[string]string{
+	repPayload, _ := json.Marshal([]api.TableData{
 		{
-			"name":  "Throttle Time (ms)",
+			Name:     "Throttle Time (ms)",
-			"value": throttleTimeMs,
+			Value:    throttleTimeMs,
+			Selector: `response.payload.throttleTimeMs`,
 		},
 		{
-			"name":  "Responses",
+			Name:     "Responses",
-			"value": string(responses),
+			Value:    string(responses),
+			Selector: `response.payload.responses`,
 		},
 	})
-	rep = append(rep, map[string]string{
+	rep = append(rep, api.SectionData{
-		"type":  api.TABLE,
+		Type:  api.TABLE,
-		"title": "Payload",
+		Title: "Payload",
-		"data":  string(repPayload),
+		Data:  string(repPayload),
 	})
 
 	return rep

tap/extensions/kafka/main.go

@@ -15,6 +15,7 @@ var _protocol api.Protocol = api.Protocol{
 	Name:            "kafka",
 	LongName:        "Apache Kafka Protocol",
 	Abbreviation:    "KAFKA",
+	Macro:           "kafka",
 	Version:         "12",
 	BackgroundColor: "#000000",
 	ForegroundColor: "#ffffff",
@@ -61,7 +62,7 @@ func (d dissecting) Dissect(b *bufio.Reader, isClient bool, tcpID *api.TcpID, co
 	}
 }
 
-func (d dissecting) Analyze(item *api.OutputChannelItem, entryId string, resolvedSource string, resolvedDestination string) *api.MizuEntry {
+func (d dissecting) Analyze(item *api.OutputChannelItem, resolvedSource string, resolvedDestination string) *api.MizuEntry {
 	request := item.Pair.Request.Payload.(map[string]interface{})
 	reqDetails := request["details"].(map[string]interface{})
 	service := "kafka"
@@ -70,76 +71,76 @@ func (d dissecting) Analyze(item *api.OutputChannelItem, entryId string, resolve
 	} else if resolvedSource != "" {
 		service = resolvedSource
 	}
-	apiKey := ApiKey(reqDetails["ApiKey"].(float64))
+	apiKey := ApiKey(reqDetails["apiKey"].(float64))
 
 	summary := ""
 	switch apiKey {
 	case Metadata:
-		_topics := reqDetails["Payload"].(map[string]interface{})["Topics"]
+		_topics := reqDetails["payload"].(map[string]interface{})["topics"]
 		if _topics == nil {
 			break
 		}
 		topics := _topics.([]interface{})
 		for _, topic := range topics {
-			summary += fmt.Sprintf("%s, ", topic.(map[string]interface{})["Name"].(string))
+			summary += fmt.Sprintf("%s, ", topic.(map[string]interface{})["name"].(string))
 		}
 		if len(summary) > 0 {
 			summary = summary[:len(summary)-2]
 		}
 		break
 	case ApiVersions:
-		summary = reqDetails["ClientID"].(string)
+		summary = reqDetails["clientID"].(string)
 		break
 	case Produce:
-		_topics := reqDetails["Payload"].(map[string]interface{})["TopicData"]
+		_topics := reqDetails["payload"].(map[string]interface{})["topicData"]
 		if _topics == nil {
 			break
 		}
 		topics := _topics.([]interface{})
 		for _, topic := range topics {
-			summary += fmt.Sprintf("%s, ", topic.(map[string]interface{})["Topic"].(string))
+			summary += fmt.Sprintf("%s, ", topic.(map[string]interface{})["topic"].(string))
 		}
 		if len(summary) > 0 {
 			summary = summary[:len(summary)-2]
 		}
 		break
 	case Fetch:
-		_topics := reqDetails["Payload"].(map[string]interface{})["Topics"]
+		_topics := reqDetails["payload"].(map[string]interface{})["topics"]
 		if _topics == nil {
 			break
 		}
 		topics := _topics.([]interface{})
 		for _, topic := range topics {
-			summary += fmt.Sprintf("%s, ", topic.(map[string]interface{})["Topic"].(string))
+			summary += fmt.Sprintf("%s, ", topic.(map[string]interface{})["topic"].(string))
 		}
 		if len(summary) > 0 {
 			summary = summary[:len(summary)-2]
 		}
 		break
 	case ListOffsets:
-		_topics := reqDetails["Payload"].(map[string]interface{})["Topics"]
+		_topics := reqDetails["payload"].(map[string]interface{})["topics"]
 		if _topics == nil {
 			break
 		}
 		topics := _topics.([]interface{})
 		for _, topic := range topics {
-			summary += fmt.Sprintf("%s, ", topic.(map[string]interface{})["Name"].(string))
+			summary += fmt.Sprintf("%s, ", topic.(map[string]interface{})["name"].(string))
 		}
 		if len(summary) > 0 {
 			summary = summary[:len(summary)-2]
 		}
 		break
 	case CreateTopics:
-		topics := reqDetails["Payload"].(map[string]interface{})["Topics"].([]interface{})
+		topics := reqDetails["payload"].(map[string]interface{})["topics"].([]interface{})
 		for _, topic := range topics {
-			summary += fmt.Sprintf("%s, ", topic.(map[string]interface{})["Name"].(string))
+			summary += fmt.Sprintf("%s, ", topic.(map[string]interface{})["name"].(string))
 		}
 		if len(summary) > 0 {
 			summary = summary[:len(summary)-2]
 		}
 		break
 	case DeleteTopics:
-		topicNames := reqDetails["TopicNames"].([]string)
+		topicNames := reqDetails["topicNames"].([]string)
 		for _, name := range topicNames {
 			summary += fmt.Sprintf("%s, ", name)
 		}
@@ -148,44 +149,48 @@ func (d dissecting) Analyze(item *api.OutputChannelItem, entryId string, resolve
 
 	request["url"] = summary
 	elapsedTime := item.Pair.Response.CaptureTime.Sub(item.Pair.Request.CaptureTime).Round(time.Millisecond).Milliseconds()
-	entryBytes, _ := json.Marshal(item.Pair)
 	return &api.MizuEntry{
-		ProtocolName:            _protocol.Name,
+		Protocol: _protocol,
-		ProtocolLongName:        _protocol.LongName,
+		Source: &api.TCP{
-		ProtocolAbbreviation:    _protocol.Abbreviation,
+			Name: resolvedSource,
-		ProtocolVersion:         _protocol.Version,
+			IP:   item.ConnectionInfo.ClientIP,
-		ProtocolBackgroundColor: _protocol.BackgroundColor,
+			Port: item.ConnectionInfo.ClientPort,
-		ProtocolForegroundColor: _protocol.ForegroundColor,
+		},
-		ProtocolFontSize:        _protocol.FontSize,
+		Destination: &api.TCP{
-		ProtocolReferenceLink:   _protocol.ReferenceLink,
+			Name: resolvedDestination,
-		EntryId:                 entryId,
+			IP:   item.ConnectionInfo.ServerIP,
-		Entry:                   string(entryBytes),
+			Port: item.ConnectionInfo.ServerPort,
-		Url:                     fmt.Sprintf("%s%s", service, summary),
+		},
-		Method:                  apiNames[apiKey],
+		Outgoing:            item.ConnectionInfo.IsOutgoing,
-		Status:                  0,
+		Request:             reqDetails,
-		RequestSenderIp:         item.ConnectionInfo.ClientIP,
+		Response:            item.Pair.Response.Payload.(map[string]interface{})["details"].(map[string]interface{}),
-		Service:                 service,
+		Url:                 fmt.Sprintf("%s%s", service, summary),
-		Timestamp:               item.Timestamp,
+		Method:              apiNames[apiKey],
-		ElapsedTime:             elapsedTime,
+		Status:              0,
-		Path:                    summary,
+		RequestSenderIp:     item.ConnectionInfo.ClientIP,
-		ResolvedSource:          resolvedSource,
+		Service:             service,
-		ResolvedDestination:     resolvedDestination,
+		Timestamp:           item.Timestamp,
-		SourceIp:                item.ConnectionInfo.ClientIP,
+		StartTime:           item.Pair.Request.CaptureTime,
-		DestinationIp:           item.ConnectionInfo.ServerIP,
+		ElapsedTime:         elapsedTime,
-		SourcePort:              item.ConnectionInfo.ClientPort,
+		Summary:             summary,
-		DestinationPort:         item.ConnectionInfo.ServerPort,
+		ResolvedSource:      resolvedSource,
-		IsOutgoing:              item.ConnectionInfo.IsOutgoing,
+		ResolvedDestination: resolvedDestination,
+		SourceIp:            item.ConnectionInfo.ClientIP,
+		DestinationIp:       item.ConnectionInfo.ServerIP,
+		SourcePort:          item.ConnectionInfo.ClientPort,
+		DestinationPort:     item.ConnectionInfo.ServerPort,
+		IsOutgoing:          item.ConnectionInfo.IsOutgoing,
 	}
 }
 
 func (d dissecting) Summarize(entry *api.MizuEntry) *api.BaseEntryDetails {
 	return &api.BaseEntryDetails{
-		Id:              entry.EntryId,
+		Id:              entry.Id,
 		Protocol:        _protocol,
 		Url:             entry.Url,
 		RequestSenderIp: entry.RequestSenderIp,
 		Service:         entry.Service,
-		Summary:         entry.Path,
+		Summary:         entry.Summary,
 		StatusCode:      entry.Status,
 		Method:          entry.Method,
 		Timestamp:       entry.Timestamp,
@@ -202,49 +207,43 @@ func (d dissecting) Summarize(entry *api.MizuEntry) *api.BaseEntryDetails {
 	}
 }
 
-func (d dissecting) Represent(entry *api.MizuEntry) (p api.Protocol, object []byte, bodySize int64, err error) {
+func (d dissecting) Represent(protoIn api.Protocol, request map[string]interface{}, response map[string]interface{}) (protoOut api.Protocol, object []byte, bodySize int64, err error) {
-	p = _protocol
+	protoOut = _protocol
 	bodySize = 0
-	var root map[string]interface{}
-	json.Unmarshal([]byte(entry.Entry), &root)
 	representation := make(map[string]interface{}, 0)
-	request := root["request"].(map[string]interface{})["payload"].(map[string]interface{})
-	response := root["response"].(map[string]interface{})["payload"].(map[string]interface{})
-	reqDetails := request["details"].(map[string]interface{})
-	resDetails := response["details"].(map[string]interface{})
 
-	apiKey := ApiKey(reqDetails["ApiKey"].(float64))
+	apiKey := ApiKey(request["apiKey"].(float64))
 
 	var repRequest []interface{}
 	var repResponse []interface{}
 	switch apiKey {
 	case Metadata:
-		repRequest = representMetadataRequest(reqDetails)
+		repRequest = representMetadataRequest(request)
-		repResponse = representMetadataResponse(resDetails)
+		repResponse = representMetadataResponse(response)
 		break
 	case ApiVersions:
-		repRequest = representApiVersionsRequest(reqDetails)
+		repRequest = representApiVersionsRequest(request)
-		repResponse = representApiVersionsResponse(resDetails)
+		repResponse = representApiVersionsResponse(response)
 		break
 	case Produce:
-		repRequest = representProduceRequest(reqDetails)
+		repRequest = representProduceRequest(request)
-		repResponse = representProduceResponse(resDetails)
+		repResponse = representProduceResponse(response)
 		break
 	case Fetch:
-		repRequest = representFetchRequest(reqDetails)
+		repRequest = representFetchRequest(request)
-		repResponse = representFetchResponse(resDetails)
+		repResponse = representFetchResponse(response)
 		break
 	case ListOffsets:
-		repRequest = representListOffsetsRequest(reqDetails)
+		repRequest = representListOffsetsRequest(request)
-		repResponse = representListOffsetsResponse(resDetails)
+		repResponse = representListOffsetsResponse(response)
 		break
 	case CreateTopics:
-		repRequest = representCreateTopicsRequest(reqDetails)
+		repRequest = representCreateTopicsRequest(request)
-		repResponse = representCreateTopicsResponse(resDetails)
+		repResponse = representCreateTopicsResponse(response)
 		break
 	case DeleteTopics:
-		repRequest = representDeleteTopicsRequest(reqDetails)
+		repRequest = representDeleteTopicsRequest(request)
-		repResponse = representDeleteTopicsResponse(resDetails)
+		repResponse = representDeleteTopicsResponse(response)
 		break
 	}
 
@@ -254,4 +253,10 @@ func (d dissecting) Represent(entry *api.MizuEntry) (p api.Protocol, object []by
 	return
 }
 
+func (d dissecting) Macros() map[string]string {
+	return map[string]string{
+		`kafka`: fmt.Sprintf(`proto.abbr == "%s"`, _protocol.Abbreviation),
+	}
+}
+
 var Dissector dissecting

tap/extensions/kafka/request.go

@@ -10,13 +10,13 @@ import (
 )
 
 type Request struct {
-	Size          int32
+	Size          int32       `json:"size"`
-	ApiKey        ApiKey
+	ApiKey        ApiKey      `json:"apiKey"`
-	ApiVersion    int16
+	ApiVersion    int16       `json:"apiVersion"`
-	CorrelationID int32
+	CorrelationID int32       `json:"correlationID"`
-	ClientID      string
+	ClientID      string      `json:"clientID"`
-	Payload       interface{}
+	Payload       interface{} `json:"payload"`
-	CaptureTime   time.Time
+	CaptureTime   time.Time   `json:"captureTime"`
 }
 
 func ReadRequest(r io.Reader, tcpID *api.TcpID, superTimer *api.SuperTimer) (apiKey ApiKey, apiVersion int16, err error) {

tap/extensions/kafka/response.go

@@ -10,10 +10,10 @@ import (
 )
 
 type Response struct {
-	Size          int32
+	Size          int32       `json:"size"`
-	CorrelationID int32
+	CorrelationID int32       `json:"correlationID"`
-	Payload       interface{}
+	Payload       interface{} `json:"payload"`
-	CaptureTime   time.Time
+	CaptureTime   time.Time   `json:"captureTime"`
 }
 
 func ReadResponse(r io.Reader, tcpID *api.TcpID, superTimer *api.SuperTimer, emitter api.Emitter) (err error) {

tap/extensions/redis/helpers.go

@@ -2,6 +2,7 @@ package main
 
 import (
 	"encoding/json"
+	"fmt"
 
 	"github.com/up9inc/mizu/tap/api"
 )
@@ -24,33 +25,38 @@ type RedisWrapper struct {
 	Details interface{} `json:"details"`
 }
 
-func representGeneric(generic map[string]interface{}) (representation []interface{}) {
+func representGeneric(generic map[string]interface{}, selectorPrefix string) (representation []interface{}) {
-	details, _ := json.Marshal([]map[string]string{
+	details, _ := json.Marshal([]api.TableData{
 		{
-			"name":  "Type",
+			Name:     "Type",
-			"value": generic["type"].(string),
+			Value:    generic["type"].(string),
+			Selector: fmt.Sprintf("%stype", selectorPrefix),
 		},
 		{
-			"name":  "Command",
+			Name:     "Command",
-			"value": generic["command"].(string),
+			Value:    generic["command"].(string),
+			Selector: fmt.Sprintf("%scommand", selectorPrefix),
 		},
 		{
-			"name":  "Key",
+			Name:     "Key",
-			"value": generic["key"].(string),
+			Value:    generic["key"].(string),
+			Selector: fmt.Sprintf("%skey", selectorPrefix),
 		},
 		{
-			"name":  "Value",
+			Name:     "Value",
-			"value": generic["value"].(string),
+			Value:    generic["value"].(string),
+			Selector: fmt.Sprintf("%svalue", selectorPrefix),
 		},
 		{
-			"name":  "Keyword",
+			Name:     "Keyword",
-			"value": generic["keyword"].(string),
+			Value:    generic["keyword"].(string),
+			Selector: fmt.Sprintf("%skeyword", selectorPrefix),
 		},
 	})
-	representation = append(representation, map[string]string{
+	representation = append(representation, api.SectionData{
-		"type":  api.TABLE,
+		Type:  api.TABLE,
-		"title": "Details",
+		Title: "Details",
-		"data":  string(details),
+		Data:  string(details),
 	})
 
 	return

tap/extensions/redis/main.go

@@ -13,6 +13,7 @@ var protocol api.Protocol = api.Protocol{
 	Name:            "redis",
 	LongName:        "Redis Serialization Protocol",
 	Abbreviation:    "REDIS",
+	Macro:           "redis",
 	Version:         "3.x",
 	BackgroundColor: "#a41e11",
 	ForegroundColor: "#ffffff",
@@ -57,9 +58,12 @@ func (d dissecting) Dissect(b *bufio.Reader, isClient bool, tcpID *api.TcpID, co
 	}
 }
 
-func (d dissecting) Analyze(item *api.OutputChannelItem, entryId string, resolvedSource string, resolvedDestination string) *api.MizuEntry {
+func (d dissecting) Analyze(item *api.OutputChannelItem, resolvedSource string, resolvedDestination string) *api.MizuEntry {
 	request := item.Pair.Request.Payload.(map[string]interface{})
+	response := item.Pair.Response.Payload.(map[string]interface{})
 	reqDetails := request["details"].(map[string]interface{})
+	resDetails := response["details"].(map[string]interface{})
+
 	service := "redis"
 	if resolvedDestination != "" {
 		service = resolvedDestination
@@ -78,45 +82,49 @@ func (d dissecting) Analyze(item *api.OutputChannelItem, entryId string, resolve
 	}
 
 	request["url"] = summary
-	entryBytes, _ := json.Marshal(item.Pair)
 	return &api.MizuEntry{
-		ProtocolName:            protocol.Name,
+		Protocol: protocol,
-		ProtocolLongName:        protocol.LongName,
+		Source: &api.TCP{
-		ProtocolAbbreviation:    protocol.Abbreviation,
+			Name: resolvedSource,
-		ProtocolVersion:         protocol.Version,
+			IP:   item.ConnectionInfo.ClientIP,
-		ProtocolBackgroundColor: protocol.BackgroundColor,
+			Port: item.ConnectionInfo.ClientPort,
-		ProtocolForegroundColor: protocol.ForegroundColor,
+		},
-		ProtocolFontSize:        protocol.FontSize,
+		Destination: &api.TCP{
-		ProtocolReferenceLink:   protocol.ReferenceLink,
+			Name: resolvedDestination,
-		EntryId:                 entryId,
+			IP:   item.ConnectionInfo.ServerIP,
-		Entry:                   string(entryBytes),
+			Port: item.ConnectionInfo.ServerPort,
-		Url:                     fmt.Sprintf("%s%s", service, summary),
+		},
-		Method:                  method,
+		Outgoing:            item.ConnectionInfo.IsOutgoing,
-		Status:                  0,
+		Request:             reqDetails,
-		RequestSenderIp:         item.ConnectionInfo.ClientIP,
+		Response:            resDetails,
-		Service:                 service,
+		Url:                 fmt.Sprintf("%s%s", service, summary),
-		Timestamp:               item.Timestamp,
+		Method:              method,
-		ElapsedTime:             0,
+		Status:              0,
-		Path:                    summary,
+		RequestSenderIp:     item.ConnectionInfo.ClientIP,
-		ResolvedSource:          resolvedSource,
+		Service:             service,
-		ResolvedDestination:     resolvedDestination,
+		Timestamp:           item.Timestamp,
-		SourceIp:                item.ConnectionInfo.ClientIP,
+		StartTime:           item.Pair.Request.CaptureTime,
-		DestinationIp:           item.ConnectionInfo.ServerIP,
+		ElapsedTime:         0,
-		SourcePort:              item.ConnectionInfo.ClientPort,
+		Summary:             summary,
-		DestinationPort:         item.ConnectionInfo.ServerPort,
+		ResolvedSource:      resolvedSource,
-		IsOutgoing:              item.ConnectionInfo.IsOutgoing,
+		ResolvedDestination: resolvedDestination,
+		SourceIp:            item.ConnectionInfo.ClientIP,
+		DestinationIp:       item.ConnectionInfo.ServerIP,
+		SourcePort:          item.ConnectionInfo.ClientPort,
+		DestinationPort:     item.ConnectionInfo.ServerPort,
+		IsOutgoing:          item.ConnectionInfo.IsOutgoing,
 	}
 
 }
 
 func (d dissecting) Summarize(entry *api.MizuEntry) *api.BaseEntryDetails {
 	return &api.BaseEntryDetails{
-		Id:              entry.EntryId,
+		Id:              entry.Id,
 		Protocol:        protocol,
 		Url:             entry.Url,
 		RequestSenderIp: entry.RequestSenderIp,
 		Service:         entry.Service,
-		Summary:         entry.Path,
+		Summary:         entry.Summary,
 		StatusCode:      entry.Status,
 		Method:          entry.Method,
 		Timestamp:       entry.Timestamp,
@@ -133,22 +141,22 @@ func (d dissecting) Summarize(entry *api.MizuEntry) *api.BaseEntryDetails {
 	}
 }
 
-func (d dissecting) Represent(entry *api.MizuEntry) (p api.Protocol, object []byte, bodySize int64, err error) {
+func (d dissecting) Represent(protoIn api.Protocol, request map[string]interface{}, response map[string]interface{}) (protoOut api.Protocol, object []byte, bodySize int64, err error) {
-	p = protocol
+	protoOut = protocol
 	bodySize = 0
-	var root map[string]interface{}
-	json.Unmarshal([]byte(entry.Entry), &root)
 	representation := make(map[string]interface{}, 0)
-	request := root["request"].(map[string]interface{})["payload"].(map[string]interface{})
+	repRequest := representGeneric(request, `request.`)
-	response := root["response"].(map[string]interface{})["payload"].(map[string]interface{})
+	repResponse := representGeneric(response, `response.`)
-	reqDetails := request["details"].(map[string]interface{})
-	resDetails := response["details"].(map[string]interface{})
-	repRequest := representGeneric(reqDetails)
-	repResponse := representGeneric(resDetails)
 	representation["request"] = repRequest
 	representation["response"] = repResponse
 	object, err = json.Marshal(representation)
 	return
 }
 
+func (d dissecting) Macros() map[string]string {
+	return map[string]string{
+		`redis`: fmt.Sprintf(`proto.abbr == "%s"`, protocol.Abbreviation),
+	}
+}
+
 var Dissector dissecting

ui/package-lock.json

@@ -1769,6 +1769,33 @@
         }
       }
     },
+    "@mapbox/rehype-prism": {
+      "version": "0.7.0",
+      "resolved": "https://registry.npmjs.org/@mapbox/rehype-prism/-/rehype-prism-0.7.0.tgz",
+      "integrity": "sha512-zSG46selA6v+3THhCatTyOt9DuTzxTIVTxTbcj15kFmxPDtjzZ5VoFVCLZfjWFouYa9PiXxcbMLLhJoVzCxh9w==",
+      "requires": {
+        "hast-util-to-string": "^1.0.4",
+        "refractor": "^3.4.0",
+        "unist-util-visit": "^2.0.3"
+      },
+      "dependencies": {
+        "prismjs": {
+          "version": "1.24.1",
+          "resolved": "https://registry.npmjs.org/prismjs/-/prismjs-1.24.1.tgz",
+          "integrity": "sha512-mNPsedLuk90RVJioIky8ANZEwYm5w9LcvCXrxHlwf4fNVSn8jEipMybMkWUyyF0JhnC+C4VcOVSBuHRKs1L5Ow=="
+        },
+        "refractor": {
+          "version": "3.4.0",
+          "resolved": "https://registry.npmjs.org/refractor/-/refractor-3.4.0.tgz",
+          "integrity": "sha512-dBeD02lC5eytm9Gld2Mx0cMcnR+zhSnsTfPpWqFaMgUMJfC9A6bcN3Br/NaXrnBJcuxnLFR90k1jrkaSyV8umg==",
+          "requires": {
+            "hastscript": "^6.0.0",
+            "parse-entities": "^2.0.0",
+            "prismjs": "~1.24.0"
+          }
+        }
+      }
+    },
     "@material-ui/core": {
       "version": "4.12.2",
       "resolved": "https://registry.npmjs.org/@material-ui/core/-/core-4.12.2.tgz",
@@ -1788,6 +1815,14 @@
         "react-transition-group": "^4.4.0"
       }
     },
+    "@material-ui/icons": {
+      "version": "4.11.2",
+      "resolved": "https://registry.npmjs.org/@material-ui/icons/-/icons-4.11.2.tgz",
+      "integrity": "sha512-fQNsKX2TxBmqIGJCSi3tGTO/gZ+eJgWmMJkgDiOfyNaunNaxcklJQFaFogYcFl0qFuaEz1qaXYXboa/bUXVSOQ==",
+      "requires": {
+        "@babel/runtime": "^7.4.4"
+      }
+    },
     "@material-ui/lab": {
       "version": "4.0.0-alpha.60",
       "resolved": "https://registry.npmjs.org/@material-ui/lab/-/lab-4.0.0-alpha.60.tgz",
@@ -2406,6 +2441,11 @@
       "resolved": "https://registry.npmjs.org/@types/parse-json/-/parse-json-4.0.0.tgz",
       "integrity": "sha512-//oorEZjL6sbPcKUaCdIGlIUeH26mgzimjBB77G6XRgnDl/L5wOnpyBGRe/Mmf5CVW3PwEBE1NjiMZ/ssFh4wA=="
     },
+    "@types/parse5": {
+      "version": "6.0.1",
+      "resolved": "https://registry.npmjs.org/@types/parse5/-/parse5-6.0.1.tgz",
+      "integrity": "sha512-ARATsLdrGPUnaBvxLhUlnltcMgn7pQG312S8ccdYlnyijabrX9RN/KN/iGj9Am96CoW8e/K9628BA7Bv4XHdrA=="
+    },
     "@types/prettier": {
       "version": "2.2.3",
       "resolved": "https://registry.npmjs.org/@types/prettier/-/prettier-2.2.3.tgz",
@@ -2629,6 +2669,26 @@
         "eslint-visitor-keys": "^2.0.0"
       }
     },
+    "@uiw/react-textarea-code-editor": {
+      "version": "1.4.12",
+      "resolved": "https://registry.npmjs.org/@uiw/react-textarea-code-editor/-/react-textarea-code-editor-1.4.12.tgz",
+      "integrity": "sha512-op0aIRxX8hLi8OLwm/23dQ2X4o2xHUoK2pLr1DWBFgNbIh4L+RM5zByNFjV8mHfi/NvE6mQLB6LtMd1qaor5MQ==",
+      "requires": {
+        "@babel/runtime": "7.15.4",
+        "@mapbox/rehype-prism": "0.7.0",
+        "rehype": "12.0.0"
+      },
+      "dependencies": {
+        "@babel/runtime": {
+          "version": "7.15.4",
+          "resolved": "https://registry.npmjs.org/@babel/runtime/-/runtime-7.15.4.tgz",
+          "integrity": "sha512-99catp6bHCaxr4sJ/DbTGgHS4+Rs2RVd2g7iOap6SLGPDknRK9ztKNsE/Fg6QhSeh1FGE5f6gHGQmvvn3I3xhw==",
+          "requires": {
+            "regenerator-runtime": "^0.13.4"
+          }
+        }
+      }
+    },
     "@webassemblyjs/ast": {
       "version": "1.9.0",
       "resolved": "https://registry.npmjs.org/@webassemblyjs/ast/-/ast-1.9.0.tgz",
@@ -3689,6 +3749,11 @@
       "resolved": "https://registry.npmjs.org/babylon/-/babylon-6.18.0.tgz",
       "integrity": "sha512-q/UEjfGJ2Cm3oKV71DJz9d25TPnq5rhBVL2Q4fA5wcC3jcrdn7+SssEybFIxwAvvP+YCsCYNKughoF33GxgycQ=="
     },
+    "bail": {
+      "version": "2.0.1",
+      "resolved": "https://registry.npmjs.org/bail/-/bail-2.0.1.tgz",
+      "integrity": "sha512-d5FoTAr2S5DSUPKl85WNm2yUwsINN8eidIdIwsOge2t33DaOfOdSmmsI11jMN3GmALCXaw+Y6HMVHDzePshFAA=="
+    },
     "balanced-match": {
       "version": "1.0.2",
       "resolved": "https://registry.npmjs.org/balanced-match/-/balanced-match-1.0.2.tgz",
@@ -4179,6 +4244,11 @@
       "resolved": "https://registry.npmjs.org/caseless/-/caseless-0.12.0.tgz",
       "integrity": "sha1-G2gcIf+EAzyCZUMJBolCDRhxUdw="
     },
+    "ccount": {
+      "version": "2.0.0",
+      "resolved": "https://registry.npmjs.org/ccount/-/ccount-2.0.0.tgz",
+      "integrity": "sha512-VOR0NWFYX65n9gELQdcpqsie5L5ihBXuZGAgaPEp/U7IOSjnPMEH6geE+2f6lcekaNEfWzAHS45mPvSo5bqsUA=="
+    },
     "chalk": {
       "version": "2.4.2",
       "resolved": "https://registry.npmjs.org/chalk/-/chalk-2.4.2.tgz",
@@ -4199,6 +4269,11 @@
       "resolved": "https://registry.npmjs.org/character-entities/-/character-entities-1.2.4.tgz",
       "integrity": "sha512-iBMyeEHxfVnIakwOuDXpVkc54HijNgCyQB2w0VfGQThle6NXn50zU6V/u+LDhxHcDUPojn6Kpga3PTAD8W1bQw=="
     },
+    "character-entities-html4": {
+      "version": "2.0.0",
+      "resolved": "https://registry.npmjs.org/character-entities-html4/-/character-entities-html4-2.0.0.tgz",
+      "integrity": "sha512-dwT2xh5ZhUAjyP96k57ilMKoTQyASaw9IAMR9U5c1lCu2RUni6O6jxfpUEdO2RcPT6TJFvr8pqsbami4Jk+2oA=="
+    },
     "character-entities-legacy": {
       "version": "1.1.4",
       "resolved": "https://registry.npmjs.org/character-entities-legacy/-/character-entities-legacy-1.1.4.tgz",
@@ -7534,11 +7609,121 @@
         "minimalistic-assert": "^1.0.1"
       }
     },
+    "hast-util-from-parse5": {
+      "version": "7.1.0",
+      "resolved": "https://registry.npmjs.org/hast-util-from-parse5/-/hast-util-from-parse5-7.1.0.tgz",
+      "integrity": "sha512-m8yhANIAccpU4K6+121KpPP55sSl9/samzQSQGpb0mTExcNh2WlvjtMwSWFhg6uqD4Rr6Nfa8N6TMypQM51rzQ==",
+      "requires": {
+        "@types/hast": "^2.0.0",
+        "@types/parse5": "^6.0.0",
+        "@types/unist": "^2.0.0",
+        "hastscript": "^7.0.0",
+        "property-information": "^6.0.0",
+        "vfile": "^5.0.0",
+        "vfile-location": "^4.0.0",
+        "web-namespaces": "^2.0.0"
+      },
+      "dependencies": {
+        "comma-separated-tokens": {
+          "version": "2.0.2",
+          "resolved": "https://registry.npmjs.org/comma-separated-tokens/-/comma-separated-tokens-2.0.2.tgz",
+          "integrity": "sha512-G5yTt3KQN4Yn7Yk4ed73hlZ1evrFKXeUW3086p3PRFNp7m2vIjI6Pg+Kgb+oyzhd9F2qdcoj67+y3SdxL5XWsg=="
+        },
+        "hast-util-parse-selector": {
+          "version": "3.1.0",
+          "resolved": "https://registry.npmjs.org/hast-util-parse-selector/-/hast-util-parse-selector-3.1.0.tgz",
+          "integrity": "sha512-AyjlI2pTAZEOeu7GeBPZhROx0RHBnydkQIXlhnFzDi0qfXTmGUWoCYZtomHbrdrheV4VFUlPcfJ6LMF5T6sQzg==",
+          "requires": {
+            "@types/hast": "^2.0.0"
+          }
+        },
+        "hastscript": {
+          "version": "7.0.2",
+          "resolved": "https://registry.npmjs.org/hastscript/-/hastscript-7.0.2.tgz",
+          "integrity": "sha512-uA8ooUY4ipaBvKcMuPehTAB/YfFLSSzCwFSwT6ltJbocFUKH/GDHLN+tflq7lSRf9H86uOuxOFkh1KgIy3Gg2g==",
+          "requires": {
+            "@types/hast": "^2.0.0",
+            "comma-separated-tokens": "^2.0.0",
+            "hast-util-parse-selector": "^3.0.0",
+            "property-information": "^6.0.0",
+            "space-separated-tokens": "^2.0.0"
+          }
+        },
+        "property-information": {
+          "version": "6.0.1",
+          "resolved": "https://registry.npmjs.org/property-information/-/property-information-6.0.1.tgz",
+          "integrity": "sha512-F4WUUAF7fMeF4/JUFHNBWDaKDXi2jbvqBW/y6o5wsf3j19wTZ7S60TmtB5HoBhtgw7NKQRMWuz5vk2PR0CygUg=="
+        },
+        "space-separated-tokens": {
+          "version": "2.0.1",
+          "resolved": "https://registry.npmjs.org/space-separated-tokens/-/space-separated-tokens-2.0.1.tgz",
+          "integrity": "sha512-ekwEbFp5aqSPKaqeY1PGrlGQxPNaq+Cnx4+bE2D8sciBQrHpbwoBbawqTN2+6jPs9IdWxxiUcN0K2pkczD3zmw=="
+        }
+      }
+    },
+    "hast-util-is-element": {
+      "version": "2.1.1",
+      "resolved": "https://registry.npmjs.org/hast-util-is-element/-/hast-util-is-element-2.1.1.tgz",
+      "integrity": "sha512-ag0fiZfRWsPiR1udvnSbaazJLGv8qd8E+/e3rW8rUZhbKG4HNJmFL4QkEceN+22BgE+uozXY30z/s+2dL6Z++g==",
+      "requires": {
+        "@types/hast": "^2.0.0",
+        "@types/unist": "^2.0.0"
+      }
+    },
     "hast-util-parse-selector": {
       "version": "2.2.5",
       "resolved": "https://registry.npmjs.org/hast-util-parse-selector/-/hast-util-parse-selector-2.2.5.tgz",
       "integrity": "sha512-7j6mrk/qqkSehsM92wQjdIgWM2/BW61u/53G6xmC8i1OmEdKLHbk419QKQUjz6LglWsfqoiHmyMRkP1BGjecNQ=="
     },
+    "hast-util-to-html": {
+      "version": "8.0.2",
+      "resolved": "https://registry.npmjs.org/hast-util-to-html/-/hast-util-to-html-8.0.2.tgz",
+      "integrity": "sha512-ipLhUTMyyJi9F/LXaNDG9BrRdshP6obCfmUZYbE/+T639IdzqAOkKN4DyrEyID0gbb+rsC3PKf0XlviZwzomhw==",
+      "requires": {
+        "@types/hast": "^2.0.0",
+        "ccount": "^2.0.0",
+        "comma-separated-tokens": "^2.0.0",
+        "hast-util-is-element": "^2.0.0",
+        "hast-util-whitespace": "^2.0.0",
+        "html-void-elements": "^2.0.0",
+        "property-information": "^6.0.0",
+        "space-separated-tokens": "^2.0.0",
+        "stringify-entities": "^4.0.0",
+        "unist-util-is": "^5.0.0"
+      },
+      "dependencies": {
+        "comma-separated-tokens": {
+          "version": "2.0.2",
+          "resolved": "https://registry.npmjs.org/comma-separated-tokens/-/comma-separated-tokens-2.0.2.tgz",
+          "integrity": "sha512-G5yTt3KQN4Yn7Yk4ed73hlZ1evrFKXeUW3086p3PRFNp7m2vIjI6Pg+Kgb+oyzhd9F2qdcoj67+y3SdxL5XWsg=="
+        },
+        "property-information": {
+          "version": "6.0.1",
+          "resolved": "https://registry.npmjs.org/property-information/-/property-information-6.0.1.tgz",
+          "integrity": "sha512-F4WUUAF7fMeF4/JUFHNBWDaKDXi2jbvqBW/y6o5wsf3j19wTZ7S60TmtB5HoBhtgw7NKQRMWuz5vk2PR0CygUg=="
+        },
+        "space-separated-tokens": {
+          "version": "2.0.1",
+          "resolved": "https://registry.npmjs.org/space-separated-tokens/-/space-separated-tokens-2.0.1.tgz",
+          "integrity": "sha512-ekwEbFp5aqSPKaqeY1PGrlGQxPNaq+Cnx4+bE2D8sciBQrHpbwoBbawqTN2+6jPs9IdWxxiUcN0K2pkczD3zmw=="
+        },
+        "unist-util-is": {
+          "version": "5.1.1",
+          "resolved": "https://registry.npmjs.org/unist-util-is/-/unist-util-is-5.1.1.tgz",
+          "integrity": "sha512-F5CZ68eYzuSvJjGhCLPL3cYx45IxkqXSetCcRgUXtbcm50X2L9oOWQlfUfDdAf+6Pd27YDblBfdtmsThXmwpbQ=="
+        }
+      }
+    },
+    "hast-util-to-string": {
+      "version": "1.0.4",
+      "resolved": "https://registry.npmjs.org/hast-util-to-string/-/hast-util-to-string-1.0.4.tgz",
+      "integrity": "sha512-eK0MxRX47AV2eZ+Lyr18DCpQgodvaS3fAQO2+b9Two9F5HEoRPhiUMNzoXArMJfZi2yieFzUBMRl3HNJ3Jus3w=="
+    },
+    "hast-util-whitespace": {
+      "version": "2.0.0",
+      "resolved": "https://registry.npmjs.org/hast-util-whitespace/-/hast-util-whitespace-2.0.0.tgz",
+      "integrity": "sha512-Pkw+xBHuV6xFeJprJe2BBEoDV+AvQySaz3pPDRUs5PNZEMQjpXJJueqrpcHIXxnWTcAGi/UOCgVShlkY6kLoqg=="
+    },
     "hastscript": {
       "version": "6.0.0",
       "resolved": "https://registry.npmjs.org/hastscript/-/hastscript-6.0.0.tgz",
@@ -7671,6 +7856,11 @@
         "terser": "^4.6.3"
       }
     },
+    "html-void-elements": {
+      "version": "2.0.0",
+      "resolved": "https://registry.npmjs.org/html-void-elements/-/html-void-elements-2.0.0.tgz",
+      "integrity": "sha512-4OYzQQsBt0G9bJ/nM9/DDsjm4+fVdzAaPJJcWk5QwA3GIAPxQEeOR0rsI8HbDHQz5Gta8pVvGnnTNSbZVEVvkQ=="
+    },
     "html-webpack-plugin": {
       "version": "4.5.0",
       "resolved": "https://registry.npmjs.org/html-webpack-plugin/-/html-webpack-plugin-4.5.0.tgz",
@@ -13470,6 +13660,14 @@
         "refractor": "^3.2.0"
       }
     },
+    "react-toastify": {
+      "version": "8.0.3",
+      "resolved": "https://registry.npmjs.org/react-toastify/-/react-toastify-8.0.3.tgz",
+      "integrity": "sha512-rv3koC7f9lKKSkdpYgo/TGzgWlrB/aaiUInF1DyV7BpiM4kyTs+uhu6/r8XDMtBY2FOIHK+FlK3Iv7OzpA/tCA==",
+      "requires": {
+        "clsx": "^1.1.1"
+      }
+    },
     "react-transition-group": {
       "version": "4.4.1",
       "resolved": "https://registry.npmjs.org/react-transition-group/-/react-transition-group-4.4.1.tgz",
@@ -13693,6 +13891,38 @@
         }
       }
     },
+    "rehype": {
+      "version": "12.0.0",
+      "resolved": "https://registry.npmjs.org/rehype/-/rehype-12.0.0.tgz",
+      "integrity": "sha512-gZcttmf9R5IYHb8AlI1rlmWqXS1yX0rSB/S5ZGJs8atfYZy2DobvH3Ic/gSzB+HL/+oOHPtBguw1TprfhxXBgQ==",
+      "requires": {
+        "@types/hast": "^2.0.0",
+        "rehype-parse": "^8.0.0",
+        "rehype-stringify": "^9.0.0",
+        "unified": "^10.0.0"
+      }
+    },
+    "rehype-parse": {
+      "version": "8.0.3",
+      "resolved": "https://registry.npmjs.org/rehype-parse/-/rehype-parse-8.0.3.tgz",
+      "integrity": "sha512-RGw0CVt+0S6KdvpE8bbP2Db9WXclQcIX7A0ufM3QFqAhTo/ddJMQrrI2j3cijlRPZlGK8R3pRgC8U5HyV76IDw==",
+      "requires": {
+        "@types/hast": "^2.0.0",
+        "hast-util-from-parse5": "^7.0.0",
+        "parse5": "^6.0.0",
+        "unified": "^10.0.0"
+      }
+    },
+    "rehype-stringify": {
+      "version": "9.0.2",
+      "resolved": "https://registry.npmjs.org/rehype-stringify/-/rehype-stringify-9.0.2.tgz",
+      "integrity": "sha512-BuVA6lAEYtOpXO2xuHLohAzz8UNoQAxAqYRqh4QEEtU39Co+P1JBZhw6wXA9hMWp+JLcmrxWH8+UKcNSr443Fw==",
+      "requires": {
+        "@types/hast": "^2.0.0",
+        "hast-util-to-html": "^8.0.0",
+        "unified": "^10.0.0"
+      }
+    },
     "relateurl": {
       "version": "0.2.7",
       "resolved": "https://registry.npmjs.org/relateurl/-/relateurl-0.2.7.tgz",
@@ -15342,6 +15572,22 @@
         }
       }
     },
+    "stringify-entities": {
+      "version": "4.0.1",
+      "resolved": "https://registry.npmjs.org/stringify-entities/-/stringify-entities-4.0.1.tgz",
+      "integrity": "sha512-gmMQxKXPWIO3NXNSPyWNhlYcBNGpPA/487D+9dLPnU4xBnIrnHdr8cv5rGJOS/1BRxEXRb7uKwg7BA36IWV7xg==",
+      "requires": {
+        "character-entities-html4": "^2.0.0",
+        "character-entities-legacy": "^2.0.0"
+      },
+      "dependencies": {
+        "character-entities-legacy": {
+          "version": "2.0.0",
+          "resolved": "https://registry.npmjs.org/character-entities-legacy/-/character-entities-legacy-2.0.0.tgz",
+          "integrity": "sha512-YwaEtEvWLpFa6Wh3uVLrvirA/ahr9fki/NUd/Bd4OR6EdJ8D22hovYQEOUCBfQfcqnC4IAMGMsHXY1eXgL4ZZA=="
+        }
+      }
+    },
     "stringify-object": {
       "version": "3.3.0",
       "resolved": "https://registry.npmjs.org/stringify-object/-/stringify-object-3.3.0.tgz",
@@ -15877,6 +16123,11 @@
       "resolved": "https://registry.npmjs.org/trim-newlines/-/trim-newlines-1.0.0.tgz",
       "integrity": "sha1-WIeWa7WCpFA6QetST301ARgVphM="
     },
+    "trough": {
+      "version": "2.0.2",
+      "resolved": "https://registry.npmjs.org/trough/-/trough-2.0.2.tgz",
+      "integrity": "sha512-FnHq5sTMxC0sk957wHDzRnemFnNBvt/gSY99HzK8F7UP5WAbvP70yX5bd7CjEQkN+TjdxwI7g7lJ6podqrG2/w=="
+    },
     "true-case-path": {
       "version": "1.0.3",
       "resolved": "https://registry.npmjs.org/true-case-path/-/true-case-path-1.0.3.tgz",
@@ -16037,6 +16288,32 @@
       "resolved": "https://registry.npmjs.org/unicode-property-aliases-ecmascript/-/unicode-property-aliases-ecmascript-1.1.0.tgz",
       "integrity": "sha512-PqSoPh/pWetQ2phoj5RLiaqIk4kCNwoV3CI+LfGmWLKI3rE3kl1h59XpX2BjgDrmbxD9ARtQobPGU1SguCYuQg=="
     },
+    "unified": {
+      "version": "10.1.0",
+      "resolved": "https://registry.npmjs.org/unified/-/unified-10.1.0.tgz",
+      "integrity": "sha512-4U3ru/BRXYYhKbwXV6lU6bufLikoAavTwev89H5UxY8enDFaAT2VXmIXYNm6hb5oHPng/EXr77PVyDFcptbk5g==",
+      "requires": {
+        "@types/unist": "^2.0.0",
+        "bail": "^2.0.0",
+        "extend": "^3.0.0",
+        "is-buffer": "^2.0.0",
+        "is-plain-obj": "^4.0.0",
+        "trough": "^2.0.0",
+        "vfile": "^5.0.0"
+      },
+      "dependencies": {
+        "is-buffer": {
+          "version": "2.0.5",
+          "resolved": "https://registry.npmjs.org/is-buffer/-/is-buffer-2.0.5.tgz",
+          "integrity": "sha512-i2R6zNFDwgEHJyQUtJEk0XFi1i0dPFn/oqjK3/vPCcDeJvW5NQ83V8QbicfF1SupOaB0h8ntgBC2YiE7dfyctQ=="
+        },
+        "is-plain-obj": {
+          "version": "4.0.0",
+          "resolved": "https://registry.npmjs.org/is-plain-obj/-/is-plain-obj-4.0.0.tgz",
+          "integrity": "sha512-NXRbBtUdBioI73y/HmOhogw/U5msYPC9DAtGkJXeFcFWSFZw0mCUsPxk/snTuJHzNKA8kLBK4rH97RMB1BfCXw=="
+        }
+      }
+    },
     "union-value": {
       "version": "1.0.1",
       "resolved": "https://registry.npmjs.org/union-value/-/union-value-1.0.1.tgz",
@@ -16082,6 +16359,38 @@
         "crypto-random-string": "^1.0.0"
       }
     },
+    "unist-util-is": {
+      "version": "4.1.0",
+      "resolved": "https://registry.npmjs.org/unist-util-is/-/unist-util-is-4.1.0.tgz",
+      "integrity": "sha512-ZOQSsnce92GrxSqlnEEseX0gi7GH9zTJZ0p9dtu87WRb/37mMPO2Ilx1s/t9vBHrFhbgweUwb+t7cIn5dxPhZg=="
+    },
+    "unist-util-stringify-position": {
+      "version": "3.0.0",
+      "resolved": "https://registry.npmjs.org/unist-util-stringify-position/-/unist-util-stringify-position-3.0.0.tgz",
+      "integrity": "sha512-SdfAl8fsDclywZpfMDTVDxA2V7LjtRDTOFd44wUJamgl6OlVngsqWjxvermMYf60elWHbxhuRCZml7AnuXCaSA==",
+      "requires": {
+        "@types/unist": "^2.0.0"
+      }
+    },
+    "unist-util-visit": {
+      "version": "2.0.3",
+      "resolved": "https://registry.npmjs.org/unist-util-visit/-/unist-util-visit-2.0.3.tgz",
+      "integrity": "sha512-iJ4/RczbJMkD0712mGktuGpm/U4By4FfDonL7N/9tATGIF4imikjOuagyMY53tnZq3NP6BcmlrHhEKAfGWjh7Q==",
+      "requires": {
+        "@types/unist": "^2.0.0",
+        "unist-util-is": "^4.0.0",
+        "unist-util-visit-parents": "^3.0.0"
+      }
+    },
+    "unist-util-visit-parents": {
+      "version": "3.1.1",
+      "resolved": "https://registry.npmjs.org/unist-util-visit-parents/-/unist-util-visit-parents-3.1.1.tgz",
+      "integrity": "sha512-1KROIZWo6bcMrZEwiH2UrXDyalAa0uqzWCxCJj6lPOvTve2WkfgCytoDTPaMnodXh1WrXOq0haVYHj99ynJlsg==",
+      "requires": {
+        "@types/unist": "^2.0.0",
+        "unist-util-is": "^4.0.0"
+      }
+    },
     "universalify": {
       "version": "2.0.0",
       "resolved": "https://registry.npmjs.org/universalify/-/universalify-2.0.0.tgz",
@@ -16306,6 +16615,42 @@
         "extsprintf": "^1.2.0"
       }
     },
+    "vfile": {
+      "version": "5.1.0",
+      "resolved": "https://registry.npmjs.org/vfile/-/vfile-5.1.0.tgz",
+      "integrity": "sha512-4o7/DJjEaFPYSh0ckv5kcYkJTHQgCKdL8ozMM1jLAxO9ox95IzveDPXCZp08HamdWq8JXTkClDvfAKaeLQeKtg==",
+      "requires": {
+        "@types/unist": "^2.0.0",
+        "is-buffer": "^2.0.0",
+        "unist-util-stringify-position": "^3.0.0",
+        "vfile-message": "^3.0.0"
+      },
+      "dependencies": {
+        "is-buffer": {
+          "version": "2.0.5",
+          "resolved": "https://registry.npmjs.org/is-buffer/-/is-buffer-2.0.5.tgz",
+          "integrity": "sha512-i2R6zNFDwgEHJyQUtJEk0XFi1i0dPFn/oqjK3/vPCcDeJvW5NQ83V8QbicfF1SupOaB0h8ntgBC2YiE7dfyctQ=="
+        }
+      }
+    },
+    "vfile-location": {
+      "version": "4.0.1",
+      "resolved": "https://registry.npmjs.org/vfile-location/-/vfile-location-4.0.1.tgz",
+      "integrity": "sha512-JDxPlTbZrZCQXogGheBHjbRWjESSPEak770XwWPfw5mTc1v1nWGLB/apzZxsx8a0SJVfF8HK8ql8RD308vXRUw==",
+      "requires": {
+        "@types/unist": "^2.0.0",
+        "vfile": "^5.0.0"
+      }
+    },
+    "vfile-message": {
+      "version": "3.0.2",
+      "resolved": "https://registry.npmjs.org/vfile-message/-/vfile-message-3.0.2.tgz",
+      "integrity": "sha512-UUjZYIOg9lDRwwiBAuezLIsu9KlXntdxwG+nXnjuQAHvBpcX3x0eN8h+I7TkY5nkCXj+cWVp4ZqebtGBvok8ww==",
+      "requires": {
+        "@types/unist": "^2.0.0",
+        "unist-util-stringify-position": "^3.0.0"
+      }
+    },
     "vm-browserify": {
       "version": "1.1.2",
       "resolved": "https://registry.npmjs.org/vm-browserify/-/vm-browserify-1.1.2.tgz",
@@ -16590,6 +16935,11 @@
         "minimalistic-assert": "^1.0.0"
       }
     },
+    "web-namespaces": {
+      "version": "2.0.0",
+      "resolved": "https://registry.npmjs.org/web-namespaces/-/web-namespaces-2.0.0.tgz",
+      "integrity": "sha512-dE7ELZRVWh0ceQsRgkjLgsAvwTuv3kcjSY/hLjqL0llleUlQBDjE9JkB9FCBY5F2mnFEwiyJoowl8+NVGHe8dw=="
+    },
     "web-vitals": {
       "version": "1.1.1",
       "resolved": "https://registry.npmjs.org/web-vitals/-/web-vitals-1.1.1.tgz",

ui/package.json

@@ -4,6 +4,7 @@
   "private": true,
   "dependencies": {
     "@material-ui/core": "^4.11.3",
+    "@material-ui/icons": "^4.11.2",
     "@material-ui/lab": "^4.0.0-alpha.60",
     "@testing-library/jest-dom": "^5.11.10",
     "@testing-library/react": "^11.2.6",
@@ -12,6 +13,7 @@
     "@types/node": "^12.20.10",
     "@types/react": "^17.0.3",
     "@types/react-dom": "^17.0.3",
+    "@uiw/react-textarea-code-editor": "^1.4.12",
     "axios": "^0.21.1",
     "jsonpath": "^1.1.1",
     "node-sass": "^5.0.0",
@@ -23,6 +25,7 @@
     "react-scripts": "4.0.3",
     "react-scrollable-feed-virtualized": "^1.4.3",
     "react-syntax-highlighter": "^15.4.3",
+    "react-toastify": "^8.0.3",
     "typescript": "^4.2.4",
     "web-vitals": "^1.1.1"
   },

ui/src/components/EntriesList.tsx

@@ -1,10 +1,7 @@
 import {EntryItem} from "./EntryListItem/EntryListItem";
-import React, {useCallback, useEffect, useMemo, useRef, useState} from "react";
+import React, {useRef} from "react";
 import styles from './style/EntriesList.module.sass';
-import spinner from './assets/spinner.svg';
 import ScrollableFeedVirtualized from "react-scrollable-feed-virtualized";
-import {StatusType} from "./Filters";
-import Api from "../helpers/api";
 import down from "./assets/downImg.svg";
 
 interface EntriesListProps {
@@ -13,114 +10,36 @@ interface EntriesListProps {
     focusedEntryId: string;
     setFocusedEntryId: (id: string) => void;
     connectionOpen: boolean;
-    noMoreDataTop: boolean;
-    setNoMoreDataTop: (flag: boolean) => void;
-    noMoreDataBottom: boolean;
-    setNoMoreDataBottom: (flag: boolean) => void;
-    methodsFilter: Array<string>;
-    statusFilter: Array<string>;
-    pathFilter: string;
-    serviceFilter: string;
     listEntryREF: any;
     onScrollEvent: (isAtBottom:boolean) => void;
     scrollableList: boolean;
+    ws: any
+    openWebSocket: any;
+    query: string;
+    updateQuery: any;
+    queriedCurrent: number;
+    queriedTotal: number;
+    startTime: number;
 }
 
-enum FetchOperator {
+export const EntriesList: React.FC<EntriesListProps> = ({entries, setEntries, focusedEntryId, setFocusedEntryId, connectionOpen, listEntryREF, onScrollEvent, scrollableList, ws, openWebSocket, query, updateQuery, queriedCurrent, queriedTotal, startTime}) => {
-    LT = "lt",
-    GT = "gt"
-}
-
-const api = new Api();
-
-export const EntriesList: React.FC<EntriesListProps> = ({entries, setEntries, focusedEntryId, setFocusedEntryId, connectionOpen, noMoreDataTop, setNoMoreDataTop, noMoreDataBottom, setNoMoreDataBottom, methodsFilter, statusFilter, pathFilter, serviceFilter, listEntryREF, onScrollEvent, scrollableList}) => {
-
-    const [loadMoreTop, setLoadMoreTop] = useState(false);
-    const [isLoadingTop, setIsLoadingTop] = useState(false);
 
     const scrollableRef = useRef(null);
 
-    useEffect(() => {
-        const list = document.getElementById('list').firstElementChild;
-        list.addEventListener('scroll', (e) => {
-            const el: any = e.target;
-            if(el.scrollTop === 0) {
-                setLoadMoreTop(true);
-            } else {
-                setLoadMoreTop(false);
-            }
-        });
-    }, []);
-
-    const filterEntries = useCallback((entry) => {
-        if(methodsFilter.length > 0 && !methodsFilter.includes(entry.method.toLowerCase())) return;
-        if(pathFilter && entry.path?.toLowerCase()?.indexOf(pathFilter) === -1) return;
-        if(serviceFilter && entry.service?.toLowerCase()?.indexOf(serviceFilter) === -1) return;
-        if(statusFilter.includes(StatusType.SUCCESS) && entry.statusCode >= 400) return;
-        if(statusFilter.includes(StatusType.ERROR) && entry.statusCode < 400) return;
-        return entry;
-    },[methodsFilter, pathFilter, statusFilter, serviceFilter])
-
-    const filteredEntries = useMemo(() => {
-        return entries.filter(filterEntries);
-    },[entries, filterEntries])
-
-    const getOldEntries = useCallback(async () => {
-        setIsLoadingTop(true);
-        const data = await api.fetchEntries(FetchOperator.LT, entries[0].timestamp);
-        setLoadMoreTop(false);
-
-        let scrollTo;
-        if(data.length === 0) {
-            setNoMoreDataTop(true);
-            scrollTo = document.getElementById("noMoreDataTop");
-        } else {
-            scrollTo = document.getElementById(filteredEntries?.[0]?.id);
-        }
-        setIsLoadingTop(false);
-        const newEntries = [...data, ...entries];
-        setEntries(newEntries);
-
-        if(scrollTo) {
-            scrollTo.scrollIntoView();
-        }
-    },[setLoadMoreTop, setIsLoadingTop, entries, setEntries, filteredEntries, setNoMoreDataTop])
-
-    useEffect(() => {
-        if(!loadMoreTop || connectionOpen || noMoreDataTop) return;
-        getOldEntries();
-    }, [loadMoreTop, connectionOpen, noMoreDataTop, getOldEntries]);
-
-    const getNewEntries = async () => {
-        const data = await api.fetchEntries(FetchOperator.GT, entries[entries.length - 1].timestamp);
-        let scrollTo;
-        if(data.length === 0) {
-            setNoMoreDataBottom(true);
-        }
-        scrollTo = document.getElementById(filteredEntries?.[filteredEntries.length -1]?.id);
-        let newEntries = [...entries, ...data];
-        setEntries(newEntries);
-        if(scrollTo) {
-            scrollTo.scrollIntoView({behavior: "smooth"});
-        }
-    }
-
     return <>
             <div className={styles.list}>
                 <div id="list" ref={listEntryREF} className={styles.list}>
-                    {isLoadingTop && <div className={styles.spinnerContainer}>
-                        <img alt="spinner" src={spinner} style={{height: 25}}/>
-                    </div>}
                     <ScrollableFeedVirtualized ref={scrollableRef} itemHeight={48} marginTop={10} onScroll={(isAtBottom) => onScrollEvent(isAtBottom)}>
-                        {noMoreDataTop && !connectionOpen && <div id="noMoreDataTop" className={styles.noMoreDataAvailable}>No more data available</div>}
+                        {false /* TODO: why there is a need for something here (not necessarily false)? */}
-                        {filteredEntries.map(entry => <EntryItem key={entry.id}
+                        {entries.map(entry => <EntryItem key={entry.id}
                                                         entry={entry}
                                                         setFocusedEntryId={setFocusedEntryId}
-                                                        isSelected={focusedEntryId === entry.id}
+                                                        isSelected={focusedEntryId === entry.id.toString()}
-                                                        style={{}}/>)}
+                                                        style={{}}
+                                                        updateQuery={updateQuery}/>)}
                     </ScrollableFeedVirtualized>
-                    {!connectionOpen && !noMoreDataBottom && <div className={styles.fetchButtonContainer}>
+                    {!connectionOpen && <div className={styles.fetchButtonContainer}>
-                        <div className={styles.styledButton} onClick={() => getNewEntries()}>Fetch more entries</div>
+                        <div className={styles.styledButton} onClick={() => {ws.close(); openWebSocket(query);}}>Reconnect</div>
                     </div>}
                     <button type="button"
                         className={`${styles.btnLive} ${scrollableList ? styles.showButton : styles.hideButton}`}
@@ -129,10 +48,10 @@ export const EntriesList: React.FC<EntriesListProps> = ({entries, setEntries, fo
                     </button>
                 </div>
 
-                {entries?.length > 0 && <div className={styles.footer}>
+                <div className={styles.footer}>
-                    <div><b>{filteredEntries?.length !== entries.length && `${filteredEntries?.length} / `} {entries?.length}</b> requests</div>
+                    <div>Displaying <b>{entries?.length}</b> results (queried <b>{queriedCurrent}</b>/<b>{queriedTotal}</b>)</div>
-                    <div>Started listening at <span style={{marginRight: 5, fontWeight: 600, fontSize: 13}}>{new Date(+entries[0].timestamp)?.toLocaleString()}</span></div>
+                    {startTime !== 0 && <div>Started listening at <span style={{marginRight: 5, fontWeight: 600, fontSize: 13}}>{new Date(startTime).toLocaleString()}</span></div>}
-                </div>}
+                </div>
             </div>
     </>;
 };

ui/src/components/EntryDetailed.tsx

@@ -3,7 +3,7 @@ import EntryViewer from "./EntryDetailed/EntryViewer";
 import {makeStyles} from "@material-ui/core";
 import Protocol from "./UI/Protocol"
 import StatusCode from "./UI/StatusCode";
-import {EndpointPath} from "./UI/EndpointPath";
+import {Summary} from "./UI/Summary";
 
 const useStyles = makeStyles(() => ({
     entryTitle: {
@@ -28,50 +28,79 @@ const useStyles = makeStyles(() => ({
 
 interface EntryDetailedProps {
     entryData: any
+    updateQuery: any
 }
 
 export const formatSize = (n: number) => n > 1000 ? `${Math.round(n / 1000)}KB` : `${n} B`;
 
-const EntryTitle: React.FC<any> = ({protocol, data, bodySize, elapsedTime}) => {
+const EntryTitle: React.FC<any> = ({protocol, data, bodySize, elapsedTime, updateQuery}) => {
     const classes = useStyles();
-    const {response} = JSON.parse(data.entry);
+    const response = data.response;
 
 
     return <div className={classes.entryTitle}>
-        <Protocol protocol={protocol} horizontal={true}/>
+        <Protocol protocol={protocol} horizontal={true} updateQuery={null}/>
         <div style={{right: "30px", position: "absolute", display: "flex"}}>
-            {response.payload && <div style={{margin: "0 18px", opacity: 0.5}}>{formatSize(bodySize)}</div>}
+            {response && <div
-            {response.payload && <div style={{marginRight: 18, opacity: 0.5}}>{Math.round(elapsedTime)}ms</div>}
+                className="queryable"
+                style={{margin: "0 18px", opacity: 0.5}}
+                onClick={() => {
+                    updateQuery(`response.bodySize == ${bodySize}`)
+                }}
+            >
+                {formatSize(bodySize)}
+            </div>}
+            {response && <div
+                className="queryable"
+                style={{marginRight: 18, opacity: 0.5}}
+                onClick={() => {
+                    updateQuery(`elapsedTime >= ${elapsedTime}`)
+                }}
+            >
+                {Math.round(elapsedTime)}ms
+            </div>}
         </div>
     </div>;
 };
 
-const EntrySummary: React.FC<any> = ({data}) => {
+const EntrySummary: React.FC<any> = ({data, updateQuery}) => {
     const classes = useStyles();
 
-    const {response, request} = JSON.parse(data.entry);
+    const response = data.response;
 
     return <div className={classes.entrySummary}>
-        {response?.payload && response.payload?.details && "status" in response.payload.details && <div style={{marginRight: 8}}>
+        {response && "status" in response && <div style={{marginRight: 8}}>
-            <StatusCode statusCode={response.payload.details.status}/>
+            <StatusCode statusCode={response.status} updateQuery={updateQuery}/>
         </div>}
         <div style={{flexGrow: 1, overflow: 'hidden'}}>
-            <EndpointPath method={request?.payload.method} path={request?.payload.url}/>
+            <Summary method={data.method} summary={data.summary} updateQuery={updateQuery}/>
         </div>
     </div>;
 };
 
-export const EntryDetailed: React.FC<EntryDetailedProps> = ({entryData}) => {
+export const EntryDetailed: React.FC<EntryDetailedProps> = ({entryData, updateQuery}) => {
     return <>
         <EntryTitle
             protocol={entryData.protocol}
             data={entryData.data}
             bodySize={entryData.bodySize}
             elapsedTime={entryData.data.elapsedTime}
+            updateQuery={updateQuery}
         />
-        {entryData.data && <EntrySummary data={entryData.data}/>}
+        {entryData.data && <EntrySummary data={entryData.data} updateQuery={updateQuery}/>}
         <>
-            {entryData.data && <EntryViewer representation={entryData.representation} isRulesEnabled={entryData.isRulesEnabled} rulesMatched={entryData.rulesMatched} contractStatus={entryData.data.contractStatus} requestReason={entryData.data.contractRequestReason} responseReason={entryData.data.contractResponseReason} contractContent={entryData.data.contractContent} elapsedTime={entryData.data.elapsedTime} color={entryData.protocol.backgroundColor}/>}
+            {entryData.data && <EntryViewer
+                representation={entryData.representation}
+                isRulesEnabled={entryData.isRulesEnabled}
+                rulesMatched={entryData.rulesMatched}
+                contractStatus={entryData.data.contractStatus}
+                requestReason={entryData.data.contractRequestReason}
+                responseReason={entryData.data.contractResponseReason}
+                contractContent={entryData.data.contractContent}
+                elapsedTime={entryData.data.elapsedTime}
+                color={entryData.protocol.backgroundColor}
+                updateQuery={updateQuery}
+            />}
         </>
     </>
 };

ui/src/components/EntryDetailed/EntrySections.tsx

@@ -9,11 +9,29 @@ import ProtobufDecoder from "protobuf-decoder";
 interface EntryViewLineProps {
     label: string;
     value: number | string;
+    updateQuery: any;
+    selector: string;
+    overrideQueryValue?: string;
 }
 
-const EntryViewLine: React.FC<EntryViewLineProps> = ({label, value}) => {
+const EntryViewLine: React.FC<EntryViewLineProps> = ({label, value, updateQuery, selector, overrideQueryValue}) => {
     return (label && value && <tr className={styles.dataLine}>
-                <td className={styles.dataKey}>{label}</td>
+                <td
+                    className={`queryable ${styles.dataKey}`}
+                    onClick={() => {
+                        if (!selector) {
+                            return
+                        } else if (overrideQueryValue) {
+                            updateQuery(`${selector} == ${overrideQueryValue}`)
+                        } else if (typeof(value) === "string") {
+                            updateQuery(`${selector} == "${JSON.stringify(value).slice(1, -1)}"`)
+                        } else {
+                            updateQuery(`${selector} == ${value}`)
+                        }
+                    }}
+                >
+                    {label}
+                </td>
                 <td>
                     <FancyTextDisplay
                         className={styles.dataValue}
@@ -62,15 +80,19 @@ export const EntrySectionContainer: React.FC<EntrySectionContainerProps> = ({tit
 interface EntryBodySectionProps {
     content: any,
     color: string,
+    updateQuery: any,
     encoding?: string,
     contentType?: string,
+    selector?: string,
 }
 
 export const EntryBodySection: React.FC<EntryBodySectionProps> = ({
     color,
+    updateQuery,
     content,
     encoding,
     contentType,
+    selector,
 }) => {
     const MAXIMUM_BYTES_TO_HIGHLIGHT = 10000; // The maximum of chars to highlight in body, in case the response can be megabytes
     const supportedLanguages = [['html', 'html'], ['json', 'json'], ['application/grpc', 'json']]; // [[indicator, languageToUse],...]
@@ -107,8 +129,8 @@ export const EntryBodySection: React.FC<EntryBodySectionProps> = ({
         {content && content?.length > 0 && <EntrySectionContainer title='Body' color={color}>
             <table>
                 <tbody>
-                    <EntryViewLine label={'Mime type'} value={contentType}/>
+                    <EntryViewLine label={'Mime type'} value={contentType} updateQuery={updateQuery} selector={selector} overrideQueryValue={`r".*"`}/>
-                    <EntryViewLine label={'Encoding'} value={encoding}/>
+                    <EntryViewLine label={'Encoding'} value={encoding} updateQuery={updateQuery} selector={selector} overrideQueryValue={`r".*"`}/>
                 </tbody>
             </table>
 
@@ -132,17 +154,23 @@ interface EntrySectionProps {
     title: string,
     color: string,
     arrayToIterate: any[],
+    updateQuery: any,
 }
 
-export const EntryTableSection: React.FC<EntrySectionProps> = ({title, color, arrayToIterate}) => {
+export const EntryTableSection: React.FC<EntrySectionProps> = ({title, color, arrayToIterate, updateQuery}) => {
     return <React.Fragment>
         {
             arrayToIterate && arrayToIterate.length > 0 ?
                 <EntrySectionContainer title={title} color={color}>
                     <table>
                         <tbody>
-                            {arrayToIterate.map(({name, value}, index) => <EntryViewLine key={index} label={name}
+                            {arrayToIterate.map(({name, value, selector}, index) => <EntryViewLine
-                                                                                            value={value}/>)}
+                                key={index}
+                                label={name}
+                                value={value}
+                                updateQuery={updateQuery}
+                                selector={selector}
+                            />)}
                         </tbody>
                     </table>
                 </EntrySectionContainer> : <span/>

ui/src/components/EntryDetailed/EntryViewer.tsx

@@ -8,7 +8,7 @@ enum SectionTypes {
     SectionBody = "body",
 }
 
-const SectionsRepresentation: React.FC<any> = ({data, color}) => {
+const SectionsRepresentation: React.FC<any> = ({data, color, updateQuery}) => {
     const sections = []
 
     if (data) {
@@ -16,12 +16,12 @@ const SectionsRepresentation: React.FC<any> = ({data, color}) => {
             switch (row.type) {
                 case SectionTypes.SectionTable:
                     sections.push(
-                        <EntryTableSection key={i} title={row.title} color={color} arrayToIterate={JSON.parse(row.data)}/>
+                        <EntryTableSection key={i} title={row.title} color={color} arrayToIterate={JSON.parse(row.data)} updateQuery={updateQuery}/>
                     )
                     break;
                 case SectionTypes.SectionBody:
                     sections.push(
-                        <EntryBodySection key={i} color={color} content={row.data} encoding={row.encoding} contentType={row.mime_type}/>
+                        <EntryBodySection key={i} color={color} content={row.data} updateQuery={updateQuery} encoding={row.encoding} contentType={row.mimeType} selector={row.selector}/>
                     )
                     break;
                 default:
@@ -33,7 +33,7 @@ const SectionsRepresentation: React.FC<any> = ({data, color}) => {
     return <>{sections}</>;
 }
 
-const AutoRepresentation: React.FC<any> = ({representation, isRulesEnabled, rulesMatched, contractStatus, requestReason, responseReason, contractContent, elapsedTime, color}) => {
+const AutoRepresentation: React.FC<any> = ({representation, isRulesEnabled, rulesMatched, contractStatus, requestReason, responseReason, contractContent, elapsedTime, color, updateQuery}) => {
     var TABS = [
         {
             tab: 'Request'
@@ -85,10 +85,10 @@ const AutoRepresentation: React.FC<any> = ({representation, isRulesEnabled, rule
                 <Tabs tabs={TABS} currentTab={currentTab} color={color} onChange={setCurrentTab} leftAligned/>
             </div>
             {currentTab === TABS[0].tab && <React.Fragment>
-                <SectionsRepresentation data={request} color={color}/>
+                <SectionsRepresentation data={request} color={color} updateQuery={updateQuery}/>
             </React.Fragment>}
             {response && currentTab === TABS[responseTabIndex].tab && <React.Fragment>
-                <SectionsRepresentation data={response} color={color}/>
+                <SectionsRepresentation data={response} color={color} updateQuery={updateQuery}/>
             </React.Fragment>}
             {isRulesEnabled && currentTab === TABS[rulesTabIndex].tab && <React.Fragment>
                 <EntryTablePolicySection title={'Rule'} color={color} latency={elapsedTime} arrayToIterate={rulesMatched ? rulesMatched : []}/>
@@ -110,9 +110,10 @@ interface Props {
     contractContent: string;
     color: string;
     elapsedTime: number;
+    updateQuery: any;
 }
 
-const EntryViewer: React.FC<Props> = ({representation, isRulesEnabled, rulesMatched, contractStatus, requestReason, responseReason, contractContent, elapsedTime, color}) => {
+const EntryViewer: React.FC<Props> = ({representation, isRulesEnabled, rulesMatched, contractStatus, requestReason, responseReason, contractContent, elapsedTime, color, updateQuery}) => {
     return <AutoRepresentation
         representation={representation}
         isRulesEnabled={isRulesEnabled}
@@ -123,6 +124,7 @@ const EntryViewer: React.FC<Props> = ({representation, isRulesEnabled, rulesMatc
         contractContent={contractContent}
         elapsedTime={elapsedTime}
         color={color}
+        updateQuery={updateQuery}
     />
 };
 

ui/src/components/EntryListItem/EntryListItem.tsx

@@ -2,7 +2,7 @@ import React from "react";
 import styles from './EntryListItem.module.sass';
 import StatusCode, {getClassification, StatusCodeClassification} from "../UI/StatusCode";
 import Protocol, {ProtocolInterface} from "../UI/Protocol"
-import {EndpointPath} from "../UI/EndpointPath";
+import {Summary} from "../UI/Summary";
 import ingoingIconSuccess from "../assets/ingoing-traffic-success.svg"
 import ingoingIconFailure from "../assets/ingoing-traffic-failure.svg"
 import ingoingIconNeutral from "../assets/ingoing-traffic-neutral.svg"
@@ -15,7 +15,7 @@ interface Entry {
     method?: string,
     summary: string,
     service: string,
-    id: string,
+    id: number,
     statusCode?: number;
     url?: string;
     timestamp: Date;
@@ -40,9 +40,10 @@ interface EntryProps {
     setFocusedEntryId: (id: string) => void;
     isSelected?: boolean;
     style: object;
+    updateQuery: any;
 }
 
-export const EntryItem: React.FC<EntryProps> = ({entry, setFocusedEntryId, isSelected, style}) => {
+export const EntryItem: React.FC<EntryProps> = ({entry, setFocusedEntryId, isSelected, style, updateQuery}) => {
     const classification = getClassification(entry.statusCode)
     const numberOfRules = entry.rules.numberOfRules
     let ingoingIcon;
@@ -115,10 +116,10 @@ export const EntryItem: React.FC<EntryProps> = ({entry, setFocusedEntryId, isSel
 
     return <>
         <div
-            id={entry.id}
+            id={entry.id.toString()}
             className={`${styles.row}
             ${isSelected && !rule && !contractEnabled ? styles.rowSelected : additionalRulesProperties}`}
-            onClick={() => setFocusedEntryId(entry.id)}
+            onClick={() => setFocusedEntryId(entry.id.toString())}
             style={{
                 border: isSelected ? `1px ${entry.protocol.backgroundColor} solid` : "1px transparent solid",
                 position: "absolute",
@@ -127,14 +128,26 @@ export const EntryItem: React.FC<EntryProps> = ({entry, setFocusedEntryId, isSel
                 width: "calc(100% - 25px)",
             }}
         >
-            <Protocol protocol={entry.protocol} horizontal={false}/>
+            <Protocol
+                protocol={entry.protocol}
+                horizontal={false}
+                updateQuery={updateQuery}
+            />
             {((entry.protocol.name === "http" && "statusCode" in entry) || entry.statusCode !== 0) && <div>
-                <StatusCode statusCode={entry.statusCode}/>
+                <StatusCode statusCode={entry.statusCode} updateQuery={updateQuery}/>
             </div>}
             <div className={styles.endpointServiceContainer}>
-                <EndpointPath method={entry.method} path={entry.summary}/>
+                <Summary method={entry.method} summary={entry.summary} updateQuery={updateQuery}/>
                 <div className={styles.service}>
-                    <span title="Service Name">{entry.service}</span>
+                    <span
+                        title="Service Name"
+                        className="queryable"
+                        onClick={() => {
+                            updateQuery(`service == "${entry.service}"`)
+                        }}
+                    >
+                        {entry.service}
+                    </span>
                 </div>
             </div>
             {
@@ -152,17 +165,53 @@ export const EntryItem: React.FC<EntryProps> = ({entry, setFocusedEntryId, isSel
                 : ""
             }
             <div className={styles.separatorRight}>
-                <span className={styles.port} title="Source Port">{entry.sourcePort}</span>
+                <span
+                    className={`queryable ${styles.port}`}
+                    title="Source Port"
+                    onClick={() => {
+                        updateQuery(`src.port == "${entry.sourcePort}"`)
+                    }}
+                >
+                    {entry.sourcePort}
+                </span>
                 {entry.isOutgoing ?
-                    <img src={outgoingIcon} alt="Ingoing traffic" title="Ingoing"/>
+                    <img
+                        src={outgoingIcon}
+                        alt="Ingoing traffic"
+                        title="Ingoing"
+                        onClick={() => {
+                            updateQuery(`outgoing == true`)
+                        }}
+                    />
                     :
-                    <img src={ingoingIcon} alt="Outgoing traffic" title="Outgoing"/>
+                    <img
+                        src={ingoingIcon}
+                        alt="Outgoing traffic"
+                        title="Outgoing"
+                        onClick={() => {
+                            updateQuery(`outgoing == false`)
+                        }}
+                    />
                 }
-                <span className={styles.port} title="Destination Port">{entry.destinationPort}</span>
+                <span
+                    className={`queryable ${styles.port}`}
+                    title="Destination Port"
+                    onClick={() => {
+                        updateQuery(`dst.port == "${entry.destinationPort}"`)
+                    }}
+                >
+                    {entry.destinationPort}
+                </span>
             </div>
             <div className={styles.timestamp}>
-                <span title="Timestamp">
+                <span
-                    {new Date(+entry.timestamp)?.toLocaleString()}
+                    title="Timestamp"
+                    className="queryable"
+                    onClick={() => {
+                        updateQuery(`timestamp >= datetime("${new Date(+entry.timestamp)?.toLocaleString("en-US", {timeZone: 'UTC' })}")`)
+                    }}
+                >
+                    {new Date(+entry.timestamp)?.toLocaleString("en-US")}
                 </span>
             </div>
         </div>

ui/src/components/Filters.tsx

@@ -1,137 +1,303 @@
-import React from "react";
+import React, {useRef, useState} from "react";
 import styles from './style/Filters.module.sass';
-import {FilterSelect} from "./UI/FilterSelect";
+import {Button, Grid, Modal, Box, Typography, Backdrop, Fade, Divider} from "@material-ui/core";
-import {TextField} from "@material-ui/core";
+import CodeEditor from '@uiw/react-textarea-code-editor';
-import {ALL_KEY} from "./UI/Select";
+import MenuBookIcon from '@material-ui/icons/MenuBook';
+import {SyntaxHighlighter} from "./UI/SyntaxHighlighter/index";
+import filterUIExample1 from "./assets/filter-ui-example-1.png"
+import filterUIExample2 from "./assets/filter-ui-example-2.png"
 
 interface FiltersProps {
-    methodsFilter: Array<string>;
+    query: string
-    setMethodsFilter: (methods: Array<string>) => void;
+    setQuery: any
-    statusFilter: Array<string>;
+    backgroundColor: string
-    setStatusFilter: (methods: Array<string>) => void;
+    ws: any
-    pathFilter: string
+    openWebSocket: (query: string) => void;
-    setPathFilter: (val: string) => void;
-    serviceFilter: string
-    setServiceFilter: (val: string) => void;
 }
 
-export const Filters: React.FC<FiltersProps> = ({methodsFilter, setMethodsFilter, statusFilter, setStatusFilter, pathFilter, setPathFilter, serviceFilter, setServiceFilter}) => {
+export const Filters: React.FC<FiltersProps> = ({query, setQuery, backgroundColor, ws, openWebSocket}) => {
-
     return <div className={styles.container}>
-        <MethodFilter methodsFilter={methodsFilter} setMethodsFilter={setMethodsFilter}/>
+        <QueryForm
-        <StatusTypesFilter statusFilter={statusFilter} setStatusFilter={setStatusFilter}/>
+            query={query}
-        <ServiceFilter serviceFilter={serviceFilter} setServiceFilter={setServiceFilter}/>
+            setQuery={setQuery}
-        <PathFilter pathFilter={pathFilter} setPathFilter={setPathFilter}/>
+            backgroundColor={backgroundColor}
+            ws={ws}
+            openWebSocket={openWebSocket}
+        />
     </div>;
 };
 
-const _toUpperCase = v => v.toUpperCase();
+interface QueryFormProps {
+    query: string
+    setQuery: any
+    backgroundColor: string
+    ws: any
+    openWebSocket: (query: string) => void;
+}
 
-const FilterContainer: React.FC = ({children}) => {
+const style = {
-    return <div className={styles.filterContainer}>
+    position: 'absolute',
-        {children}
+    top: '50%',
-    </div>;
+    left: '50%',
+    transform: 'translate(-50%, -50%)',
+    width: '80vw',
+    bgcolor: 'background.paper',
+    borderRadius: '5px',
+    boxShadow: 24,
+    p: 4,
+    color: '#000',
 };
 
-enum HTTPMethod {
+export const QueryForm: React.FC<QueryFormProps> = ({query, setQuery, backgroundColor, ws, openWebSocket}) => {
-    GET = "get",
-    PUT = "put",
-    POST = "post",
-    DELETE = "delete",
-    OPTIONS="options",
-    PATCH = "patch"
-}
 
-interface MethodFilterProps {
+    const formRef = useRef<HTMLFormElement>(null);
-    methodsFilter: Array<string>;
-    setMethodsFilter: (methods: Array<string>) => void;
-}
 
-const MethodFilter: React.FC<MethodFilterProps> = ({methodsFilter, setMethodsFilter}) => {
+    const [openModal, setOpenModal] = useState(false);
 
-    const methodClicked = (val) => {
+    const handleOpenModal = () => setOpenModal(true);
-        if(val === ALL_KEY) {
+    const handleCloseModal = () => setOpenModal(false);
-            setMethodsFilter([]);
+
-            return;
+    const handleChange = async (e) => {
-        }
+        setQuery(e.target.value);
-        if(methodsFilter.includes(val)) {
-            setMethodsFilter(methodsFilter.filter(method => method !== val))
-        } else {
-            setMethodsFilter([...methodsFilter, val]);
-        }
     }
 
-    return <FilterContainer>
+    const handleSubmit = (e) => {
-        <FilterSelect
+        ws.close()
-            items={Object.values(HTTPMethod)}
+        openWebSocket(query)
-            allowMultiple={true}
+        e.preventDefault();
-            value={methodsFilter}
-            onChange={(val) => methodClicked(val)}
-            transformDisplay={_toUpperCase}
-            label={"Methods"}
-        />
-    </FilterContainer>;
-};
-
-export enum StatusType {
-    SUCCESS = "success",
-    ERROR = "error"
-}
-
-interface StatusTypesFilterProps {
-    statusFilter: Array<string>;
-    setStatusFilter: (methods: Array<string>) => void;
-}
-
-const StatusTypesFilter: React.FC<StatusTypesFilterProps> = ({statusFilter, setStatusFilter}) => {
-
-    const statusClicked = (val) => {
-        if(val === ALL_KEY) {
-            setStatusFilter([]);
-            return;
-        }
-        setStatusFilter([val]);
     }
 
-    return <FilterContainer>
+    return <>
-        <FilterSelect
+        <form
-            items={Object.values(StatusType)}
+            ref={formRef}
-            allowMultiple={true}
+            onSubmit={handleSubmit}
-            value={statusFilter}
+            style={{
-            onChange={(val) => statusClicked(val)}
+                width: '100%',
-            transformDisplay={_toUpperCase}
+            }}
-            label="Status"
+        >
-        />
+            <Grid container spacing={2}>
-    </FilterContainer>;
+                <Grid
-};
+                    item
+                    xs={8}
+                    style={{
+                        maxHeight: '25vh',
+                        overflowY: 'auto',
+                    }}
+                >
+                    <label>
+                        <CodeEditor
+                            value={query}
+                            language="py"
+                            placeholder="Mizu Filter Syntax"
+                            onChange={handleChange}
+                            padding={8}
+                            style={{
+                                fontSize: 14,
+                                backgroundColor: `${backgroundColor}`,
+                                fontFamily: 'ui-monospace,SFMono-Regular,SF Mono,Consolas,Liberation Mono,Menlo,monospace',
+                            }}
+                        />
+                    </label>
+                </Grid>
+                <Grid item xs={4}>
+                    <Button type="submit" variant="contained" style={{margin: "2px 0px 0px 0px"}}>Apply</Button>
+                    <Button
+                        title="Open Filtering Guide (Cheatsheet)"
+                        variant="contained"
+                        style={{margin: "2px 0px 0px 10px", minWidth: "26px"}}
+                        onClick={handleOpenModal}
+                    >
+                        <MenuBookIcon fontSize="inherit"></MenuBookIcon>
+                    </Button>
+                </Grid>
+            </Grid>
+        </form>
 
-interface PathFilterProps {
+        <Modal
-    pathFilter: string;
+            aria-labelledby="transition-modal-title"
-    setPathFilter: (val: string) => void;
+            aria-describedby="transition-modal-description"
+            open={openModal}
+            onClose={handleCloseModal}
+            closeAfterTransition
+            BackdropComponent={Backdrop}
+            BackdropProps={{
+                timeout: 500,
+            }}
+            style={{overflow: 'auto'}}
+        >
+            <Fade in={openModal}>
+                <Box sx={style}>
+                    <Typography id="modal-modal-title" variant="h5" component="h2" style={{textAlign: 'center'}}>
+                        Filtering Guide (Cheatsheet)
+                    </Typography>
+                    <Typography id="modal-modal-description">
+                        <p>Mizu has a rich filtering syntax that let's you query the results both flexibly and efficiently.</p>
+                        <p>Here are some examples that you can try;</p>
+                    </Typography>
+                    <Grid container>
+                        <Grid item xs style={{margin: "10px"}}>
+                            <Typography id="modal-modal-description">
+                                This is a simple query that matches to HTTP packets with request path "/catalogue":
+                            </Typography>
+                            <SyntaxHighlighter
+                                isWrapped={false}
+                                showLineNumbers={false}
+                                code={`http and request.path == "/catalogue"`}
+                                language="python"
+                            />
+                            <Typography id="modal-modal-description">
+                                The same query can be negated for HTTP path and written like this:
+                            </Typography>
+                            <SyntaxHighlighter
+                                isWrapped={false}
+                                showLineNumbers={false}
+                                code={`http and request.path != "/catalogue"`}
+                                language="python"
+                            />
+                            <Typography id="modal-modal-description">
+                                The syntax supports regular expressions. Here is a query that matches the HTTP requests that send JSON to a server:
+                            </Typography>
+                            <SyntaxHighlighter
+                                isWrapped={false}
+                                showLineNumbers={false}
+                                code={`http and request.headers["Accept"] == r"application/json.*"`}
+                                language="python"
+                            />
+                            <Typography id="modal-modal-description">
+                                Here is another query that matches HTTP responses with status code 4xx:
+                            </Typography>
+                            <SyntaxHighlighter
+                                isWrapped={false}
+                                showLineNumbers={false}
+                                code={`http and response.status == r"4.*"`}
+                                language="python"
+                            />
+                            <Typography id="modal-modal-description">
+                                The same exact query can be as integer comparison:
+                            </Typography>
+                            <SyntaxHighlighter
+                                isWrapped={false}
+                                showLineNumbers={false}
+                                code={`http and response.status >= 400`}
+                                language="python"
+                            />
+                            <Typography id="modal-modal-description">
+                                The results can be queried based on their timestamps:
+                            </Typography>
+                            <SyntaxHighlighter
+                                isWrapped={false}
+                                showLineNumbers={false}
+                                code={`timestamp < datetime("10/28/2021, 9:13:02 PM")`}
+                                language="python"
+                            />
+                        </Grid>
+                        <Divider orientation="vertical" flexItem />
+                        <Grid item xs style={{margin: "10px"}}>
+                            <Typography id="modal-modal-description">
+                                Since Mizu supports various protocols like gRPC, AMQP, Kafka and Redis. It's possible to write complex queries that match multiple protocols like this:
+                            </Typography>
+                            <SyntaxHighlighter
+                                isWrapped={false}
+                                showLineNumbers={false}
+                                code={`(http and request.method == "PUT") or (amqp and request.queue.startsWith("test"))\n or (kafka and response.payload.errorCode == 2) or (redis and request.key == "example")\n or (grpc and request.headers[":path"] == r".*foo.*")`}
+                                language="python"
+                            />
+                            <Typography id="modal-modal-description">
+                                By clicking the UI elements in both left-pane and right-pane, you can automatically select a field and update the query:
+                            </Typography>
+                            <img
+                                src={filterUIExample1}
+                                width={600}
+                                alt="Clicking to UI elements (left-pane)"
+                                title="Clicking to UI elements (left-pane)"
+                            />
+                            <Typography id="modal-modal-description">
+                                Such that; clicking this in left-pane, would append the query below:
+                            </Typography>
+                            <SyntaxHighlighter
+                                isWrapped={false}
+                                showLineNumbers={false}
+                                code={`and service == "http://carts.sock-shop"`}
+                                language="python"
+                            />
+                            <Typography id="modal-modal-description">
+                                Another queriable UI element example, this time from the right-pane:
+                            </Typography>
+                            <img
+                                src={filterUIExample2}
+                                width={300}
+                                alt="Clicking to UI elements (right-pane)"
+                                title="Clicking to UI elements (right-pane)"
+                            />
+                            <Typography id="modal-modal-description">
+                                A query that compares one selector to another is also a valid query:
+                            </Typography>
+                            <SyntaxHighlighter
+                                isWrapped={false}
+                                showLineNumbers={false}
+                                code={`http and (request.query["x"] == response.headers["y"]\n or response.content.text.contains(request.query["x"]))`}
+                                language="python"
+                            />
+                        </Grid>
+                        <Divider orientation="vertical" flexItem />
+                        <Grid item xs style={{margin: "10px"}}>
+                            <Typography id="modal-modal-description">
+                                There are a few helper methods included the in the filter language* to help building queries more easily.
+                            </Typography>
+                            <br></br>
+                            <Typography id="modal-modal-description">
+                                true if the given selector's value starts with the string:
+                            </Typography>
+                            <SyntaxHighlighter
+                                isWrapped={false}
+                                showLineNumbers={false}
+                                code={`request.path.startsWith("something")`}
+                                language="python"
+                            />
+                            <Typography id="modal-modal-description">
+                                true if the given selector's value ends with the string:
+                            </Typography>
+                            <SyntaxHighlighter
+                                isWrapped={false}
+                                showLineNumbers={false}
+                                code={`request.path.endsWith("something")`}
+                                language="python"
+                            />
+                            <Typography id="modal-modal-description">
+                                true if the given selector's value contains the string:
+                            </Typography>
+                            <SyntaxHighlighter
+                                isWrapped={false}
+                                showLineNumbers={false}
+                                code={`request.path.contains("something")`}
+                                language="python"
+                            />
+                            <Typography id="modal-modal-description">
+                                returns the UNIX timestamp which is the equivalent of the time that's provided by the string. Invalid input evaluates to false:
+                            </Typography>
+                            <SyntaxHighlighter
+                                isWrapped={false}
+                                showLineNumbers={false}
+                                code={`timestamp >= datetime("10/19/2021, 6:29:02 PM")`}
+                                language="python"
+                            />
+                            <Typography id="modal-modal-description">
+                                limits the number of records that are streamed back as a result of a query. Always evaluates to true:
+                            </Typography>
+                            <SyntaxHighlighter
+                                isWrapped={false}
+                                showLineNumbers={false}
+                                code={`and limit(100)`}
+                                language="python"
+                            />
+                        </Grid>
+                    </Grid>
+                    <br></br>
+                    <Typography id="modal-modal-description" style={{fontSize: 12, fontStyle: 'italic'}}>
+                        *The filtering functionality is provided through <b>Basenine</b> database server. Please refer to <a href="https://github.com/up9inc/basenine/wiki/BFL-Syntax-Reference"><b>BFL Syntax Reference</b></a> for more information.
+                    </Typography>
+                </Box>
+            </Fade>
+        </Modal>
+    </>
 }
-
-const PathFilter: React.FC<PathFilterProps> = ({pathFilter, setPathFilter}) => {
-
-    return <FilterContainer>
-        <div className={styles.filterLabel}>Path</div>
-        <div>
-            <TextField value={pathFilter} variant="outlined" className={styles.filterText} style={{minWidth: '150px'}} onChange={(e: any) => setPathFilter(e.target.value)}/>
-        </div>
-    </FilterContainer>;
-};
-
-interface ServiceFilterProps {
-    serviceFilter: string;
-    setServiceFilter: (val: string) => void;
-}
-
-const ServiceFilter: React.FC<ServiceFilterProps> = ({serviceFilter, setServiceFilter}) => {
-
-    return <FilterContainer>
-        <div className={styles.filterLabel}>Service</div>
-        <div>
-            <TextField value={serviceFilter} variant="outlined" className={styles.filterText} style={{minWidth: '150px'}} onChange={(e: any) => setServiceFilter(e.target.value)}/>
-        </div>
-    </FilterContainer>;
-};
-

ui/src/components/TrafficPage.tsx

@@ -10,6 +10,8 @@ import pauseIcon from './assets/pause.svg';
 import variables from '../variables.module.scss';
 import {StatusBar} from "./UI/StatusBar";
 import Api, {MizuWebsocketURL} from "../helpers/api";
+import { ToastContainer, toast } from 'react-toastify';
+import 'react-toastify/dist/ReactToastify.css';
 
 const useLayoutStyles = makeStyles(() => ({
     details: {
@@ -34,7 +36,6 @@ const useLayoutStyles = makeStyles(() => ({
 enum ConnectionStatus {
     Closed,
     Connected,
-    Paused
 }
 
 interface TrafficPageProps {
@@ -52,25 +53,52 @@ export const TrafficPage: React.FC<TrafficPageProps> = ({setAnalyzeStatus, onTLS
     const [focusedEntryId, setFocusedEntryId] = useState(null);
     const [selectedEntryData, setSelectedEntryData] = useState(null);
     const [connection, setConnection] = useState(ConnectionStatus.Closed);
-    const [noMoreDataTop, setNoMoreDataTop] = useState(false);
-    const [noMoreDataBottom, setNoMoreDataBottom] = useState(false);
-
-    const [methodsFilter, setMethodsFilter] = useState([]);
-    const [statusFilter, setStatusFilter] = useState([]);
-    const [pathFilter, setPathFilter] = useState("");
-    const [serviceFilter, setServiceFilter] = useState("");
 
     const [tappingStatus, setTappingStatus] = useState(null);
 
     const [disableScrollList, setDisableScrollList] = useState(false);
 
+    const [query, setQueryDefault] = useState("");
+    const [queryBackgroundColor, setQueryBackgroundColor] = useState("#f5f5f5");
+
+    const [queriedCurrent, setQueriedCurrent] = useState(0);
+    const [queriedTotal, setQueriedTotal] = useState(0);
+
+    const [startTime, setStartTime] = useState(0);
+
+    const setQuery = async (query) => {
+        if (!query) {
+            setQueryBackgroundColor("#f5f5f5")
+        } else {
+            const data = await api.validateQuery(query);
+            if (data.valid) {
+                setQueryBackgroundColor("#d2fad2")
+            } else {
+                setQueryBackgroundColor("#fad6dc")
+            }
+        }
+        setQueryDefault(query)
+    }
+
+    const updateQuery = (addition) => {
+        if (query) {
+            setQuery(`${query} and ${addition}`)
+        } else {
+            setQuery(addition)
+        }
+    }
+
     const ws = useRef(null);
 
     const listEntry = useRef(null);
 
-    const openWebSocket = () => {
+    const openWebSocket = (query) => {
+        setEntries([])
         ws.current = new WebSocket(MizuWebsocketURL);
-        ws.current.onopen = () => setConnection(ConnectionStatus.Connected);
+        ws.current.onopen = () => {
+            ws.current.send(query)
+            setConnection(ConnectionStatus.Connected);
+        }
         ws.current.onclose = () => setConnection(ConnectionStatus.Closed);
     }
 
@@ -81,11 +109,7 @@ export const TrafficPage: React.FC<TrafficPageProps> = ({setAnalyzeStatus, onTLS
             switch (message.messageType) {
                 case "entry":
                     const entry = message.data
-                    if (connection === ConnectionStatus.Paused) {
+                    if (!focusedEntryId) setFocusedEntryId(entry.id.toString())
-                        setNoMoreDataBottom(false)
-                        return;
-                    }
-                    if (!focusedEntryId) setFocusedEntryId(entry.id)
                     let newEntries = [...entries];
                     setEntries([...newEntries, entry])
                     if(listEntry.current) {
@@ -103,6 +127,25 @@ export const TrafficPage: React.FC<TrafficPageProps> = ({setAnalyzeStatus, onTLS
                 case "outboundLink":
                     onTLSDetected(message.Data.DstIP);
                     break;
+                case "toast":
+                    toast[message.data.type](message.data.text, {
+                        position: "bottom-right",
+                        theme: "colored",
+                        autoClose: message.data.autoClose,
+                        hideProgressBar: false,
+                        closeOnClick: true,
+                        pauseOnHover: true,
+                        draggable: true,
+                        progress: undefined,
+                    });
+                    break;
+                case "queryMetadata":
+                    setQueriedCurrent(message.data.current)
+                    setQueriedTotal(message.data.total)
+                    break;
+                case "startTime":
+                    setStartTime(message.data);
+                    break;
                 default:
                     console.error(`unsupported websocket message type, Got: ${message.messageType}`)
             }
@@ -111,7 +154,7 @@ export const TrafficPage: React.FC<TrafficPageProps> = ({setAnalyzeStatus, onTLS
 
     useEffect(() => {
         (async () => {
-            openWebSocket();
+            openWebSocket("rlimit(100)");
             try{
                 const tapStatusResponse = await api.tapStatus();
                 setTappingStatus(tapStatusResponse);
@@ -139,14 +182,17 @@ export const TrafficPage: React.FC<TrafficPageProps> = ({setAnalyzeStatus, onTLS
     }, [focusedEntryId])
 
     const toggleConnection = () => {
-        setConnection(connection === ConnectionStatus.Connected ? ConnectionStatus.Paused : ConnectionStatus.Connected);
+        if (connection === ConnectionStatus.Connected) {
+            ws.current.close();
+        } else {
+            openWebSocket(query);
+            setConnection(ConnectionStatus.Connected);
+        }
     }
 
     const getConnectionStatusClass = (isContainer) => {
         const container = isContainer ? "Container" : "";
         switch (connection) {
-            case ConnectionStatus.Paused:
-                return "orangeIndicator" + container;
             case ConnectionStatus.Connected:
                 return "greenIndicator" + container;
             default:
@@ -156,8 +202,6 @@ export const TrafficPage: React.FC<TrafficPageProps> = ({setAnalyzeStatus, onTLS
 
     const getConnectionTitle = () => {
         switch (connection) {
-            case ConnectionStatus.Paused:
-                return "traffic paused";
             case ConnectionStatus.Connected:
                 return "connected, waiting for traffic"
             default:
@@ -176,8 +220,10 @@ export const TrafficPage: React.FC<TrafficPageProps> = ({setAnalyzeStatus, onTLS
     return (
         <div className="TrafficPage">
             <div className="TrafficPageHeader">
-                {connection !== ConnectionStatus.Closed && <img style={{cursor: "pointer", marginRight: 15, height: 30}} alt="pause"
+                <img className="playPauseIcon" style={{visibility: connection === ConnectionStatus.Connected ? "visible" : "hidden"}} alt="pause"
-                     src={connection === ConnectionStatus.Connected ? pauseIcon : playIcon} onClick={toggleConnection}/>}
+                    src={pauseIcon} onClick={toggleConnection}/>
+                <img className="playPauseIcon" style={{position: "absolute", visibility: connection === ConnectionStatus.Connected ? "hidden" : "visible"}} alt="play"
+                     src={playIcon} onClick={toggleConnection}/>
                 <div className="connectionText">
                     {getConnectionTitle()}
                     <div className={"indicatorContainer " + getConnectionStatusClass(true)}>
@@ -185,42 +231,51 @@ export const TrafficPage: React.FC<TrafficPageProps> = ({setAnalyzeStatus, onTLS
                     </div>
                 </div>
             </div>
-            {entries.length > 0 && <div className="TrafficPage-Container">
+            {<div className="TrafficPage-Container">
                 <div className="TrafficPage-ListContainer">
-                    <Filters methodsFilter={methodsFilter}
+                    <Filters
-                                setMethodsFilter={setMethodsFilter}
+                        query={query}
-                                statusFilter={statusFilter}
+                        setQuery={setQuery}
-                                setStatusFilter={setStatusFilter}
+                        backgroundColor={queryBackgroundColor}
-                                pathFilter={pathFilter}
+                        ws={ws.current}
-                                setPathFilter={setPathFilter}
+                        openWebSocket={openWebSocket}
-                                serviceFilter={serviceFilter}
-                                setServiceFilter={setServiceFilter}
                     />
                     <div className={styles.container}>
-                        <EntriesList entries={entries}
+                        <EntriesList
-                                        setEntries={setEntries}
+                            entries={entries}
-                                        focusedEntryId={focusedEntryId}
+                            setEntries={setEntries}
-                                        setFocusedEntryId={setFocusedEntryId}
+                            focusedEntryId={focusedEntryId}
-                                        connectionOpen={connection === ConnectionStatus.Connected}
+                            setFocusedEntryId={setFocusedEntryId}
-                                        noMoreDataBottom={noMoreDataBottom}
+                            connectionOpen={connection === ConnectionStatus.Connected}
-                                        setNoMoreDataBottom={setNoMoreDataBottom}
+                            listEntryREF={listEntry}
-                                        noMoreDataTop={noMoreDataTop}
+                            onScrollEvent={onScrollEvent}
-                                        setNoMoreDataTop={setNoMoreDataTop}
+                            scrollableList={disableScrollList}
-                                        methodsFilter={methodsFilter}
+                            ws={ws.current}
-                                        statusFilter={statusFilter}
+                            openWebSocket={openWebSocket}
-                                        pathFilter={pathFilter}
+                            query={query}
-                                        serviceFilter={serviceFilter}
+                            updateQuery={updateQuery}
-                                        listEntryREF={listEntry}
+                            queriedCurrent={queriedCurrent}
-                                        onScrollEvent={onScrollEvent}
+                            queriedTotal={queriedTotal}
-                                        scrollableList={disableScrollList}
+                            startTime={startTime}
                         />
                     </div>
                 </div>
                 <div className={classes.details}>
-                    {selectedEntryData && <EntryDetailed entryData={selectedEntryData}/>}
+                    {selectedEntryData && <EntryDetailed entryData={selectedEntryData} updateQuery={updateQuery}/>}
                 </div>
             </div>}
             {tappingStatus?.pods != null && <StatusBar tappingStatus={tappingStatus}/>}
+            <ToastContainer
+                position="bottom-right"
+                autoClose={5000}
+                hideProgressBar={false}
+                newestOnTop={false}
+                closeOnClick
+                rtl={false}
+                pauseOnFocusLoss
+                draggable
+                pauseOnHover
+            />
         </div>
     )
 };

ui/src/components/UI/EndpointPath.tsx

@@ -1,15 +0,0 @@
-import miscStyles from "./style/misc.module.sass";
-import React from "react";
-import styles from './style/EndpointPath.module.sass';
-
-interface EndpointPathProps {
-    method: string,
-    path: string
-}
-
-export const EndpointPath: React.FC<EndpointPathProps> = ({method, path}) => {
-    return <div className={styles.container}>
-        {method && <span title="Method" className={`${miscStyles.protocol} ${miscStyles.method}`}>{method}</span>}
-        {path && <div title="Summary" className={styles.path}>{path}</div>}
-    </div>
-};

ui/src/components/UI/FilterSelect.tsx

@@ -1,28 +0,0 @@
-import React from "react";
-import { MenuItem } from '@material-ui/core';
-import style from './style/FilterSelect.module.sass';
-import { Select, SelectProps } from "./Select";
-
-interface FilterSelectProps extends SelectProps {
-    items: string[];
-    value: string | string[];
-    onChange: (string) => void;
-    label?: string;
-    allowMultiple?: boolean;
-    transformDisplay?: (string) => string;
-}
-
-export const FilterSelect: React.FC<FilterSelectProps> = ({items, value, onChange, label, allowMultiple= false, transformDisplay}) => {
-    return <Select
-        value={value}
-        multiple={allowMultiple}
-        label={label}
-        onChange={onChange}
-        transformDisplay={transformDisplay}
-        labelOnTop={true}
-        labelClassName={style.SelectLabel}
-        trimItemsWhenMultiple={true}
-    >
-        {items?.map(item => <MenuItem key={item} value={item}><span className='uppercase'>{item}</span></MenuItem>)}
-    </Select>
-};

ui/src/components/UI/Protocol.tsx

@@ -4,7 +4,8 @@ import styles from './style/Protocol.module.sass';
 export interface ProtocolInterface {
     name: string
     longName: string
-    abbreviation: string
+    abbr: string
+    macro: string
     backgroundColor: string
     foregroundColor: string
     fontSize: number
@@ -16,9 +17,10 @@ export interface ProtocolInterface {
 interface ProtocolProps {
     protocol: ProtocolInterface
     horizontal: boolean
+    updateQuery: any
 }
 
-const Protocol: React.FC<ProtocolProps> = ({protocol, horizontal}) => {
+const Protocol: React.FC<ProtocolProps> = ({protocol, horizontal, updateQuery}) => {
     if (horizontal) {
         return <a target="_blank" rel="noopener noreferrer" href={protocol.referenceLink}>
             <span
@@ -28,25 +30,26 @@ const Protocol: React.FC<ProtocolProps> = ({protocol, horizontal}) => {
                     color: protocol.foregroundColor,
                     fontSize: 13,
                 }}
-                title={protocol.abbreviation}
+                title={protocol.abbr}
             >
                 {protocol.longName}
             </span>
         </a>
     } else {
-        return <a target="_blank" rel="noopener noreferrer" href={protocol.referenceLink}>
+        return <span
-            <span
+            className={`${styles.base} ${styles.vertical}`}
-                className={`${styles.base} ${styles.vertical}`}
+            style={{
-                style={{
+                backgroundColor: protocol.backgroundColor,
-                    backgroundColor: protocol.backgroundColor,
+                color: protocol.foregroundColor,
-                    color: protocol.foregroundColor,
+                fontSize: protocol.fontSize,
-                    fontSize: protocol.fontSize,
+            }}
-                }}
+            title={protocol.longName}
-                title={protocol.longName}
+            onClick={() => {
-            >
+                updateQuery(protocol.macro)
-                {protocol.abbreviation}
+            }}
-            </span>
+        >
-        </a>
+            {protocol.abbr}
+        </span>
     }
 };
 

ui/src/components/UI/StatusCode.tsx

@@ -9,16 +9,21 @@ export enum StatusCodeClassification {
 
 interface EntryProps {
     statusCode: number
+    updateQuery: any
 }
 
-const StatusCode: React.FC<EntryProps> = ({statusCode}) => {
+const StatusCode: React.FC<EntryProps> = ({statusCode, updateQuery}) => {
 
     const classification = getClassification(statusCode)
 
     return <span
         title="Status Code"
-        className={`${styles[classification]} ${styles.base}`}>
+        className={`queryable ${styles[classification]} ${styles.base}`}
-            {statusCode}
+        onClick={() => {
+            updateQuery(`response.status == ${statusCode}`)
+        }}
+    >
+        {statusCode}
     </span>
 };
 

ui/src/components/UI/Summary.tsx

@@ -0,0 +1,32 @@
+import miscStyles from "./style/misc.module.sass";
+import React from "react";
+import styles from './style/Summary.module.sass';
+
+interface SummaryProps {
+    method: string
+    summary: string
+    updateQuery: any
+}
+
+export const Summary: React.FC<SummaryProps> = ({method, summary, updateQuery}) => {
+    return <div className={styles.container}>
+        {method && <span
+            title="Method"
+            className={`queryable ${miscStyles.protocol} ${miscStyles.method}`}
+            onClick={() => {
+                updateQuery(`method == "${method}"`)
+            }}
+        >
+            {method}
+        </span>}
+        {summary && <div
+            title="Summary"
+            className={`queryable ${styles.summary}`}
+            onClick={() => {
+                updateQuery(`summary == "${summary}"`)
+            }}
+        >
+            {summary}
+        </div>}
+    </div>
+};

ui/src/components/UI/SyntaxHighlighter/highlighterStyle.ts

@@ -112,7 +112,7 @@ export const highlighterStyle = {
         "color": "#C6C5FE"
     },
     "operator": {
-        "color": "#EDEDED"
+        "color": "#A1A1A1"
     },
     "entity": {
         "color": "#fdab2b",

ui/src/components/UI/style/Protocol.module.sass

@@ -6,7 +6,6 @@
     background-color: #000
     color: #fff
     margin-left: -8px
-    margin-bottom: -4px
 
 .vertical
     line-height: 22px

ui/src/components/UI/style/Summary.module.sass

@@ -2,7 +2,7 @@
     display: flex
     align-items: center
 
-.path
+.summary
     text-overflow: ellipsis
     overflow: hidden
     white-space: nowrap

ui/src/components/style/EntriesList.module.sass

@@ -38,15 +38,6 @@
   border: 1px solid #627ef7
   background-color: rgba(255, 255, 255, 0.06)
 
-.spinnerContainer
-  display: flex
-  justify-content: center
-  margin-bottom: 10px
-
-.noMoreDataAvailable
-  text-align: center
-  font-weight: 600
-  color: $secondary-font-color
 .fetchButtonContainer
   width: 100%
   display: flex

ui/src/components/style/Filters.module.sass

@@ -4,9 +4,6 @@
     display: flex
     flex-direction: row
     align-items: center
-    min-height: 3rem
-    overflow-y: hidden
-    overflow-x: auto
     padding: .5rem 0
     border-bottom: 1px solid #BCC6DD
     margin-right: 20px
@@ -29,8 +26,8 @@
   input
     padding: 4px 12px
     background: $main-background-color
-    border-radius: 12px
+    border-radius: 4px
-    font-size: 12px
+    font-size: 14px
     border: 1px solid #BCC6DD
   fieldset
     border: none

ui/src/components/style/TrafficPage.sass

@@ -110,3 +110,8 @@
     align-items: center
     height: 17px
     font-size: 16px
+
+.playPauseIcon
+    cursor: pointer
+    margin-right: 15px
+    height: 30px

ui/src/helpers/api.js

@@ -29,13 +29,8 @@ export default class Api {
         return response.data;
     }
 
-    getEntry = async (entryId) => {
+    getEntry = async (id) => {
-        const response = await this.client.get(`/entries/${entryId}`);
+        const response = await this.client.get(`/entries/${id}`);
-        return response.data;
-    }
-
-    fetchEntries = async (operator, timestamp) => {
-        const response = await this.client.get(`/entries?limit=50&operator=${operator}&timestamp=${timestamp}`);
         return response.data;
     }
 
@@ -48,4 +43,11 @@ export default class Api {
         const response = await this.client.get("/status/auth");
         return response.data;
     }
+
+    validateQuery = async (query) => {
+        const form = new FormData();
+        form.append('query', query)
+        const response = await this.client.post(`/query/validate`, form);
+        return response.data;
+    }
 }

ui/src/index.sass

@@ -22,6 +22,11 @@ code
 .uppercase
   text-transform: uppercase
 
+.queryable
+  cursor: pointer
+  &:hover
+    text-decoration: underline
+
 /****
  * Button
  ***/
@@ -31,11 +36,13 @@ button
   &:not(.MuiFab-root)
     &.MuiButtonBase-root
       box-sizing: border-box
-      font-weight: 500
+      font-weight: 600
       line-height: 1
-      border-radius: 20px
+      border-radius: 4px
       letter-spacing: 0.02857em
-      text-transform: uppercase
+      background-color: $blue-color
+      color: #fff
+      text-transform: none
       img:not(.custom)
         max-width: 13px
         max-height: 13px