mirror of
https://github.com/kubeshark/kubeshark.git
synced 2025-09-06 21:11:11 +00:00
Migrate from SQLite to Basenine and introduce a new filtering syntax (#279)
* Fix the OOMKilled error by calling `debug.FreeOSMemory` periodically * Remove `MAX_NUMBER_OF_GOROUTINES` environment variable * Change the line * Increase the default value of `TCP_STREAM_CHANNEL_TIMEOUT_MS` to `10000` * Write the client and integrate to the new real-time database * Refactor the WebSocket implementaiton for `/ws` * Adapt the UI to the new filtering system * Fix the rest of the issues in the UI * Increase the buffer of the scanner * Implement accessing single records * Increase the buffer of another scanner * Populate `Request` and `Response` fields of `MizuEntry` * Add syntax highlighting for the query * Add database to `Dockerfile` * Fix some issues * Update the `realtime_dbms` Git module commit hash * Upgrade Gin version and print the query string * Revert "Upgrade Gin version and print the query string" This reverts commitaa09f904ee. * Use WebSocket's itself to query instead of the query string * Fix some errors related to conversion to HAR * Fix the issues caused by the latest merge * Fix the build error * Fix PR validation GitHub workflow * Replace the git submodule with latest Basenine version `0.1.0` Remove `realtime_client.go` and use the official client library `github.com/up9inc/basenine/client/go` instead. * Move Basenine host and port constants to `shared` module * Reliably execute and wait for Basenine to become available * Upgrade Basenine version * Properly close WebSocket and data channel * Fix the issues caused by the recent merge commit * Clean up the TypeScript code * Update `.gitignore` * Limit the database size * Add `Macros` method signature to `Dissector` interface and set the macros provided by the protocol extensions * Run `go mod tidy` on `agent` * Upgrade `github.com/up9inc/basenine/client/go` version * Implement a mechanism to update the query using click events in the UI and use it for protocol macros * Update the query on click to timestamps * Fix some issues in the WebSocket and channel handling * Update the query on clicks to status code * Update the query on clicks to method, path and service * Update the query on clicks to is outgoing, source and destination ports * Add an API endpoint to validate the query against syntax errors * Move the query background color state into `TrafficPage` * Fix the logic in `setQuery` * Display a toast message in case of a syntax error in the query * Remove a call to `fmt.Printf` * Upgrade Basenine version to `0.1.3` * Fix an issue related to getting `MAX_ENTRIES_DB_BYTES` environment variable * Have the `path` key in request details, in HTTP * Rearrange the HTTP headers for the querying * Do the same thing for `cookies` and `queryString` * Update the query on click to table elements Add the selectors for `TABLE` type representations in HTTP extension. * Update the query on click to `bodySize` and `elapsedTime` in `EntryTitle` * Add the selectors for `TABLE` type representations in AMQP extension * Add the selectors for `TABLE` type representations in Kafka extension * Add the selectors for `TABLE` type representations in Redis extension * Define a struct in `tap/api.go` for the section representation data * Add the selectors for `BODY` type representations * Add `request.path` to the HTTP request details * Change the summary string's field name from `path` to `summary` * Introduce `queryable` CSS class for queryable UI elements and underline them on hover * Instead of `N requests` at the bottom, make it `Displaying N results (queried X/Y)` and live update the values Upgrade Basenine version to `0.2.0`. * Verify the sha256sum of Basenine executable inside `Dockerfile` * Pass the start time to web UI through WebSocket and always show the `EntriesList` footer * Pipe the `stderr` of Basenine as well * Fix the layout issues related to `CodeEditor` in the UI * Use the correct `shasum` command in `Dockerfile` * Upgrade Basenine version to `0.2.1` * Limit the height of `CodeEditor` container * Remove `Paused` enum `ConnectionStatus` in UI * Fix the issue caused by the recent merge * Add the filtering guide (cheatsheet) * Update open cheatsheet button's title * Update cheatsheet content * Remove the old SQLite code, adapt the `--analyze` related code to Basenine * Change the method signature of `NewEntry` * Change the method signature of `Represent` * Introduce `HTTPPair` field in `MizuEntry` specific to HTTP * Remove `Entry`, `EntryId` and `EstimatedSizeBytes` fields from `MizuEntry` Also remove the `getEstimatedEntrySizeBytes` method. * Remove `gorm.io/gorm` dependency * Remove unused `sensitiveDataFiltering` folder * Increase the left margin of open cheatsheet button * Add `overflow: auto` to the cheatsheet `Modal` * Fix `GetEntry` method * Fix the macro for gRPC * Fix an interface conversion in case of AMQP * Fix two more interface conversion errors in AMQP * Make the `syncEntriesImpl` method blocking * Fix a grammar mistake in the cheatsheet * Adapt to the changes in the recent merge commit * Improve the cheatsheet text * Always display the timestamp in `en-US` * Upgrade Basenine version to `0.2.2` * Fix the order of closing Basenine connections and channels * Don't close the Basenine channels at all * Upgrade Basenine version to `0.2.3` * Set the initial filter to `rlimit(100)` * Make Basenine persistent * Upgrade Basenine version to `0.2.4` * Update `debug.Dockerfile` * Fix a failing test * Upgrade Basenine version to `0.2.5` * Revert "Do not show play icon when disconnected (#428)" This reverts commit8af2e562f8. * Upgrade Basenine version to `0.2.6` * Make all non-informative things informative * Make `100` a constant * Use `===` in JavaScript no matter what * Remove a forgotten `console.log` * Add a comment and update the `query` in `syncEntriesImpl` * Don't call `panic` in `GetEntry` * Replace `panic` calls in `startBasenineServer` with `logger.Log.Panicf` * Remove unnecessary `\n` characters in the logs
This commit is contained in:
M. Mert Yıldıran
GitHub
@@ -3,8 +3,8 @@ module mizuserver
|
||||
go 1.16
|
||||
|
||||
require (
|
||||
github.com/antelman107/net-wait-go v0.0.0-20210623112055-cf684aebda7b
|
||||
github.com/djherbis/atime v1.0.0
|
||||
github.com/fsnotify/fsnotify v1.4.9
|
||||
github.com/getkin/kin-openapi v0.76.0
|
||||
github.com/gin-contrib/static v0.0.1
|
||||
github.com/gin-gonic/gin v1.7.2
|
||||
@@ -16,13 +16,12 @@ require (
|
||||
github.com/op/go-logging v0.0.0-20160315200505-970db520ece7
|
||||
github.com/orcaman/concurrent-map v0.0.0-20210106121528-16402b402231
|
||||
github.com/patrickmn/go-cache v2.1.0+incompatible
|
||||
github.com/up9inc/basenine/client/go v0.0.0-20211106180626-0193408db715
|
||||
github.com/up9inc/mizu/shared v0.0.0
|
||||
github.com/up9inc/mizu/tap v0.0.0
|
||||
github.com/up9inc/mizu/tap/api v0.0.0
|
||||
github.com/yalp/jsonpath v0.0.0-20180802001716-5cc68e5049a0
|
||||
go.mongodb.org/mongo-driver v1.7.1
|
||||
gorm.io/driver/sqlite v1.1.4
|
||||
gorm.io/gorm v1.21.8
|
||||
golang.org/x/text v0.3.5 // indirect
|
||||
k8s.io/api v0.21.2
|
||||
k8s.io/apimachinery v0.21.2
|
||||
k8s.io/client-go v0.21.2
|
||||
|
||||
76
agent/go.sum
76
agent/go.sum
@@ -52,6 +52,8 @@ github.com/alecthomas/template v0.0.0-20190718012654-fb15b899a751/go.mod h1:LOuy
|
||||
github.com/alecthomas/units v0.0.0-20151022065526-2efee857e7cf/go.mod h1:ybxpYRFXyAe+OPACYpWeL0wqObRcbAqCMya13uyzqw0=
|
||||
github.com/alecthomas/units v0.0.0-20190717042225-c3de453c63f4/go.mod h1:ybxpYRFXyAe+OPACYpWeL0wqObRcbAqCMya13uyzqw0=
|
||||
github.com/andreyvit/diff v0.0.0-20170406064948-c7f18ee00883/go.mod h1:rCTlJbsFo29Kk6CurOXKm700vrz8f0KW0JNfpkRJY/8=
|
||||
github.com/antelman107/net-wait-go v0.0.0-20210623112055-cf684aebda7b h1:8m+eVxVVDDyJFidv7Ck1OwqnDaQR6pTSRGlCC2Dnw0A=
|
||||
github.com/antelman107/net-wait-go v0.0.0-20210623112055-cf684aebda7b/go.mod h1:+tQQjzrp2501Nd6JXrb9s/XsNvFK3ZbxOnCdQl/vDRo=
|
||||
github.com/armon/circbuf v0.0.0-20150827004946-bbbad097214e/go.mod h1:3U/XgcO3hCbHZ8TKRvWD2dDTCfh9M9ya+I9JpbB7O8o=
|
||||
github.com/armon/consul-api v0.0.0-20180202201655-eb2c6b5be1b6/go.mod h1:grANhF5doyWs3UAsr3K4I6qtAmlQcZDesFNEHPZAzj8=
|
||||
github.com/armon/go-metrics v0.0.0-20180917152333-f0300d1749da/go.mod h1:Q73ZrmVTwzkszR9V5SSuryQ31EELlFMUz1kKyl939pY=
|
||||
@@ -94,7 +96,6 @@ github.com/djherbis/atime v1.0.0 h1:ySLvBAM0EvOGaX7TI4dAM5lWj+RdJUCKtGSEHN8SGBg=
|
||||
github.com/djherbis/atime v1.0.0/go.mod h1:5W+KBIuTwVGcqjIfaTwt+KSYX1o6uep8dtevevQP/f8=
|
||||
github.com/docker/distribution v2.7.1+incompatible/go.mod h1:J2gT2udsDAN96Uj4KfcMRqY0/ypR+oyYUYmja8H+y+w=
|
||||
github.com/docker/go-units v0.3.3/go.mod h1:fgPhTUdO+D/Jk86RDLlptpiXQzgHJF7gydDDbaIK4Dk=
|
||||
github.com/docker/go-units v0.4.0 h1:3uh0PgVws3nIA0Q+MwDC8yjEPf9zjRfZZWXZYDct3Tw=
|
||||
github.com/docker/go-units v0.4.0/go.mod h1:fgPhTUdO+D/Jk86RDLlptpiXQzgHJF7gydDDbaIK4Dk=
|
||||
github.com/docopt/docopt-go v0.0.0-20180111231733-ee0de3bc6815/go.mod h1:WwZ+bS3ebgob9U8Nd0kOddGdZWjyMGR8Wziv+TBNwSE=
|
||||
github.com/elazarl/goproxy v0.0.0-20180725130230-947c36da3153/go.mod h1:/Zj4wYkgs4iZTTu3o/KG3Itv/qCCa8VVMlb3i9OVuzc=
|
||||
@@ -111,7 +112,6 @@ github.com/fatih/color v1.7.0/go.mod h1:Zm6kSWBoL9eyXnKyktHP6abPY2pDugNf5Kwzbycv
|
||||
github.com/form3tech-oss/jwt-go v3.2.2+incompatible h1:TcekIExNqud5crz4xD2pavyTgWiPvpYe4Xau31I0PRk=
|
||||
github.com/form3tech-oss/jwt-go v3.2.2+incompatible/go.mod h1:pbq4aXjuKjdthFRnoDwaVPLA+WlJuPGy+QneDUgJi2k=
|
||||
github.com/fsnotify/fsnotify v1.4.7/go.mod h1:jwhsz4b93w/PPRr/qN1Yymfu8t87LnFCMoQvtojpjFo=
|
||||
github.com/fsnotify/fsnotify v1.4.9 h1:hsms1Qyu0jgnwNXIxa+/V/PDsU6CfLf6CNO8H7IWoS4=
|
||||
github.com/fsnotify/fsnotify v1.4.9/go.mod h1:znqG4EE+3YCdAaPaxE2ZRY/06pZUdp0tY4IgpuI1SZQ=
|
||||
github.com/fvbommel/sortorder v1.0.1/go.mod h1:uk88iVf1ovNn1iLfgUVU2F9o5eO30ui720w+kxuqRs0=
|
||||
github.com/getkin/kin-openapi v0.76.0 h1:j77zg3Ec+k+r+GA3d8hBoXpAc6KX9TbBPrwQGBIy2sY=
|
||||
@@ -195,31 +195,7 @@ github.com/go-playground/validator/v10 v10.4.1/go.mod h1:nlOn6nFhuKACm19sB/8EGNn
|
||||
github.com/go-playground/validator/v10 v10.5.0 h1:X9rflw/KmpACwT8zdrm1upefpvdy6ur8d1kWyq6sg3E=
|
||||
github.com/go-playground/validator/v10 v10.5.0/go.mod h1:xm76BBt941f7yWdGnI2DVPFFg1UK3YY04qifoXU3lOk=
|
||||
github.com/go-stack/stack v1.8.0/go.mod h1:v0f6uXyyMGvRgIKkXu+yp6POWl0qKG85gN/melR3HDY=
|
||||
github.com/gobuffalo/attrs v0.0.0-20190224210810-a9411de4debd/go.mod h1:4duuawTqi2wkkpB4ePgWMaai6/Kc6WEz83bhFwpHzj0=
|
||||
github.com/gobuffalo/depgen v0.0.0-20190329151759-d478694a28d3/go.mod h1:3STtPUQYuzV0gBVOY3vy6CfMm/ljR4pABfrTeHNLHUY=
|
||||
github.com/gobuffalo/depgen v0.1.0/go.mod h1:+ifsuy7fhi15RWncXQQKjWS9JPkdah5sZvtHc2RXGlg=
|
||||
github.com/gobuffalo/envy v1.6.15/go.mod h1:n7DRkBerg/aorDM8kbduw5dN3oXGswK5liaSCx4T5NI=
|
||||
github.com/gobuffalo/envy v1.7.0/go.mod h1:n7DRkBerg/aorDM8kbduw5dN3oXGswK5liaSCx4T5NI=
|
||||
github.com/gobuffalo/flect v0.1.0/go.mod h1:d2ehjJqGOH/Kjqcoz+F7jHTBbmDb38yXA598Hb50EGs=
|
||||
github.com/gobuffalo/flect v0.1.1/go.mod h1:8JCgGVbRjJhVgD6399mQr4fx5rRfGKVzFjbj6RE/9UI=
|
||||
github.com/gobuffalo/flect v0.1.3/go.mod h1:8JCgGVbRjJhVgD6399mQr4fx5rRfGKVzFjbj6RE/9UI=
|
||||
github.com/gobuffalo/genny v0.0.0-20190329151137-27723ad26ef9/go.mod h1:rWs4Z12d1Zbf19rlsn0nurr75KqhYp52EAGGxTbBhNk=
|
||||
github.com/gobuffalo/genny v0.0.0-20190403191548-3ca520ef0d9e/go.mod h1:80lIj3kVJWwOrXWWMRzzdhW3DsrdjILVil/SFKBzF28=
|
||||
github.com/gobuffalo/genny v0.1.0/go.mod h1:XidbUqzak3lHdS//TPu2OgiFB+51Ur5f7CSnXZ/JDvo=
|
||||
github.com/gobuffalo/genny v0.1.1/go.mod h1:5TExbEyY48pfunL4QSXxlDOmdsD44RRq4mVZ0Ex28Xk=
|
||||
github.com/gobuffalo/gitgen v0.0.0-20190315122116-cc086187d211/go.mod h1:vEHJk/E9DmhejeLeNt7UVvlSGv3ziL+djtTr3yyzcOw=
|
||||
github.com/gobuffalo/gogen v0.0.0-20190315121717-8f38393713f5/go.mod h1:V9QVDIxsgKNZs6L2IYiGR8datgMhB577vzTDqypH360=
|
||||
github.com/gobuffalo/gogen v0.1.0/go.mod h1:8NTelM5qd8RZ15VjQTFkAW6qOMx5wBbW4dSCS3BY8gg=
|
||||
github.com/gobuffalo/gogen v0.1.1/go.mod h1:y8iBtmHmGc4qa3urIyo1shvOD8JftTtfcKi+71xfDNE=
|
||||
github.com/gobuffalo/here v0.6.0/go.mod h1:wAG085dHOYqUpf+Ap+WOdrPTp5IYcDAs/x7PLa8Y5fM=
|
||||
github.com/gobuffalo/logger v0.0.0-20190315122211-86e12af44bc2/go.mod h1:QdxcLw541hSGtBnhUc4gaNIXRjiDppFGaDqzbrBd3v8=
|
||||
github.com/gobuffalo/mapi v1.0.1/go.mod h1:4VAGh89y6rVOvm5A8fKFxYG+wIW6LO1FMTG9hnKStFc=
|
||||
github.com/gobuffalo/mapi v1.0.2/go.mod h1:4VAGh89y6rVOvm5A8fKFxYG+wIW6LO1FMTG9hnKStFc=
|
||||
github.com/gobuffalo/packd v0.0.0-20190315124812-a385830c7fc0/go.mod h1:M2Juc+hhDXf/PnmBANFCqx4DM3wRbgDvnVWeG2RIxq4=
|
||||
github.com/gobuffalo/packd v0.1.0/go.mod h1:M2Juc+hhDXf/PnmBANFCqx4DM3wRbgDvnVWeG2RIxq4=
|
||||
github.com/gobuffalo/packr/v2 v2.0.9/go.mod h1:emmyGweYTm6Kdper+iywB6YK5YzuKchGtJQZ0Odn4pQ=
|
||||
github.com/gobuffalo/packr/v2 v2.2.0/go.mod h1:CaAwI0GPIAv+5wKLtv8Afwl+Cm78K/I/VCm/3ptBN+0=
|
||||
github.com/gobuffalo/syncx v0.0.0-20190224160051-33c29581e754/go.mod h1:HhnNqWY95UYwwW3uSASeV7vtgYkT2t16hJgV3AEPUpw=
|
||||
github.com/gogo/protobuf v1.1.1/go.mod h1:r8qH/GZQm5c6nD/R0oafs1akxWv10x8SbQlK7atdtwQ=
|
||||
github.com/gogo/protobuf v1.2.1/go.mod h1:hp+jE20tsWTFYpLwKvXlhS1hjn+gTNwPg2I6zVXpSg4=
|
||||
github.com/gogo/protobuf v1.3.2 h1:Ov1cvc58UF3b5XjBnZv7+opcTcQFZebYjWzi34vdm4Q=
|
||||
@@ -251,7 +227,6 @@ github.com/golang/protobuf v1.4.1/go.mod h1:U8fpvMrcmy5pZrNK1lt4xCsGvpyWQ/VVv6QD
|
||||
github.com/golang/protobuf v1.4.2/go.mod h1:oDoupMAO8OvCJWAcko0GGGIgR6R6ocIYbsSw735rRwI=
|
||||
github.com/golang/protobuf v1.4.3 h1:JjCZWpVbqXDqFVmTfYWEVTMIYrL/NPdPSCHPJ0T/raM=
|
||||
github.com/golang/protobuf v1.4.3/go.mod h1:oDoupMAO8OvCJWAcko0GGGIgR6R6ocIYbsSw735rRwI=
|
||||
github.com/golang/snappy v0.0.1/go.mod h1:/XxbfmMg8lxefKM7IXC3fBNl/7bRcc72aCRzEWrmP2Q=
|
||||
github.com/golangplus/testing v0.0.0-20180327235837-af21d9c3145e/go.mod h1:0AA//k/eakGydO4jKRoRL2j92ZKSzTgj9tclaCrvXHk=
|
||||
github.com/google/btree v0.0.0-20180813153112-4030bb1f1f0c/go.mod h1:lNA+9X1NB3Zf8V7Ke586lFgjr2dZNuvo3lPJSGZ5JPQ=
|
||||
github.com/google/btree v1.0.0/go.mod h1:lNA+9X1NB3Zf8V7Ke586lFgjr2dZNuvo3lPJSGZ5JPQ=
|
||||
@@ -321,12 +296,6 @@ github.com/ianlancetaylor/demangle v0.0.0-20181102032728-5e5cf60278f6/go.mod h1:
|
||||
github.com/imdario/mergo v0.3.5 h1:JboBksRwiiAJWvIYJVo46AfV+IAIKZpfrSzVKj42R4Q=
|
||||
github.com/imdario/mergo v0.3.5/go.mod h1:2EnlNZ0deacrJVfApfmtdGgDfMuh/nq6Ok1EcJh5FfA=
|
||||
github.com/inconshreveable/mousetrap v1.0.0/go.mod h1:PxqpIevigyE2G7u3NXJIT2ANytuPF1OarO4DADm73n8=
|
||||
github.com/jinzhu/inflection v1.0.0 h1:K317FqzuhWc8YvSVlFMCCUb36O/S9MCKRDI7QkRKD/E=
|
||||
github.com/jinzhu/inflection v1.0.0/go.mod h1:h+uFLlag+Qp1Va5pdKtLDYj+kHp5pxUVkryuEj+Srlc=
|
||||
github.com/jinzhu/now v1.1.1/go.mod h1:d3SSVoowX0Lcu0IBviAWJpolVfI5UJVZZ7cO71lE/z8=
|
||||
github.com/jinzhu/now v1.1.2 h1:eVKgfIdy9b6zbWBMgFpfDPoAMifwSZagU9HmEU6zgiI=
|
||||
github.com/jinzhu/now v1.1.2/go.mod h1:d3SSVoowX0Lcu0IBviAWJpolVfI5UJVZZ7cO71lE/z8=
|
||||
github.com/joho/godotenv v1.3.0/go.mod h1:7hK45KPybAkOC6peb+G5yklZfMxEjkZhHbwpqxOKXbg=
|
||||
github.com/jonboulle/clockwork v0.1.0/go.mod h1:Ii8DK3G1RaLaWxj9trq07+26W01tbo22gdxWY5EU2bo=
|
||||
github.com/json-iterator/go v1.1.6/go.mod h1:+SdeFBvtyEkXs7REEP0seUULqWtbJapLOCVDaaPEHmU=
|
||||
github.com/json-iterator/go v1.1.9/go.mod h1:KdQUCv79m/52Kvf8AW2vK1V8akMuk1QjK/uOdHXbAo4=
|
||||
@@ -336,14 +305,10 @@ github.com/jstemmer/go-junit-report v0.0.0-20190106144839-af01ea7f8024/go.mod h1
|
||||
github.com/jstemmer/go-junit-report v0.9.1/go.mod h1:Brl9GWCQeLvo8nXZwPNNblvFj/XSXhF0NWZEnDohbsk=
|
||||
github.com/jtolds/gls v4.20.0+incompatible/go.mod h1:QJZ7F/aHp+rZTRtaJ1ow/lLfFfVYBRgL+9YlvaHOwJU=
|
||||
github.com/julienschmidt/httprouter v1.2.0/go.mod h1:SYymIcj16QtmaHHD7aYtjjsJG7VTCxuUUipMqKk8s4w=
|
||||
github.com/karrick/godirwalk v1.8.0/go.mod h1:H5KPZjojv4lE+QYImBI8xVtrBRgYrIVsaRPx4tDPEn4=
|
||||
github.com/karrick/godirwalk v1.10.3/go.mod h1:RoGL9dQei4vP9ilrpETWE8CLOZ1kiN0LhBygSwrAsHA=
|
||||
github.com/kisielk/errcheck v1.1.0/go.mod h1:EZBBE59ingxPouuu3KfxchcWSUPOHkagtvWXihfKN4Q=
|
||||
github.com/kisielk/errcheck v1.5.0/go.mod h1:pFxgyoBC7bSaBwPgfKdkLd5X25qrDl4LWUI2bnpBCr8=
|
||||
github.com/kisielk/gotool v1.0.0/go.mod h1:XhKaO+MFFWcvkIS/tQcRk01m1F5IRFswLeQ+oQHNcck=
|
||||
github.com/klauspost/compress v1.9.5/go.mod h1:RyIbtBH6LamlWaDj8nUwkbUhJ87Yi3uG0guNDohfE1A=
|
||||
github.com/konsorten/go-windows-terminal-sequences v1.0.1/go.mod h1:T0+1ngSBFLxvqU3pZ+m/2kptfBszLMUkC4ZK/EgS/cQ=
|
||||
github.com/konsorten/go-windows-terminal-sequences v1.0.2/go.mod h1:T0+1ngSBFLxvqU3pZ+m/2kptfBszLMUkC4ZK/EgS/cQ=
|
||||
github.com/kr/logfmt v0.0.0-20140226030751-b84e30acd515/go.mod h1:+0opPa2QZZtGFBFZlji/RkVcI2GknAs/DXo4wKdlNEc=
|
||||
github.com/kr/pretty v0.1.0/go.mod h1:dAy3ld7l9f0ibDNOQOHHMYYIIbhfbHSm3C4ZsoJORNo=
|
||||
github.com/kr/pretty v0.2.0/go.mod h1:ipq/a2n7PKx3OHsz4KJII5eveXtPO4qwEXGdVfWzfnI=
|
||||
@@ -364,16 +329,12 @@ github.com/mailru/easyjson v0.0.0-20190614124828-94de47d64c63/go.mod h1:C1wdFJiN
|
||||
github.com/mailru/easyjson v0.0.0-20190626092158-b2ccc519800e/go.mod h1:C1wdFJiN94OJF2b5HbByQZoLdCWB1Yqtg26g4irojpc=
|
||||
github.com/mailru/easyjson v0.7.0 h1:aizVhC/NAAcKWb+5QsU1iNOZb4Yws5UO2I+aIprQITM=
|
||||
github.com/mailru/easyjson v0.7.0/go.mod h1:KAzv3t3aY1NaHWoQz1+4F1ccyAH66Jk7yos7ldAVICs=
|
||||
github.com/markbates/oncer v0.0.0-20181203154359-bf2de49a0be2/go.mod h1:Ld9puTsIW75CHf65OeIOkyKbteujpZVXDpWK6YGZbxE=
|
||||
github.com/markbates/pkger v0.17.1/go.mod h1:0JoVlrol20BSywW79rN3kdFFsE5xYM+rSCQDXbLhiuI=
|
||||
github.com/markbates/safe v1.0.1/go.mod h1:nAqgmRi7cY2nqMc92/bSEeQA+R4OheNU2T1kNSCBdG0=
|
||||
github.com/mattn/go-colorable v0.0.9/go.mod h1:9vuHe8Xs5qXnSaW/c/ABM9alt+Vo+STaOChaDxuIBZU=
|
||||
github.com/mattn/go-isatty v0.0.3/go.mod h1:M+lRXTBqGeGNdLjl/ufCoiOlB5xdOkqRJdNxMWT7Zi4=
|
||||
github.com/mattn/go-isatty v0.0.12 h1:wuysRhFDzyxgEmMf5xjvJ2M9dZoWAXNNr5LSBS7uHXY=
|
||||
github.com/mattn/go-isatty v0.0.12/go.mod h1:cbi8OIDigv2wuxKPP5vlRcQ1OAZbq2CE4Kysco4FUpU=
|
||||
github.com/mattn/go-runewidth v0.0.7/go.mod h1:H031xJmbD/WCDINGzjvQ9THkh0rPKHF+m2gUSrubnMI=
|
||||
github.com/mattn/go-sqlite3 v1.14.5 h1:1IdxlwTNazvbKJQSxoJ5/9ECbEeaTTyeU7sEAZ5KKTQ=
|
||||
github.com/mattn/go-sqlite3 v1.14.5/go.mod h1:WVKg1VTActs4Qso6iwGbiFih2UIHo0ENGwNd0Lj+XmI=
|
||||
github.com/matttproud/golang_protobuf_extensions v1.0.1/go.mod h1:D8He9yQNgCq6Z5Ld7szi9bcBfOoFv/3dc6xSMkL2PC0=
|
||||
github.com/matttproud/golang_protobuf_extensions v1.0.2-0.20181231171920-c182affec369/go.mod h1:BSXmuO+STAnVfrANrmjBb36TMTDstsz7MSK+HVaYKv4=
|
||||
github.com/miekg/dns v1.0.14/go.mod h1:W1PPwlIAgtquWBMBEV9nkV9Cazfe8ScdGz/Lj7v3Nrg=
|
||||
@@ -395,7 +356,6 @@ github.com/modern-go/reflect2 v0.0.0-20180701023420-4b7aa43c6742/go.mod h1:bx2lN
|
||||
github.com/modern-go/reflect2 v1.0.1 h1:9f412s+6RmYXLWZSEzVVgPGK7C2PphHj5RJrvfx9AWI=
|
||||
github.com/modern-go/reflect2 v1.0.1/go.mod h1:bx2lNnkwVCuqBIxFjflWJWanXIb3RllmbCylyMrvgv0=
|
||||
github.com/monochromegane/go-gitignore v0.0.0-20200626010858-205db1a8cc00/go.mod h1:Pm3mSP3c5uWn86xMLZ5Sa7JB9GsEZySvHYXCTK4E9q4=
|
||||
github.com/montanaflynn/stats v0.0.0-20171201202039-1bf9dbcd8cbe/go.mod h1:wL8QJuTMNUDYhXwkmfOly8iTdp5TEcJFWZD2D7SIkUc=
|
||||
github.com/munnerz/goautoneg v0.0.0-20120707110453-a547fc61f48d/go.mod h1:+n7T8mK8HuQTcFwEeznm/DIxMOiR9yIdICNftLE1DvQ=
|
||||
github.com/mwitkow/go-conntrack v0.0.0-20161129095857-cc309e4a2223/go.mod h1:qRWi+5nqEBWmkhHvq77mSJWrCKwh8bxhgT7d/eI7P4U=
|
||||
github.com/mxk/go-flowrate v0.0.0-20140419014527-cca7078d478f h1:y5//uYreIhSUg3J1GEMiLbxo1LJaP8RfCpH6pymGZus=
|
||||
@@ -419,7 +379,6 @@ github.com/patrickmn/go-cache v2.1.0+incompatible h1:HRMgzkcYKYpi3C8ajMPV8OFXaaR
|
||||
github.com/patrickmn/go-cache v2.1.0+incompatible/go.mod h1:3Qf8kWWT7OJRJbdiICTKqZju1ZixQ/KpMGzzAfe6+WQ=
|
||||
github.com/pborman/uuid v1.2.0/go.mod h1:X/NO0urCmaxf9VXbdlT7C2Yzkj2IKimNn4k+gtPdI/k=
|
||||
github.com/pelletier/go-toml v1.2.0/go.mod h1:5z9KED0ma1S8pY6P1sdut58dfprrGBbd/94hg7ilaic=
|
||||
github.com/pelletier/go-toml v1.7.0/go.mod h1:vwGMzjaWMwyfHwgIBhI2YUM4fB6nL6lVAvS1LBMMhTE=
|
||||
github.com/peterbourgon/diskv v2.0.1+incompatible/go.mod h1:uqqh8zWWbv1HBMNONnaR/tNboyR3/BZd58JJSHlUSCU=
|
||||
github.com/pkg/errors v0.8.0/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0=
|
||||
github.com/pkg/errors v0.8.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0=
|
||||
@@ -447,8 +406,6 @@ github.com/prometheus/procfs v0.1.3/go.mod h1:lV6e/gmhEcM9IjHGsFOCxxuZ+z1YqCvr4O
|
||||
github.com/prometheus/procfs v0.2.0/go.mod h1:lV6e/gmhEcM9IjHGsFOCxxuZ+z1YqCvr4OA4YeYWdaU=
|
||||
github.com/prometheus/tsdb v0.7.1/go.mod h1:qhTCs0VvXwvX/y3TZrWD7rabWM+ijKTux40TwIPHuXU=
|
||||
github.com/rogpeppe/fastuuid v0.0.0-20150106093220-6724a57986af/go.mod h1:XWv6SoW27p1b0cqNHllgS5HIMJraePCO15w5zCzIWYg=
|
||||
github.com/rogpeppe/go-internal v1.1.0/go.mod h1:M8bDsm7K2OlrFYOpmOWEs/qY81heoFRclV5y23lUDJ4=
|
||||
github.com/rogpeppe/go-internal v1.2.2/go.mod h1:M8bDsm7K2OlrFYOpmOWEs/qY81heoFRclV5y23lUDJ4=
|
||||
github.com/rogpeppe/go-internal v1.3.0/go.mod h1:M8bDsm7K2OlrFYOpmOWEs/qY81heoFRclV5y23lUDJ4=
|
||||
github.com/russross/blackfriday v1.5.2/go.mod h1:JO/DiYxRf+HjHt06OyowR9PTA263kcR/rfWxYHBV53g=
|
||||
github.com/russross/blackfriday/v2 v2.0.1/go.mod h1:+Rmxgy9KzJVeS9/2gXHxylqXiyQDYRxCVz55jmeOWTM=
|
||||
@@ -458,8 +415,6 @@ github.com/sergi/go-diff v1.0.0/go.mod h1:0CfEIISq7TuYL3j771MWULgwwjU+GofnZX9QAm
|
||||
github.com/sergi/go-diff v1.1.0/go.mod h1:STckp+ISIX8hZLjrqAeVduY0gWCT9IjLuqbuNXdaHfM=
|
||||
github.com/shurcooL/sanitized_anchor_name v1.0.0/go.mod h1:1NzhyTcUVG4SuEtjjoZeVRXNmyL/1OwPU0+IJeTBvfc=
|
||||
github.com/sirupsen/logrus v1.2.0/go.mod h1:LxeOpSwHxABJmUn/MG1IvRgCAasNZTLOkJPxbbu5VWo=
|
||||
github.com/sirupsen/logrus v1.4.0/go.mod h1:LxeOpSwHxABJmUn/MG1IvRgCAasNZTLOkJPxbbu5VWo=
|
||||
github.com/sirupsen/logrus v1.4.1/go.mod h1:ni0Sbl8bgC9z8RoU9G6nDWqqs/fq4eDPysMBDgk/93Q=
|
||||
github.com/sirupsen/logrus v1.4.2/go.mod h1:tLMulIdttU9McNUspp0xgXVQah82FyeX6MwdIuYE2rE=
|
||||
github.com/smartystreets/assertions v0.0.0-20180927180507-b2de0cb4f26d/go.mod h1:OnSkiWE9lh6wB0YB77sQom3nweQdgAjqCqsofrRNTgc=
|
||||
github.com/smartystreets/goconvey v1.6.4/go.mod h1:syvi0/a8iFYH4r/RixwvyeAJjdLS9QV7WQ/tjFTllLA=
|
||||
@@ -468,7 +423,6 @@ github.com/spaolacci/murmur3 v0.0.0-20180118202830-f09979ecbc72/go.mod h1:JwIasO
|
||||
github.com/spf13/afero v1.1.2/go.mod h1:j4pytiNVoe2o6bmDsKpLACNPDBIoEAkihy7loJ1B0CQ=
|
||||
github.com/spf13/afero v1.2.2/go.mod h1:9ZxEEn6pIJ8Rxe320qSDBk6AsU0r9pR7Q4OcevTdifk=
|
||||
github.com/spf13/cast v1.3.0/go.mod h1:Qx5cxh0v+4UWYiBimWS+eyWzqEqokIECu5etghLkUJE=
|
||||
github.com/spf13/cobra v0.0.3/go.mod h1:1l0Ry5zgKvJasoi3XT1TypsSe7PqH0Sj9dhYf7v3XqQ=
|
||||
github.com/spf13/cobra v1.0.0/go.mod h1:/6GTrnGXV9HjY+aR4k0oJ5tcvakLuG6EuKReYlHNrgE=
|
||||
github.com/spf13/cobra v1.1.1/go.mod h1:WnodtKOvamDL/PwE2M4iKs8aMDBZ5Q5klgD3qfVJQMI=
|
||||
github.com/spf13/jwalterweatherman v1.0.0/go.mod h1:cQK4TGJAtQXfYWX+Ddv3mKDzgVb68N+wFjFa4jdeBTo=
|
||||
@@ -485,8 +439,9 @@ github.com/stretchr/testify v1.2.2/go.mod h1:a8OnRcib4nhh0OaRAV+Yts87kKdq0PP7pXf
|
||||
github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI=
|
||||
github.com/stretchr/testify v1.4.0/go.mod h1:j7eGeouHqKxXV5pUuKE4zz7dFj8WfuZ+81PSLYec5m4=
|
||||
github.com/stretchr/testify v1.5.1/go.mod h1:5W2xD1RspED5o8YsWQXVCued0rvSQ+mT+I5cxcmMvtA=
|
||||
github.com/stretchr/testify v1.6.1 h1:hDPOHmpOpP40lSULcqw7IrRb/u7w6RpDC9399XyoNd0=
|
||||
github.com/stretchr/testify v1.6.1/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
|
||||
github.com/stretchr/testify v1.7.0 h1:nwc3DEeHmmLAfoZucVR881uASk0Mfjw8xYJ99tb5CcY=
|
||||
github.com/stretchr/testify v1.7.0/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
|
||||
github.com/subosito/gotenv v1.2.0/go.mod h1:N0PQaV/YGNqwC0u51sEeR/aUtSLEXKX9iv69rRypqCw=
|
||||
github.com/tidwall/pretty v1.0.0/go.mod h1:XNkn88O1ChpSDQmQeStsy+sBenx6DDtFZJxhVysOjyk=
|
||||
github.com/tmc/grpc-websocket-proxy v0.0.0-20190109142713-0ad062ec5ee5/go.mod h1:ncp9v5uamzpCO7NfCPTXjqaC+bZgJeR0sMTm6dMHP7U=
|
||||
@@ -495,26 +450,22 @@ github.com/ugorji/go v1.1.7 h1:/68gy2h+1mWMrwZFeD1kQialdSzAb432dtpeJ42ovdo=
|
||||
github.com/ugorji/go v1.1.7/go.mod h1:kZn38zHttfInRq0xu/PH0az30d+z6vm202qpg1oXVMw=
|
||||
github.com/ugorji/go/codec v1.1.7 h1:2SvQaVZ1ouYrrKKwoSk2pzd4A9evlKJb9oTL+OaLUSs=
|
||||
github.com/ugorji/go/codec v1.1.7/go.mod h1:Ax+UKWsSmolVDwsd+7N3ZtXu+yMGCf907BLYF3GoBXY=
|
||||
github.com/up9inc/basenine/client/go v0.0.0-20211106180626-0193408db715 h1:3RNTMQZO/4g5gRn4R98cPwCjCrsMklmcOS0g+QwCh5c=
|
||||
github.com/up9inc/basenine/client/go v0.0.0-20211106180626-0193408db715/go.mod h1:SvJGPoa/6erhUQV7kvHBwM/0x5LyO6XaG2lUaCaKiUI=
|
||||
github.com/vektah/gqlparser v1.1.2/go.mod h1:1ycwN7Ij5njmMkPPAOaRFY4rET2Enx7IkVv3vaXspKw=
|
||||
github.com/vishvananda/netns v0.0.0-20210104183010-2eb08e3e575f h1:p4VB7kIXpOQvVn1ZaTIVp+3vuYAXFe3OJEvjbUYJLaA=
|
||||
github.com/vishvananda/netns v0.0.0-20210104183010-2eb08e3e575f/go.mod h1:DD4vA1DwXk04H54A1oHXtwZmA0grkVMdPxx/VGLCah0=
|
||||
github.com/xdg-go/pbkdf2 v1.0.0/go.mod h1:jrpuAogTd400dnrH08LKmI/xc1MbPOebTwRqcT5RDeI=
|
||||
github.com/xdg-go/scram v1.0.2/go.mod h1:1WAq6h33pAW+iRreB34OORO2Nf7qel3VV3fjBj+hCSs=
|
||||
github.com/xdg-go/stringprep v1.0.2/go.mod h1:8F9zXuvzgwmyT5DUm4GUfZGDdT3W+LCvS6+da4O5kxM=
|
||||
github.com/xiang90/probing v0.0.0-20190116061207-43a291ad63a2/go.mod h1:UETIi67q53MR2AWcXfiuqkDkRtnGDLqkBTpCHuJHxtU=
|
||||
github.com/xlab/treeprint v0.0.0-20181112141820-a009c3971eca/go.mod h1:ce1O1j6UtZfjr22oyGxGLbauSBp2YVXpARAosm7dHBg=
|
||||
github.com/xordataexchange/crypt v0.0.3-0.20170626215501-b2862e3d0a77/go.mod h1:aYKd//L2LvnjZzWKhF00oedf4jCCReLcmhLdhm1A27Q=
|
||||
github.com/yalp/jsonpath v0.0.0-20180802001716-5cc68e5049a0 h1:6fRhSjgLCkTD3JnJxvaJ4Sj+TYblw757bqYgZaOq5ZY=
|
||||
github.com/yalp/jsonpath v0.0.0-20180802001716-5cc68e5049a0/go.mod h1:/LWChgwKmvncFJFHJ7Gvn9wZArjbV5/FppcK2fKk/tI=
|
||||
github.com/youmark/pkcs8 v0.0.0-20181117223130-1be2e3e5546d/go.mod h1:rHwXgn7JulP+udvsHwJoVG1YGAP6VLg4y9I5dyZdqmA=
|
||||
github.com/yuin/goldmark v1.1.27/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74=
|
||||
github.com/yuin/goldmark v1.2.1/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74=
|
||||
go.etcd.io/bbolt v1.3.2/go.mod h1:IbVyRI1SCnLcuJnV2u8VeU0CEYM7e686BmAb1XKL+uU=
|
||||
go.mongodb.org/mongo-driver v1.0.3/go.mod h1:u7ryQJ+DOzQmeO7zB6MHyr8jkEQvC8vH7qLUO4lqsUM=
|
||||
go.mongodb.org/mongo-driver v1.1.1/go.mod h1:u7ryQJ+DOzQmeO7zB6MHyr8jkEQvC8vH7qLUO4lqsUM=
|
||||
go.mongodb.org/mongo-driver v1.1.2/go.mod h1:u7ryQJ+DOzQmeO7zB6MHyr8jkEQvC8vH7qLUO4lqsUM=
|
||||
go.mongodb.org/mongo-driver v1.7.1 h1:jwqTeEM3x6L9xDXrCxN0Hbg7vdGfPBOTIkr0+/LYZDA=
|
||||
go.mongodb.org/mongo-driver v1.7.1/go.mod h1:Q4oFMbo1+MSNqICAdYMlC/zSTrwCogR4R8NzkI+yfU8=
|
||||
go.opencensus.io v0.21.0/go.mod h1:mSImk1erAIZhrmZN+AvHh14ztQfjbGwt4TtuofqLduU=
|
||||
go.opencensus.io v0.22.0/go.mod h1:+kGneAE2xo2IficOXnaByMWTGM9T73dGwxeWcUqIpI8=
|
||||
go.opencensus.io v0.22.2/go.mod h1:yxeiOL68Rb0Xd1ddK5vPZ/oVn4vY4Ynel7k9FzqtOIw=
|
||||
@@ -527,13 +478,11 @@ golang.org/x/crypto v0.0.0-20180904163835-0709b304e793/go.mod h1:6SG95UA2DQfeDnf
|
||||
golang.org/x/crypto v0.0.0-20181029021203-45a5f77698d3/go.mod h1:6SG95UA2DQfeDnfUPMdvaQW0Q7yPrPDi9nlGo2tz2b4=
|
||||
golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w=
|
||||
golang.org/x/crypto v0.0.0-20190320223903-b7391e95e576/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w=
|
||||
golang.org/x/crypto v0.0.0-20190422162423-af44ce270edf/go.mod h1:WFFai1msRO1wXaEeE5yQxYXgSfI8pQAWXbQop6sCtWE=
|
||||
golang.org/x/crypto v0.0.0-20190510104115-cbcb75029529/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI=
|
||||
golang.org/x/crypto v0.0.0-20190605123033-f99c8df09eb5/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI=
|
||||
golang.org/x/crypto v0.0.0-20190611184440-5c40567a22f8/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI=
|
||||
golang.org/x/crypto v0.0.0-20190617133340-57b3e21c3d56/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI=
|
||||
golang.org/x/crypto v0.0.0-20191011191535-87dc89f01550/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI=
|
||||
golang.org/x/crypto v0.0.0-20200302210943-78000ba7a073/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto=
|
||||
golang.org/x/crypto v0.0.0-20200622213623-75b288015ac9/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto=
|
||||
golang.org/x/crypto v0.0.0-20201002170205-7f63de1d35b0/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto=
|
||||
golang.org/x/crypto v0.0.0-20210220033148-5ea612d1eb83 h1:/ZScEX8SfEmUGRHs0gxpqteO5nfNW6axyZbBdw9A12g=
|
||||
@@ -611,7 +560,6 @@ golang.org/x/sync v0.0.0-20180314180146-1d60e4601c6f/go.mod h1:RxMgew5VJxzue5/jJ
|
||||
golang.org/x/sync v0.0.0-20181108010431-42b317875d0f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
|
||||
golang.org/x/sync v0.0.0-20181221193216-37e7f081c4d4/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
|
||||
golang.org/x/sync v0.0.0-20190227155943-e225da77a7e6/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
|
||||
golang.org/x/sync v0.0.0-20190412183630-56d357773e84/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
|
||||
golang.org/x/sync v0.0.0-20190423024810-112230192c58/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
|
||||
golang.org/x/sync v0.0.0-20190911185100-cd5d95a43a6e/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
|
||||
golang.org/x/sync v0.0.0-20201020160332-67f06af15bc9/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
|
||||
@@ -625,13 +573,10 @@ golang.org/x/sys v0.0.0-20181116152217-5ac8a444bdc5/go.mod h1:STP8DvDyc/dI5b8T5h
|
||||
golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
|
||||
golang.org/x/sys v0.0.0-20190312061237-fead79001313/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
|
||||
golang.org/x/sys v0.0.0-20190321052220-f7bb7a8bee54/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
|
||||
golang.org/x/sys v0.0.0-20190403152447-81d4e9dc473e/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
|
||||
golang.org/x/sys v0.0.0-20190412213103-97732733099d/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
|
||||
golang.org/x/sys v0.0.0-20190419153524-e8e3143a4f4a/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
|
||||
golang.org/x/sys v0.0.0-20190422165155-953cdadca894/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
|
||||
golang.org/x/sys v0.0.0-20190502145724-3ef323f4f1fd/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
|
||||
golang.org/x/sys v0.0.0-20190507160741-ecd444e8653b/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
|
||||
golang.org/x/sys v0.0.0-20190531175056-4c3a928424d2/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
|
||||
golang.org/x/sys v0.0.0-20190606165138-5da285871e9c/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
|
||||
golang.org/x/sys v0.0.0-20190616124812-15dcb6c0061f/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
|
||||
golang.org/x/sys v0.0.0-20190624142023-c5567b49c5d0/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
|
||||
@@ -685,13 +630,9 @@ golang.org/x/tools v0.0.0-20190311212946-11955173bddd/go.mod h1:LCzVGOaR6xXOjkQ3
|
||||
golang.org/x/tools v0.0.0-20190312151545-0bb0c0a6e846/go.mod h1:LCzVGOaR6xXOjkQ3onu1FJEFr0SW1gC7cKk1uF8kGRs=
|
||||
golang.org/x/tools v0.0.0-20190312170243-e65039ee4138/go.mod h1:LCzVGOaR6xXOjkQ3onu1FJEFr0SW1gC7cKk1uF8kGRs=
|
||||
golang.org/x/tools v0.0.0-20190328211700-ab21143f2384/go.mod h1:LCzVGOaR6xXOjkQ3onu1FJEFr0SW1gC7cKk1uF8kGRs=
|
||||
golang.org/x/tools v0.0.0-20190329151228-23e29df326fe/go.mod h1:LCzVGOaR6xXOjkQ3onu1FJEFr0SW1gC7cKk1uF8kGRs=
|
||||
golang.org/x/tools v0.0.0-20190416151739-9c9e1878f421/go.mod h1:LCzVGOaR6xXOjkQ3onu1FJEFr0SW1gC7cKk1uF8kGRs=
|
||||
golang.org/x/tools v0.0.0-20190420181800-aa740d480789/go.mod h1:LCzVGOaR6xXOjkQ3onu1FJEFr0SW1gC7cKk1uF8kGRs=
|
||||
golang.org/x/tools v0.0.0-20190425150028-36563e24a262/go.mod h1:RgjU9mgBXZiqYHBnxXauZ1Gv1EHHAz9KjViQ78xBX0Q=
|
||||
golang.org/x/tools v0.0.0-20190506145303-2d16b83fe98c/go.mod h1:RgjU9mgBXZiqYHBnxXauZ1Gv1EHHAz9KjViQ78xBX0Q=
|
||||
golang.org/x/tools v0.0.0-20190524140312-2c0ae7006135/go.mod h1:RgjU9mgBXZiqYHBnxXauZ1Gv1EHHAz9KjViQ78xBX0Q=
|
||||
golang.org/x/tools v0.0.0-20190531172133-b3315ee88b7d/go.mod h1:/rFqwRUd4F7ZHNgwSSTFct+R/Kf4OFW1sUzUTQQTgfc=
|
||||
golang.org/x/tools v0.0.0-20190606124116-d0a3d012864b/go.mod h1:/rFqwRUd4F7ZHNgwSSTFct+R/Kf4OFW1sUzUTQQTgfc=
|
||||
golang.org/x/tools v0.0.0-20190614205625-5aca471b1d59/go.mod h1:/rFqwRUd4F7ZHNgwSSTFct+R/Kf4OFW1sUzUTQQTgfc=
|
||||
golang.org/x/tools v0.0.0-20190617190820-da514acc4774/go.mod h1:/rFqwRUd4F7ZHNgwSSTFct+R/Kf4OFW1sUzUTQQTgfc=
|
||||
@@ -805,11 +746,6 @@ gopkg.in/yaml.v2 v2.4.0/go.mod h1:RDklbk79AGWmwhnvt/jBztapEOGDOx6ZbXqjP6csGnQ=
|
||||
gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
|
||||
gopkg.in/yaml.v3 v3.0.0-20210107192922-496545a6307b h1:h8qDotaEPuJATrMmW04NCwg7v22aHH28wwpauUhK9Oo=
|
||||
gopkg.in/yaml.v3 v3.0.0-20210107192922-496545a6307b/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
|
||||
gorm.io/driver/sqlite v1.1.4 h1:PDzwYE+sI6De2+mxAneV9Xs11+ZyKV6oxD3wDGkaNvM=
|
||||
gorm.io/driver/sqlite v1.1.4/go.mod h1:mJCeTFr7+crvS+TRnWc5Z3UvwxUN1BGBLMrf5LA9DYw=
|
||||
gorm.io/gorm v1.20.7/go.mod h1:0HFTzE/SqkGTzK6TlDPPQbAYCluiVvhzoA1+aVyzenw=
|
||||
gorm.io/gorm v1.21.8 h1:2CEwZSzogdhsKPlJ9OvBKTdlWIpELXb6HbfLfMNhSYI=
|
||||
gorm.io/gorm v1.21.8/go.mod h1:F+OptMscr0P2F2qU97WT1WimdH9GaQPoDW7AYd5i2Y0=
|
||||
gotest.tools/v3 v3.0.2/go.mod h1:3SzNCllyD9/Y+b5r9JIKQ474KzkZyqLqEfYqMsX94Bk=
|
||||
gotest.tools/v3 v3.0.3/go.mod h1:Z7Lb0S5l+klDB31fvDQX8ss/FlKDxtlFlw3Oa8Ymbl8=
|
||||
honnef.co/go/tools v0.0.0-20190102054323-c2f93a96b099/go.mod h1:rf3lG4BRIbNafJWhAfAdb/ePZxsR/4RtNHQocxwk9r4=
|
||||
|
||||
@@ -6,13 +6,10 @@ import (
|
||||
"errors"
|
||||
"flag"
|
||||
"fmt"
|
||||
"github.com/up9inc/mizu/shared/kubernetes"
|
||||
"io/ioutil"
|
||||
v1 "k8s.io/api/core/v1"
|
||||
"mizuserver/pkg/api"
|
||||
"mizuserver/pkg/config"
|
||||
"mizuserver/pkg/controllers"
|
||||
"mizuserver/pkg/database"
|
||||
"mizuserver/pkg/models"
|
||||
"mizuserver/pkg/providers"
|
||||
"mizuserver/pkg/routes"
|
||||
@@ -20,6 +17,7 @@ import (
|
||||
"mizuserver/pkg/utils"
|
||||
"net/http"
|
||||
"os"
|
||||
"os/exec"
|
||||
"os/signal"
|
||||
"path"
|
||||
"path/filepath"
|
||||
@@ -28,10 +26,15 @@ import (
|
||||
"syscall"
|
||||
"time"
|
||||
|
||||
"github.com/up9inc/mizu/shared/kubernetes"
|
||||
v1 "k8s.io/api/core/v1"
|
||||
|
||||
"github.com/antelman107/net-wait-go/wait"
|
||||
"github.com/gin-contrib/static"
|
||||
"github.com/gin-gonic/gin"
|
||||
"github.com/gorilla/websocket"
|
||||
"github.com/op/go-logging"
|
||||
basenine "github.com/up9inc/basenine/client/go"
|
||||
"github.com/up9inc/mizu/shared"
|
||||
"github.com/up9inc/mizu/shared/logger"
|
||||
"github.com/up9inc/mizu/tap"
|
||||
@@ -49,10 +52,12 @@ var harsDir = flag.String("hars-dir", "", "Directory to read hars from")
|
||||
var extensions []*tapApi.Extension // global
|
||||
var extensionsMap map[string]*tapApi.Extension // global
|
||||
|
||||
var startTime int64
|
||||
|
||||
const (
|
||||
socketConnectionRetries = 10
|
||||
socketConnectionRetries = 10
|
||||
socketConnectionRetryDelay = time.Second * 2
|
||||
socketHandshakeTimeout = time.Second * 2
|
||||
socketHandshakeTimeout = time.Second * 2
|
||||
)
|
||||
|
||||
func main() {
|
||||
@@ -109,7 +114,8 @@ func main() {
|
||||
|
||||
go pipeTapChannelToSocket(socketConnection, filteredOutputItemsChannel)
|
||||
} else if *apiServerMode {
|
||||
database.InitDataBase(config.Config.AgentDatabasePath)
|
||||
startBasenineServer(shared.BasenineHost, shared.BaseninePort)
|
||||
startTime = time.Now().UnixNano() / int64(time.Millisecond)
|
||||
api.StartResolving(*namespace)
|
||||
|
||||
outputItemsChannel := make(chan *tapApi.OutputChannelItem)
|
||||
@@ -142,6 +148,53 @@ func main() {
|
||||
logger.Log.Info("Exiting")
|
||||
}
|
||||
|
||||
func startBasenineServer(host string, port string) {
|
||||
cmd := exec.Command("basenine", "-addr", host, "-port", port, "-persistent")
|
||||
cmd.Dir = config.Config.AgentDatabasePath
|
||||
cmd.Stdout = os.Stdout
|
||||
cmd.Stderr = os.Stderr
|
||||
err := cmd.Start()
|
||||
if err != nil {
|
||||
logger.Log.Panicf("Failed starting Basenine: %v", err)
|
||||
}
|
||||
|
||||
if !wait.New(
|
||||
wait.WithProto("tcp"),
|
||||
wait.WithWait(200*time.Millisecond),
|
||||
wait.WithBreak(50*time.Millisecond),
|
||||
wait.WithDeadline(5*time.Second),
|
||||
wait.WithDebug(true),
|
||||
).Do([]string{fmt.Sprintf("%s:%s", host, port)}) {
|
||||
logger.Log.Panicf("Basenine is not available: %v", err)
|
||||
}
|
||||
|
||||
// Make a channel to gracefully exit Basenine.
|
||||
channel := make(chan os.Signal)
|
||||
signal.Notify(channel, os.Interrupt, syscall.SIGTERM)
|
||||
|
||||
// Handle the channel.
|
||||
go func() {
|
||||
<-channel
|
||||
cmd.Process.Signal(syscall.SIGTERM)
|
||||
}()
|
||||
|
||||
// Limit the database size to default 200MB
|
||||
err = basenine.Limit(host, port, config.Config.MaxDBSizeBytes)
|
||||
if err != nil {
|
||||
logger.Log.Panicf("Error while limiting database size: %v", err)
|
||||
}
|
||||
|
||||
for _, extension := range extensions {
|
||||
macros := extension.Dissector.Macros()
|
||||
for macro, expanded := range macros {
|
||||
err = basenine.Macro(host, port, macro, expanded)
|
||||
if err != nil {
|
||||
logger.Log.Panicf("Error while adding a macro: %v", err)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func loadExtensions() {
|
||||
dir, _ := filepath.Abs(filepath.Dir(os.Args[0]))
|
||||
extensionsDir := path.Join(dir, "./extensions/")
|
||||
@@ -200,7 +253,8 @@ func hostApi(socketHarOutputChannel chan<- *tapApi.OutputChannelItem) {
|
||||
app.Use(static.ServeRoot("/", "./site"))
|
||||
app.Use(CORSMiddleware()) // This has to be called after the static middleware, does not work if its called before
|
||||
|
||||
api.WebSocketRoutes(app, &eventHandlers)
|
||||
api.WebSocketRoutes(app, &eventHandlers, startTime)
|
||||
routes.QueryRoutes(app)
|
||||
routes.EntriesRoutes(app)
|
||||
routes.MetadataRoutes(app)
|
||||
routes.StatusRoutes(app)
|
||||
@@ -361,7 +415,7 @@ func dialSocketWithRetry(socketAddress string, retryAmount int, retryDelay time.
|
||||
socketConnection, _, err := dialer.Dial(socketAddress, nil)
|
||||
if err != nil {
|
||||
if i < retryAmount {
|
||||
logger.Log.Infof("socket connection to %s failed: %v, retrying %d out of %d in %d seconds...", socketAddress, err, i, retryAmount, retryDelay / time.Second)
|
||||
logger.Log.Infof("socket connection to %s failed: %v, retrying %d out of %d in %d seconds...", socketAddress, err, i, retryAmount, retryDelay/time.Second)
|
||||
time.Sleep(retryDelay)
|
||||
}
|
||||
} else {
|
||||
@@ -371,8 +425,7 @@ func dialSocketWithRetry(socketAddress string, retryAmount int, retryDelay time.
|
||||
return nil, lastErr
|
||||
}
|
||||
|
||||
|
||||
func startMizuTapperSyncer(ctx context.Context) (*kubernetes.MizuTapperSyncer, error){
|
||||
func startMizuTapperSyncer(ctx context.Context) (*kubernetes.MizuTapperSyncer, error) {
|
||||
provider, err := kubernetes.NewProviderInCluster()
|
||||
if err != nil {
|
||||
return nil, err
|
||||
|
||||
@@ -5,7 +5,6 @@ import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"mizuserver/pkg/database"
|
||||
"mizuserver/pkg/holder"
|
||||
"mizuserver/pkg/providers"
|
||||
"os"
|
||||
@@ -14,15 +13,16 @@ import (
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"go.mongodb.org/mongo-driver/bson/primitive"
|
||||
|
||||
"github.com/google/martian/har"
|
||||
"github.com/up9inc/mizu/shared"
|
||||
"github.com/up9inc/mizu/shared/logger"
|
||||
tapApi "github.com/up9inc/mizu/tap/api"
|
||||
|
||||
"mizuserver/pkg/models"
|
||||
"mizuserver/pkg/resolver"
|
||||
"mizuserver/pkg/utils"
|
||||
|
||||
basenine "github.com/up9inc/basenine/client/go"
|
||||
)
|
||||
|
||||
var k8sResolver *resolver.Resolver
|
||||
@@ -99,6 +99,12 @@ func startReadingChannel(outputItems <-chan *tapApi.OutputChannelItem, extension
|
||||
panic("Channel of captured messages is nil")
|
||||
}
|
||||
|
||||
connection, err := basenine.NewConnection(shared.BasenineHost, shared.BaseninePort)
|
||||
if err != nil {
|
||||
panic(err)
|
||||
}
|
||||
connection.InsertMode()
|
||||
|
||||
disableOASValidation := false
|
||||
ctx := context.Background()
|
||||
doc, contractContent, router, err := loadOAS(ctx)
|
||||
@@ -112,13 +118,13 @@ func startReadingChannel(outputItems <-chan *tapApi.OutputChannelItem, extension
|
||||
|
||||
extension := extensionsMap[item.Protocol.Name]
|
||||
resolvedSource, resolvedDestionation := resolveIP(item.ConnectionInfo)
|
||||
mizuEntry := extension.Dissector.Analyze(item, primitive.NewObjectID().Hex(), resolvedSource, resolvedDestionation)
|
||||
mizuEntry := extension.Dissector.Analyze(item, resolvedSource, resolvedDestionation)
|
||||
baseEntry := extension.Dissector.Summarize(mizuEntry)
|
||||
mizuEntry.EstimatedSizeBytes = getEstimatedEntrySizeBytes(mizuEntry)
|
||||
mizuEntry.Base = baseEntry
|
||||
if extension.Protocol.Name == "http" {
|
||||
if !disableOASValidation {
|
||||
var httpPair tapApi.HTTPRequestResponsePair
|
||||
json.Unmarshal([]byte(mizuEntry.Entry), &httpPair)
|
||||
json.Unmarshal([]byte(mizuEntry.HTTPPair), &httpPair)
|
||||
|
||||
contract := handleOAS(ctx, doc, router, httpPair.Request.Payload.RawRequest, httpPair.Response.Payload.RawResponse, contractContent)
|
||||
baseEntry.ContractStatus = contract.Status
|
||||
@@ -128,18 +134,18 @@ func startReadingChannel(outputItems <-chan *tapApi.OutputChannelItem, extension
|
||||
mizuEntry.ContractContent = contract.Content
|
||||
}
|
||||
|
||||
var pair tapApi.RequestResponsePair
|
||||
json.Unmarshal([]byte(mizuEntry.Entry), &pair)
|
||||
harEntry, err := utils.NewEntry(&pair)
|
||||
harEntry, err := utils.NewEntry(mizuEntry.Request, mizuEntry.Response, mizuEntry.StartTime, mizuEntry.ElapsedTime)
|
||||
if err == nil {
|
||||
rules, _, _ := models.RunValidationRulesState(*harEntry, mizuEntry.Service)
|
||||
baseEntry.Rules = rules
|
||||
}
|
||||
}
|
||||
database.CreateEntry(mizuEntry)
|
||||
|
||||
baseEntryBytes, _ := models.CreateBaseEntryWebSocketMessage(baseEntry)
|
||||
BroadcastToBrowserClients(baseEntryBytes)
|
||||
data, err := json.Marshal(mizuEntry)
|
||||
if err != nil {
|
||||
panic(err)
|
||||
}
|
||||
connection.SendText(string(data))
|
||||
}
|
||||
}
|
||||
|
||||
@@ -171,21 +177,3 @@ func CheckIsServiceIP(address string) bool {
|
||||
}
|
||||
return k8sResolver.CheckIsServiceIP(address)
|
||||
}
|
||||
|
||||
// gives a rough estimate of the size this will take up in the db, good enough for maintaining db size limit accurately
|
||||
func getEstimatedEntrySizeBytes(mizuEntry *tapApi.MizuEntry) int {
|
||||
sizeBytes := len(mizuEntry.Entry)
|
||||
sizeBytes += len(mizuEntry.EntryId)
|
||||
sizeBytes += len(mizuEntry.Service)
|
||||
sizeBytes += len(mizuEntry.Url)
|
||||
sizeBytes += len(mizuEntry.Method)
|
||||
sizeBytes += len(mizuEntry.RequestSenderIp)
|
||||
sizeBytes += len(mizuEntry.ResolvedDestination)
|
||||
sizeBytes += len(mizuEntry.ResolvedSource)
|
||||
sizeBytes += 8 // Status bytes (sqlite integer is always 8 bytes)
|
||||
sizeBytes += 8 // Timestamp bytes
|
||||
sizeBytes += 8 // SizeBytes bytes
|
||||
sizeBytes += 1 // IsOutgoing bytes
|
||||
|
||||
return sizeBytes
|
||||
}
|
||||
|
||||
@@ -1,13 +1,18 @@
|
||||
package api
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"mizuserver/pkg/models"
|
||||
"net/http"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"github.com/gin-gonic/gin"
|
||||
"github.com/gorilla/websocket"
|
||||
basenine "github.com/up9inc/basenine/client/go"
|
||||
"github.com/up9inc/mizu/shared"
|
||||
"github.com/up9inc/mizu/shared/debounce"
|
||||
"github.com/up9inc/mizu/shared/logger"
|
||||
)
|
||||
@@ -39,17 +44,17 @@ func init() {
|
||||
connectedWebsockets = make(map[int]*SocketConnection, 0)
|
||||
}
|
||||
|
||||
func WebSocketRoutes(app *gin.Engine, eventHandlers EventHandlers) {
|
||||
func WebSocketRoutes(app *gin.Engine, eventHandlers EventHandlers, startTime int64) {
|
||||
app.GET("/ws", func(c *gin.Context) {
|
||||
websocketHandler(c.Writer, c.Request, eventHandlers, false)
|
||||
websocketHandler(c.Writer, c.Request, eventHandlers, false, startTime)
|
||||
})
|
||||
app.GET("/wsTapper", func(c *gin.Context) {
|
||||
websocketHandler(c.Writer, c.Request, eventHandlers, true)
|
||||
websocketHandler(c.Writer, c.Request, eventHandlers, true, startTime)
|
||||
})
|
||||
}
|
||||
|
||||
func websocketHandler(w http.ResponseWriter, r *http.Request, eventHandlers EventHandlers, isTapper bool) {
|
||||
conn, err := websocketUpgrader.Upgrade(w, r, nil)
|
||||
func websocketHandler(w http.ResponseWriter, r *http.Request, eventHandlers EventHandlers, isTapper bool, startTime int64) {
|
||||
ws, err := websocketUpgrader.Upgrade(w, r, nil)
|
||||
if err != nil {
|
||||
logger.Log.Errorf("Failed to set websocket upgrade: %v", err)
|
||||
return
|
||||
@@ -59,23 +64,103 @@ func websocketHandler(w http.ResponseWriter, r *http.Request, eventHandlers Even
|
||||
|
||||
connectedWebsocketIdCounter++
|
||||
socketId := connectedWebsocketIdCounter
|
||||
connectedWebsockets[socketId] = &SocketConnection{connection: conn, lock: &sync.Mutex{}, eventHandlers: eventHandlers, isTapper: isTapper}
|
||||
connectedWebsockets[socketId] = &SocketConnection{connection: ws, lock: &sync.Mutex{}, eventHandlers: eventHandlers, isTapper: isTapper}
|
||||
|
||||
websocketIdsLock.Unlock()
|
||||
|
||||
var connection *basenine.Connection
|
||||
var isQuerySet bool
|
||||
|
||||
// `!isTapper` means it's a connection from the web UI
|
||||
if !isTapper {
|
||||
connection, err = basenine.NewConnection(shared.BasenineHost, shared.BaseninePort)
|
||||
if err != nil {
|
||||
panic(err)
|
||||
}
|
||||
}
|
||||
|
||||
data := make(chan []byte)
|
||||
meta := make(chan []byte)
|
||||
|
||||
defer func() {
|
||||
data <- []byte(basenine.CloseChannel)
|
||||
meta <- []byte(basenine.CloseChannel)
|
||||
connection.Close()
|
||||
socketCleanup(socketId, connectedWebsockets[socketId])
|
||||
}()
|
||||
|
||||
eventHandlers.WebSocketConnect(socketId, isTapper)
|
||||
|
||||
startTimeBytes, _ := models.CreateWebsocketStartTimeMessage(startTime)
|
||||
BroadcastToBrowserClients(startTimeBytes)
|
||||
|
||||
for {
|
||||
_, msg, err := conn.ReadMessage()
|
||||
_, msg, err := ws.ReadMessage()
|
||||
if err != nil {
|
||||
logger.Log.Errorf("Error reading message, socket id: %d, error: %v", socketId, err)
|
||||
break
|
||||
}
|
||||
eventHandlers.WebSocketMessage(socketId, msg)
|
||||
|
||||
if !isTapper && !isQuerySet {
|
||||
query := string(msg)
|
||||
err = basenine.Validate(shared.BasenineHost, shared.BaseninePort, query)
|
||||
if err != nil {
|
||||
toastBytes, _ := models.CreateWebsocketToastMessage(&models.ToastMessage{
|
||||
Type: "error",
|
||||
AutoClose: 5000,
|
||||
Text: fmt.Sprintf("Syntax error: %s", err.Error()),
|
||||
})
|
||||
BroadcastToBrowserClients(toastBytes)
|
||||
break
|
||||
}
|
||||
|
||||
isQuerySet = true
|
||||
|
||||
handleDataChannel := func(c *basenine.Connection, data chan []byte) {
|
||||
for {
|
||||
bytes := <-data
|
||||
|
||||
if string(bytes) == basenine.CloseChannel {
|
||||
return
|
||||
}
|
||||
|
||||
var dataMap map[string]interface{}
|
||||
err = json.Unmarshal(bytes, &dataMap)
|
||||
|
||||
base := dataMap["base"].(map[string]interface{})
|
||||
base["id"] = uint(dataMap["id"].(float64))
|
||||
|
||||
baseEntryBytes, _ := models.CreateBaseEntryWebSocketMessage(base)
|
||||
BroadcastToBrowserClients(baseEntryBytes)
|
||||
}
|
||||
}
|
||||
|
||||
handleMetaChannel := func(c *basenine.Connection, meta chan []byte) {
|
||||
for {
|
||||
bytes := <-meta
|
||||
|
||||
if string(bytes) == basenine.CloseChannel {
|
||||
return
|
||||
}
|
||||
|
||||
var metadata *basenine.Metadata
|
||||
err = json.Unmarshal(bytes, &metadata)
|
||||
if err != nil {
|
||||
logger.Log.Debugf("Error recieving metadata: %v\n", err.Error())
|
||||
}
|
||||
|
||||
metadataBytes, _ := models.CreateWebsocketQueryMetadataMessage(metadata)
|
||||
BroadcastToBrowserClients(metadataBytes)
|
||||
}
|
||||
}
|
||||
|
||||
go handleDataChannel(connection, data)
|
||||
go handleMetaChannel(connection, meta)
|
||||
|
||||
connection.Query(query, data, meta)
|
||||
} else {
|
||||
eventHandlers.WebSocketMessage(socketId, msg)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -2,14 +2,17 @@ package controllers
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"github.com/gin-gonic/gin"
|
||||
tapApi "github.com/up9inc/mizu/tap/api"
|
||||
"mizuserver/pkg/database"
|
||||
"mizuserver/pkg/models"
|
||||
"mizuserver/pkg/utils"
|
||||
"mizuserver/pkg/validation"
|
||||
"net/http"
|
||||
"strconv"
|
||||
|
||||
"github.com/gin-gonic/gin"
|
||||
|
||||
basenine "github.com/up9inc/basenine/client/go"
|
||||
"github.com/up9inc/mizu/shared"
|
||||
"github.com/up9inc/mizu/shared/logger"
|
||||
tapApi "github.com/up9inc/mizu/tap/api"
|
||||
)
|
||||
|
||||
var extensionsMap map[string]*tapApi.Extension // global
|
||||
@@ -18,68 +21,36 @@ func InitExtensionsMap(ref map[string]*tapApi.Extension) {
|
||||
extensionsMap = ref
|
||||
}
|
||||
|
||||
func GetEntries(c *gin.Context) {
|
||||
entriesFilter := &models.EntriesFilter{}
|
||||
|
||||
if err := c.BindQuery(entriesFilter); err != nil {
|
||||
c.JSON(http.StatusBadRequest, err)
|
||||
}
|
||||
err := validation.Validate(entriesFilter)
|
||||
func Error(c *gin.Context, err error) bool {
|
||||
if err != nil {
|
||||
c.JSON(http.StatusBadRequest, err)
|
||||
logger.Log.Errorf("Error getting entry: %v", err)
|
||||
c.Error(err)
|
||||
c.AbortWithStatusJSON(http.StatusInternalServerError, gin.H{"error": true, "msg": err.Error()})
|
||||
return true // signal that there was an error and the caller should return
|
||||
}
|
||||
|
||||
order := database.OperatorToOrderMapping[entriesFilter.Operator]
|
||||
operatorSymbol := database.OperatorToSymbolMapping[entriesFilter.Operator]
|
||||
var entries []tapApi.MizuEntry
|
||||
database.GetEntriesTable().
|
||||
Order(fmt.Sprintf("timestamp %s", order)).
|
||||
Where(fmt.Sprintf("timestamp %s %v", operatorSymbol, entriesFilter.Timestamp)).
|
||||
Limit(entriesFilter.Limit).
|
||||
Find(&entries)
|
||||
|
||||
if len(entries) > 0 && order == database.OrderDesc {
|
||||
// the entries always order from oldest to newest - we should reverse
|
||||
utils.ReverseSlice(entries)
|
||||
}
|
||||
|
||||
baseEntries := make([]tapApi.BaseEntryDetails, 0)
|
||||
for _, entry := range entries {
|
||||
baseEntryDetails := tapApi.BaseEntryDetails{}
|
||||
if err := models.GetEntry(&entry, &baseEntryDetails); err != nil {
|
||||
continue
|
||||
}
|
||||
|
||||
var pair tapApi.RequestResponsePair
|
||||
json.Unmarshal([]byte(entry.Entry), &pair)
|
||||
harEntry, err := utils.NewEntry(&pair)
|
||||
if err == nil {
|
||||
rules, _, _ := models.RunValidationRulesState(*harEntry, entry.Service)
|
||||
baseEntryDetails.Rules = rules
|
||||
}
|
||||
|
||||
baseEntries = append(baseEntries, baseEntryDetails)
|
||||
}
|
||||
|
||||
c.JSON(http.StatusOK, baseEntries)
|
||||
return false // no error, can continue
|
||||
}
|
||||
|
||||
func GetEntry(c *gin.Context) {
|
||||
var entryData tapApi.MizuEntry
|
||||
database.GetEntriesTable().
|
||||
Where(map[string]string{"entryId": c.Param("entryId")}).
|
||||
First(&entryData)
|
||||
id, _ := strconv.Atoi(c.Param("id"))
|
||||
var entry tapApi.MizuEntry
|
||||
bytes, err := basenine.Single(shared.BasenineHost, shared.BaseninePort, id)
|
||||
if Error(c, err) {
|
||||
return // exit
|
||||
}
|
||||
err = json.Unmarshal(bytes, &entry)
|
||||
if Error(c, err) {
|
||||
return // exit
|
||||
}
|
||||
|
||||
extension := extensionsMap[entryData.ProtocolName]
|
||||
protocol, representation, bodySize, _ := extension.Dissector.Represent(&entryData)
|
||||
extension := extensionsMap[entry.Protocol.Name]
|
||||
protocol, representation, bodySize, _ := extension.Dissector.Represent(entry.Protocol, entry.Request, entry.Response)
|
||||
|
||||
var rules []map[string]interface{}
|
||||
var isRulesEnabled bool
|
||||
if entryData.ProtocolName == "http" {
|
||||
var pair tapApi.RequestResponsePair
|
||||
json.Unmarshal([]byte(entryData.Entry), &pair)
|
||||
harEntry, _ := utils.NewEntry(&pair)
|
||||
_, rulesMatched, _isRulesEnabled := models.RunValidationRulesState(*harEntry, entryData.Service)
|
||||
if entry.Protocol.Name == "http" {
|
||||
harEntry, _ := utils.NewEntry(entry.Request, entry.Response, entry.StartTime, entry.ElapsedTime)
|
||||
_, rulesMatched, _isRulesEnabled := models.RunValidationRulesState(*harEntry, entry.Service)
|
||||
isRulesEnabled = _isRulesEnabled
|
||||
inrec, _ := json.Marshal(rulesMatched)
|
||||
json.Unmarshal(inrec, &rules)
|
||||
@@ -89,7 +60,7 @@ func GetEntry(c *gin.Context) {
|
||||
Protocol: protocol,
|
||||
Representation: string(representation),
|
||||
BodySize: bodySize,
|
||||
Data: entryData,
|
||||
Data: entry,
|
||||
Rules: rules,
|
||||
IsRulesEnabled: isRulesEnabled,
|
||||
})
|
||||
|
||||
31
agent/pkg/controllers/query_controller.go
Normal file
31
agent/pkg/controllers/query_controller.go
Normal file
@@ -0,0 +1,31 @@
|
||||
package controllers
|
||||
|
||||
import (
|
||||
"net/http"
|
||||
|
||||
"github.com/gin-gonic/gin"
|
||||
basenine "github.com/up9inc/basenine/client/go"
|
||||
"github.com/up9inc/mizu/shared"
|
||||
)
|
||||
|
||||
type ValidateResponse struct {
|
||||
Valid bool `json:"valid"`
|
||||
Message string `json:"message"`
|
||||
}
|
||||
|
||||
func PostValidate(c *gin.Context) {
|
||||
query := c.PostForm("query")
|
||||
valid := true
|
||||
message := ""
|
||||
|
||||
err := basenine.Validate(shared.BasenineHost, shared.BaseninePort, query)
|
||||
if err != nil {
|
||||
valid = false
|
||||
message = err.Error()
|
||||
}
|
||||
|
||||
c.JSON(http.StatusOK, ValidateResponse{
|
||||
Valid: valid,
|
||||
Message: message,
|
||||
})
|
||||
}
|
||||
@@ -1,78 +0,0 @@
|
||||
package database
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"mizuserver/pkg/utils"
|
||||
"time"
|
||||
|
||||
"gorm.io/driver/sqlite"
|
||||
"gorm.io/gorm"
|
||||
"gorm.io/gorm/logger"
|
||||
|
||||
tapApi "github.com/up9inc/mizu/tap/api"
|
||||
)
|
||||
|
||||
const (
|
||||
OrderDesc = "desc"
|
||||
OrderAsc = "asc"
|
||||
LT = "lt"
|
||||
GT = "gt"
|
||||
TimeFormat = "2006-01-02 15:04:05.000000000"
|
||||
)
|
||||
|
||||
var (
|
||||
DB *gorm.DB
|
||||
IsDBLocked = false
|
||||
OperatorToSymbolMapping = map[string]string{
|
||||
LT: "<",
|
||||
GT: ">",
|
||||
}
|
||||
OperatorToOrderMapping = map[string]string{
|
||||
LT: OrderDesc,
|
||||
GT: OrderAsc,
|
||||
}
|
||||
)
|
||||
|
||||
var DBPath string
|
||||
|
||||
func GetEntriesTable() *gorm.DB {
|
||||
return DB.Table("mizu_entries")
|
||||
}
|
||||
|
||||
func CreateEntry(entry *tapApi.MizuEntry) {
|
||||
if IsDBLocked {
|
||||
return
|
||||
}
|
||||
GetEntriesTable().Create(entry)
|
||||
}
|
||||
|
||||
func InitDataBase(databasePath string) *gorm.DB {
|
||||
DBPath = databasePath
|
||||
DB, _ = gorm.Open(sqlite.Open(databasePath), &gorm.Config{
|
||||
Logger: &utils.TruncatingLogger{LogLevel: logger.Warn, SlowThreshold: 500 * time.Millisecond},
|
||||
})
|
||||
_ = DB.AutoMigrate(&tapApi.MizuEntry{}) // this will ensure table is created
|
||||
go StartEnforcingDatabaseSize()
|
||||
return DB
|
||||
}
|
||||
|
||||
func GetEntriesFromDb(timeFrom time.Time, timeTo time.Time, protocolName *string) []tapApi.MizuEntry {
|
||||
order := OrderDesc
|
||||
protocolNameCondition := "1 = 1"
|
||||
if protocolName != nil {
|
||||
protocolNameCondition = fmt.Sprintf("protocolName = '%s'", *protocolName)
|
||||
}
|
||||
|
||||
var entries []tapApi.MizuEntry
|
||||
GetEntriesTable().
|
||||
Where(protocolNameCondition).
|
||||
Where(fmt.Sprintf("created_at BETWEEN '%s' AND '%s'", timeFrom.Format(TimeFormat), timeTo.Format(TimeFormat))).
|
||||
Order(fmt.Sprintf("timestamp %s", order)).
|
||||
Find(&entries)
|
||||
|
||||
if len(entries) > 0 {
|
||||
// the entries always order from oldest to newest so we should revers
|
||||
utils.ReverseSlice(entries)
|
||||
}
|
||||
return entries
|
||||
}
|
||||
@@ -1,102 +0,0 @@
|
||||
package database
|
||||
|
||||
import (
|
||||
"mizuserver/pkg/config"
|
||||
"os"
|
||||
"time"
|
||||
|
||||
"github.com/fsnotify/fsnotify"
|
||||
"github.com/up9inc/mizu/shared/debounce"
|
||||
"github.com/up9inc/mizu/shared/logger"
|
||||
"github.com/up9inc/mizu/shared/units"
|
||||
tapApi "github.com/up9inc/mizu/tap/api"
|
||||
)
|
||||
|
||||
const percentageOfMaxSizeBytesToPrune = 15
|
||||
|
||||
func StartEnforcingDatabaseSize() {
|
||||
watcher, err := fsnotify.NewWatcher()
|
||||
if err != nil {
|
||||
logger.Log.Fatalf("Error creating filesystem watcher for db size enforcement: %v\n", err)
|
||||
return
|
||||
}
|
||||
|
||||
checkFileSizeDebouncer := debounce.NewDebouncer(5*time.Second, func() {
|
||||
checkFileSize(config.Config.MaxDBSizeBytes)
|
||||
})
|
||||
|
||||
go func() {
|
||||
for {
|
||||
select {
|
||||
case event, ok := <-watcher.Events:
|
||||
if !ok {
|
||||
return // closed channel
|
||||
}
|
||||
if event.Op == fsnotify.Write {
|
||||
checkFileSizeDebouncer.SetOn()
|
||||
}
|
||||
case err, ok := <-watcher.Errors:
|
||||
if !ok {
|
||||
return // closed channel
|
||||
}
|
||||
logger.Log.Errorf("filesystem watcher encountered error:%v", err)
|
||||
}
|
||||
}
|
||||
}()
|
||||
|
||||
err = watcher.Add(DBPath)
|
||||
if err != nil {
|
||||
logger.Log.Fatalf("Error adding %s to filesystem watcher for db size enforcement: %v\n", DBPath, err)
|
||||
}
|
||||
}
|
||||
|
||||
func checkFileSize(maxSizeBytes int64) {
|
||||
fileStat, err := os.Stat(DBPath)
|
||||
if err != nil {
|
||||
logger.Log.Errorf("Error checking %s file size: %v", DBPath, err)
|
||||
} else {
|
||||
if fileStat.Size() > maxSizeBytes {
|
||||
pruneOldEntries(fileStat.Size())
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func pruneOldEntries(currentFileSize int64) {
|
||||
// sqlite locks the database while delete or VACUUM are running and sqlite is terrible at handling its own db lock while a lot of inserts are attempted, we prevent a significant bottleneck by handling the db lock ourselves here
|
||||
IsDBLocked = true
|
||||
defer func() { IsDBLocked = false }()
|
||||
|
||||
amountOfBytesToTrim := currentFileSize / (100 / percentageOfMaxSizeBytesToPrune)
|
||||
|
||||
rows, err := GetEntriesTable().Limit(10000).Order("id").Rows()
|
||||
if err != nil {
|
||||
logger.Log.Errorf("Error getting 10000 first db rows: %v", err)
|
||||
return
|
||||
}
|
||||
|
||||
entryIdsToRemove := make([]uint, 0)
|
||||
bytesToBeRemoved := int64(0)
|
||||
for rows.Next() {
|
||||
if bytesToBeRemoved >= amountOfBytesToTrim {
|
||||
break
|
||||
}
|
||||
var entry tapApi.MizuEntry
|
||||
err = DB.ScanRows(rows, &entry)
|
||||
if err != nil {
|
||||
logger.Log.Errorf("Error scanning db row: %v", err)
|
||||
continue
|
||||
}
|
||||
|
||||
entryIdsToRemove = append(entryIdsToRemove, entry.ID)
|
||||
bytesToBeRemoved += int64(entry.EstimatedSizeBytes)
|
||||
}
|
||||
|
||||
if len(entryIdsToRemove) > 0 {
|
||||
GetEntriesTable().Where(entryIdsToRemove).Delete(tapApi.MizuEntry{})
|
||||
// VACUUM causes sqlite to shrink the db file after rows have been deleted, the db file will not shrink without this
|
||||
DB.Exec("VACUUM")
|
||||
logger.Log.Errorf("Removed %d rows and cleared %s", len(entryIdsToRemove), units.BytesToHumanReadable(bytesToBeRemoved))
|
||||
} else {
|
||||
logger.Log.Error("Found no rows to remove when pruning")
|
||||
}
|
||||
}
|
||||
@@ -2,12 +2,12 @@ package models
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"mizuserver/pkg/rules"
|
||||
|
||||
tapApi "github.com/up9inc/mizu/tap/api"
|
||||
|
||||
"mizuserver/pkg/rules"
|
||||
|
||||
"github.com/google/martian/har"
|
||||
basenine "github.com/up9inc/basenine/client/go"
|
||||
"github.com/up9inc/mizu/shared"
|
||||
"github.com/up9inc/mizu/tap"
|
||||
)
|
||||
@@ -16,15 +16,9 @@ func GetEntry(r *tapApi.MizuEntry, v tapApi.DataUnmarshaler) error {
|
||||
return v.UnmarshalData(r)
|
||||
}
|
||||
|
||||
type EntriesFilter struct {
|
||||
Limit int `form:"limit" validate:"required,min=1,max=200"`
|
||||
Operator string `form:"operator" validate:"required,oneof='lt' 'gt'"`
|
||||
Timestamp int64 `form:"timestamp" validate:"required,min=1"`
|
||||
}
|
||||
|
||||
type WebSocketEntryMessage struct {
|
||||
*shared.WebSocketMessageMetadata
|
||||
Data *tapApi.BaseEntryDetails `json:"data,omitempty"`
|
||||
Data map[string]interface{} `json:"data,omitempty"`
|
||||
}
|
||||
|
||||
type WebSocketTappedEntryMessage struct {
|
||||
@@ -42,7 +36,28 @@ type AuthStatus struct {
|
||||
Model string `json:"model"`
|
||||
}
|
||||
|
||||
func CreateBaseEntryWebSocketMessage(base *tapApi.BaseEntryDetails) ([]byte, error) {
|
||||
type ToastMessage struct {
|
||||
Type string `json:"type"`
|
||||
AutoClose uint `json:"autoClose"`
|
||||
Text string `json:"text"`
|
||||
}
|
||||
|
||||
type WebSocketToastMessage struct {
|
||||
*shared.WebSocketMessageMetadata
|
||||
Data *ToastMessage `json:"data,omitempty"`
|
||||
}
|
||||
|
||||
type WebSocketQueryMetadataMessage struct {
|
||||
*shared.WebSocketMessageMetadata
|
||||
Data *basenine.Metadata `json:"data,omitempty"`
|
||||
}
|
||||
|
||||
type WebSocketStartTimeMessage struct {
|
||||
*shared.WebSocketMessageMetadata
|
||||
Data int64 `json:"data"`
|
||||
}
|
||||
|
||||
func CreateBaseEntryWebSocketMessage(base map[string]interface{}) ([]byte, error) {
|
||||
message := &WebSocketEntryMessage{
|
||||
WebSocketMessageMetadata: &shared.WebSocketMessageMetadata{
|
||||
MessageType: shared.WebSocketMessageTypeEntry,
|
||||
@@ -72,6 +87,36 @@ func CreateWebsocketOutboundLinkMessage(base *tap.OutboundLink) ([]byte, error)
|
||||
return json.Marshal(message)
|
||||
}
|
||||
|
||||
func CreateWebsocketToastMessage(base *ToastMessage) ([]byte, error) {
|
||||
message := &WebSocketToastMessage{
|
||||
WebSocketMessageMetadata: &shared.WebSocketMessageMetadata{
|
||||
MessageType: shared.WebSocketMessageTypeToast,
|
||||
},
|
||||
Data: base,
|
||||
}
|
||||
return json.Marshal(message)
|
||||
}
|
||||
|
||||
func CreateWebsocketQueryMetadataMessage(base *basenine.Metadata) ([]byte, error) {
|
||||
message := &WebSocketQueryMetadataMessage{
|
||||
WebSocketMessageMetadata: &shared.WebSocketMessageMetadata{
|
||||
MessageType: shared.WebSocketMessageTypeQueryMetadata,
|
||||
},
|
||||
Data: base,
|
||||
}
|
||||
return json.Marshal(message)
|
||||
}
|
||||
|
||||
func CreateWebsocketStartTimeMessage(base int64) ([]byte, error) {
|
||||
message := &WebSocketStartTimeMessage{
|
||||
WebSocketMessageMetadata: &shared.WebSocketMessageMetadata{
|
||||
MessageType: shared.WebSocketMessageTypeStartTime,
|
||||
},
|
||||
Data: base,
|
||||
}
|
||||
return json.Marshal(message)
|
||||
}
|
||||
|
||||
// ExtendedHAR is the top level object of a HAR log.
|
||||
type ExtendedHAR struct {
|
||||
Log *ExtendedLog `json:"log"`
|
||||
|
||||
@@ -2,16 +2,10 @@ package providers_test
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"mizuserver/pkg/config"
|
||||
"mizuserver/pkg/database"
|
||||
"mizuserver/pkg/providers"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func init() {
|
||||
database.InitDataBase(config.DefaultDatabasePath)
|
||||
}
|
||||
|
||||
func TestNoEntryAddedCount(t *testing.T) {
|
||||
entriesStats := providers.GetGeneralStats()
|
||||
|
||||
|
||||
@@ -1,14 +1,14 @@
|
||||
package routes
|
||||
|
||||
import (
|
||||
"github.com/gin-gonic/gin"
|
||||
"mizuserver/pkg/controllers"
|
||||
|
||||
"github.com/gin-gonic/gin"
|
||||
)
|
||||
|
||||
// EntriesRoutes defines the group of har entries routes.
|
||||
func EntriesRoutes(ginApp *gin.Engine) {
|
||||
routeGroup := ginApp.Group("/entries")
|
||||
|
||||
routeGroup.GET("/", controllers.GetEntries) // get entries (base/thin entries)
|
||||
routeGroup.GET("/:entryId", controllers.GetEntry) // get single (full) entry
|
||||
routeGroup.GET("/:id", controllers.GetEntry) // get single (full) entry
|
||||
}
|
||||
|
||||
13
agent/pkg/routes/query_routes.go
Normal file
13
agent/pkg/routes/query_routes.go
Normal file
@@ -0,0 +1,13 @@
|
||||
package routes
|
||||
|
||||
import (
|
||||
"mizuserver/pkg/controllers"
|
||||
|
||||
"github.com/gin-gonic/gin"
|
||||
)
|
||||
|
||||
func QueryRoutes(ginApp *gin.Engine) {
|
||||
routeGroup := ginApp.Group("/query")
|
||||
|
||||
routeGroup.POST("/validate", controllers.PostValidate)
|
||||
}
|
||||
@@ -1,10 +0,0 @@
|
||||
package sensitiveDataFiltering
|
||||
|
||||
const maskedFieldPlaceholderValue = "[REDACTED]"
|
||||
|
||||
//these values MUST be all lower case and contain no `-` or `_` characters
|
||||
var personallyIdentifiableDataFields = []string{"token", "authorization", "authentication", "cookie", "userid", "password",
|
||||
"username", "user", "key", "passcode", "pass", "auth", "authtoken", "jwt",
|
||||
"bearer", "clientid", "clientsecret", "redirecturi", "phonenumber",
|
||||
"zip", "zipcode", "address", "country", "firstname", "lastname",
|
||||
"middlename", "fname", "lname", "birthdate"}
|
||||
@@ -7,15 +7,16 @@ import (
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"io/ioutil"
|
||||
"mizuserver/pkg/database"
|
||||
"mizuserver/pkg/utils"
|
||||
"net/http"
|
||||
"net/url"
|
||||
"regexp"
|
||||
"strings"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"github.com/google/martian/har"
|
||||
basenine "github.com/up9inc/basenine/client/go"
|
||||
"github.com/up9inc/mizu/shared"
|
||||
"github.com/up9inc/mizu/shared/logger"
|
||||
tapApi "github.com/up9inc/mizu/tap/api"
|
||||
@@ -23,6 +24,7 @@ import (
|
||||
|
||||
const (
|
||||
AnalyzeCheckSleepTime = 5 * time.Second
|
||||
SentCountLogInterval = 100
|
||||
)
|
||||
|
||||
type GuestToken struct {
|
||||
@@ -204,44 +206,62 @@ func syncEntriesImpl(token string, model string, envPrefix string, uploadInterva
|
||||
analyzeInformation.AnalyzeDestination = envPrefix
|
||||
analyzeInformation.SentCount = 0
|
||||
|
||||
sleepTime := time.Second * time.Duration(uploadIntervalSec)
|
||||
// "http or grpc" filter indicates that we're only interested in HTTP and gRPC entries
|
||||
query := "http or grpc"
|
||||
|
||||
var timeFrom time.Time
|
||||
protocolFilter := "http"
|
||||
logger.Log.Infof("Getting entries from the database\n")
|
||||
|
||||
for {
|
||||
timeTo := time.Now()
|
||||
logger.Log.Infof("Getting entries from %v, to %v\n", timeFrom.Format(time.RFC3339Nano), timeTo.Format(time.RFC3339Nano))
|
||||
entriesArray := database.GetEntriesFromDb(timeFrom, timeTo, &protocolFilter)
|
||||
var connection *basenine.Connection
|
||||
var err error
|
||||
connection, err = basenine.NewConnection(shared.BasenineHost, shared.BaseninePort)
|
||||
if err != nil {
|
||||
panic(err)
|
||||
}
|
||||
|
||||
if len(entriesArray) > 0 {
|
||||
result := make([]har.Entry, 0)
|
||||
for _, data := range entriesArray {
|
||||
var pair tapApi.RequestResponsePair
|
||||
if err := json.Unmarshal([]byte(data.Entry), &pair); err != nil {
|
||||
continue
|
||||
}
|
||||
harEntry, err := utils.NewEntry(&pair)
|
||||
if err != nil {
|
||||
continue
|
||||
}
|
||||
if data.ResolvedSource != "" {
|
||||
harEntry.Request.Headers = append(harEntry.Request.Headers, har.Header{Name: "x-mizu-source", Value: data.ResolvedSource})
|
||||
}
|
||||
if data.ResolvedDestination != "" {
|
||||
harEntry.Request.Headers = append(harEntry.Request.Headers, har.Header{Name: "x-mizu-destination", Value: data.ResolvedDestination})
|
||||
harEntry.Request.URL = utils.SetHostname(harEntry.Request.URL, data.ResolvedDestination)
|
||||
}
|
||||
data := make(chan []byte)
|
||||
meta := make(chan []byte)
|
||||
|
||||
// go's default marshal behavior is to encode []byte fields to base64, python's default unmarshal behavior is to not decode []byte fields from base64
|
||||
if harEntry.Response.Content.Text, err = base64.StdEncoding.DecodeString(string(harEntry.Response.Content.Text)); err != nil {
|
||||
continue
|
||||
}
|
||||
defer func() {
|
||||
data <- []byte(basenine.CloseChannel)
|
||||
meta <- []byte(basenine.CloseChannel)
|
||||
connection.Close()
|
||||
}()
|
||||
|
||||
result = append(result, *harEntry)
|
||||
handleDataChannel := func(wg *sync.WaitGroup, connection *basenine.Connection, data chan []byte) {
|
||||
defer wg.Done()
|
||||
for {
|
||||
dataBytes := <-data
|
||||
|
||||
if string(dataBytes) == basenine.CloseChannel {
|
||||
return
|
||||
}
|
||||
|
||||
logger.Log.Infof("About to upload %v entries\n", len(result))
|
||||
var dataMap map[string]interface{}
|
||||
err = json.Unmarshal(dataBytes, &dataMap)
|
||||
|
||||
result := make([]har.Entry, 0)
|
||||
var entry tapApi.MizuEntry
|
||||
if err := json.Unmarshal([]byte(dataBytes), &entry); err != nil {
|
||||
continue
|
||||
}
|
||||
harEntry, err := utils.NewEntry(entry.Request, entry.Response, entry.StartTime, entry.ElapsedTime)
|
||||
if err != nil {
|
||||
continue
|
||||
}
|
||||
if entry.ResolvedSource != "" {
|
||||
harEntry.Request.Headers = append(harEntry.Request.Headers, har.Header{Name: "x-mizu-source", Value: entry.ResolvedSource})
|
||||
}
|
||||
if entry.ResolvedDestination != "" {
|
||||
harEntry.Request.Headers = append(harEntry.Request.Headers, har.Header{Name: "x-mizu-destination", Value: entry.ResolvedDestination})
|
||||
harEntry.Request.URL = utils.SetHostname(harEntry.Request.URL, entry.ResolvedDestination)
|
||||
}
|
||||
|
||||
// go's default marshal behavior is to encode []byte fields to base64, python's default unmarshal behavior is to not decode []byte fields from base64
|
||||
if harEntry.Response.Content.Text, err = base64.StdEncoding.DecodeString(string(harEntry.Response.Content.Text)); err != nil {
|
||||
continue
|
||||
}
|
||||
|
||||
result = append(result, *harEntry)
|
||||
|
||||
body, jMarshalErr := json.Marshal(result)
|
||||
if jMarshalErr != nil {
|
||||
@@ -273,18 +293,33 @@ func syncEntriesImpl(token string, model string, envPrefix string, uploadInterva
|
||||
logger.Log.Info("Stopping sync entries")
|
||||
logger.Log.Fatal(postErr)
|
||||
}
|
||||
analyzeInformation.SentCount += len(entriesArray)
|
||||
logger.Log.Infof("Finish uploading %v entries to %s\n", len(entriesArray), GetTrafficDumpUrl(envPrefix, model))
|
||||
analyzeInformation.SentCount += 1
|
||||
|
||||
logger.Log.Infof("Uploaded %v entries until now", analyzeInformation.SentCount)
|
||||
} else {
|
||||
logger.Log.Infof("Nothing to upload")
|
||||
if analyzeInformation.SentCount%SentCountLogInterval == 0 {
|
||||
logger.Log.Infof("Uploaded %v entries until now", analyzeInformation.SentCount)
|
||||
}
|
||||
}
|
||||
|
||||
logger.Log.Infof("Sleeping for %v...\n", sleepTime)
|
||||
time.Sleep(sleepTime)
|
||||
timeFrom = timeTo
|
||||
}
|
||||
|
||||
handleMetaChannel := func(wg *sync.WaitGroup, connection *basenine.Connection, meta chan []byte) {
|
||||
defer wg.Done()
|
||||
for {
|
||||
metaBytes := <-meta
|
||||
|
||||
if string(metaBytes) == basenine.CloseChannel {
|
||||
return
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
var wg sync.WaitGroup
|
||||
go handleDataChannel(&wg, connection, data)
|
||||
go handleMetaChannel(&wg, connection, meta)
|
||||
wg.Add(2)
|
||||
|
||||
connection.Query(query, data, meta)
|
||||
|
||||
wg.Wait()
|
||||
}
|
||||
|
||||
func UpdateAnalyzeStatus(callback func(data []byte)) {
|
||||
|
||||
@@ -10,7 +10,6 @@ import (
|
||||
|
||||
"github.com/google/martian/har"
|
||||
"github.com/up9inc/mizu/shared/logger"
|
||||
"github.com/up9inc/mizu/tap/api"
|
||||
)
|
||||
|
||||
// Keep it because we might want cookies in the future
|
||||
@@ -120,13 +119,11 @@ func BuildPostParams(rawParams []interface{}) []har.Param {
|
||||
return params
|
||||
}
|
||||
|
||||
func NewRequest(request *api.GenericMessage) (harRequest *har.Request, err error) {
|
||||
reqDetails := request.Payload.(map[string]interface{})["details"].(map[string]interface{})
|
||||
func NewRequest(request map[string]interface{}) (harRequest *har.Request, err error) {
|
||||
headers, host, scheme, authority, path, _ := BuildHeaders(request["_headers"].([]interface{}))
|
||||
cookies := make([]har.Cookie, 0) // BuildCookies(request["_cookies"].([]interface{}))
|
||||
|
||||
headers, host, scheme, authority, path, _ := BuildHeaders(reqDetails["headers"].([]interface{}))
|
||||
cookies := make([]har.Cookie, 0) // BuildCookies(reqDetails["cookies"].([]interface{}))
|
||||
|
||||
postData, _ := reqDetails["postData"].(map[string]interface{})
|
||||
postData, _ := request["postData"].(map[string]interface{})
|
||||
mimeType, _ := postData["mimeType"]
|
||||
if mimeType == nil || len(mimeType.(string)) == 0 {
|
||||
mimeType = "text/html"
|
||||
@@ -138,7 +135,7 @@ func NewRequest(request *api.GenericMessage) (harRequest *har.Request, err error
|
||||
}
|
||||
|
||||
queryString := make([]har.QueryString, 0)
|
||||
for _, _qs := range reqDetails["queryString"].([]interface{}) {
|
||||
for _, _qs := range request["_queryString"].([]interface{}) {
|
||||
qs := _qs.(map[string]interface{})
|
||||
queryString = append(queryString, har.QueryString{
|
||||
Name: qs["name"].(string),
|
||||
@@ -146,7 +143,7 @@ func NewRequest(request *api.GenericMessage) (harRequest *har.Request, err error
|
||||
})
|
||||
}
|
||||
|
||||
url := fmt.Sprintf("http://%s%s", host, reqDetails["url"].(string))
|
||||
url := fmt.Sprintf("http://%s%s", host, request["url"].(string))
|
||||
if strings.HasPrefix(mimeType.(string), "application/grpc") {
|
||||
url = fmt.Sprintf("%s://%s%s", scheme, authority, path)
|
||||
}
|
||||
@@ -157,9 +154,9 @@ func NewRequest(request *api.GenericMessage) (harRequest *har.Request, err error
|
||||
}
|
||||
|
||||
harRequest = &har.Request{
|
||||
Method: reqDetails["method"].(string),
|
||||
Method: request["method"].(string),
|
||||
URL: url,
|
||||
HTTPVersion: reqDetails["httpVersion"].(string),
|
||||
HTTPVersion: request["httpVersion"].(string),
|
||||
HeadersSize: -1,
|
||||
BodySize: int64(bytes.NewBufferString(postDataText).Len()),
|
||||
QueryString: queryString,
|
||||
@@ -175,13 +172,11 @@ func NewRequest(request *api.GenericMessage) (harRequest *har.Request, err error
|
||||
return
|
||||
}
|
||||
|
||||
func NewResponse(response *api.GenericMessage) (harResponse *har.Response, err error) {
|
||||
resDetails := response.Payload.(map[string]interface{})["details"].(map[string]interface{})
|
||||
func NewResponse(response map[string]interface{}) (harResponse *har.Response, err error) {
|
||||
headers, _, _, _, _, _status := BuildHeaders(response["_headers"].([]interface{}))
|
||||
cookies := make([]har.Cookie, 0) // BuildCookies(response["_cookies"].([]interface{}))
|
||||
|
||||
headers, _, _, _, _, _status := BuildHeaders(resDetails["headers"].([]interface{}))
|
||||
cookies := make([]har.Cookie, 0) // BuildCookies(resDetails["cookies"].([]interface{}))
|
||||
|
||||
content, _ := resDetails["content"].(map[string]interface{})
|
||||
content, _ := response["content"].(map[string]interface{})
|
||||
mimeType, _ := content["mimeType"]
|
||||
if mimeType == nil || len(mimeType.(string)) == 0 {
|
||||
mimeType = "text/html"
|
||||
@@ -200,9 +195,11 @@ func NewResponse(response *api.GenericMessage) (harResponse *har.Response, err e
|
||||
Size: int64(len(bodyText)),
|
||||
}
|
||||
|
||||
status := int(resDetails["status"].(float64))
|
||||
status := int(response["status"].(float64))
|
||||
if strings.HasPrefix(mimeType.(string), "application/grpc") {
|
||||
status, err = strconv.Atoi(_status)
|
||||
if _status != "" {
|
||||
status, err = strconv.Atoi(_status)
|
||||
}
|
||||
if err != nil {
|
||||
logger.Log.Errorf("Failed converting status to int %s (%v,%+v)", err, err, err)
|
||||
return nil, errors.New("failed converting response status to int for HAR")
|
||||
@@ -210,9 +207,9 @@ func NewResponse(response *api.GenericMessage) (harResponse *har.Response, err e
|
||||
}
|
||||
|
||||
harResponse = &har.Response{
|
||||
HTTPVersion: resDetails["httpVersion"].(string),
|
||||
HTTPVersion: response["httpVersion"].(string),
|
||||
Status: status,
|
||||
StatusText: resDetails["statusText"].(string),
|
||||
StatusText: response["statusText"].(string),
|
||||
HeadersSize: -1,
|
||||
BodySize: int64(bytes.NewBufferString(bodyText).Len()),
|
||||
Headers: headers,
|
||||
@@ -222,34 +219,33 @@ func NewResponse(response *api.GenericMessage) (harResponse *har.Response, err e
|
||||
return
|
||||
}
|
||||
|
||||
func NewEntry(pair *api.RequestResponsePair) (*har.Entry, error) {
|
||||
harRequest, err := NewRequest(&pair.Request)
|
||||
func NewEntry(request map[string]interface{}, response map[string]interface{}, startTime time.Time, elapsedTime int64) (*har.Entry, error) {
|
||||
harRequest, err := NewRequest(request)
|
||||
if err != nil {
|
||||
logger.Log.Errorf("Failed converting request to HAR %s (%v,%+v)", err, err, err)
|
||||
return nil, errors.New("failed converting request to HAR")
|
||||
}
|
||||
|
||||
harResponse, err := NewResponse(&pair.Response)
|
||||
harResponse, err := NewResponse(response)
|
||||
if err != nil {
|
||||
logger.Log.Errorf("Failed converting response to HAR %s (%v,%+v)", err, err, err)
|
||||
return nil, errors.New("failed converting response to HAR")
|
||||
}
|
||||
|
||||
totalTime := pair.Response.CaptureTime.Sub(pair.Request.CaptureTime).Round(time.Millisecond).Milliseconds()
|
||||
if totalTime < 1 {
|
||||
totalTime = 1
|
||||
if elapsedTime < 1 {
|
||||
elapsedTime = 1
|
||||
}
|
||||
|
||||
harEntry := har.Entry{
|
||||
StartedDateTime: pair.Request.CaptureTime,
|
||||
Time: totalTime,
|
||||
StartedDateTime: startTime,
|
||||
Time: elapsedTime,
|
||||
Request: harRequest,
|
||||
Response: harResponse,
|
||||
Cache: &har.Cache{},
|
||||
Timings: &har.Timings{
|
||||
Send: -1,
|
||||
Wait: -1,
|
||||
Receive: totalTime,
|
||||
Receive: elapsedTime,
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
@@ -1,60 +0,0 @@
|
||||
package utils
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"time"
|
||||
|
||||
loggerShared "github.com/up9inc/mizu/shared/logger"
|
||||
"gorm.io/gorm/logger"
|
||||
"gorm.io/gorm/utils"
|
||||
)
|
||||
|
||||
// TruncatingLogger implements the gorm logger.Interface interface. Its purpose is to act as gorm's logger while truncating logs to a max of 50 characters to minimise the performance impact
|
||||
type TruncatingLogger struct {
|
||||
LogLevel logger.LogLevel
|
||||
SlowThreshold time.Duration
|
||||
}
|
||||
|
||||
func (truncatingLogger *TruncatingLogger) LogMode(logLevel logger.LogLevel) logger.Interface {
|
||||
truncatingLogger.LogLevel = logLevel
|
||||
return truncatingLogger
|
||||
}
|
||||
|
||||
func (truncatingLogger *TruncatingLogger) Info(_ context.Context, message string, __ ...interface{}) {
|
||||
if truncatingLogger.LogLevel < logger.Info {
|
||||
return
|
||||
}
|
||||
loggerShared.Log.Errorf("gorm info: %.150s", message)
|
||||
}
|
||||
|
||||
func (truncatingLogger *TruncatingLogger) Warn(_ context.Context, message string, __ ...interface{}) {
|
||||
if truncatingLogger.LogLevel < logger.Warn {
|
||||
return
|
||||
}
|
||||
loggerShared.Log.Errorf("gorm warning: %.150s", message)
|
||||
}
|
||||
|
||||
func (truncatingLogger *TruncatingLogger) Error(_ context.Context, message string, __ ...interface{}) {
|
||||
if truncatingLogger.LogLevel < logger.Error {
|
||||
return
|
||||
}
|
||||
loggerShared.Log.Errorf("gorm error: %.150s", message)
|
||||
}
|
||||
|
||||
func (truncatingLogger *TruncatingLogger) Trace(ctx context.Context, begin time.Time, fc func() (string, int64), err error) {
|
||||
if truncatingLogger.LogLevel == logger.Silent {
|
||||
return
|
||||
}
|
||||
elapsed := time.Since(begin)
|
||||
if err != nil {
|
||||
sql, rows := fc() // copied into every condition as this is a potentially heavy operation best done only when necessary
|
||||
truncatingLogger.Error(ctx, fmt.Sprintf("Error in %s: %v - elapsed: %fs affected rows: %d, sql: %s", utils.FileWithLineNum(), err, elapsed.Seconds(), rows, sql))
|
||||
} else if truncatingLogger.LogLevel >= logger.Warn && elapsed > truncatingLogger.SlowThreshold {
|
||||
sql, rows := fc()
|
||||
truncatingLogger.Warn(ctx, fmt.Sprintf("Slow sql query - elapse: %fs rows: %d, sql: %s", elapsed.Seconds(), rows, sql))
|
||||
} else if truncatingLogger.LogLevel >= logger.Info {
|
||||
sql, rows := fc()
|
||||
truncatingLogger.Info(ctx, fmt.Sprintf("Sql query - elapse: %fs rows: %d, sql: %s", elapsed.Seconds(), rows, sql))
|
||||
}
|
||||
}
|
||||
Reference in New Issue
Block a user