The API traffic analyzer for Kubernetes providing real-time K8s protocol-level visibility, capturing and monitoring all traffic and payloads going in, out and across containers, pods, nodes and clusters. Inspired by Wireshark, purposely built for Kubernetes
Go to file
David Levanon e696851261
Tapper Refactor (#396)
* introduce tcp_assembler and tcp_packet_source - the motivation is to … (#380)

* add passive-tapper main tester (#353)

* add passive-tapper main tester

* add errors to go.sum of mizu agent

* disable host mode for tester - to avoid filterAuthorities

* rename main to tester

* build extenssions as part of the tester launch

* add a README to the tester

* solving go.mod and .sum conflicts with addition of go-errors

* trivial warning fixes (#354)

* add passive-tapper main tester

* trivial warning fixes

* add errors to go.sum of mizu agent

* disable host mode for tester - to avoid filterAuthorities

* tcp streams map (#355)

* add passive-tapper main tester

* trivial warning fixes

* add errors to go.sum of mizu agent

* tcp streams map

* disable host mode for tester - to avoid filterAuthorities

* set tcp streams map for tcp stream factory

* change rlog to mizu logger

* errors map (#356)

* add passive-tapper main tester

* trivial warning fixes

* add errors to go.sum of mizu agent

* tcp streams map

* disable host mode for tester - to avoid filterAuthorities

* set tcp streams map for tcp stream factory

* errors map

* change int to uint - errorsmap

* change from int to uint

* Change errorsMap.nErrors to uint.

* change errors map to mizu logger instead of rlog

* init mizu logger in tester + fix errormap declaration

Co-authored-by: Nimrod Gilboa Markevich <nimrod@up9.com>

* move own ips to tcp stream factory (#358)

* add passive-tapper main tester

* trivial warning fixes

* add errors to go.sum of mizu agent

* tcp streams map

* disable host mode for tester - to avoid filterAuthorities

* set tcp streams map for tcp stream factory

* errors map

* move own ips to tcp stream factory

* Feature/tapper refactor i/move own ips to tcp stream factory (#379)

* add passive-tapper main tester

* trivial warning fixes

* add errors to go.sum of mizu agent

* tcp streams map

* disable host mode for tester - to avoid filterAuthorities

* set tcp streams map for tcp stream factory

* errors map

* move own ips to tcp stream factory

* fix ownips compilation issue

* introduce tcp_assembler and tcp_packet_source - the motivation is to split the actual packet sources from the assembler, so we can have a single thread for the assembly which is separated from packet source threads

* make struts private at this point - planning to move some packages to their own package so we can utilize encapsulation

* move context to tcp_assembly + fix error check of tcp source packet

* use param instead of gloab flag for ipdefrag

* fix a typo

* remove unused pid param

Co-authored-by: Nimrod Gilboa Markevich <nimrod@up9.com>

* extract stats functions out of the main tapping function (#381)

* add passive-tapper main tester (#353)

* add passive-tapper main tester

* add errors to go.sum of mizu agent

* disable host mode for tester - to avoid filterAuthorities

* rename main to tester

* build extenssions as part of the tester launch

* add a README to the tester

* solving go.mod and .sum conflicts with addition of go-errors

* trivial warning fixes (#354)

* add passive-tapper main tester

* trivial warning fixes

* add errors to go.sum of mizu agent

* disable host mode for tester - to avoid filterAuthorities

* tcp streams map (#355)

* add passive-tapper main tester

* trivial warning fixes

* add errors to go.sum of mizu agent

* tcp streams map

* disable host mode for tester - to avoid filterAuthorities

* set tcp streams map for tcp stream factory

* change rlog to mizu logger

* errors map (#356)

* add passive-tapper main tester

* trivial warning fixes

* add errors to go.sum of mizu agent

* tcp streams map

* disable host mode for tester - to avoid filterAuthorities

* set tcp streams map for tcp stream factory

* errors map

* change int to uint - errorsmap

* change from int to uint

* Change errorsMap.nErrors to uint.

* change errors map to mizu logger instead of rlog

* init mizu logger in tester + fix errormap declaration

Co-authored-by: Nimrod Gilboa Markevich <nimrod@up9.com>

* move own ips to tcp stream factory (#358)

* add passive-tapper main tester

* trivial warning fixes

* add errors to go.sum of mizu agent

* tcp streams map

* disable host mode for tester - to avoid filterAuthorities

* set tcp streams map for tcp stream factory

* errors map

* move own ips to tcp stream factory

* Feature/tapper refactor i/move own ips to tcp stream factory (#379)

* add passive-tapper main tester

* trivial warning fixes

* add errors to go.sum of mizu agent

* tcp streams map

* disable host mode for tester - to avoid filterAuthorities

* set tcp streams map for tcp stream factory

* errors map

* move own ips to tcp stream factory

* fix ownips compilation issue

* introduce tcp_assembler and tcp_packet_source - the motivation is to split the actual packet sources from the assembler, so we can have a single thread for the assembly which is separated from packet source threads

* make struts private at this point - planning to move some packages to their own package so we can utilize encapsulation

* extract stats functions out of the main tapping function

* move context to tcp_assembly + fix error check of tcp source packet

* use param instead of gloab flag for ipdefrag

Co-authored-by: Nimrod Gilboa Markevich <nimrod@up9.com>

* Feature/tapper refactor i/internal tapper stats (#384)

* add passive-tapper main tester (#353)

* add passive-tapper main tester

* add errors to go.sum of mizu agent

* disable host mode for tester - to avoid filterAuthorities

* rename main to tester

* build extenssions as part of the tester launch

* add a README to the tester

* solving go.mod and .sum conflicts with addition of go-errors

* trivial warning fixes (#354)

* add passive-tapper main tester

* trivial warning fixes

* add errors to go.sum of mizu agent

* disable host mode for tester - to avoid filterAuthorities

* tcp streams map (#355)

* add passive-tapper main tester

* trivial warning fixes

* add errors to go.sum of mizu agent

* tcp streams map

* disable host mode for tester - to avoid filterAuthorities

* set tcp streams map for tcp stream factory

* change rlog to mizu logger

* errors map (#356)

* add passive-tapper main tester

* trivial warning fixes

* add errors to go.sum of mizu agent

* tcp streams map

* disable host mode for tester - to avoid filterAuthorities

* set tcp streams map for tcp stream factory

* errors map

* change int to uint - errorsmap

* change from int to uint

* Change errorsMap.nErrors to uint.

* change errors map to mizu logger instead of rlog

* init mizu logger in tester + fix errormap declaration

Co-authored-by: Nimrod Gilboa Markevich <nimrod@up9.com>

* move own ips to tcp stream factory (#358)

* add passive-tapper main tester

* trivial warning fixes

* add errors to go.sum of mizu agent

* tcp streams map

* disable host mode for tester - to avoid filterAuthorities

* set tcp streams map for tcp stream factory

* errors map

* move own ips to tcp stream factory

* Feature/tapper refactor i/move own ips to tcp stream factory (#379)

* add passive-tapper main tester

* trivial warning fixes

* add errors to go.sum of mizu agent

* tcp streams map

* disable host mode for tester - to avoid filterAuthorities

* set tcp streams map for tcp stream factory

* errors map

* move own ips to tcp stream factory

* fix ownips compilation issue

* introduce tcp_assembler and tcp_packet_source - the motivation is to split the actual packet sources from the assembler, so we can have a single thread for the assembly which is separated from packet source threads

* make struts private at this point - planning to move some packages to their own package so we can utilize encapsulation

* extract stats functions out of the main tapping function

* move context to tcp_assembly + fix error check of tcp source packet

* use param instead of gloab flag for ipdefrag

* introduce internal tapper stats

* minor fixes for errors map and internal stats

Co-authored-by: Nimrod Gilboa Markevich <nimrod@up9.com>

* Feature/tapper refactor i/diagnose package (#386)

* add passive-tapper main tester (#353)

* add passive-tapper main tester

* add errors to go.sum of mizu agent

* disable host mode for tester - to avoid filterAuthorities

* rename main to tester

* build extenssions as part of the tester launch

* add a README to the tester

* solving go.mod and .sum conflicts with addition of go-errors

* trivial warning fixes (#354)

* add passive-tapper main tester

* trivial warning fixes

* add errors to go.sum of mizu agent

* disable host mode for tester - to avoid filterAuthorities

* tcp streams map (#355)

* add passive-tapper main tester

* trivial warning fixes

* add errors to go.sum of mizu agent

* tcp streams map

* disable host mode for tester - to avoid filterAuthorities

* set tcp streams map for tcp stream factory

* change rlog to mizu logger

* errors map (#356)

* add passive-tapper main tester

* trivial warning fixes

* add errors to go.sum of mizu agent

* tcp streams map

* disable host mode for tester - to avoid filterAuthorities

* set tcp streams map for tcp stream factory

* errors map

* change int to uint - errorsmap

* change from int to uint

* Change errorsMap.nErrors to uint.

* change errors map to mizu logger instead of rlog

* init mizu logger in tester + fix errormap declaration

Co-authored-by: Nimrod Gilboa Markevich <nimrod@up9.com>

* move own ips to tcp stream factory (#358)

* add passive-tapper main tester

* trivial warning fixes

* add errors to go.sum of mizu agent

* tcp streams map

* disable host mode for tester - to avoid filterAuthorities

* set tcp streams map for tcp stream factory

* errors map

* move own ips to tcp stream factory

* Feature/tapper refactor i/move own ips to tcp stream factory (#379)

* add passive-tapper main tester

* trivial warning fixes

* add errors to go.sum of mizu agent

* tcp streams map

* disable host mode for tester - to avoid filterAuthorities

* set tcp streams map for tcp stream factory

* errors map

* move own ips to tcp stream factory

* fix ownips compilation issue

* introduce tcp_assembler and tcp_packet_source - the motivation is to split the actual packet sources from the assembler, so we can have a single thread for the assembly which is separated from packet source threads

* make struts private at this point - planning to move some packages to their own package so we can utilize encapsulation

* extract stats functions out of the main tapping function

* move context to tcp_assembly + fix error check of tcp source packet

* use param instead of gloab flag for ipdefrag

* introduce internal tapper stats

* minor fixes for errors map and internal stats

* move errors map + app stats + internal stats + periodic tasks to diagnose package

* initialize tapper internal stats

Co-authored-by: Nimrod Gilboa Markevich <nimrod@up9.com>

* move tcp packet source to its packet (#387)

* add passive-tapper main tester (#353)

* add passive-tapper main tester

* add errors to go.sum of mizu agent

* disable host mode for tester - to avoid filterAuthorities

* rename main to tester

* build extenssions as part of the tester launch

* add a README to the tester

* solving go.mod and .sum conflicts with addition of go-errors

* trivial warning fixes (#354)

* add passive-tapper main tester

* trivial warning fixes

* add errors to go.sum of mizu agent

* disable host mode for tester - to avoid filterAuthorities

* tcp streams map (#355)

* add passive-tapper main tester

* trivial warning fixes

* add errors to go.sum of mizu agent

* tcp streams map

* disable host mode for tester - to avoid filterAuthorities

* set tcp streams map for tcp stream factory

* change rlog to mizu logger

* errors map (#356)

* add passive-tapper main tester

* trivial warning fixes

* add errors to go.sum of mizu agent

* tcp streams map

* disable host mode for tester - to avoid filterAuthorities

* set tcp streams map for tcp stream factory

* errors map

* change int to uint - errorsmap

* change from int to uint

* Change errorsMap.nErrors to uint.

* change errors map to mizu logger instead of rlog

* init mizu logger in tester + fix errormap declaration

Co-authored-by: Nimrod Gilboa Markevich <nimrod@up9.com>

* move own ips to tcp stream factory (#358)

* add passive-tapper main tester

* trivial warning fixes

* add errors to go.sum of mizu agent

* tcp streams map

* disable host mode for tester - to avoid filterAuthorities

* set tcp streams map for tcp stream factory

* errors map

* move own ips to tcp stream factory

* Feature/tapper refactor i/move own ips to tcp stream factory (#379)

* add passive-tapper main tester

* trivial warning fixes

* add errors to go.sum of mizu agent

* tcp streams map

* disable host mode for tester - to avoid filterAuthorities

* set tcp streams map for tcp stream factory

* errors map

* move own ips to tcp stream factory

* fix ownips compilation issue

* introduce tcp_assembler and tcp_packet_source - the motivation is to split the actual packet sources from the assembler, so we can have a single thread for the assembly which is separated from packet source threads

* make struts private at this point - planning to move some packages to their own package so we can utilize encapsulation

* extract stats functions out of the main tapping function

* move context to tcp_assembly + fix error check of tcp source packet

* use param instead of gloab flag for ipdefrag

* introduce internal tapper stats

* minor fixes for errors map and internal stats

* move errors map + app stats + internal stats + periodic tasks to diagnose package

* move tcp packet source to its packet

* initialize tapper internal stats

Co-authored-by: Nimrod Gilboa Markevich <nimrod@up9.com>

* Fix coding style

* Remove `tap/internal_stats.go`

* make channel between input and assembler blocking - to preserve the same behaviour we have before the refactor

Co-authored-by: Nimrod Gilboa Markevich <nimrod@up9.com>
Co-authored-by: M. Mert Yildiran <mehmet@up9.com>
2021-10-25 13:59:06 +03:00
.github build m1 and windows (#395) 2021-10-24 15:10:46 +03:00
acceptanceTests Fix the go.mod of acceptanceTests (#371) 2021-10-18 16:35:10 +03:00
agent fixed naming of latency to response time (#388) 2021-10-21 12:45:17 +03:00
assets Feature/testing contributing doc (#197) 2021-08-11 09:59:14 +03:00
cli build m1 and windows (#395) 2021-10-24 15:10:46 +03:00
devops Move all docs to docs folder and clean project root (#278) 2021-09-15 11:53:23 +03:00
docs Add OAS contract monitoring support (#325) 2021-10-19 14:24:22 +03:00
examples/roles Namespace restricted mode (#147) 2021-08-05 10:28:31 +03:00
shared TRA-3803 handle k8s watch timeouts (#372) 2021-10-19 14:41:37 +03:00
tap Tapper Refactor (#396) 2021-10-25 13:59:06 +03:00
ui fixed naming of latency to response time (#388) 2021-10-21 12:45:17 +03:00
.dockerignore Refactor Mizu, define an extension API and add new protocols: AMQP, Kafka (#224) 2021-09-02 14:34:06 +03:00
.gitignore Fix the memory exhaustion by optimizing max. AMQP message size and GOGC (#257) 2021-09-09 17:45:37 +03:00
codecov.yml codecov yml for tests threshold (#214) 2021-08-15 12:19:00 +03:00
debug.Dockerfile Renamed collector, aggregator to api server, api folder to agent (#133) 2021-07-22 17:17:17 +03:00
Dockerfile Add OAS contract monitoring support (#325) 2021-10-19 14:24:22 +03:00
LICENSE Initial commit 2021-04-19 13:29:56 +03:00
Makefile Move all docs to docs folder and clean project root (#278) 2021-09-15 11:53:23 +03:00
README.md Add k8s version requirements to README (#389) 2021-10-21 15:29:26 +03:00

Mizu: The API Traffic Viewer for Kubernetes

The API Traffic Viewer for Kubernetes

A simple-yet-powerful API traffic viewer for Kubernetes enabling you to view all API communication between microservices to help your debug and troubleshoot regressions.

Think TCPDump and Chrome Dev Tools combined.

Simple UI

Features

  • Simple and powerful CLI
  • Real-time view of all HTTP requests, REST and gRPC API calls
  • No installation or code instrumentation
  • Works completely on premises

Requirements

A Kubernetes server version of 1.16.0 or higher is required.

Download

Download Mizu for your platform and operating system

Latest Stable Release

  • for MacOS - Intel
curl -Lo mizu \
https://github.com/up9inc/mizu/releases/latest/download/mizu_darwin_amd64 \
&& chmod 755 mizu
  • for Linux - Intel 64bit
curl -Lo mizu \
https://github.com/up9inc/mizu/releases/latest/download/mizu_linux_amd64 \
&& chmod 755 mizu

SHA256 checksums are available on the Releases page

Development (unstable) Build

Pick one from the Releases page

Kubeconfig & Permissions

While mizumost often works out of the box, you can influence its behavior:

  1. [OPTIONAL] Set KUBECONFIG environment variable to your Kubernetes configuration. If this is not set, Mizu assumes that configuration is at ${HOME}/.kube/config
  2. mizu assumes user running the command has permissions to create resources (such as pods, services, namespaces) on your Kubernetes cluster (no worries - mizu resources are cleaned up upon termination)

For detailed list of k8s permissions see PERMISSIONS document

How to Run

  1. Find pods you'd like to tap to in your Kubernetes cluster
  2. Run mizu tap or mizu tap PODNAME
  3. Open browser on http://localhost:8899/mizu or as instructed in the CLI
  4. Watch the API traffic flowing
  5. Type ^C to stop

Examples

Run mizu help for usage options

To tap all pods in current namespace -

 $ kubectl get pods 
 NAME                            READY   STATUS    RESTARTS   AGE
 carts-66c77f5fbb-fq65r          2/2     Running   0          20m
 catalogue-5f4cb7cf5-7zrmn       2/2     Running   0          20m
 front-end-649fc5fd6-kqbtn       2/2     Running   0          20m
 ..

 $ mizu tap
 +carts-66c77f5fbb-fq65r
 +catalogue-5f4cb7cf5-7zrmn
 +front-end-649fc5fd6-kqbtn
 Web interface is now available at http://localhost:8899
 ^C

To tap specific pod -

 $ kubectl get pods 
 NAME                            READY   STATUS    RESTARTS   AGE
 front-end-649fc5fd6-kqbtn       2/2     Running   0          7m
 ..

 $ mizu tap front-end-649fc5fd6-kqbtn
 +front-end-649fc5fd6-kqbtn
 Web interface is now available at http://localhost:8899
 ^C

To tap multiple pods using regex -

 $ kubectl get pods 
 NAME                            READY   STATUS    RESTARTS   AGE
 carts-66c77f5fbb-fq65r          2/2     Running   0          20m
 catalogue-5f4cb7cf5-7zrmn       2/2     Running   0          20m
 front-end-649fc5fd6-kqbtn       2/2     Running   0          20m
 ..

 $ mizu tap "^ca.*"
 +carts-66c77f5fbb-fq65r
 +catalogue-5f4cb7cf5-7zrmn
 Web interface is now available at http://localhost:8899
 ^C

Configuration

Mizu can work with config file which should be stored in ${HOME}/.mizu/config.yaml (macOS: ~/.mizu/config.yaml)
In case no config file found, defaults will be used
In case of partial configuration defined, all other fields will be used with defaults
You can always override the defaults or config file with CLI flags

To get the default config params run mizu config
To generate a new config file with default values use mizu config -r

Telemetry

By default, mizu reports usage telemetry. It can be disabled by adding a line of telemetry: false in the ${HOME}/.mizu/config.yaml file

Advanced Usage

Namespace-Restricted Mode

Some users have permission to only manage resources in one particular namespace assigned to them By default mizu tap creates a new namespace mizu for all of its Kubernetes resources. In order to instead install Mizu in an existing namespace, set the mizu-resources-namespace config option

If mizu-resources-namespace is set to a value other than the default mizu, Mizu will operate in a Namespace-Restricted mode. It will only tap pods in mizu-resources-namespace. This way Mizu only requires permissions to the namespace set by mizu-resources-namespace. The user must set the tapped namespace to the same namespace by using the --namespace flag or by setting tap.namespaces in the config file

Setting mizu-resources-namespace=mizu resets Mizu to its default behavior

User agent filtering

User-agent filtering (like health checks) - can be configured using command-line options:

$ mizu tap "^ca.*" --set tap.ignored-user-agents=kube-probe --set tap.ignored-user-agents=prometheus
+carts-66c77f5fbb-fq65r
+catalogue-5f4cb7cf5-7zrmn
Web interface is now available at http://localhost:8899
^C

Any request that contains User-Agent header with one of the specified values (kube-probe or prometheus) will not be captured

Traffic validation rules

This feature allows you to define set of simple rules, and test the traffic against them. Such validation may test response for specific JSON fields, headers, etc.

Please see TRAFFIC RULES page for more details and syntax.

OpenAPI Specification (OAS) Contract Monitoring

An OAS/Swagger file can contain schemas under parameters and responses fields. With --contract catalogue.yaml CLI option, you can pass your API description to Mizu and the traffic will automatically be validated against the contracts.

Please see CONTRACT MONITORING page for more details and syntax.

How to Run local UI

  • run from mizu/agent go run main.go --hars-read --hars-dir <folder>

  • copy Har files into the folder from last command

  • change MizuWebsocketURL and apiURL in api.js file

  • run from mizu/ui - npm run start

  • open browser on localhost:3000