diff --git a/Development-Workflows.md b/Development-Workflows.md new file mode 100644 index 0000000..70b5121 --- /dev/null +++ b/Development-Workflows.md @@ -0,0 +1,263 @@ +In this tutorial we'll explain two development workflows for [Mizu](https://github.com/up9inc/mizu): + +- Standard development workflow that you build a Docker image, publish into Docker Hub and pull it to Kubernetes. The workflow is only necessary if you're developing some Kubernetes specific feature. +- Local machine development workflow that you run Mizu Agent on your machine with `sudo` privileges. Which is much faster than the standard development workflow and can be used if you're **not** developing a Kubernetes specific feature. + +## Standard Development Workflow + +Whenever you make changes in the `agent` and `tap` packages or protocol extensions to test it; you first need to build a Docker image: + +```shell +$ docker build . -t mertyildiran/mizuagent:latest +``` + +Then you need to push that image into Docker Hub: + +```shell +$ docker push mertyildiran/mizuagent:latest +``` + +Set the `agent-image` to `mertyildiran/mizuagent:latest` in the Mizu config YAML (on path `~/.mizu/config.yaml`): + +```shell +mertyildiran@Corsair:~/.mizu$ cat config.yaml +agent-image: mertyildiran/mizuagent:latest +telemetry: false +``` + +_It will make Mizu to use your custom Mizu Agent Docker image as the API server and tapper Daemon Set._ + +You also need to build the CLI program: + +```shell +$ make cli +``` + +Start minikube: + +```shell +$ minikube start +``` + +Deploy `hello-node`: + +```shell +$ kubectl create deployment hello-node --image=k8s.gcr.io/echoserver:1.4 +$ kubectl expose deployment hello-node --type=LoadBalancer --port=8080 +``` + +You then look at the pod name: + +```shell +$ kubectl get pod -A +NAMESPACE NAME READY STATUS RESTARTS AGE +default hello-node-7567d9fdc9-f9n7r 1/1 Running 3 6d14h +kube-system coredns-558bd4d5db-qkz86 1/1 Running 15 31d +kube-system etcd-minikube 1/1 Running 15 31d +kube-system kube-apiserver-minikube 1/1 Running 15 31d +kube-system kube-controller-manager-minikube 1/1 Running 15 31d +kube-system kube-proxy-62xdc 1/1 Running 15 31d +kube-system kube-scheduler-minikube 1/1 Running 15 31d +kube-system storage-provisioner 1/1 Running 28 31d +kubernetes-dashboard dashboard-metrics-scraper-7976b667d4-vsmmv 1/1 Running 15 31d +kubernetes-dashboard kubernetes-dashboard-6fcdf4f6d-9z5xj 1/1 Running 24 31d +``` + +Then you tap into the pod with Mizu: + +```shell +./cli/bin/mizu__ tap hello-node-7567d9fdc9-f9n7r +``` + +Then you wait for the pull of the image: + +```shell +$ kubectl get pod -A +NAMESPACE NAME READY STATUS RESTARTS AGE +default hello-node-7567d9fdc9-f9n7r 1/1 Running 3 6d14h +kube-system coredns-558bd4d5db-qkz86 1/1 Running 15 31d +kube-system etcd-minikube 1/1 Running 15 31d +kube-system kube-apiserver-minikube 1/1 Running 15 31d +kube-system kube-controller-manager-minikube 1/1 Running 15 31d +kube-system kube-proxy-62xdc 1/1 Running 15 31d +kube-system kube-scheduler-minikube 1/1 Running 15 31d +kube-system storage-provisioner 1/1 Running 28 31d +kubernetes-dashboard dashboard-metrics-scraper-7976b667d4-vsmmv 1/1 Running 15 31d +kubernetes-dashboard kubernetes-dashboard-6fcdf4f6d-9z5xj 1/1 Running 24 31d +mizu mizu-api-server 1/1 Running 0 15s +mizu mizu-tapper-daemon-set-z7w2m 1/1 Running 0 15s +``` + +You look at the Daemon Set logs to see if the program running as expected: + +```shell +$ kubectl logs mizu-tapper-daemon-set-z7w2m -n mizu +2021/08/23 10:43:31 Loading extension: amqp.so +2021/08/23 10:43:31 Initializing AMQP extension... +2021/08/23 10:43:31 Extension Properties: &{Protocol:{Name:amqp LongName:Advanced Message Queuing Protocol 0-9-1 Abbreviation:AMQP Version:0-9-1 BackgroundColor:#ff6600 ForegroundColor:#ffffff FontSize:12 ReferenceLink:https://www.rabbitmq.com/amqp-0-9-1-reference.html Ports:[5671 5672]} Path:/app/extensions/amqp.so Plug:0xc0002deb40 Dissector:0x7fd399bd0070} +2021/08/23 10:43:31 Loading extension: http.so +2021/08/23 10:43:31 Initializing HTTP extension. +2021/08/23 10:43:31 Extension Properties: &{Protocol:{Name:http LongName:Hypertext Transfer Protocol -- HTTP/1.1 Abbreviation:HTTP Version:1.1 BackgroundColor:#205cf5 ForegroundColor:#ffffff FontSize:12 ReferenceLink:https://datatracker.ietf.org/doc/html/rfc2616 Ports:[80 8080 50051]} Path:/app/extensions/http.so Plug:0xc0002de9f0 Dissector:0x7fd399851860} +2021/08/23 10:43:31 Loading extension: kafka.so +2021/08/23 10:43:31 Initializing Kafka extension... +2021/08/23 10:43:31 Extension Properties: &{Protocol:{Name:kafka LongName:Apache Kafka Protocol Abbreviation:KAFKA Version:12 BackgroundColor:#000000 ForegroundColor:#ffffff FontSize:11 ReferenceLink:https://kafka.apache.org/protocol Ports:[9092]} Path:/app/extensions/kafka.so Plug:0xc0002df0e0 Dissector:0x7fd399185940} +2021/08/23 10:43:31 All extension ports: [5671 5672 80 8080 50051 9092] +2021-08-23T10:43:31Z INFO : Filtering for the following authorities: [172.17.0.5] +2021-08-23T10:43:31Z INFO : Received empty/no APP_PORTS env var! only listening to ports: [5671 5672 80 8080 50051 9092] +2021/08/23 10:43:31 passive_tapper.go:254: App Ports: [] +Failed connecting to websocket server: dial tcp 10.98.144.224:80: connect: connection refused, (dial tcp 10.98.144.224:80: connect: connection refused,dial tcp 10.98.144.224:80: connect: connection refused) +2021-08-23T10:43:31Z INFO : Starting to read packets +2021-08-23T10:43:31Z INFO : Assembler options: maxBufferedPagesTotal=5000, maxBufferedPagesPerConnection=5000 +``` + +_Notice HTTP, AMQP and Kafka extensions are loaded._ + +Visit http://localhost:8899/mizu + +Run `minikube service hello-node` to generate some traffic. + +Now you should be seeing the traffic in your browser: + +![](https://i.ibb.co/f84yprB/Screenshot-from-2021-08-23-13-44-52.png) + +You look at the Daemon Set logs again to see if something went wrong: + +```shell +$ kubectl logs mizu-tapper-daemon-set-z7w2m -n mizu +2021/08/23 10:43:31 Loading extension: amqp.so +2021/08/23 10:43:31 Initializing AMQP extension... +2021/08/23 10:43:31 Extension Properties: &{Protocol:{Name:amqp LongName:Advanced Message Queuing Protocol 0-9-1 Abbreviation:AMQP Version:0-9-1 BackgroundColor:#ff6600 ForegroundColor:#ffffff FontSize:12 ReferenceLink:https://www.rabbitmq.com/amqp-0-9-1-reference.html Ports:[5671 5672]} Path:/app/extensions/amqp.so Plug:0xc0002deb40 Dissector:0x7fd399bd0070} +2021/08/23 10:43:31 Loading extension: http.so +2021/08/23 10:43:31 Initializing HTTP extension. +2021/08/23 10:43:31 Extension Properties: &{Protocol:{Name:http LongName:Hypertext Transfer Protocol -- HTTP/1.1 Abbreviation:HTTP Version:1.1 BackgroundColor:#205cf5 ForegroundColor:#ffffff FontSize:12 ReferenceLink:https://datatracker.ietf.org/doc/html/rfc2616 Ports:[80 8080 50051]} Path:/app/extensions/http.so Plug:0xc0002de9f0 Dissector:0x7fd399851860} +2021/08/23 10:43:31 Loading extension: kafka.so +2021/08/23 10:43:31 Initializing Kafka extension... +2021/08/23 10:43:31 Extension Properties: &{Protocol:{Name:kafka LongName:Apache Kafka Protocol Abbreviation:KAFKA Version:12 BackgroundColor:#000000 ForegroundColor:#ffffff FontSize:11 ReferenceLink:https://kafka.apache.org/protocol Ports:[9092]} Path:/app/extensions/kafka.so Plug:0xc0002df0e0 Dissector:0x7fd399185940} +2021/08/23 10:43:31 All extension ports: [5671 5672 80 8080 50051 9092] +2021-08-23T10:43:31Z INFO : Filtering for the following authorities: [172.17.0.5] +2021-08-23T10:43:31Z INFO : Received empty/no APP_PORTS env var! only listening to ports: [5671 5672 80 8080 50051 9092] +2021/08/23 10:43:31 passive_tapper.go:254: App Ports: [] +Failed connecting to websocket server: dial tcp 10.98.144.224:80: connect: connection refused, (dial tcp 10.98.144.224:80: connect: connection refused,dial tcp 10.98.144.224:80: connect: connection refused) +2021-08-23T10:43:31Z INFO : Starting to read packets +2021-08-23T10:43:31Z INFO : Assembler options: maxBufferedPagesTotal=5000, maxBufferedPagesPerConnection=5000 +2021/08/23 10:44:31 passive_tapper.go:362: 1m0.001427106s (errors: 452, errTypes:2) - Errors Summary: map[FSM-rejection:134 OptionChecker-rejection:318] +2021/08/23 10:44:31 passive_tapper.go:372: mem: 25554360, goroutines: 13, unmatched messages: +2021/08/23 10:44:31 passive_tapper.go:380: cleaner - flushed connections: 0, closed connections: 0, deleted messages: 0 +2021/08/23 10:44:31 passive_tapper.go:388: app stats - {"processedBytes":5817724,"packetsCount":10216,"tcpPacketsCount":10164,"reassembledTcpPayloadsCount":0,"tlsConnectionsCount":0,"matchedPairs":0} +``` + +_Everything is OK!_ + +You CTRL+C to close Mizu CLI and wait for it to remove the resources: + +```shell +$ ./cli/bin/mizu__ tap hello-node-7567d9fdc9-f9n7r +Mizu will store up to 200MB of traffic, old traffic will be cleared once the limit is reached. +Tapping pods in namespaces "default" ++hello-node-7567d9fdc9-f9n7r +Mizu is available at http://localhost:8899/mizu +^C +Removing mizu resources +Update available! 0.0.0 -> 0.11.0 (https://github.com/up9inc/mizu/releases/tag/0.11.0) +``` + +### Conclusion + +It took several minutes to test even the tiniest change. So this method should be used only if you're developing something Kubernetes specific. + +It also requires you to develop the Kubernetes deployment configuration for the feature that you want to test. Which means that multiply the development by a factor of two at least. + +## Local Machine Development Workflow + +This method involves doing everything on your local machine without requiring any Kubernetes deployment. + +Save this script to a file named `dev.sh`: + +```bash +#!/bin/bash + +rm entries.db +rm -rf pprof/* && make clean && make agent +sudo setcap cap_net_raw,cap_net_admin=eip ./agent/build/mizuagent +./agent/build/mizuagent --api-server & \ +PID1=$! && \ +sleep 0.5 && \ +GOGC=12800 NODE_NAME=dev TAPPED_ADDRESSES_PER_HOST='{"dev": ["localhost"]}' \ + ./agent/build/mizuagent \ + -i any \ + --tap \ + --api-server-address ws://localhost:8899/wsTapper \ + --nodefrag & \ +PID2=$! && \ +read -r -d '' _ CTRL+C + +Run the React app in another terminal: + +```shell +$ cd ui/ && cp .env.example .env && npm install && npm start +``` + +That's all. Now you have Mizu listenning your local machine. + +### Testing HTTP, gRPC, AMQP, Kafka protocols together + +> See [this Gist](https://gist.github.com/mertyildiran/7286261b3b4c9345f4f51fe30a955f16) for the example scripts. + +Open another terminal, run the following to have RabbitMQ and Kafka available on your network: + +```shell +$ docker run -d -it --rm --name rabbitmq --net=host rabbitmq:latest && sleep 10 +$ docker run -d -it --rm --name kafka --net=host up9inc/mockintosh:self-contained-kafka && sleep 2 +``` + +Start a gRPC server example: + +```shell +$ cd ~/Downloads/ && git clone -b v1.38.0 https://github.com/grpc/grpc --depth 1 +$ cd ~/Downloads/grpc/examples/python/route_guide && python route_guide_server.py +``` + +Save the `.go` and `.py` files that are provided by this Gist into a directory. + +In the same directory, save this script to a file named `run.sh`: + +```bash +#!/bin/bash + +go run produce.go & +go run consume.go & +python3 example.py & +python3 ~/Downloads/grpc/examples/python/route_guide/route_guide_client.py & +go run create_topics.go & +go run list_topics.go & +``` + +These Go and Python programs will generate the traffic for the protocols: HTTP, gRPC, AMQP, Kafka + +Make it executable `chmod +x run.sh` and run `./run.sh` + +Now you should be seeing the traffic in your browser: + +![](https://i.ibb.co/mcwphdT/Screenshot-from-2021-08-23-14-51-08.png) + +### Conclusion + +Testing a TCP packet sniffer/analyzer on your local machine is much a more robust development workflow than building a Docker image and deploying to a Kubernetes cluster. + +The time for the trial and error cycle is reduced from several minutes to a few seconds. + +This method cannot be used for the features that are Kubernetes specific. \ No newline at end of file