github/kubeshark
1
0
Fork 0
mirror of https://github.com/kubeshark/kubeshark.git synced 2025-09-07 13:32:29 +00:00
Code Issues Packages Projects Releases Wiki Activity

Refactor Mizu, define an extension API and add new protocols: AMQP, Kafka (#224)

Browse Source 
* Separate HTTP related code into `extensions/http` as a Go plugin

* Move `extensions` folder into `tap` folder

* Move HTTP files into `tap/extensions/lib` for now

* Replace `orcaman/concurrent-map` with `sync.Map`

* Remove `grpc_assembler.go`

* Remove `github.com/up9inc/mizu/tap/extensions/http/lib`

* Add a build script to automatically build extensions from a known path and load them

* Start to define the extension API

* Implement the `run()` function for the TCP stream

* Add support of defining multiple ports to the extension API

* Set the extension name inside the extension

* Declare the `Dissect` function in the extension API

* Dissect HTTP request from inside the HTTP extension

* Make the distinction of outbound and inbound ports

* Dissect HTTP response from inside the HTTP extension

* Bring back the HTTP request-response pair matcher

* Return a `*api.RequestResponsePair` from the dissection

* Bring back the gRPC-HTTP/2 parser

* Fix the issues in `handleHTTP1ClientStream` and `handleHTTP1ServerStream`

* Call a function pointer to emit dissected data back to the `tap` package

* roee changes -
trying to fix agent to work with the "api" object) - ***still not working***

* small mistake in the conflicts

* Fix the issues that are introduced by the merge conflict

* Add `Emitter` interface to the API and send `OutputChannelItem`(s) to `OutputChannel`

* Fix the `HTTP1` handlers

* Set `ConnectionInfo` in HTTP handlers

* Fix the `Dockerfile` to build the extensions

* remove some unwanted code

* no message

* Re-enable `getStreamProps` function

* Migrate back from `gopacket/tcpassembly` to `gopacket/reassembly`

* Introduce `HTTPPayload` struct and `HTTPPayloader` interface to `MarshalJSON()` all the data structures that are returned by the HTTP protocol

* Read `socketHarOutChannel` instead of `filteredHarChannel`

* Connect `OutputChannelItem` to the last WebSocket means that finally the web UI started to work again

* Add `.env.example` to React app

* Marshal and unmarshal `*http.Request`, `*http.Response` pairs

* Move `loadExtensions` into `main.go` and map extensions into `extensionsMap`

* Add `Summarize()` method to the `Dissector` interface

* Add `Analyze` method to the `Dissector` interface and `MizuEntry` to the extension API

* Add `Protocol` struct and make it effect the UI

* Refactor `BaseEntryDetails` struct and display the source and destination ports in the UI

* Display the protocol name inside the details layout

* Add `Represent` method to the `Dissector` interface and manipulate the UI through this method

* Make the protocol color affect the details layout color and write protocol abbreviation vertically

* Remove everything HTTP related from the `tap` package and make the extension system fully functional

* Fix the TypeScript warnings

* Bring in the files related AMQP into `amqp` directory

* Add `--nodefrag` flag to the tapper and bring in the main AMQP code

* Implement the AMQP `BasicPublish` and fix some issues in the UI when the response payload is missing

* Implement `representBasicPublish` method

* Fix several minor issues

* Implement the AMQP `BasicDeliver`

* Implement the AMQP `QueueDeclare`

* Implement the AMQP `ExchangeDeclare`

* Implement the AMQP `ConnectionStart`

* Implement the AMQP `ConnectionClose`

* Implement the AMQP `QueueBind`

* Implement the AMQP `BasicConsume`

* Fix an issue in `ConnectionStart`

* Fix a linter error

* Bring in the files related Kafka into `kafka` directory

* Fix the build errors in Kafka Go files

* Implement `Dissect` method of Kafka and adapt request-response pair matcher to asynchronous client-server stream

* Do the "Is reversed?" checked inside `getStreamProps` and fix an issue in Kafka `Dissect` method

* Implement `Analyze`, `Summarize` methods of Kafka

* Implement the representations for Kafka `Metadata`, `RequestHeader` and `ResponseHeader`

* Refactor the AMQP and Kafka implementations to create the summary string only inside the `Analyze` method

* Implement the representations for Kafka `ApiVersions`

* Implement the representations for Kafka `Produce`

* Implement the representations for Kafka `Fetch`

* Implement the representations for Kafka `ListOffsets`, `CreateTopics` and `DeleteTopics`

* Fix the encoding of AMQP `BasicPublish` and `BasicDeliver` body

* Remove the unnecessary logging

* Remove more logging

* Introduce `Version` field to `Protocol` struct for dynamically switching the HTTP protocol to HTTP/2

* Fix the issues in analysis and representation of HTTP/2 (gRPC) protocol

* Fix the issues in summary section of details layout for HTTP/2 (gRPC) protocol

* Fix the read errors that freezes the sniffer in HTTP and Kafka

* Fix the issues in HTTP POST data

* Fix one more issue in HTTP POST data

* Fix an infinite loop in Kafka

* Fix another freezing issue in Kafka

* Revert "UI Infra - Support multiple entry types + refactoring (#211)"

This reverts commit f74a52d4dc.

* Fix more issues that are introduced by the merge

* Fix the status code in the summary section

* adding the cleaner again (why we removed it?).
add TODO: on the extension loop .

* fix dockerfile (remove deleting .env file) - it is found in dockerignore and fails to build if the file not exists

* fix GetEntrties ("/entries" endpoint) - working with "tapApi.BaseEntryDetail" (moved from shared)

* Fix an issue in the UI summary section

* Refactor the protocol payload structs

* Fix a log message in the passive tapper

* Adapt `APP_PORTS` environment variable to the new extension system and change its format to `APP_PORTS='{"http": ["8001"]}' `

* Revert "fix dockerfile (remove deleting .env file) - it is found in dockerignore and fails to build if the file not exists"

This reverts commit 4f514ae1f4.

* Bring in the necessary changes from f74a52d4dc

* Open the API server URL in the web browser as soon as Mizu is ready

* Make the TCP reader consists of a single Go routine (instead of two) and try to dissect in both client and server mode by rewinding

* Swap `TcpID` without overwriting it

* Sort extension by priority

* Try to dissect with looping through all the extensions

* fix getStreamProps function.
(it should be passed from CLI as it was before).

* Turn TCP reader back into two Goroutines (client and server)

* typo

* Learn `isClient` from the TCP stream

* Set `viewer` style `overflow: "auto"`

* Fix the memory leaks in AMQP and Kafka dissectors

* Revert some of the changes in be7c65eb6d

* Remove `allExtensionPorts` since it's no longer needed

* Remove `APP_PORTS` since it's no longer needed

* Fix all of the minor issues in the React code

* Check Kafka header size and fail-fast

* Break the dissectors loop upon a successful dissection

* Don't break the dissector loop. Protocols might collide

* Improve the HTTP request-response counter (still not perfect)

* Make the HTTP request-response counter perfect

* Revert "Revert some of the changes in be7c65eb6d3fb657a059707da3ca559937e59739"

This reverts commit 08e7d786d8.

* Bring back `filterItems` and `isHealthCheckByUserAgent` functions

* Remove some development artifacts

* remove unused and commented lines that are not relevant

* Fix the performance in TCP stream factory. Make it create two `tcpReader`(s) per extension

* Change a log to debug

* Make `*api.CounterPair` a field of `tcpReader`

* Set `isTapTarget` to always `true` again since `filterAuthorities` implementation has problems

* Remove a variable that's only used for logging even though not introduced by this branch

* Bring back the `NumberOfRules` field of `ApplicableRules` struct

* Remove the unused `NewEntry` function

* Move `k8sResolver == nil` check to a more appropriate place

* default healthChecksUserAgentHeaders should be empty array (like the default config value)

* remove spam console.log

* Rules button cause app to crash (access the service via incorrect property)

* Ignore all .env* files in docker build.

* Better caching in dockerfile: only copy go.mod before go mod download.

* Check for errors while loading an extension

* Add a comment about why `Protocol` is not a pointer

* Bring back the call to `deleteOlderThan`

* Remove the `nil` check

* Reduce the maximum allowed AMQP message from 128MB to 1MB

* Fix an error that only occurs when a Kafka broker is initiating

* Revert the change in b2abd7b990

* Fix the service name resolution in all protocols

* Remove the `anydirection` flag and fix the issue in `filterAuthorities`

* Pass `sync.Map` by reference to `deleteOlderThan` method

* Fix the packet capture issue in standalone mode that's introduced by the removal of `anydirection`

* Temporarily resolve the memory exhaustion in AMQP

* Fix a nil pointer dereference error

* Fix the CLI build error

* Fix a memory leak that's identified by `pprof`

Co-authored-by: Roee Gadot <roee.gadot@up9.com>
Co-authored-by: Nimrod Gilboa Markevich <nimrod@up9.com>
This commit is contained in:
M. Mert Yıldıran
2021-09-02 14:34:06 +03:00
committed by GitHub
parent 17fa163ee3
commit 366c1d0c6c
111 changed files with 14396 additions and 1947 deletions
Download Patch File Download Diff File Expand all files Collapse all files

480
tap/extensions/kafka/protocol.go Normal file
View File

@@ -0,0 +1,480 @@
package main
import (
"fmt"
"io"
"net"
"reflect"
"strconv"
"strings"
)
// Message is an interface implemented by all request and response types of the
// kafka protocol.
//
// This interface is used mostly as a safe-guard to provide a compile-time check
// for values passed to functions dealing kafka message types.
type Message interface {
ApiKey() ApiKey
}
type ApiKey int16
func (k ApiKey) String() string {
if i := int(k); i >= 0 && i < len(apiNames) {
return apiNames[i]
}
return strconv.Itoa(int(k))
}
func (k ApiKey) MinVersion() int16 { return k.apiType().minVersion() }
func (k ApiKey) MaxVersion() int16 { return k.apiType().maxVersion() }
func (k ApiKey) SelectVersion(minVersion, maxVersion int16) int16 {
min := k.MinVersion()
max := k.MaxVersion()
switch {
case min > maxVersion:
return min
case max < maxVersion:
return max
default:
return maxVersion
}
}
func (k ApiKey) apiType() apiType {
if i := int(k); i >= 0 && i < len(apiTypes) {
return apiTypes[i]
}
return apiType{}
}
const (
Produce ApiKey = 0
Fetch ApiKey = 1
ListOffsets ApiKey = 2
Metadata ApiKey = 3
LeaderAndIsr ApiKey = 4
StopReplica ApiKey = 5
UpdateMetadata ApiKey = 6
ControlledShutdown ApiKey = 7
OffsetCommit ApiKey = 8
OffsetFetch ApiKey = 9
FindCoordinator ApiKey = 10
JoinGroup ApiKey = 11
Heartbeat ApiKey = 12
LeaveGroup ApiKey = 13
SyncGroup ApiKey = 14
DescribeGroups ApiKey = 15
ListGroups ApiKey = 16
SaslHandshake ApiKey = 17
ApiVersions ApiKey = 18
CreateTopics ApiKey = 19
DeleteTopics ApiKey = 20
DeleteRecords ApiKey = 21
InitProducerId ApiKey = 22
OffsetForLeaderEpoch ApiKey = 23
AddPartitionsToTxn ApiKey = 24
AddOffsetsToTxn ApiKey = 25
EndTxn ApiKey = 26
WriteTxnMarkers ApiKey = 27
TxnOffsetCommit ApiKey = 28
DescribeAcls ApiKey = 29
CreateAcls ApiKey = 30
DeleteAcls ApiKey = 31
DescribeConfigs ApiKey = 32
AlterConfigs ApiKey = 33
AlterReplicaLogDirs ApiKey = 34
DescribeLogDirs ApiKey = 35
SaslAuthenticate ApiKey = 36
CreatePartitions ApiKey = 37
CreateDelegationToken ApiKey = 38
RenewDelegationToken ApiKey = 39
ExpireDelegationToken ApiKey = 40
DescribeDelegationToken ApiKey = 41
DeleteGroups ApiKey = 42
ElectLeaders ApiKey = 43
IncrementalAlterConfigs ApiKey = 44
AlterPartitionReassignments ApiKey = 45
ListPartitionReassignments ApiKey = 46
OffsetDelete ApiKey = 47
DescribeClientQuotas ApiKey = 48
AlterClientQuotas ApiKey = 49
numApis = 50
)
var apiNames = [numApis]string{
Produce: "Produce",
Fetch: "Fetch",
ListOffsets: "ListOffsets",
Metadata: "Metadata",
LeaderAndIsr: "LeaderAndIsr",
StopReplica: "StopReplica",
UpdateMetadata: "UpdateMetadata",
ControlledShutdown: "ControlledShutdown",
OffsetCommit: "OffsetCommit",
OffsetFetch: "OffsetFetch",
FindCoordinator: "FindCoordinator",
JoinGroup: "JoinGroup",
Heartbeat: "Heartbeat",
LeaveGroup: "LeaveGroup",
SyncGroup: "SyncGroup",
DescribeGroups: "DescribeGroups",
ListGroups: "ListGroups",
SaslHandshake: "SaslHandshake",
ApiVersions: "ApiVersions",
CreateTopics: "CreateTopics",
DeleteTopics: "DeleteTopics",
DeleteRecords: "DeleteRecords",
InitProducerId: "InitProducerId",
OffsetForLeaderEpoch: "OffsetForLeaderEpoch",
AddPartitionsToTxn: "AddPartitionsToTxn",
AddOffsetsToTxn: "AddOffsetsToTxn",
EndTxn: "EndTxn",
WriteTxnMarkers: "WriteTxnMarkers",
TxnOffsetCommit: "TxnOffsetCommit",
DescribeAcls: "DescribeAcls",
CreateAcls: "CreateAcls",
DeleteAcls: "DeleteAcls",
DescribeConfigs: "DescribeConfigs",
AlterConfigs: "AlterConfigs",
AlterReplicaLogDirs: "AlterReplicaLogDirs",
DescribeLogDirs: "DescribeLogDirs",
SaslAuthenticate: "SaslAuthenticate",
CreatePartitions: "CreatePartitions",
CreateDelegationToken: "CreateDelegationToken",
RenewDelegationToken: "RenewDelegationToken",
ExpireDelegationToken: "ExpireDelegationToken",
DescribeDelegationToken: "DescribeDelegationToken",
DeleteGroups: "DeleteGroups",
ElectLeaders: "ElectLeaders",
IncrementalAlterConfigs: "IncrementalAlterConfigs",
AlterPartitionReassignments: "AlterPartitionReassignments",
ListPartitionReassignments: "ListPartitionReassignments",
OffsetDelete: "OffsetDelete",
DescribeClientQuotas: "DescribeClientQuotas",
AlterClientQuotas: "AlterClientQuotas",
}
type messageType struct {
version int16
flexible bool
gotype reflect.Type
decode decodeFunc
encode encodeFunc
}
func (t *messageType) new() Message {
return reflect.New(t.gotype).Interface().(Message)
}
type apiType struct {
requests []messageType
responses []messageType
}
func (t apiType) minVersion() int16 {
if len(t.requests) == 0 {
return 0
}
return t.requests[0].version
}
func (t apiType) maxVersion() int16 {
if len(t.requests) == 0 {
return 0
}
return t.requests[len(t.requests)-1].version
}
var apiTypes [numApis]apiType
// Register is automatically called by sub-packages are imported to install a
// new pair of request/response message types.
func Register(req, res Message) {
k1 := req.ApiKey()
k2 := res.ApiKey()
if k1 != k2 {
panic(fmt.Sprintf("[%T/%T]: request and response API keys mismatch: %d != %d", req, res, k1, k2))
}
apiTypes[k1] = apiType{
requests: typesOf(req),
responses: typesOf(res),
}
}
func typesOf(v interface{}) []messageType {
return makeTypes(reflect.TypeOf(v).Elem())
}
func makeTypes(t reflect.Type) []messageType {
minVersion := int16(-1)
maxVersion := int16(-1)
// All future versions will be flexible (according to spec), so don't need to
// worry about maxes here.
minFlexibleVersion := int16(-1)
forEachStructField(t, func(_ reflect.Type, _ index, tag string) {
forEachStructTag(tag, func(tag structTag) bool {
if minVersion < 0 || tag.MinVersion < minVersion {
minVersion = tag.MinVersion
}
if maxVersion < 0 || tag.MaxVersion > maxVersion {
maxVersion = tag.MaxVersion
}
if tag.TagID > -2 && (minFlexibleVersion < 0 || tag.MinVersion < minFlexibleVersion) {
minFlexibleVersion = tag.MinVersion
}
return true
})
})
types := make([]messageType, 0, (maxVersion-minVersion)+1)
for v := minVersion; v <= maxVersion; v++ {
flexible := minFlexibleVersion >= 0 && v >= minFlexibleVersion
types = append(types, messageType{
version: v,
gotype: t,
flexible: flexible,
decode: decodeFuncOf(t, v, flexible, structTag{}),
encode: encodeFuncOf(t, v, flexible, structTag{}),
})
}
return types
}
type structTag struct {
MinVersion int16
MaxVersion int16
Compact bool
Nullable bool
TagID int
}
func forEachStructTag(tag string, do func(structTag) bool) {
if tag == "-" {
return // special case to ignore the field
}
forEach(tag, '|', func(s string) bool {
tag := structTag{
MinVersion: -1,
MaxVersion: -1,
// Legitimate tag IDs can start at 0. We use -1 as a placeholder to indicate
// that the message type is flexible, so that leaves -2 as the default for
// indicating that there is no tag ID and the message is not flexible.
TagID: -2,
}
var err error
forEach(s, ',', func(s string) bool {
switch {
case strings.HasPrefix(s, "min="):
tag.MinVersion, err = parseVersion(s[4:])
case strings.HasPrefix(s, "max="):
tag.MaxVersion, err = parseVersion(s[4:])
case s == "tag":
tag.TagID = -1
case strings.HasPrefix(s, "tag="):
tag.TagID, err = strconv.Atoi(s[4:])
case s == "compact":
tag.Compact = true
case s == "nullable":
tag.Nullable = true
default:
err = fmt.Errorf("unrecognized option: %q", s)
}
return err == nil
})
if err != nil {
panic(fmt.Errorf("malformed struct tag: %w", err))
}
if tag.MinVersion < 0 && tag.MaxVersion >= 0 {
panic(fmt.Errorf("missing minimum version in struct tag: %q", s))
}
if tag.MaxVersion < 0 && tag.MinVersion >= 0 {
panic(fmt.Errorf("missing maximum version in struct tag: %q", s))
}
if tag.MinVersion > tag.MaxVersion {
panic(fmt.Errorf("invalid version range in struct tag: %q", s))
}
return do(tag)
})
}
func forEach(s string, sep byte, do func(string) bool) bool {
for len(s) != 0 {
p := ""
i := strings.IndexByte(s, sep)
if i < 0 {
p, s = s, ""
} else {
p, s = s[:i], s[i+1:]
}
if !do(p) {
return false
}
}
return true
}
func forEachStructField(t reflect.Type, do func(reflect.Type, index, string)) {
for i, n := 0, t.NumField(); i < n; i++ {
f := t.Field(i)
if f.PkgPath != "" && f.Name != "_" {
continue
}
kafkaTag, ok := f.Tag.Lookup("kafka")
if !ok {
kafkaTag = "|"
}
do(f.Type, indexOf(f), kafkaTag)
}
}
func parseVersion(s string) (int16, error) {
if !strings.HasPrefix(s, "v") {
return 0, fmt.Errorf("invalid version number: %q", s)
}
i, err := strconv.ParseInt(s[1:], 10, 16)
if err != nil {
return 0, fmt.Errorf("invalid version number: %q: %w", s, err)
}
if i < 0 {
return 0, fmt.Errorf("invalid negative version number: %q", s)
}
return int16(i), nil
}
func dontExpectEOF(err error) error {
switch err {
case nil:
return nil
case io.EOF:
return io.ErrUnexpectedEOF
default:
return err
}
}
type Broker struct {
ID int32
Host string
Port int32
Rack string
}
func (b Broker) String() string {
return net.JoinHostPort(b.Host, itoa(b.Port))
}
func (b Broker) Format(w fmt.State, v rune) {
switch v {
case 'd':
io.WriteString(w, itoa(b.ID))
case 's':
io.WriteString(w, b.String())
case 'v':
io.WriteString(w, itoa(b.ID))
io.WriteString(w, " ")
io.WriteString(w, b.String())
if b.Rack != "" {
io.WriteString(w, " ")
io.WriteString(w, b.Rack)
}
}
}
func itoa(i int32) string {
return strconv.Itoa(int(i))
}
type Topic struct {
Name string
Error int16
Partitions map[int32]Partition
}
type Partition struct {
ID int32
Error int16
Leader int32
Replicas []int32
ISR []int32
Offline []int32
}
// BrokerMessage is an extension of the Message interface implemented by some
// request types to customize the broker assignment logic.
type BrokerMessage interface {
// Given a representation of the kafka cluster state as argument, returns
// the broker that the message should be routed to.
Broker(Cluster) (Broker, error)
}
// GroupMessage is an extension of the Message interface implemented by some
// request types to inform the program that they should be routed to a group
// coordinator.
type GroupMessage interface {
// Returns the group configured on the message.
Group() string
}
// PreparedMessage is an extension of the Message interface implemented by some
// request types which may need to run some pre-processing on their state before
// being sent.
type PreparedMessage interface {
// Prepares the message before being sent to a kafka broker using the API
// version passed as argument.
Prepare(apiVersion int16)
}
// Splitter is an interface implemented by messages that can be split into
// multiple requests and have their results merged back by a Merger.
type Splitter interface {
// For a given cluster layout, returns the list of messages constructed
// from the receiver for each requests that should be sent to the cluster.
// The second return value is a Merger which can be used to merge back the
// results of each request into a single message (or an error).
Split(Cluster) ([]Message, Merger, error)
}
// Merger is an interface implemented by messages which can merge multiple
// results into one response.
type Merger interface {
// Given a list of message and associated results, merge them back into a
// response (or an error). The results must be either Message or error
// values, other types should trigger a panic.
Merge(messages []Message, results []interface{}) (Message, error)
}
// Result converts r to a Message or and error, or panics if r could be be
// converted to these types.
func Result(r interface{}) (Message, error) {
switch v := r.(type) {
case Message:
return v, nil
case error:
return nil, v
default:
panic(fmt.Errorf("BUG: result must be a message or an error but not %T", v))
}
}