Commit Graph

96 Commits

Author SHA1 Message Date
M. Mert Yıldıran
2ad4838cf5
Add Go crypto/tls eBPF tracer for TLS connections (#1120)
* Run `go generate tls_tapper.go`

* Add `golang_uprobes.c`

* Add Golang hooks and offsets

* Add `golangConnection` struct and implement `pollGolangReadWrite` method

* Upgrade `github.com/cilium/ebpf` version to `v0.8.1`

* Fix the linter error

* Move map related stuff to `maps.h` and run `go generate tls_tapper.go`

* Remove unused parameter

* Add an environment variable to test Golang locally

* Replace `Libssl` occurrences with `Ssllib` for consistency

* Fix exe path finding

* Temporarily disable OpenSSL

* Fix the mixed offsets and dissection preparation

* Change the read symbol from `net/http.(*persistConn).Read` to `crypto/tls.(*Conn).Read`

* Remove `len` and `cap` fields

* Fix the indent

* Fix the read data address

* Make `golang_dial_writes` key `__u64` and include the PID

* Fix the read data address one more time

* Temporarily disable the PCAP capture

* Add a uprobe for `net/http.(*gzipReader).Read` to read chunked HTTP response body

* Cancel `golang_crypto_tls_read_uprobe` if it's a gzip read

* Make hash map names more meaningful

* Pass the connection address from `write` to `gzip` through a common address between `gzip` and `dial`

* Fix the probed line number links

* Add `golangReader` struct and implement its `Read` method

* Have a single counter pair and request response matcher per Golang connection

* Add `MIZU_GLOBAL_GOLANG_PATH` environment variable

* `NULL` terminate the bytes with `unix.ByteSliceToString`

* Temporarily reject the gzip chunks

* Add malformed TODOs

* Revert "`NULL` terminate the bytes with `unix.ByteSliceToString`"

This reverts commit 7ee7ef7e44.

* Bring back `len` and `cap` fields

* Set `len` and `cap` in `golang_net_http_gzipreader_read_uprobe` as well

* Remove two `TODO`s

* Fix the `key_gzip` offsets

* Compress if it's gzip chunk (probably wrong!)

* Revert "Compress if it's gzip chunk (probably wrong!)"

This reverts commit 094a7c3da4.

* Remove `golang_net_http_gzipreader_read_uprobe`

* Read constant 4KiB

* Use constant read length

* Get the correct len of bytes (saw the second entry)

* Set all buffer sizes to `CHUNK_SIZE`

* Remove a `TODO`

* Revert "Temporarily disable the PCAP capture"

This reverts commit a2da15ef2d.

* Update `golang_crypto_tls_read_uprobe`

* Set the `reader` field of `tlsStream` to fix a `nil pointer dereference` error

* Don't export any fields of `golangConnection`

* Close the reader when we drop the connection

* Add a tracepoint for `sys_enter_close` to detect socket closes

* Rename `socket` struct to `golang_socket`

* Call `should_tap` in Golang uprobes

* Add `log_error` calls

* Revert "Temporarily disable OpenSSL"

This reverts commit f54d9a453f.

* Fix linter

* Revert "Revert "Temporarily disable OpenSSL""

This reverts commit 2433d867af.

* Change `golang_read_writes` map type from `BPF_RINGBUF` to `BPF_PERF_OUTPUT`

* Rename `golang_read_write` to `golang_event`

* Define an error

* Add comments

* Revert "Revert "Revert "Temporarily disable OpenSSL"""

This reverts commit e5a1de9c71.

* Fix `pollGolang`

* Revert "Revert "Revert "Revert "Temporarily disable OpenSSL""""

This reverts commit 6e1bd5d4f3.

* Fix `panic: send on closed channel`

* Revert "Revert "Revert "Revert "Revert "Temporarily disable OpenSSL"""""

This reverts commit 57d0584655.

* Use `findLibraryByPid`

* Revert "Revert "Revert "Revert "Revert "Revert "Temporarily disable OpenSSL""""""

This reverts commit 46f3d290b0.

* Revert "Revert "Revert "Revert "Revert "Revert "Revert "Temporarily disable OpenSSL"""""""

This reverts commit 775c833c06.

* Log tapping Golang

* Fix `Poll`

* Refactor `golang_net_http_dialconn_uprobe`

* Remove an excess error check

* Fix `can only use path@version syntax with 'go get' and 'go install' in module-aware mode` error in `tap/tlstapper/bpf-builder/build.sh`

* Unify Golang and OpenSSL under a single perf event buffer and `tls_chunk` struct

* Generate `tlsTapperChunkType` type (enum) as well

* Use kernel page size for the `sys_closes` perf buffer

* Fix the linter error

* Fix `MIZU_GLOBAL_GOLANG_PID` environment variable's functionality

* Rely on tracepoints for file descriptor retrieval in Golang implementation

* Remove the unnecessary changes

* Move common functions into `common.c`

* Declare `lookup_ssl_info` function to reduce duplication

* Fix linter

* Add comments and TODOs

* Remove `MIZU_GLOBAL_GOLANG_PATH` environment variable

* Update the object files

* Fix indentation

* Update object files

* Add `go_abi_internal.h`

* Fix `lookup_ssl_info`

* Convert indentation to spaces

* Add header guard comment

* Add more comments

* Find the `ret` instructions using Capstone Engine and `uprobe` the `return` statements

* Implement `get_fd_from_tcp_conn` function

* Separate SSL contexts to OpenSSL and Go

* Move `get_count_bytes` from `common.c` to `openssl_uprobes.c`

* Rename everything contains Golang to Go

* Reduce duplication in `go_uprobes.c`

* Update the comments

* Install Capstone in CI and Docker native builds

* Update `devops/install-capstone.sh`

* Add Capstone to AArch64 cross-compilation target

* Fix some of the issues on ARM64

* Delete the map element in `_ex_urpobe`

* Remove an unsued `LOG_` macro

* Rename `aquynh` to `capstone-engine`

* Add comment

* Revert "Fix some of the issues on ARM64"

This reverts commit 0b3eceddf4.

* Revert "Revert "Fix some of the issues on ARM64""

This reverts commit 681534ada1.

* Update object files

* Remove unnecessary return

* Increase timeout

* #run_acceptance_tests

* #run_acceptance_tests

* Fix the `arm64v8` sourced builds

* #run_acceptance_tests
2022-06-12 14:14:41 +03:00
Igor Gov
8f64fdaa61
Alert to slack if release action fails (#1117)
* Trigger mizu ent stg deployment on release
2022-05-29 15:16:45 +03:00
gadotroee
08ae2bf6d7
Move "es Lint" to different step and add go lint on logger module (#1064) 2022-05-02 10:01:41 +03:00
Igor Gov
e71a12d399
Introducing eslint (#1048)
* Introducing eslint
2022-04-28 11:46:00 +03:00
gadotroee
171b72831f
Add docker buildx acceptance tests image build (#1039) 2022-04-25 13:06:46 +03:00
Igor Gov
1e167f2757
Trigger by commit message acceptance test on PR to develop (#1017) 2022-04-19 09:12:30 +03:00
gadotroee
149e86d050
Run unit tests when tap/api changes (#1016)
* update test files paths
2022-04-19 08:51:32 +03:00
Igor Gov
7cfe506897
Remove auto issue close if stale too long (#999)
* Remove auto issue close if stale too long
2022-04-13 18:58:02 +03:00
Nimrod Gilboa Markevich
62cbcf8857
Exclude labels with the enhancement label from becoming stale (#997) 2022-04-13 16:17:49 +03:00
Nimrod Gilboa Markevich
2de7107c0a
Use author instead of commiter in slack alerts (#992) 2022-04-12 10:48:50 +03:00
RoyUP9
66ba778384
Fixed golint modified files (#969) 2022-04-04 15:32:22 +03:00
Igor Gov
a97b5b3b38
Add conditional Go lint validation to CI (#967) 2022-04-04 14:35:47 +03:00
Nimrod Gilboa Markevich
aa8dcc5f5c
Format commit message as code to handle multi line messages (#963)
Co-authored-by: gadotroee <55343099+gadotroee@users.noreply.github.com>
2022-04-03 22:10:43 +03:00
Igor Gov
8eeb0e54c9 Changing unit tests workflow timeout to 30 minutes 2022-03-30 11:52:47 +03:00
Andrey Pokhilko
97db24aeba
OAS: rework data feeding + sampleIDs (#917)
* Call OAS feeder

* Don't call old OAS code

* Rework calls

* Work on it

* Put back rules

* Make it compile

* start thinking of test

* Compiles

* Save

* Fixes

* Save

* Fixing

* Trying to fake conn

* add timeout

* Test timeout

* Fix tests

* Only build OAS for HTTP entries

* Remove some dead code

* Adding SampleIDs

* Cosmetics

* lint

* Revert rename

* Sample ID for content

* Cleanuo

* Add more sample IDs

* Checking hypothesis

* Move assignment place a bit

* Cosmetics

* Update test.yml

Co-authored-by: undera <undera@undera-old-desktop.home>
Co-authored-by: Igor Gov <iggvrv@gmail.com>
2022-03-30 11:14:25 +03:00
Igor Gov
e867b7d0f1
Build ui-common part of CI (#914)
* Build ui-common always locally
2022-03-29 14:14:52 +03:00
Nimrod Gilboa Markevich
1f6e539590
Add commit message and committer to acceptance tests slack alert (#946)
* Add commit message and committer username to slack alerts
* Use name instead of username
* Use name and email
2022-03-29 09:15:42 +03:00
Igor Gov
41a7587088
Remove redundant Google auth from test workflow (#911)
* Remove google auth for test workflow
2022-03-23 11:51:42 +02:00
Igor Gov
617fb89ca5
Build custom branch Github action (#890)
* Build custom branch github action #build_and_publish_custom_image

* #build_and_publish_custom_image

* #build_and_publish_custom_image

* #build_and_publish_custom_image

* #build_and_publish_custom_image

* #build_and_publish_custom_image

* .
2022-03-14 13:15:28 +02:00
Igor Gov
1cbd9cb199
Adding dev latest tag for each pre-release docker (#888) 2022-03-13 09:47:49 +02:00
Igor Gov
23c1b66855
Adding dev latest tag for each pre-release docker (#885) 2022-03-10 17:26:56 +02:00
Igor Gov
5e90d67b0e
Run PR validation check only when needed & use docker cache during build (#876)
* Improve PR validation checks
2022-03-06 15:03:43 +02:00
RoyUP9
5af0c5a9e9
Refactor to acceptance tests setup (#875) 2022-03-03 17:22:26 +02:00
Nimrod Gilboa Markevich
c5471c501b
Add acceptance tests for k8s permissions (#834) 2022-03-01 12:22:34 +02:00
M. Mert Yıldıran
69ee8752d0
Create and push the latest tag if the branch is main (stable) (#831)
* Fix the styling issue in the badges

* Create and push the `latest` tag if the branch is `main` (stable)

* Build the for `latest` tag as well
2022-02-19 18:09:18 +03:00
M. Mert Yıldıran
db427d91cc
Add unit tests for HTTP dissector (#767)
* Add unit tests for HTTP dissector

* Ignore some fields on test environment

* Remove Git patches

* Don't have indent in the expected JSONs

* Fix more issues and update expected JSONs

* Refactor the test environment lookup

* Add a Makefile

* Include HTTP tests into the CI

* Fix the linting errors

* Fix another linting error

* Update the expected JSONs

* Sort `PostData.Params` params as well

* Move expected JSONs into `expect/dissect`

* Add `TestAnalyze` method

* Add `TestRepresent` method

* Add `TestRegister`, `TestMacros` and `TestPing` methods

* Test extensions first

* Remove expected JSONs

* Remove `bin` directory and update `Makefile` rules

* Update `.gitignore`

* Fix skipping download functionality in the Makefile

* Run `go mod tidy`

* Fix the race condition in the tests

* Revert "Test extensions first"

This reverts commit b8350cf139.

* Make `TEST_UPDATE` env lookup one-liner

* Update .github/workflows/test.yml

Co-authored-by: gadotroee <55343099+gadotroee@users.noreply.github.com>

* Add a newline

* Replace `ls` with `find`

Co-authored-by: gadotroee <55343099+gadotroee@users.noreply.github.com>
2022-02-09 13:34:52 +03:00
Igor Gov
2c72c27bc9
Removing concurrency group from acceptance test workflow (#771) 2022-02-08 13:09:56 +02:00
Igor Gov
98ca1b2840
Moving to go 1.17 (#766)
* Updating to go1.17
2022-02-08 11:58:08 +02:00
Igor Gov
5b53252033
CI: test share module (#763) 2022-02-07 07:16:28 +02:00
Igor Gov
bfa275d08d
Upload codecov report for develop and main branches (#761) 2022-02-06 21:08:52 +02:00
Igor Gov
0af1bc021d
Fixing warning during release workflow (#760)
* Fixing warning during release workflow

Co-authored-by: Igor Gov <igor.govorov1@gmail.com>
2022-02-06 20:52:15 +02:00
Igor Gov
bd253e1ff7
Fix cli build param ver (#758)
Co-authored-by: Igor Gov <igor.govorov1@gmail.com>
2022-02-06 20:35:41 +02:00
gadotroee
379885845b
Update release.yml (#756) 2022-02-06 15:19:30 +02:00
Igor Gov
ad6fb844aa
Moving from Semver to major.patch versioning (#755)
* Moving from Semver to major.patch versions

* version check should work with new version forma

Co-authored-by: Igor Gov <igor.govorov1@gmail.com>
2022-02-06 15:01:36 +02:00
Igor Gov
0a2e55f7bc
Fix lint errors file not found (#740)
Co-authored-by: Igor Gov <igor.govorov1@gmail.com>
2022-02-01 14:17:56 +02:00
Igor Gov
602225bb36
Adding go lint to more modules (#738) 2022-02-01 12:08:55 +02:00
Igor Gov
0f6c56986f
Technical depth: Adding Go linter to CI (#734) 2022-02-01 08:47:26 +02:00
Igor Gov
82db4acb7d
Build agent docker image during CI (#725) 2022-01-31 14:54:35 +02:00
M. Mert Yıldıran
a9a61edd50
Add ARM64 and cross-compilation support to the agent image (#659)
* modified Dockerfile to work for both amd64 (Intel) and arm64 (M1)

* added changelog

* Update `Dockerfile` to have `ARCH` build argument

* Remove `docs/CHANGES.md`

* Upgrade the Basenine version from `v0.3.0` to `v0.4.6`

* Update `publish.yml` to have `ARCH` build argument

* Switch `BasenineImageRepo` to Docker Hub

* Have separate build arguments for `ARCH` and `GOARCH`

* Upgrade the Basenine version from `v0.4.6` to `v0.4.10`

* Oops forgot to update the 10th duplicated shell script

* Fix the oopsie and reduce duplications

* Fix `Dockerfile`

* Fix the incompatibility issue between Go plugins and gold linker in Alpine inside `Dockerfile`

* Fix `asm: xxhash_amd64.s:120: when dynamic linking, R15 is clobbered by a global variable access` error

* Update `Dockerfile` to have cross-compilation on an AMD64 machine

Also revert changes in the shell scripts

* Delete `debug.Dockerfile`

* Create a custom base (`debian:buster-slim` based) image for the shipped image

* Replace `mertyildiran/debian-pcap` with `up9inc/debian-pcap`

* Upgrade Basenine version to `v0.4.12`

* Use `debian:stable-slim` as the base

* Fix the indentation in the `Dockerfile`

* Update `publish.yml`

* Enable `publish.yml` for `feature/multiarch_build` branch

* Tag correctly and set `ARCH` Docker argument

* Remove the lines that are forgotten to be removed from the shell scripts

* Add `MizuAgentImageRepo` constant and use it as default `AgentImage` value

* Bring back `Set up Cloud SDK` step to `Build the CLI and publish` job

* Build ARM64 CLI for Linux as well

* Revert "Enable `publish.yml` for `feature/multiarch_build` branch"

This reverts commit d30be4c1f0.

* Revert Go 1.17 upgrade

* Remove `build_extensions_debug.sh` as well

* Make the `Dockerfile` to compile the agent statically

* Statically link the protocol extensions

* Fix `Dockerfile`

* Bring back `-s -w` flags

* Verify the signatures of the downloads in `dockcross/linux-arm64-musl`

* Revert modifications in some shell scripts

* Make the `BUILDARCH` and `TARGETARCH` separation in the `Dockerfile`

* Separate cross-compilation builder image into a separate repo named `up9inc/linux-arm64-musl-go-libpcap`

* Fill the shell script and specify the tag for `dockcross/linux-arm64-musl`

* Remove the unnecessary dependencies from `builder-native-base`

* Improve the comments in the `Dockerfile`

* Upgrade Basenine version to `v0.4.13`

* Fix `Dockerfile`

* Revert "Revert "Enable `publish.yml` for `feature/multiarch_build` branch""

This reverts commit 303e466bdc.

* Revert "Revert "Revert "Enable `publish.yml` for `feature/multiarch_build` branch"""

This reverts commit 0fe252bbdb.

* Remove `push-docker-debug` from the `Makefile`

* Rename `publish.yml` to `release.yml`

Co-authored-by: Alex Haiut <alex@up9.com>
2022-01-25 21:24:50 +03:00
gadotroee
bcea6cdc49
Update publish.yml (#679) 2022-01-23 16:18:50 +02:00
gadotroee
1a2697dd0d
Images to docker hub (#676) 2022-01-23 12:41:47 +02:00
lirazyehezkel
18d90cdf36
Support node 16 (#673)
* upgrade node-sass

* upgrade axios

* update dockerfile

Co-authored-by: gadotroee <55343099+gadotroee@users.noreply.github.com>
2022-01-20 18:41:00 +02:00
RoyUP9
5ca3107422
Added build ui to pr validation flow (#655) 2022-01-16 17:43:18 +02:00
Igor Gov
4de795e463
Alert on acceptance tests failure (#537) 2021-12-16 11:04:36 +02:00
gadotroee
31d95c6557
Auto close inactive issues (#441) 2021-11-08 14:49:25 +02:00
Igor Gov
2c29903910
Run snyk on all project with ignore current issues (#415) 2021-10-28 21:59:42 +03:00
Igor Gov
f49e29045c
Adding Snyk to CI (#414) 2021-10-28 16:37:11 +03:00
RoyUP9
4e50e17d81
build m1 and windows (#395) 2021-10-24 15:10:46 +03:00
M. Mert Yıldıran
754f385865
Improve formatting in bug_report.md issue template (#352) 2021-10-15 14:14:51 +03:00
Igor Gov
b9d2e671c7
Move all docs to docs folder and clean project root (#278) 2021-09-15 11:53:23 +03:00